The Troubled International Tests Data

About two years ago, I was first introduced to the Program for International Student Assessment (PISA), when taking a class on education and globalization. PISA is a worldwide study by the Organization for Economic Co-operation and Development (OECD) in member and non-member nations of 15-year-old school pupils' scholastic performance on mathematics, science, and reading. First carried out in 2000, PISA assessments have been following a triennial cycle, testing students from a growing number of countries (in 2000, 43 countries participated; the number rose to 65 in 2012) and producing reports after a-year-and-a-half analysis period. Last December, educators, policy makers and academics around the world anxiously waited for OECD to release its report on PISA 2012 results. The ranking, once again, ignited heated debates and discussions in the international education community.

I have remained skeptical about PISA since the first encounter. Personally, I have had detrimental experiences with standardized tests. Although I believe that testing could be used as a method to help teachers monitor student learning, I tend to take any standardized tests that go beyond the purpose of understanding individual students’ development with a grain of salt. Test results analyzed and reported in a collective manner (like what OECD is doing) take education systems and the discourses around them out of their social, political and economic contexts and lead to inaccurate and dehumanized understanding and policies talks around education reform. Over the years, I have kept an eye on the PISA data and discussions revolving around it. What I have found is that more and more educators and policy makers have spoken  about the troubling interpretation of international tests data.

In Daniel Koretz’s book, Measuring Up: What Educational Testing Really Tells Us, he discussed the trickiness of interpreting international test data. First off, the term “international mean” used by many media outlets and policy makers to compare American students’ performance to a seemingly “objective” standard is a fallacy. “International mean” varies from test to test depending on the participating nations. Secondly, more specific comparison between countries could still be inconsistent. Depending on the test content and ranking system, countries perform differently on different test. It is dangerous to draw conclusions based on an individual source. Koretz warned people not to “treat either one of these assessments as the definitive answer.”

Koretz did however make the point that it is relative safe to conclude that countries like Japan ranking above the U.S. with a large margin consistently perform better. But many people take such conclusions too far by claiming that East Asian countries have a better education system or better math/science curriculum without shedding a critical light on those tests. When the 2012 PISA report came out last December, Shanghai impressed the world with its students ranking number one again (after its initial appearance on PISA 2009 report) in all three areas PISA tests. PISA has turned Shanghai into this magic land whose superb student achievement educators around the world envy. When I was studying abroad in France last spring semester, PISA test was briefly discussed in my Comparative Education class. The professor was absolutely impressed by Shanghai’s scores and commented that “amazing things are happening there.” Not only did I have to disagree with her comment, I was also alarmed by such a misleading impression. Back then, I was not informed enough to explain to my professor why her interpretation of the data was incomprehensive and misguided. A year later, when the new report comes out, I was able to send my professors articles written by educators around the world who are critically examining PISA results and speaking up against the misleading information propelled by PISA report.

Tom Loveless enlightens the public of OECD-PISA’s negligenceof the migrant student population in Shanghai, which makes the Shanghai data a skewed depiction of the reality. Due to China’s kukou system that ties people to their geographic origins, many children of families that have moved to the city from rural areas of China are excluded from Shanghai school system. Loveless claims that those children were abandoned by OECD-PISA assessment. Their plight is brushed under the rug while Shanghai enjoys international praise for its education success.  Benjamin Riley, a Fulbright public policy fellow, also sheds doubt on the PISA data. Coming from a more technical perspective, he questioned the methodology used by PISA to analyze its data. He pointed us to the critique of PISA by Svend Kreiner, one of the world’s leading experts on the Rasch model, who argues that the OECD is using the Rasch model incorrectly. I am incapable of psychometrician analysis, but such critique should not be taken lightly.

I want to close this blog with a quote from Daniel Koretz’s book: “Scores describe some of what students can do, but they don’t describe all they can do, and they don’t explain why they can or cannot do it.” With that said, I believe standardized tests (both international and state-mandated ones) have enjoyed overly-exaggerated validity and attention over the past few decades. It is time to refocus and reframe our discussion on education reform.