Blog: The Educated Reporter

Getting Ready for PISA Scores: Background and Story Tips

Rankings may be human catnip for news readers, but they rarely tell the whole story. As education journalists gear up for a season of new reports that detail how much U.S. students know, tips on what the forthcoming PISA scores say — and don’t say — are in order.

EWA tapped two education assessment experts – Andreas Schleicher of the Organization for Economic Co-operation and Development and Marc Tucker of The National Center on Education and the Economy – for a webinar to talk about what they hope reporters will be asking once the Program for International Student Assessment (PISA) scores are posted in early December. And though the NAEP district scores (TUDA) weren’t on the docket, lessons gleaned from PISA can still be applied to the district-comparing NAEP scores.

So what is PISA? It’s a two-hour exam that’s administered every three years in over 60 countries, enough to represent nearly 90 percent of the world’s economy, according to the OECD. Roughly half a million students, comprised of a representative sample of fifteen-year-olds from each participating country, take PISA. The students and their school leaders also share information on socio-economics and educational programs. In some countries, parents also participate in a survey.

What does it test? The international assessment looks for student comprehension of subject matter in reading, math and science. Like the similarly tasked NAEP, PISA aims to “assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations,” Schleicher says. Problem-solving and financial literacy also factor into the mix of items that appear on the assessment.

Why care? The PISA results are a “global metric,” says Schleicher. The OECD calls PISA not just a snapshot of where students are academically, but also a glimpse into the future. Indeed, last month’s OECD assessment of adult skills closely correlates with how students fared on PISA the first time it was issued in the late 1990s. They can also spotlight a country’s strength and weaknesses overtime. The U.S., for example, is unique in the education skills gap between its low-income and better off students, Schleicher notes. There’s far greater compression among skills in top performing countries like Finland. Tucker, meanwhile, chides U.S. education stakeholders for arguing the U.S. would score just as highly as top performing countries if we close our achievement gap. PISA results distribute the scores by the level of proficiency for each subject. Leading PISA countries have many more of their pupils, even low-income students, performing at the top quartile of achievement than the U.S.

Because the PISA scores come with profiles of how each education system functions, reporters have an opportunity to compare U.S. practices to those of leading countries. In the top performing countries, there’s a “very strong emphasis on academic aspiration,” Schleicher says. “[These countries] have very little tolerance for failure but have a lot of opportunities and means to innovate.” In Finland for example, Schleicher says 30 percent of instruction occurs outside the classroom. “Teachers have a lot of opportunities to help students who are struggling,” he says.

Schleicher also says the PISA scores over time have shown tracking students based on skill and social background sets a country back. He points to the recent international adult skills assessment that largely shows countries that track students have a less knowledgeable workforce overall.

The reliance on entrance exams at elite schools, honors classes in middle school, and the AP system are some examples of tracking in the U.S. Other countries are more deliberate, assigning students to courses of study that lock them into a career of blue-collar work.

What’s new for this test? For U.S. journalists, the PISA test will drill down to the state level, allowing for fair comparisons to smaller education systems. Shanghai, a perennial achiever on PISA, has a population of over 20 million—bigger than most U.S. states. Still, there are limits to cross-national comparisons. Tucker warns that a substantial number of Shanghai students do not receive full schooling and may not be captured in the PISA scores. Migration between provinces in China waives a family’s right to basic services, like education. For workers moving from the impoverished west to the booming east, this can spell trouble for the school-aged children in tow.

What doesn’t it show? Webinar host and EWA public editor Emily Richmond started things off by summarizing, “it can tell us where students stand, [ but] that’s not the same thing as telling us how they got there, or what interventions worked out to help them improve.” There are some caveats to this rule, however. Schleicher pointed to his native Germany as one example of a country responding with public policy initiatives after a bad PISA showing. The country scored surprisingly low on the 2000 PISA results, kicking in a flurry of education reforms similar in approach to the Common Core State Standards in the U.S. By Tucker’s estimate, it took Germany six years to conceive and rollout the new academic changes. He and Schleicher say those changes went a long way toward improving Germany’s PISA results by 2009.

Top scoring PISA countries have what in common? Schleicher provided the chart below to illustrate the dual trends common among top PISA countries. Systems that allow for a lot of flexibility but expect a lot from its teachers tend to generate the highest PISA scores. High expectations with teeth coupled with a tight leash don’t bear as high results, Schleicher says, because teachers and school leaders need to come up with their own curricula to match the expectations set by national leaders.

Something of a surprise to many U.S. readers may be the low stock in small classrooms. Schleicher says the top performing PISA countries make, “effective spending choices that prioritize high quality teachers over smaller classes.” In South Korea, teachers are recruited from the top 3 or 5 percent of postsecondary students. In Finland, it’s the top decile. Tucker says that the U.S. plucks its teachers from the bottom third of postsecondary students. But recruitment and several years of graduate work don’t tell the whole story: Schleicher says teachers in the leading PISA countries are more accountable to students and peers rather than higher-ranking authorities. There’s also less hierarchy within schools, he says.

Tucker says that the main education priorities of the left and right don’t even comprise the top-10 student learning improvement measures among leading PISA countries. He has in mind spending more money and the reduction of class size as pet issues of the left, and school choice and teacher accountability measures tied to student performance from the right. “We are pursuing an agenda that is largely a distraction from what we have to do to match the performance of the top performing countries,” Tucker said about the U.S. education reform movement.

What should reporters be asking? Best for last, right?

Beyond reporting on the rankings, Tucker recommends reporters see if U.S. 15-year-olds improved this time around compared to the 2009 results (which were released in 2010).

He also encourages reporters to see if achievement gaps have widened.

Is the U.S. continuing to outspend nearly all the tested countries in primary and secondary education per pupil? Earlier I wrote Tucker cast doubt on the effects of spending more money to improve student learning. Are we getting more efficient with our dollars? A word of caution, though: This year’s results may overstate national spending in some countries more than others, depending on whether they used stimulus money to spur the economy during the global recession.

Are districts integrating the new, tougher standards into their curricula? Tucker says top countries don’t lean on standards to drive their education systems. He says top PISA countries do a better job of synchronizing standards with curricula. One way they do this? With tougher, far more expensive assessments. Tucker estimates top PISA countries spend five times more on assessments than states will on the new Common Core-aligned assessments, which have already generated headlines for their sticker shock.

Ask your district whether they hire the top teachers and if they’d support efforts to recruit teachers from the upper echelon of college talent.

Abroad, the PISA scores garner many headlines and spur months of debate, Tucker says. He encourages reporters to ask district leaders if a) they’re familiar with the PISA scores and b) if they are familiar with the education systems of the top performing PISA countries.

Tucker fears the U.S. is too dismissive of the lessons that can be learned from education systems abroad. If reporters can use their stories to let readers know there’s another approach to educating students, “that would be an enormous victory,” he said.

 



Have a question, comment or concern for the Educated Reporter? Contact Emily Richmond. Follow her on Twitter @EWAEmily.

Read other Educated Reporter articles.