Five Questions for … University of Virginia Prof. Thomas Dee, on New Study Showing SIG School Progress in California
A new study by University of Virginia researcher Thomas S. Dee has found early evidence that the federal School Improvement Grant program had a positive impact on student achievement in the California schools that were awarded the competitive funds. Dee compared schools that were “just eligible” for the funds with those that were “just ineligible,” with both groups sharing largely similar baseline characteristics. At the “just eligible” schools, the SIG-funded reforms closed 23 percent of the achievement gap when it came to meeting the state’s performance targets on standardized tests. The gains were largely concentrated among schools that opted for the “turnaround” reform model, which required replacing the principal and at least 50 percent of the staff. He spoke with EWA about the study.
1. Was there anything particularly unexpected in the findings?
I was surprised that there were positive effects at all within the first year. My presumption had been that schools sought out this money largely because they were in fiscal crisis. I worried that the local buy-in to the required reforms would be poor and that the implementation would be uneven. Instead, what I found were sizable first-year improvements in school performance as a result of the SIG-funded reforms.
2. How much weight should be given to one year’s worth of data?
That’s clearly an important caveat: We don’t know how effective California will be at sustaining these early gains. We also don’t yet have good evidence on how SIGs may be influencing student outcomes in other states. I’m in the process of studying that. Until we can gather more information, we have to be cautious about how we judge this historic effort. Nonetheless, these early indicators are encouraging. The clear inference from my study is that the combination of SIG funding and whole-school reforms catalyzed meaningful improvements in California’s lowest-achieving schools.
3. Is there a way to identify whether a specific reform – such as replacing the principal or adding more student support services – had a larger effect?
The theory behind this turnaround effort seems to be that we need to move aggressively on multiple fronts to turn around chronically underperforming schools. That includes new leadership, teacher evaluations, data-driven and differentiated instruction as well as wraparound services to support students. Because schools generally implemented these reforms components simultaneously, we can’t convincingly point to one initiative say that it was the element that really leveraged school improvement. However, we should still continue to explore how schools are implementing these reforms, both to guide supports and possible refinements and to get some sense of what might be worth replicating in other low-achieving schools
4. What do you know about the schools that showed improvement after implementing SIG-funded reforms?
We know that the majority of them, like most of the SIG schools nationally, opted for the “transformation” model, where they implement instructional reforms, provide “wraparound” student supports and bring in new leadership. In California, 60 percent of SIG recipients chose transformation while a third chose the “turnaround” model, which also required dramatic teacher turnover.
I also find evidence that SIG recipients hired more teachers, lowering pupil-teacher ratios by roughly 5. Furthermore, the mix of teacher turnover and new hiring implied that the teachers in these schools had less experience. Receipt of a SIG award implied that average level of teacher experience fell by 2 years.
Another interesting finding involves cost effectiveness. We shouldn’t lose sight of the fact that these reforms are expensive. The typical school got $1.5 million, or $1,500 per pupil. And what will happen in these schools when the funding is no longer available is an open question. Nonetheless, my first-year evidence suggests that these reforms seem fairly cost-effective. Specifically, the first-year gains I found (i.e., 0.1 standard deviations in student test scores) compares quite favorably to the cost effectiveness of class-size reductions.
5. Early studies at the national level are providing mixed reviews of the SIG program. What’s your take on how it’s doing?
Observing the early discourse around the SIG effort underscores for me how we now live in exceptionally compressed and connected cycles of news and commentary. I am concerned that there has been some rush to judgment (both on the positive and negative side) that simply isn’t yet justified by the available evidence. For now, I think our focus should instead be on supporting and refining the extensive reform efforts we have asked these schools to undertake.
We should also position ourselves to understand how these reforms have changed school and teacher practices and whether these reforms are really generating sustained improvements in other states. As that evidence accumulates, my hope is that our overall assessment of this bold initiative will privilege the best available evidence.
I emphasize this because I think the stakes here are high. Reforms like those supported by the SIGs are also being encouraged by other federal initiatives such as NCLB waivers and the Race To The Top. I also view this SIG effort as part of a broader debate about the capacity of schools alone to reduce inequality. On the one hand you have the “no excuses” crowd arguing that a reorganization of leadership and practices can make schools highly effective. On the other side you have the “better and bolder” crowd arguing we need wraparound services that support student readiness to learn, and address all the problems they bring to the schoolhouse door.
What’s interesting to me about the SIG-funded reforms is they’re an amalgam, combining elements of both camps – there is staff retraining and new leadership, along with substantial funding for more student support. That, to me, is one of the most fascinating aspects to the whole endeavor.
This post originally appeared on EWA’s now-defunct online community, EdMedia Commons. Old content from EMC will appear in the Ed Beat archives.