Are value-added estimations reliable or stable?

Value-added modeling measures individual students’ performance on tests over time, using prior test scores to predict future outcomes. Statistical controls attempt to screen out factors such as race, family background, and the effect of peers, so as to attribute the remaining variation in student academic outcomes to schools and teachers.

At the level of individual teachers, such estimates vary considerably, pointing to differences in teachers’ levels of skill.

Some scholars say that of the measures of teacher effectiveness studied so far, value-added appears to be among the most promising. In one study, for instance, researchers used value-added estimates of teachers to predict the student-achievement patterns of some 3,000 students in 78 classrooms, and then randomly assigned teachers to these classrooms. The value-added models, while not perfect, were significant predictors of actual outcomes. [1]

At the same time, researchers have discovered that a host of factors contribute to measurement error in these estimates. These problems include the nonrandom assignment of students and teachers to schools and classrooms[2]; different effect sizes or results based on the statistical models used[3]; differences in the tests that supply the underlying data[4]; the seeming instability of estimates of particular teachers from year to year[5]; and the fade-out of teacher effects.[6]

In general, the variance in year-to-year estimates of individual teachers’ performance could indicate measurement error. Some of these problems, like the problem of tracking and instability in the estimates, seem to be ameliorated by using additional years of student data for each teacher, though researchers continue to debate this issue.[7]

The implications of these problems, both for policy and for research, are difficult to parse, and policy experts continue to debate the use of value-added as a component of teacher evaluations and for other purposes.

Finally, the research on teacher quality suggests that other school factors may affect how effective teachers appear to be in these types of calculations. One study found that up to a quarter of the estimate of an individual teacher’s value-added score depended on whether teachers were a good “match” for a particular school. Its author postulated that such factors as whether the teacher’s teaching philosophy meshed with the school’s culture and the choice of curricula might contribute to this match effect.[8]

Many teacher groups argue that value-added measures fail to take into account the considerable role of school and district leadership. Researchers are still investigating the role of principals as distinct from teachers, but it is difficult to disentangle the two.

Teachers’ peers may also influence their effectiveness. At least one study has found that a teacher appears to improve when surrounded by more-effective colleagues. [9] But a second paper looking at this question found no consistent evidence that teachers who hold National Board Certification, an independent honor that teachers go through a rigorous process to obtain, have an impact on the effectiveness of their peers.[10]

For a longer discussion of the issue of value-added measurement and its place in policy, see Harris 2011.

It can be said:

Value-added models appear to pick up some differences in teacher quality, but they can be influenced by a number of factors, such as the statistical controls selected. They may also be affected by the characteristics of schools and peers. The impact of unmeasured factors in schools, such as principals and choice of curriculum, is less clear.

Education Writers Association

2122 P St NW, Suite 201, Washington, DC 20037 • Phone: (202) 452-9830 • E-mail: ewa@ewa.org