More on Problems with Value-Added Evaluation of Teachers

Why is there such a push to evaluate teachers using value-added analysis? asks Michael Marder (University of Texas/Austin) in this Kappa Delta Pi Record article. Because many governors, state education commissioners, and federal legislators have embraced these points:

• Teachers are the single most important in-school factor in student achievement.
• Teachers’ background, degrees, and certifications do not tell if they are effective.
• Conventional evaluation systems give almost all teachers good ratings.
• It’s possible to identify the best and worst teachers by analyzing students’ state test scores. 

“If you held these beliefs… wouldn’t you act?” asks Marder. And action is coming. Within the next two years, many public school teachers’ annual evaluations will include calculations using student growth on standardized tests. The same is true of principals.

“Using computers to evaluate teachers based on student test scores is more difficult than it seems,” says Marder. Here are two seriously flawed methods of using student achievement:

• Grading teachers based on their students’ test scores – for example, giving a math teacher a grade of A, B, C, D, or F depending on whether 90 percent, 80 percent, 70 percent,  60 percent, or 50 percent of students passed a state exam, or on whether the class average was 90, 80, etc. Either method would be unfair, because some teachers might have classes that entered with higher achievement than others, giving them a head start.
• Grading teachers based on how much their students’ scores increase during the year. But this is unfair to teachers whose students enter the year with high achievement – it’s almost impossible for them to get strong gains because students have already topped out. In addition, it’s very difficult to design tests that measure during-the-year and year-to-year gains.

“Value-added modeling is a genuinely serious attempt to grapple with the difficulties,” says Marder. It does so by calculating how well students should do by the end of the year and then 

evaluating teachers on whether they exceeded, met, or did not meet those statistical goals. 

“To design a computer program that creates a custom expectation for the classroom of each teacher is not an easy task,” says Marder. “The program has to take many things into account… The calculations are so complex that only a handful of specialists know how to carry them out.” But are they valid for the individual teacher? Many concerns have been raised about missing information such as student mobility, large variations in results from year to year, the need for many years of data to get reliable results, and the lack of suitable pretests in some subject areas. That’s why educators and policymakers who understand value-added calculations – advocates and skeptics – agree that the results should be used with caution. “However,” says Marder, “because numbers in official printouts are so specific and appear so authoritative, it will prove problematic in practice to prevent them from dominating decisions about promotion and dismissal.” 

He tackles the complexities by asking a simple question: “What are the data you can find on a student that most accurately predict how much the student’s score will change over the next year?” The most important predictor is the student’s score the previous year. “This effect is huge,” says Marder. Thus, the surest way for a teacher to boost his or her value-added score would be to swap a few students based on their previous achievement levels. 

“There are other large effects floating around,” continues Marder. At certain grade levels, especially those with high stakes promotion tests, students consistently make bigger gains that are unconnected to teacher performance. And then there’s poverty, race, and differences between the best and worst teachers. It turns out that teachers are one of the smallest factors in test-score gains or losses. 

The bottom line: Marder doesn’t trust the value-added formulas. Every one he has looked at “begins with a bit of technical hastiness” as it tries to account for the many factors operating inside and outside the school, he says. “Make a technical mistake in accounting for them, then attribute that mistake to the teacher, and the results are wrong.” 

“It is tempting to automate a process that previously has been the province of human judgment,” Marder concludes. “But judgment is always present: if not in each detailed decision, then in the rules of automation. Automating a decision does not make it right. Computers are consistent, but not necessarily correct… Expert advice on value-added modeling always says that it should at most be used as a component of evaluation, in combination with other factors. Indeed. It provides information. It can flag real problems. But it has a limited view. And like the humans that created it, it is fallible.” 

“Measuring Teacher Quality with Value-Added Modeling” by Michael Marder in Kappa Delta Pi Record, October-December 2012 (Vol. 48, #4, p. 156-161), 

http://www.tandfonline.com/toc/ukdr20/48/4 

From the Marshall Memo #461