The Downside of Value-Added Evaluation of Teachers

In this troubling Teachers College Record article, Leo Casey (Albert Shanker Institute) analyzes the way the New York City Department of Education used value-added student test data to evaluate the effectiveness of 18,000 teachers based on their students’ state test scores from 2007 to 2010. The Teacher Data Reports were initially going to be used in a low-stakes manner – to help teachers develop as professionals – but the DOE changed its mind and began to use them as one factor in tenure decisions. The teachers’ union went to court to prevent the reports from being publicly released, but a judge ruled that they were public information and they were released to newspapers and other media outlets. In the ensuing publicity, a sixth-grade teacher in Queens was named by the New York Post as the “worst teacher” in New York City. 

Casey argues that the value-added reports were inaccurate at three levels. First, the state test scores on which they were based had recently been criticized by Harvard testing expert Daniel Koretz (hired by the State Education Department to study testing policies) as suffering from “score inflation, a lowering of standards, or both.” Second, even when they are based on higher-quality state tests, value-added measures have “disturbingly high rates of error,” says Casey. Third, it’s difficult to separate the effect of a particular teacher from classroom factors and school effects, especially when looking at only one year of performance. For example, the Queens teacher who had such an abysmally low value-added report was, according to her principal, colleagues, students, and parents, an excellent teacher. The low score was due to the fact that she taught ESL to small classes of high-need recent immigrants over several years, plus the school’s grade configuration. Because of these statistical quirks, the scores presented a completely inaccurate picture. 

According to the Department’s own calculations, teachers’ score reports had a margin of error of 53 percentile points in English language arts and 35 points in math. One-third of the scores were so imprecise that the Department said it couldn’t use them. “Yet the DOE chose to ignore these problems in a push to produce data which could be used in the annual evaluations of the greatest number of teachers,” says Casey. “The sheer magnitude of these numbers takes us into the realm of the statistically surreal… It was, to put it simply, a demonstration of professional malpractice in the realm of testing.” 

The outcry from teachers, psychometricians, and others was intense, and in June of 2012, the New York state legislature and governor enacted legislation that prohibited the public disclosure of teacher evaluations (although parents can be told the rating of their own children’s teachers). 

Why did intelligent and well-meaning city and DOE leaders embark on this course? asks Casey. He attributes it to the theory of action espoused by the Mayor, Chancellor, and other reform advocates around the country: public schools are a monopoly that needs to be subjected to the discipline of the marketplace. “The solution to all that ails public schools, therefore, is to remake them in the image and likeness of a competitive business,” explains Casey. “Just as private businesses rise and fall on their ability to compete in the marketplace, as measured by the ‘bottom line’ of their profit balance sheet, schools need to live or die on their ability to compete with each other, based on an educational ‘bottom line.’ If ‘bad’ schools die and new ‘good’ schools are created in their stead, the productivity of education improves.” Test scores were determined to be the best “bottom line” and teachers were “stacked” from the highest to the lowest performing according to a calculation of how much value they had added to their students’ test scores. Others could learn from this unfortunate experience, says Casey.

“The Will to Quantify: The ‘Bottom Line’ in the Market Model of Education Reform” by Leo Casey in Teachers College Record, September 2013 (Vol. 115, #9, p. 1-7), no free e-link

From the Marshall Memo #506