Another Look at the Los Angeles Value-Added Data on Teachers

From the Marshall Memo #443

In this National Education Policy Center brief, Derek Briggs and Ben Domingue of the University of Colorado/Boulder reexamine the value-added analysis of Los Angeles teachers conducted by Richard Buddin, a senior economist at the RAND Corporation, and published by the Los Angeles Times in August 2010. Here are Buddin’s key questions and Briggs and Domingue’s comments:

• How much does quality vary from teacher to teacher? Buddin found significant variation in students’ reading and math test scores depending on which teachers they had. Briggs and Domingue agree with this finding. In fact, they found a slightly larger teacher effect. 

• How important are teachers’ qualifications to their students’ results? Buddin found that years of experience, advanced degrees, having a full teaching credential, race, and gender had only a weak association with student results. Briggs and Domingue disagree. Based on their reanalysis of the data, they found that some factors did play a role, with more-experienced and better-qualified teachers getting better results. 

Another important question with the Buddin data is whether it’s possible to pin down an individual teacher’s effectiveness based on student test scores. Computing individual value-added scores for teachers presupposes that students are randomly assigned to different teachers. Briggs and Domingue found that students are not randomly assigned – especially in reading. “If students are non-randomly assigned to teachers in ways that systemically advantage some teachers and disadvantage others (e.g., stronger students tending to be in certain teachers’ classrooms),” they say, “then these advantages and disadvantages will show up whether one looks at past teachers, present teachers, or future teachers. That is, the model’s outputs result, at least in part, from this bias, in addition to the teacher effectiveness the model is hoping to capture.” 

So how accurate were the 5-4-3-2-1 teacher ratings published in the Los Angeles Times? Briggs and Domingue used the same test and demographic data to construct a more sophisticated estimate of each teacher’s value-add and compared it to Buddin’s. The result: in reading, only 46.4% of teachers had the same effectiveness rating in both studies. In math, 60.8% of teachers had the same rating. According to Briggs and Domingue, 22 percent of reading teachers and 14 percent of math teachers were misclassified. There were lots of false positives (teachers rated effective who were really average) and false negatives (teachers rated as ineffective who were really average). 

Briggs and Domingue believe that the value-added ratings reported by the LA Times were inaccurate, unfair to many teachers, and failed to provide useful feedback to help mediocre and ineffective teachers improve. Proponents of value-added information on teachers argue that it’s better than the teacher-evaluation data we have now. True, but that’s not saying much. And it doesn’t mean that opponents of value-added data are supporters of the status quo. 

“The use of standardized test scores to evaluate teachers involves making difficult choices in which there are invariably some tradeoffs between decisions that might be optimal from the perspective of estimating an unbiased causal effect,” say Briggs and Domingue, “but not optimal from the perspective of crafting an educational accountability polity with a coherent theory of action. The obligation for those with expertise in statistical and econometric methods is to be explicit and transparent about these choices, so that policymakers and administrators have the information they need to weigh the costs and benefits, and so that all stakeholders have an entry point to the policy debate.

“The Buddin white paper presents a picture that implies a ‘have your cake and eat it too’ scenario,” Briggs and Domingue conclude: “that from a technical standpoint we know how to validly isolate the causal effect of a teacher, and from a policy standpoint we know how to create an incentive structure that winnows away the ineffective teachers while rewarding the effective ones enough to encourage new ones to enter the field. This picture is an illusion. Causal inference may well be the holy grail of quantitative research in the social sciences, but it should not be proclaimed lightly. When the causal language of teacher ‘effects’ or ‘effectiveness’ is casually applied to the estimates from a value-added model simply because it conditions on a prior year test score, it trivializes the entire enterprise. And instead of promoting discussion among parents, teachers and school administrators about what students are and are not learning in their classrooms, it seems much more likely to shut them down.” 

“Due Diligence and the Evaluation of Teachers: A Review of the Value-Added Analysis Underlying the Effectiveness Rankings of Los Angeles Unified School District Teachers by the Los Angeles Times” by Derek Briggs and Ben Domingue, February 2011, National Education Policy Center, http://nepc.colorado.edu/publication/due-diligence