Major Problems with Value-Added Test-Score Evaluation of Teachers

From the Marshall Memo #427

“Practitioners, researchers, and policy makers agree that most current teacher evaluation systems do little to help teachers improve or to support personnel decision making.” So say Linda Darling-Hammond and Edward Haertel (Stanford University), Audrey Amrein-Beardsley (Arizona State University), and Jesse Rothstein (University of California/Berkeley) in this Kappan article. They note the growing consensus that teachers’ contributions to student learning should be part of the evaluation process, but strongly criticize the use of value-added test score data to evaluate individual teachers. Here are their reasons:

• Value-added models don’t accurately measure a teacher’s impact on students. That’s because gains in student achievement are influenced by a number of factors in addition to the individual teacher: class size; curriculum materials; instructional time; availability of specialists and tutors; availability of books, computers, and science labs; home and community supports or challenges; individual students’ needs and abilities, health, and attendance; peer culture and achievement; prior teachers and schooling; other current teachers (for example, art, music, computer, phys. ed., and pullout Title I or ELL instructors); summer learning experiences; and the specific tests used, which may or may not measure all areas of the curriculum and are notoriously inaccurate in measuring achievement that is well above or below grade level. Value-added models attempt to control for non-teacher factors by projecting student-achievement gains based on demographic factors, but they can’t take the full range of variables into account and are therefore inaccurate in isolating the true impact of each teacher.

• Value-added models are inconsistent. Ratings of teacher effectiveness differ substantially from class to class, from year to year, and from one statistical model to another. In a study of five school districts, some teachers who scored in the bottom quintile one year moved to the top quintile the next, and some teachers who scored in the top quintile moved to the bottom. A re-analysis of the 2011 teacher ratings published in the Los Angeles Times found that 40-55% of teachers had markedly different scores when a different metric was used.

• Value-added ratings assume random assignment of students to classrooms. But in the real world, students aren’t randomly assigned, and studies have shown that teachers’ value-add ratings are directly affected by the number of students in their classes who have poor attendance, are homeless, have severe problems at home, have special needs, or are English language learners. “This variability raises concerns that using such ratings for evaluating teachers could create disincentives for teachers to serve high-need students,” say the authors.

• Value-added ratings can’t disentangle the many influences on student progress. Some teachers are effective in certain areas of the curriculum and not in others, and their ratings depend on which curriculum areas are emphasized in the assessment. Some teachers are better at preparing students for state tests but less effective in preparing them for success in the next grade or course, so using value-added ratings can put a premium on narrow test preparation rather than deeper preparation for future success. One school had an 8^th-grade science teacher who was getting low value-added scores and a 6^th-grade teacher who was getting high scores. The principal flipped their assignments and their ratings flipped too. “The notion that there is a stable ‘teacher effect’ that’s a function of the teacher’s teaching ability or effectiveness is called into question if the specific class or grade-level assignment is a stronger predictor of the value-added rating than the teacher,” say the authors.

So are value-added models good for anything? Yes, say Darling-Hammond, Haertel, Amrein-Beardsley, and Rothstein – if they’re used to look at groups of teachers and specific teaching practices that can be used in a more thoughtful approach to teacher evaluation. These include:

• Teachers understanding subject matter deeply and flexibly;
• Connecting what is to be learned to students’ prior knowledge and experience;
• Creating effective scaffolds and supports for learning;
• Using instructional strategies that help students make connections, apply what they’re learning, practice new skills, and monitor their own learning;
• Continuously assessing student learning and using data to adapt teaching;
• Providing clear standards, constant feedback, and opportunities to revise work;
• Managing a collaborative classroom in which students have a voice.

These and other effective practices have been studied and are being incorporated into the teacher-evaluation process in a number of districts. Best practices are most accurately measured by multiple classroom visits with timely and meaningful feedback to teachers. Presenting evidence of student learning is also part of the mix in some districts. “With these features in place,” conclude the authors, “evaluation can become a more useful part of a productive teaching and learning system, supporting accurate information about teachers, helpful feedback, and well-grounded personnel decisions.” 

“Evaluating Teacher Evaluation” by Linda Darling-Hammond, Edward Haertel, Audrey Amrein-Beardsley, and Jesse Rothstein in Phi Delta Kappan, March 2012 (Vol. 93, #6, p. 8-15), http://www.kappanmagazine.org; Darling-Hammond can be reached at ldh@stanford.edu.