Ravitch: Problems with the big teacher evaluation study

This was written by education historian Diane Ravitch for her Bridging Differences blog, which she co-authors with Deborah Meier on the Education Week website. Ravitch and Meier exchange letters about what matters most in education. Ravitch, a research professor at New York University, is the author of the bestselling “The Death and Life of the Great American School System,” an important critique of the flaws in the modern school reform movement that she just updated.

Dear Deborah,

Days ago, three economists released a study that created a great deal of controversy. Raj Chetty and John N. Friedman of Harvard University and Jonah Rockoff of Columbia University studied the school records and income tax records of 2.5 million students in a major urban district (probably New York City) over a 20-year period. They concluded that good teachers cause students to get higher test scores, which lead in turn to higher lifetime earnings, fewer teen pregnancies, and higher college-going rates.

The study was reported on Page One of The New York Times, covered on the PBS Newshour, and lauded by Nicholas Kristof in The Times. While the study itself did not have specific policy recommendations, one of the authors told The Times: “The message is to fire people sooner rather than later.”

The study seemed to vindicate supporters of value-added assessment. It was certainly good news for Erik Hanushek of the Hoover Institution, who has been arguing for several years that the key to improving education is to fire the bottom 5 to 10 percent of teachers based on the test scores of their students. In theory, if a “bad” teacher is replaced by an average teacher, then scores go up.

As you can well imagine, the study had immediate political ramifications. Conservative Republican governors immediately embraced the study as justification for abolishing tenure and any other job protections for teachers.

Bloggers quickly chimed in, and here is a list of the best posts, compiled by blogger extraordinaire and Sacramento, Calif., teacher Larry Ferlazzo.

Here are some obvious conclusions from the study: Teachers are really important. They make a lasting difference in the lives of their students. Some teachers are better than other teachers. Some are better at raising students’ test scores.

The problems of the study are not technical, but educational.

The Chetty-Friedman-Rockoff analysis points us to an education system in which tests become even more consequential than they are now. Teachers would work in school systems with no job protection, and their jobs would depend on the rise or fall of their students’ test scores.

Most teachers do not teach tested subjects, so it is not clear how they would be rated. But those who teach reading or mathematics in grades 3-8 would have to pay close attention to the tests. They would spend extra time preparing students to take them, even more than they do now.

There would be even less time in our schools than now for the arts, history, civics, geography, the sciences, foreign languages, health, and physical education. There would be less time to read challenging literature. There would be less time for science experiments. There would be less time for field trips to museums or historical sites. There would be less time for anything other than getting ready for the state tests.

There would be less time for extracurricular activities. There would be less time for chorus or band or dramatics or painting or film-making. There would be less time to read books, whether novels or histories.

None of these things is directly related to raising test scores.

What matters most is getting the right answer on the test. Divergent thinking would be discouraged because divergent thinking might produce wrong answers. So would originality, creativity, ingenuity, or any other display of independence or critical thinking.

We can expect that some teachers will find ways to avoid teaching the most challenging students and to avoid the most difficult schools and districts. Isn’t that the way incentives work?

When you put all these likely outcomes together, it’s hard to imagine that we will have better education for more kids. We might or might not have higher test scores, but at what cost? Under these circumstances, who will want to teach? Is there a large pool of average, good, or great teachers waiting in the wings?

It’s not surprising that students who get higher test scores are likelier to go to college and eventually have a higher income. But, according to economist Bruce Baker of Rutgers University, it is not so simple to identify which teachers produced these good outcomes. Baker writes: “... just because teacher [value-added] scores in a massive data set show variance does not mean that we can identify with any level of precision or accuracy which individual teachers (plucking single points from a massive scatter plot) are ‘good’ and which are ‘bad.’ Therein exists one of the major fallacies of moving from large scale econometric analysis to micro level human resource management.” (my italics)

It is surprising, in light of all the publicity, that the differences produced by the high value-added teachers are relatively small. Baker shows that the income gains are only about $250 a year over a 40-year working span for each of the students.

As Baker writes: “One of the big quotes in The New York Times article is: ‘Replacing a poor teacher with an average one would raise a single classroom’s lifetime earnings by about $266,000, the economists estimate.’ This comes straight from the research paper. BUT ... let’s break that down. It’s a whole classroom of kids. Let’s say ... for rounding purposes, 26.6 kids if this is a large urban district like NYC. Let’s say we’re talking about earning careers from age 25 to 65 or about 40 years. So, $266,000/26.6 = $10,000 lifetime additional earnings per individual. Hmmm ... no longer catchy headline stuff. Now, per year? $10,000/40 = $250. Yep, about $250 per year.”

Now, to clear up any doubt, let me make it clear that I don’t believe any school should hire or retain incompetent or “bad” teachers. If teachers can’t teach, they should be fired. No one who is incompetent should be awarded due process rights. Teachers who are having problems should be evaluated by their (hopefully, experienced) principal and peers, offered help, and if they don’t or can’t improve, they should be terminated.

The most peculiar aspect of the study is its concluding paragraph. It is not at all consonant with their public statements about “firing sooner rather than later,” nor with the policy agendas that are being built around the assumption that they recommend laying off more teachers and instituting merit pay. They conclude:

“While these calculations show that good teachers have great value, they do not by themselves have implications for optimal teacher salaries or merit pay policies. The most important lesson of this study is that finding policies to raise the quality of teaching — whether via the use of value-added measures, changes in salary structure, or teacher training — is likely to have substantial economic and social benefits in the long run.” No one could disagree with that statement, certainly not me.

As for me, I prefer deliberate efforts to raise entry standards into teaching, to improve teacher preparation, and to ensure that every school has a significant number of experienced teachers who are masters of their craft. That seems to be what the high-performing nations do. The goal would be to make teaching a prestigious profession, rather than a job that any college graduate — with only minimal preparation — can do.