The new Harvard-Columbia study provides important information about the relationship between student test scores and longer-run outcomes. But there is much that we still don’t understand. We need careful study of pilot programs, not to remake our education system.

One unheralded new result is that teachers’ effectiveness changes over time. The study shows that some teachers are ineffective at first but improve as they age, while others start strong but then burn out. Policy design must account for this. How many teachers who are fired early on for poor student achievement would improve given the chance? And how many who get early raises will continue to draw them through years of later coasting? Calculations that firing a poor teacher saves $2.5 million entirely ignore this factor.

Second, how much variation in teacher effectiveness do test impacts miss? We now know that test score impacts correlate with longer-run outcomes. But how strong is this correlation? My work suggests that it may be weak. Are there other measures – perhaps the textured classroom observations studied in last week’s Gates Foundation report – that would better predict long-run impacts?

Third, and most important: How much will test score measures deteriorate when careers depend on them? A longstanding principle of social science is that measures that work well in low-stakes settings can be badly distorted when the stakes rise. Teachers will ignore non-tested subjects, avoid students who will hurt their scores, overemphasize test preparation, suspend poor performers on test day, and even cheat outright. This will limit the value of test scores for identifying effective teachers, and will hurt student learning.

It is quite possible for a value added-based evaluation system to do more harm than good. To date, pilot studies have not been promising. We should not race ahead without careful policy development and testing.