The Measures of Effective Teaching Project Issues Its Final Report

In this report, lead authors Steven Cantrell and Thomas Kane say, “Teaching is too complex for any single measure of performance to capture it accurately.” The best approach, they believe, is for principals to combine data from three sources:

• Classroom observations
• Student perception surveys
• Student achievement gains

MET researchers looked into a fourth metric – teachers’ content knowledge (as measured by the Content Knowledge for Teaching tests) – and found that it didn’t pass their test for validity. “MET project teachers who performed better on the CKT tests were not substantively more effective in improving student achievement on the outcomes we measured,” they say. “This was true whether student achievement was measured using state tests or the supplemental assessments of higher-order thinking skills.” Teachers’ content knowledge is important, but current metrics aren’t robust enough to be part of MET’s calculations of teacher effectiveness.

In the report, Cantrell and Kane answer three questions that arise from MET’s basic recommendations:

• Can we accurately predict which teachers will get better student results? The short answer is yes. MET researchers identified teachers in several districts who earned good evaluations using the three criteria above; arranged for random assignment of students for 2010-11 at specific grade levels; and found that students in higher-rated teachers’ classrooms did significantly better than students whose teachers had lower ratings. “In addition,” report Cantrell and Kane, “we found that more-effective teachers not only caused students to perform better on state tests, but they also caused students to score higher on other, more cognitively challenging assessments in math and English… Great teaching does make a difference.”

• How much weight should be placed on each of the three measures? MET researchers found that weighting teachers’ past state test-score gains more heavily created the strongest correlation with their current students’ state test-score gains. But looking at students’ results on more-sophisticated tests (similar to those that will be used to assess Common Core standards), they found that different weighting schemes all produced about the same correlations. And when it came to reliability, heavy weighting of state test scores was the least accurate. The MET team also worried that weighting test scores too heavily would create perverse incentives for test prep and deemphasizing other important parts of the curriculum. All this led Cantrell and Kane to advocate for a balanced weighting of the three elements, with test-score gains counting for between 33 and 50 percent. 

One thing is clear: evaluating teachers by combining classroom observations, student perception surveys, and student achievement gains is far better than what most schools are currently using to determine teachers’ salaries and make retention decisions (years of teaching experience and possession of a master’s degree). “On every student outcome – the state tests, supplemental tests, students’ self-reported level of effort, and enjoyment in class – the teachers who excelled on the composite measure had better outcomes than those with high levels of teaching experience or a master’s degree,” say Cantrell and Kane. 

What about a hypothetical district that uses only classroom observations to evaluate teachers? Once again, the combined three-way evaluation approach is significantly more accurate. “Even with four full classroom observations (two by one observer and two by another), conducted by observers trained and certified by the Educational Testing Service, the observation-only model performed far worse than any of our multiple measures composites,” say the authors. 

• How can we maximize the accuracy of teacher supervision? “Classroom observations can be powerful tools for professional growth,” say Cantrell and Kane. “But for observations to be of value, they must reliably reflect what teachers do throughout the year, as opposed to the subjective impressions of a particular observer or some unusual aspect of a particular lesson. Teachers need to know they are being observed by the right people, with the right skills, and a sufficient number of times to produce trustworthy results.” 

The MET team found that multiple short observations increase reliability and make more efficient use of administrators’ time. In fact, observers watching the first 15 minutes of a class saw a significant amount of what they would have seen staying for the full class. Researchers found that bringing in observers from outside the school is a useful check against “in-school bias”, and that peer observers can provide useful insights. “Our results also suggest,” say Cantrell and Kane, “that it is important to have at least one or two full-length observations, given that some aspects of teaching scored on the Framework for Teaching (Danielson’s instrument) were frequently not observed during the first 15 minutes of class.” 

[The basic model proposed by the MET study makes perfect sense, but I continue to have concerns about four areas: (a) Cantrell and Kane seem not to have considered the possibility of principals making multiple short visits to the beginning, middle, and end of teachers’ classes during the year, thereby getting an accurate sampling of full lessons; (b) They are wedded to the traditional model of teacher evaluation and don’t take into account the distortions introduced by announced classroom observations (the dog-and-pony show); (c) They haven’t explored the downside of principals using rubrics to give feedback to teachers after each observation, nor how principals can effectively coach and develop teachers during the year; and (d) They haven’t factored in possible misuses of test-score evaluation of teachers when districts use less-sophisticated assessments than were possible in the MET study. See my November 2012 Educational Leadership article for further discussion of these points (at www.marshallmemo.com, click on Kim Bio/Publications):  K.M.]

“Ensuring Fair and Reliable Measures of Effective Teaching: Culminating Findings from the MET Project’s Three-Year Study” by Steven Cantrell and Thomas Kane, January 2013; full findings and related reports are available at www.metproject.org. 

From the Marshall Memo #468