Using Student Survey Data As Part of Teachers’ Evaluations

From the Marshall Memo #453

In this thoughtful article in The Atlantic, Amanda Ripley reports on the increasing use of anonymous student surveys as one factor in teachers’ evaluations. Last school year, about 250,000 students took surveys designed to capture what they thought of their teachers and the classroom culture they created. Research has shown, says Ripley, that “if you asked kids the right questions, they could identify, with uncanny accuracy, their most – and least – effective teachers… Their survey answers, it turned out, were more reliable than any other known measure of teacher performance – including classroom observations and student test-score growth.”

Several school districts are experimenting with student surveys and deciding whether the results should be used for high-stakes decisions about teachers – pay, promotion, or dismissal. Ripley was given permission to observe the survey process in the Washington, D.C. schools, a district that is also using more-intensive classroom visits and student test-score gains as part of the evaluation process. Use of test scores hasn’t been well received by teachers in D.C. and other districts. “Most teachers do not consider test-score data a fair measure of what students have learned,” says Ripley. “Complex algorithms that adjust for students’ income and race have made test-score assessments more fair – but are widely resented, contested, or misunderstood by teachers.” In addition, only about 15% of D.C. teachers have test-score data, begging the question of how the other 85% will be judged. Even if all teachers had value-added data, there’s the question of how the feedback will help improve instruction. Surveys, on the other hand, “focus on the means, not the ends,” says Ripley, “giving teachers tangible ideas about what they can fix right now, straight from the minds of the people who sit in front of them all day long.”

A decade ago, Harvard economics professor Ron Ferguson began to study the black-white achievement gap in Shaker Heights, Ohio. He suspected that important forces were at work in classrooms that teachers were unaware of, and began to administer early versions of student surveys. To his surprise, students of all racial groups had quite similar assessments of their teachers. “In one classroom, kids said they worked hard, paid attention, and corrected their mistakes,” says Ripley. “ They liked being there, and they believed that the teacher cared about them. In the next classroom, the very same kids reported that the teacher had trouble explaining things and didn’t notice when students failed to understand a lesson.” School officials in Shaker Heights found the data exceptionally helpful, but back at Harvard, other academicians ignored Ferguson’s work. 

Then the Measures of Effective Teaching (MET) Project, headed by Ferguson’s colleague Thomas Kane (see summary #1 above), decided to use a shorter version of the student survey as one of three factors in its study of teacher evaluation. Looking at the data, MET researchers found that math teachers who received the highest ratings from their students delivered about six additional months of achievement. “Students were better than trained adult observers at evaluating teachers,” says Ripley. “This wasn’t because they were smarter but because they had months to form an opinion, as opposed to 30 minutes. And there were dozens of them, as opposed to a single principal. Even if one kid had a grudge against a teacher or just blew off the survey, his response alone couldn’t sway the average.” 

Ferguson looked at 199,000 surveys and found that fewer than half of one percent of students show evidence of not taking the questions seriously. The overwhelming majority of students find the questions interesting; in addition, students can’t always tell which is the “right” answer, so even students who are trying to ding a teacher might not know how to do so. Student survey data turn out to be much less volatile than test-score gains; they deliver clear feedback to teachers across classrooms and from year to year. 

A large part of the validity of surveys lies in the questions Ferguson and his colleagues have written and revised over the years. Students are not asked whether they like their teachers or whether their teachers are nice. Instead, they are asked to rate their teachers anonymously on  five-point scale with statements like these:

• Students in this class treat the teacher with respect.
• My classmates behave the way my teacher wants them to.
• Our class stays busy and doesn’t waste time.
• In this class, we learn a lot almost every day.
• In this class, we learn to correct our mistakes.

Ferguson’s and MET’s research shows that caring is not the most important correlate of student achievement (as many teachers believe); what lines up with student achievement gains is control over the classroom and academic rigor and challenge. “As most of us remember from our own school days,” says Ripley, “those two conditions did not always coexist: some teachers had high levels of control, but low levels of rigor.” 

Having attained academic visibility via MET, student surveys have become a hot item in the national teacher-evaluation debate. Last year, Memphis became the first district to tie student survey data to teacher evaluation (counting 5 percent). The New Teacher Project used student surveys to evaluate 460 of the program’s 1,006 teachers. In Pittsburgh, all students took the survey last year, and the district may override union objections and use the data as part of teachers’ evaluations. Thomas Kane advocates counting survey data for 20-30 percent of teachers’ evaluations – a big enough proportion to be taken seriously but not so large that teachers are tempted to game the system by pandering to students or pressuring them to give high ratings. 

Ferguson is discouraged by how few teachers look at their survey data – only about one-third so far. Perhaps more teachers would if the stakes were higher. He urges school districts to give the survey multiple times before making it count toward high-stakes evaluations. 

Interestingly, Ferguson and other Harvard professors are evaluated partly on student surveys, which are a factor in salary discussions and promotion reviews. He says he dreads looking at the data after every course he teaches, but admits there are areas where he has made changes in his pedagogy in response to student feedback – and others where he’s flat-out rejected students’ suggestions. 

What matters with surveys – in universities and K-12 schools – is the quality of the questions, how carefully the surveys are administered (students have to be sure their comments are anonymous), and what instructors do with the data. Ferguson’s research shows little variation between students’ opinions across racial lines and between students who got high grades and low grades. But more-affluent students tend to be more critical of their teachers than students from lower-SES homes. Ferguson also believes students as young as kindergarten give quite reliable data if the questions are read aloud and their regular teacher is not in the classroom.

How do students react to being asked to evaluate their teachers? Almost all take the surveys seriously. “They should’ve done this since I was in eighth grade,” said one D.C. high-school student. But there’s some skepticism about how the results will be used. Another high-school student in the nation’s capital said, “Everybody knows the good teachers from the ones who don’t really want to be in the job… I care about the change the results bring. If I come back in five years and some crappy teacher is still sitting at that crappy desk, then what was the point of the survey?”

“Why Kids Should Grade Teachers” by Amanda Ripley in The Atlantic, October 2012, 

http://bit.ly/RNtChK