Policymakers, do your homework on evaluation reform

Posted by Barnett Berry on Monday, 06/17/2013

Center for Teacher Quality

Last week, Stephen Sawchuk of Education Week reported on the “race” to implement the new teacher evaluation system in New York City. The system has been marked by “years of bitter disagreement” among political and union leaders over the purpose of the new policy as well as the criteria for determining the quality of teaching.

The debate over using test scores (and a value-added or student-growth statistical model) to determine teacher effectiveness continues to rage, and implementation problems abound.

You should check out the differing critiques of Diane Ravitch and Rick Hess on using test scores alone as a basis for teacher evaluation, as well as the case made by education economist Dan Goldhaber, insisting that flawed systems of measuring teachers on the basis of student results are better than the status quo.

Ravitch, Hess, and Goldhaber make important points for policymakers, the public, and practitioners to consider. But the real story—and the potential policy disaster of teacher evaluation reform in New York City—is not in the compromise over test scores but in the implementation timeline.

The city has three months to get ready for a uniform evaluation system for 75,000 teachers.

That’s right: NYC principals and teachers have three months to implement the complex system of teacher evaluation, using multiple measures of student learning. Large numbers of administrators will need to be trained to use more detailed classroom observation tools, as well as to interpret a diverse set of data points in valid and reliable ways.

Even a district official, David Weiner (deputy chancellor for talent, innovation, and development), admitted that the challenge was enormous: "It's a large cultural change for a lot of schools, and they'll be asked to do things they've never been asked to do before.”

Looking back to move forward
I wonder if school reformers of today have studied the rich history of failed school reforms of the past. Looking back at research I drew on when teaching graduate students—more than two decades ago—I came up with a few lessons worth remembering:

• Policy success is most directly impacted by two broad factors—local capacity and will (McLaughlin, 1987);
• Schools should not adopt a single design, but co-construct their models to fit their environments (Olson, 1999);
• People’s participation in various communities and relationships is essential to implementation (McLaughlin and Talbert, 2001); and
• It takes three to eight years to successfully implement comprehensive school reform (Rand, 2004).

In particular, I would encourage readers to return to the Rand Change Agent Studies of the late 1970s, in which Paul Berman and colleagues shared what did not work for successful implementation:

• Reliance on outside consultants;
• Prepackaged management approaches; and
• One-shot, pre-implementation training.

Berman and colleagues also shared what did work:

• Concrete, teacher-specific, and extended training;
• Teacher participation in project decisions; and
• Local development of project materials.

If you only read one study
But perhaps more than anything else, I encourage today’s reformers and union leaders to read Arthur E. Wise’s 1984 study of effective teacher evaluation practices, which alerted policymakers and practitioners to four areas of focus:

• Organizational commitment: Investing in oversight to ensure fidelity;
• Evaluator competence: Including teachers as peer reviewers;
• Teacher-administrator collaboration: Focusing on joint ownership; and
• Strategic compatibility: Ensuring personalization over standardization of the reviews.

Dr. Wise’s study was released 30 years ago. He found that most school districts, like those today, are uncertain about which instrumentation to use, how frequently teachers should be assessed, and how data informs other elements of the system. Successful teacher evaluation systems begin with district leadership allowing time for implementation:

“With regard to commitment, all four case study districts recognize that the key obstacle to successful evaluation is time—or, more precisely the lack of it—for observing, conferring with, and especially assisting teachers who most need intensive help. (Successful) districts create time for evaluation.”

Recently, the NYC union fought for and won “teacher participation” in the evaluation process, and the district has doubled the amount of professional-development funds they send to schools that could be used to support local implementation. But the timeline is ungracious at best. There appears to be a gross neglect by the district of the need to take the time to scale their system of teacher evaluation.

Charlotte Danielson, whose teaching-observation framework was bought by the district, was reported as saying, "There aren't enough consultants in the world, or enough money in the world to pay the consultants, to do that training face to face in three months.”

But as the change agent studies of the 1970s and Wise’s evaluation study of the 1980s remind us, the implementation of complex school reforms requires less traditional training and more capacity-building at the school level so that those whose work is to be judged can use the evidence to improve their practice.

It seems that this has been a very hard lesson for today’s school reformers to learn. I wonder why?