“All the Research Says…” – but Does It Really?

From the Marshall Memo #428

In this helpful American School Board Journal column, author/consultant Douglas Reeves has advice on being a critical consumer of educational research: 

• Independence – There should be five levels, says Reeves: (a) Ideally, the conclusions should come from several different researchers who are genuinely independent from one another, even competing; (b) conclusions are more compelling when they come from different samples and not “samples of convenience” – schools, students, and teachers who happened to be available; (c) it’s ideal if different researchers used different methods – quantitative, qualitative, meta-analyses; (d) conclusions are more robust if they come from different venues – east coast/west coast, urban/suburban/rural, etc.; (e) Data should come from different student populations – high and low poverty, ELLs, etc. – so the conclusions stem from teaching and leadership, not demographics.

• Precise terminology – A number of labels – “professional learning communities”, “differentiated instruction”, “response to intervention” – are used loosely among educators and researchers. “When evaluating competing research claims,” says Reeves, “leaders and policymakers must distinguish such labels from actual implementation.”

• Candor – “Beware of the researcher who is always right,” says Reeves. “Every credible researcher – bar none – will have no difficulty in finding examples of misguided hypotheses and misdirected conclusions. The best researchers will have published those failures and acknowledged them in front of peers and the public.” 

• Replication – “Research is littered with personal narratives of the heroic teacher or principal who succeeded against the odds,” says Reeves. But it’s convincing only if the approach has been used successfully in different places and different conditions.  

“Research Shows” by Douglas Reeves in American School Board Journal, April 2012 (Vol. 199, #4, p. 36-37), http://www.asbj.com; Reeves is at dreeves@leadandlearn.com.