A new study by researchers at MIT, Harvard, and Brown cast doubt on the value of pursuing higher scores on standardized tests as an end in themselves.

Since this has been the highest goal of federal policy since 2002, when No Child Left Behind was signed into law, the study raises questions about the billions spent on testing, test preparation, evaluating teachers and schools by test scores, firing teachers and principals because of test scores, and closing schools based on test scores.

Are test scores the Golden Fleece? No.

Yet with the release of every NAEP test or every international test, the media go into a frenzy, and Arne Duncan leads a national day of high anxiety and breast beating about our nation’s imminent peril because test scores did not rise as much as they should.

The new study raises the question of how much those standardized test scores mean.

The study found:

In a study of nearly 1,400 eighth-graders in the Boston public school system, the researchers found that some schools have successfully raised their students’ scores on the Massachusetts Comprehensive Assessment System (MCAS). However, those schools had almost no effect on students’ performance on tests of fluid intelligence skills, such as working memory capacity, speed of information processing, and ability to solve abstract problems.

The researchers calculated how much of the variation in MCAS scores was due to the school that students attended. For MCAS scores in English, schools accounted for 24 percent of the variation, and they accounted for 34 percent of the math MCAS variation. However, the schools accounted for very little of the variation in fluid cognitive skills — less than 3 percent for all three skills combined.
…
Even stronger evidence came from a comparison of about 200 students who had entered a lottery for admittance to a handful of Boston’s oversubscribed charter schools, many of which achieve strong improvement in MCAS scores. The researchers found that students who were randomly selected to attend high-performing charter schools did significantly better on the math MCAS than those who were not chosen, but there was no corresponding increase in fluid intelligence scores.

U.S. News describes “fluid intelligence”:

Those skills are described as fluid because they require using logical thinking and problem solving in novel situations, rather than recalling previously learned facts and skills.

“It doesn’t seem like you get these skills for free in the way that you might hope, just by doing a lot of studying and being a good student,” said the study’s senior author, John Gabrieli, in a statement.

What improving test scores does do, Gabrieli said, is raise students’ “crystallized intelligence” – the ability to access information from long-term memory to use acquired knowledge and skills.

The importance – or lack thereof – of standardized tests has been widely debated by educators and state policymakers. While some argue that testing is important to track students’ performance and progress, others say there is a culture of over-testing in the United States.

He added that “crystallized intelligence” is important, but should not be the only goal of schooling:

But Gabrieli, a professor of brain and cognitive sciences at MIT, said improving crystallized skills – such as recalling previously learned facts – is still important, and that the findings should also be used to push educational policymakers to add practices that help enhance cognitive abilities as well.

“It’s valuable to push up the crystallized abilities, because if you can do more math, if you can read a paragraph and answer comprehension questions, all those things are positive,” he said in the statement. “Schools can improve crystallized abilities, and now it might be a priority to see if there are some methods for enhancing the fluid ones as well.”

“Fluid intelligence” seems to be the “higher-order thinking skills” that policymakers claim to value. But they are not measured by standardized tests.