Here is a third post in a debate on The Answer Sheet about international test scores and whether they tell us anything important about the U.S. public education system.

The conversation began with a post I wrote last week titled “The fetishization of international test scores” which looked to the upcoming release of 2012 PISA test scores on Dec. 3 and said we place too much attention on these scores. A few days ago I published a dissent from Marc Tucker, resident of the non-profit National Center on Education and the Economy. Both pieces mentioned a report released early this year by Martin Carnoy, education professor at  Stanford University’s Graduate School of Education, Richard Rothstein, research associate at the Economic Policy Institute, which raised questions about whether the average scores in the 2009 PISA were reported lower than they should have been.

In this new post, Carnoy and Rothstein take issue with Tucker’s characterization of their report and further the discussion about international test scores. It appeared on the website of the Economic Policy Institute,  a non-profit organization created to broaden the discussion about economic policy to include the interests of low- and middle-income workers.

By Martin Carnoy and Richard Rothstein

Earlier this year, we published an analysis of international test score data in which we showed that these data hide many complex issues, and that glib conclusions regarding the meaning and policy implications of international test data should be avoided. We showed that it is more appropriate to compare student performance across countries by comparing students with similar social class backgrounds, and we showed that comparative information is more useful if it includes test data trends over time as well as levels in the current year. We also presented apparent anomalies in test data (for example, periods in which performance on one international test goes up but performance on another international test, purporting to measure the same subject, goes down, or carelessness in sampling methodology) that should caution analysts from relying too heavily on test score data.

Upon the release of our report, we were attacked by several promoters of the conventional idea that international test data show that American schools are in collapse and are threatening our economic security. Prominent among these was Marc Tucker, president of one of the leading education-scold organizations, the National Center on Education and the Economy. Tucker attacked our report without having bothered to read it, and was subsequently forced to issue an apology for misrepresenting our findings (“We misstated the conclusions presented by Martin Carnoy and Richard Rothstein in the report described in this newsletter. We believe we have stated those conclusions accurately here, and apologize to the authors for the error.”).

Now, two weeks before release of new international test score data, he has again attacked our earlier report, again based on his own misrepresentations of what the report actually says. The occasion of his current critique is Valerie Strauss’ Answer Sheet blog.

To be precise, his first critique is not of what our report says, but of what he claims the report “comes close to saying,” or “suggests.” Referring to the Program for International Student Assessment, Tucker writes:

Carnoy and Rothstein argue that the scores of American students on PISA should be corrected for the unusually large proportion of students from low-income students in the sample. They come close to saying that, if you just took those students out, we would perform very well.

And, he adds:

So it is not the case that, if you took the students from low-income families out of the calculation, the United States would be a top performer, as Carnoy and Rothstein suggest.

In fact, our report says nothing of the sort. It doesn’t even “come close” to saying that and does not “suggest” it. Indeed, it illustrates the opposite. For reading, what the report showed is that if you adjusted U.S. test scores for our unusually large proportion of disadvantaged students (we make no reference whatsoever to “low-income” students and refer only to students from families with few books in their homes, a proxy for academic resources—something correlated with income, but not totally so), and re-calculated U.S. scores as though our social class distribution was similar to that of the top-scoring countries of Canada, Finland, and Korea, U.S. students would still not perform very well, but the gap between U.S. average scores and those of the three top scoring countries would be narrower. We also show that with respect to similar post-industrial countries, after standardizing for social class distribution, U.S. students’ average performance is mostly superior. You can see this conclusion if you look at Table 3A (page 14), and the surrounding discussion, of our report.

For math, we likewise show that the gap would be narrower between the U.S. and top scoring countries after standardizing for social class distribution, but the U.S. average would still be substantially below the average of the top-scoring countries, and still below the average of similar post-industrial countries. You can see this conclusion if you look at Table 3C (page 16), and the surrounding discussion, of our report.

What about “coming close to saying” that if you got rid of our disadvantaged students, we would perform very well? Our tables, and surrounding discussion, clearly show that in both reading and math, the most advantaged U.S. students perform less well than the most advantaged students in the top scoring countries. You can see this conclusion if you look at Table 5 (page 22), and the surrounding discussion, of our report. We also show that with respect to similar post-industrial countries, in reading, the most advantaged U.S. students perform better than students in Germany, a little better than students in the U.K., and a little worse than students in France. In math the most advantaged U.S. students perform about the same as students in the U.K., but less well than students in France and Germany. You can see these conclusions if you look at Table 4 (page 22), and the surrounding discussion, of our report.

Unlike Tucker, we don’t consider the fact that our most advantaged students are not internationally top performers to be a harbinger of economic crisis, but it is difficult to have a rational discussion of the relationship of test scores to economic competitiveness with so-called experts who refuse to read what we say but nonetheless feel free to attack it.

Tucker adds:

Carnoy and Rothstein would have you believe that our schools are doing the best that could be done by anyone, but they are just being overwhelmed by kids from low-income backgrounds.

Note that Tucker doesn’t actually accuse us of saying this, but only of “having you believe it.” In fact, we say nothing of the sort. What we would actually have you believe is what we actually say: that rapid improvements over time in the performance of disadvantaged U.S. students but less rapid improvements for advantaged students, means that policymakers should place more emphasis on school improvements for middle-class and advantaged students, and not focus almost all of their improvement efforts on disadvantaged students (see discussion on page 7 of our report).

Next, Tucker observes that an argument he has (falsely) attributed to us “is a very odd argument. No one is going to give the United States a pass in the global economy because so many of our students come from low-income families.” This is a very odd critique. We never suggested that anyone is “going to give the United States a pass in the global economy.” In fact, our report makes no claims whatsoever about the economic implications of international test score comparisons. What we do believe, however (see the discussion on page 7 of our report), is that if one (not the only) reason for relatively low U.S. test scores is that we have an unusually large percentage of disadvantaged children, and if disadvantaged children in every country perform less well than advantaged children (more about this below), then we would be wise to spend more effort worrying about remedying the disadvantage and less time faulting schools for failing to overcome problems over which they have no control. If one (not the only) reason for relatively low U.S. test scores is that too many children come from homes where they are not read to, have exposure to less sophisticated vocabulary, gain less background knowledge that is the foundation for literacy, and as a result come to school not ready to learn, it might make sense to provide children with high quality early childhood care and education, and with high quality after school and summer experiences, so they can come to school with the pre-literacy and background knowledge that will leave them more ready to learn.

Tucker then goes on to make some astonishing misrepresentations of the data:

The fact that socio-economic background is a much better predictor of academic success in the United States than in most countries in the PISA sample shows that our schools do less than schools in other countries to lift students from low-income families out of poverty.

This is no fact, but fiction. In our report, we calculate the difference in performance between more advantaged and less advantaged children in the seven countries for which we did detailed analysis: Canada, Finland, and Korea (three top-scoring countries), France, Germany, and the United Kingdom (three post-industrial countries that are similar to the United States), and the United States. We show that there is a test score gap between advantaged and disadvantaged children in each country, that the gap in the U.S. is smaller than the gap in the three post-industrial countries, and that it is only slightly larger than the gap in the three top-scoring countries (and, in a curious anomaly, the reading gap between the most advantaged and least advantaged students is substantially smaller in the U.S. than in Korea). For the data, see Tables 4 and 5 on page 22, and the surrounding discussion, of our report. As we said earlier, we don’t agree with Tucker that this is a harbinger of economic crisis, but it is difficult to have a rational discussion of the relationship of test scores to economic competitiveness with so-called experts who refuse to read what we say but nonetheless feel free to attack it.

Tucker then writes:  

Since the 1970s, access to education in the United States and improvements in the quality of education came to a crashing halt. Both have been flat for 40 years.

This is breathtaking fiction. As one of us has written previously, based on census data, with regard to access to education, in the last four decades, the percentage of college graduates in the United States has nearly doubled. In 1970, 16 percent of young adults (ages 25 to 29) were college graduates. Today, it is 31 percent. The improvement has been across the board: the share of African-American young adults who are college graduates has gone from 10 percent to 19 percent; for whites it has gone from 17 percent to 37 percent. A “crashing halt”? Can we and should we do better? The answer to “can we do better” is “probably” and to “should we do better” is “yes.” But we will never figure out how to do better if those who promote themselves as education experts, and whom the media anoint as such, have so little regard for facts. No social problem can be addressed unless we understand it accurately. Implementing policies without regard to the facts could easily result in destroying whatever we have been doing that has resulted in the improvements we’ve actually accomplished.

What about the quality of education? Here, too, the facts are straightforward. As our report shows (see Tables 20 and 21 on pages 58-59) with further detail in a report one of us previously issued, the only longer term (longitudinal) measure of student achievement that is available is the National Assessment of Educational Progress (NAEP). NAEP provides trends for 4th, 8th, and 12th graders, disaggregated by race, ethnicity, and poverty, since the 1970s in basic skills in math and reading (called the “Long Term Trend NAEP”) and since about 1990 for 4th and 8th graders in slightly more sophisticated math and reading skills (called the “Main NAEP”). On these exams, American students have improved substantially, in some cases phenomenally. In general, the improvements have been greatest for African-American students, and among these, for the most disadvantaged. The improvements have been greatest for both black and white 4th and 8th graders in math. Improvements have been less great but still substantial for black 4th and 8th graders in reading and for black 12th graders in both math and reading. Improvements have been modest for whites in 12th grade math and at all three grade levels in reading. There is no way to interpret these trends as a “crashing halt.” Again, should we do better? Yes. Can we expect to do better if we refuse to understand how we have been doing? Unlikely.

Finally, there is the straw-man dodge. Tucker says:

Anyone who believes in this day and age that national education performance is irrelevant to national economic performance is whistling Dixie.

Not being whistlers, we certainly believe that national education performance is relevant to national economic performance. But we are not convinced that it is the most important relevant factor. Our view is that our education system is probably sufficient, or close to being sufficient, for successful national economic performance. It is likely that improved educational outcomes could improve national economic performance somewhat. But there are many, many other factors that are as or more relevant to our national economic performance—a macro-economic policy that is insufficiently stimulative, a decaying infrastructure, tax policies that reward speculation more than production, an over-extended military, declining investment in research, a wasteful and inefficient health care system, and many others.

In her blog, Valerie Strauss observed: “Public schools have long been blamed when something happens to challenge the country’s standing in the world.” She might have added that one of the most outspoken long-time blamers has been Marc Tucker.

In a June 2007 article in Phi Delta Kappan, Lawrence Mishel, along with one of us, discussed Tucker’s 1990 report, America’s Choice: High Skills or Low Wages (Commission on the Skills of the American Workforce). In his report, Tucker’s commission bemoaned what it considered disastrous American school quality in the 1970s and 1980s that threatened our national economic future. Our response noted Tucker’s charge that inadequate skills, which resulted from flawed schools, had caused industrial productivity to “slow to a crawl” and would, without radical school reform, lead to a condition of permanently low wages. Yet within a few years of Tucker’s complaint, and about 20 years after the deteriorated school outputs that Tucker denounced, Americans’ ability to master technological change generated an extraordinary leap in productivity. This acceleration, exceeding that of other advanced countries, was accomplished by the very same work force that Tucker’s commission said imperiled our future. And it created new wealth that could have supported a steady increase in Americans’ standard of living. That the fruits of American productivity growth have been distributed to the wealthiest of Americans, mostly in the form of profits, is a scandal that can hardly be attributed to our elementary and secondary schools.