CLEVELAND, Ohio - The school voucher programs that some federal and state officials want to expand have mixed test results in Ohio that make it unclear how much more students learn than if they had stayed in their local public schools.

Ohio's voucher programs, which give families grants to help pay tuition at private schools, have a low bar to clear to look successful.

Neither the state's main voucher program, EdChoice, and a Cleveland-only program are competing with high-scoring suburban districts. Both were created to let families avoid schools the state considered to be failing, so they only have to beat the lowest-rated schools.

But the private schools receiving voucher dollars have mixed results, even when compared to these "failing" public schools.

Voucher students do well on reading tests. Students in both the Cleveland and statewide EdChoice program score significantly better on all state reading tests between grades 3 and 8 than students at the public schools they left behind, state data shows.

But the private schools falter in math, with their voucher students scoring lower - sometimes by a hair, sometimes by a lot - on four of the six state math tests for the same grades.

It's a trend that has held for several years. The Plain Dealer reported that result for Cleveland in 2011, then the state found the same pattern both in Cleveland and statewide in 2015 and the last round of state tests for 2015-16 confirmed a similar pattern for Cleveland again.

See detailed charts below.

The state's analysis for 2015-16 is not yet available.

Further, a detailed analysis last year - one commissioned by a major pro-school choice group - suggests that voucher students in Ohio score worse in math and English than if they had just stayed in their public school.  

It's a very nuanced and technical study that avoids looking at a key group of public schools - the very worst-performing ones - for statistical validity reasons. Northwestern University professor David Figlio also doesn't quantify how much ground students lose by going to voucher schools in terms that make sense to the general public.

David Figlio discusses his voucher study at the City Club last year.City Club of Cleveland 

But Figlio found that while voucher students were typically better-off financially and stronger academically than students they left behind, they did worse after going to private schools than comparable students that stayed in public school.

"The kids that were going to the private schools were doing worse," he said. "The kids who participated in the program did considerably worse compared to the closest-looking kids in similar schools."

The study was commissioned by the Fordham Institute, a leading advocate for school choice in Ohio and nationally.

"Even when its findings are unexpected and painful, rigorous, disinterested evaluation remains the best way to prod improvements and make progress toward the program's goals," Fordham's Ohio director Chad Aldis wrote in the foreword to Figlio's report.

Click here to see Figlio discussing the study at the City Club of Cleveland.

Value of vouchers up for debate

The effectiveness of school voucher programs is already part of a national debate, with President Donald Trump and Betsy DeVos, his secretary of Education, proposing spending an extra $1 billion on school choice efforts.

Vouchers will also be up for debate soon here in Ohio, with a bill from State Sen. Matt Huffman to expand Ohio's voucher programs that just started having its first hearings in Columbus.

With the test score data mixed, Huffman and other voucher supporters are not basing their case for expanding the programs on academic grounds. Huffman says the value of private schools for families is not all about earning better marks on tests.

"I don't think the only measure of the success of a school is the tests a student takes," said Huffman.

He said things like student behavior, academic programs, sports teams and other extra-curriculars, and even distance from home are all "other better positive attributes of the environment that a child is in."

Much like DeVos, Huffman wants parents to be able to make their own choice.

"Let the marketplace work," he said.

Catholic school supporters, who have a lot at stake, agreed with much of Huffman's position. Religious schools receive 97 percent of Ohio's voucher dollars, of which all but 2 percent goes to Christian schools. Catholic schools dominate the recipients, but exact percentages are not clear.

The Catholic Diocese of Cleveland declined to address why Catholic schools in Cleveland would do so well in reading compared to the Cleveland schools, but worse in math.

"Catholic schools do not base a student's performance solely on test scores but take a holistic approach to a student's education and formation," the diocese said in a written statement.

The Catholic Conference of Ohio, a statewide association of bishops, seconded the statement from the diocese.

Is education under a voucher better, or just different?

Cleveland schools CEO Eric Gordon, whose district is the most affected by vouchers in Ohio, said he supports school choice efforts like vouchers and charter schools, but only if they help students have better learning experiences, not just different ones.

"If there's a pretty high assurance that my kids are getting a better education somewhere else, I can't in good conscience oppose that," Gordon said.

But he said the data doesn't show that voucher schools do any better. Though Catholic schools encourage a "toxic nostalgia" that they are better than public schools, Gordon said, "The data doesn't support it."

Just having a different experience in school, he said, is not enough to "gamble again, spend lots and lots of dollars and not move the needle for children."

"That's my bottom line," Gordon said.

Why the data is limited

State law requires all K-8 voucher students to take the same state math and English tests as public school students, even if other students at their private schools do not. That provides some reasonable data for the comparisons we shared above.

There are limitations, though.

-       High school comparisons are more complicated, thanks to admissions tests, new state tests for high school students and private schools lobbying the state to avoid the new high school tests.

Click here for more on the high schools.

-       Since voucher programs are limited to students in Cleveland or students who attend "failing" schools in other districts, comparisons of voucher schools to suburban districts are hard. There just are not a lot of students from many suburban schools that use vouchers .

-       Some students also receive vouchers because of low family income, but their scores are just starting to show up in test scores at younger grades.

And, as Gordon noted, many report card measures or key aspects of schools are not counted by the state for voucher schools.

 -        "Performance Index," the composite of test scores across multiple subjects and grades the state uses as its catch-all measure of how much students know.

-        "Value-added," a measure that highlights how much academic progress students make in a year. Value-added can show if a school is doing a good job with students  by catching up those who are behind their peers, or just coasting with advanced students.

-       Socio-economic challenges like poverty and income that have a strong relationship to test scores across Ohio and nationally.

-       The number of special education students or students learning English as a second language at a school.

-       Student attendance rates and disciplinary statistics, like the number of suspensions each year.

-       And, for high schools, the graduation rate at each school.

For the Cleveland school district's Gordon, all of those factors help show if a school is good or not. But he is frustrated that parents can't see that hard data from voucher schools.

He praised Huffman's bill for including a value-added calculation for voucher schools, just in a very broad way. His bill would call for a single calculation for all voucher students in Ohio, not for each school.

Gordon called that a "step in the right direction," but wants the state to go further to show real comparisons between schools.

"Those are all great sources," he said. "Let's build a data system that looks at it."

Cleveland voucher students' scores vs. the district

Here ia a comparison of the proficiency rates of K-8 Cleveland voucher students to Cleveland school district students on state tests for the 2015-16 school year. Voucher students had higher scores on all reading tests, but district students scored better on four of the six math tests shown here.

How EdChoice students scored vs students in the "failing" schools they left

Here are state calculations from the 2013-14 school year of how students in the statewide EdChoice voucher program performed on state tests compared to students in the public schools they would have attended, all labeled as failing by the state.

Note that 5th and 6th grade math scores are slightly higher for district schools, but are tied when rounded off.