A Status Report on Computer Scoring of Writing

In this article in Education Week Technology Counts, Caralee Adams reports on the progress being made with artificial-intelligence scoring of students’ written work. Georgia teacher Jeff Pence says that grading 140 essays from his seventh-grade English students used to be a two-week project, and that kept him from giving frequent writing assignments. This year he’s been experimenting with Pearson’s WriteToLearn program and has been pleasantly surprised. “It doesn’t tell them what to do, but it points out where issues may exist,” says Pence. “I feel it’s pretty accurate. Is it perfect? No. But when I reach the 67^th essay, I’m not real accurate, either. As a team, we are pretty good.” Now he’s asking students to write a weekly essay and is able to give them helpful feedback.

Computer scoring of writing was first developed in the 1970s, and encountered heavy criticism for doing little more than counting words and word length. The technology took another step in the 1990s with the advent of the Internet and improved computer storage capacity. And in the last few years, programs have developed the capacity to evaluate language, grammar, mechanics, and style, detect plagiarism, and give students quantitative and qualitative feedback. In 2013, the Hewlett Foundation sponsored an open-source competition to spur innovation in the field, and commercial publishers and teams of scientists took part and shared ideas. “It’s a hot topic,” says David Williamson of Educational Testing Service. “There are a lot of researchers and academia and industry looking into this, and that’s a good thing.” The biggest challenges are coming up with a universal definition of good writing and programming computers to evaluate a writer’s “voice.” 

These are some of the companies, universities, and states using computer scoring of writing:

• ETS has a program to score writing in the GMAT (Graduate Management Admission Test).
• ETS has developed the Criterion Online Writing Evaluation System for grades 4-12.
• West Virginia is using a CTB/McGraw-Hill program to score statewide reading language arts assessments in grades 3-11 (a comparability study showed that the software and a trained teacher performed better than two trained teachers).
• ACT has developed Compass to score exams for community college placement.
• Pearson has the WriteToLearn program.
• Pearson also has a program to score GED tests for high-school diplomas.
• Lightside has developed an open-source system that can find the strongest and weakest sections of writing and give students specific feedback. It is being piloted in schools in New York and Pennsylvania.
• EdX has developed automated software to grade higher-education open-response questions. 

But a number of major testing initiatives are taking a wait-and-see approach. The College Board has yet to embrace computer scoring for SAT writing sections, and the ACT is also holding off for its college-entrance exams. Both the PARCC and Smarter Balanced consortia (which are developing tests for the Common Core) are assessing computer scoring of student writing to see if it meets their criteria. 

One outspoken skeptic is Les Perelman, the retired director of MIT’s Writing Across the Curriculum program. He believes the programs still rely too much on mechanical algorithms and can’t compete with human judgment. Perelman has proved his point by having students submit gibberish to computer scoring programs and get high scores. “The real danger of this is that it can really dumb down education,” he said. “It will make teachers teach students to write long, meaningless sentences and not care that much about actual content.”

“Automating Writing Evaluations” by Caralee Adams in Education Week Technology Counts, Mar. 13, 2014 (Vol. 33, #25, p. 13, 15), www.edweek.org 

From the Marshall Memo #528