Can Software Grade Students’ Essays?

In this front-page, above-the-fold story in The New York Times, John Markoff reports on a new software program that can, say its developers at EdX, use artificial intelligence to grade student essays and short written answers almost instantaneously – and then allow students to rewrite their essays and resubmit them in hopes of a better grade. Before instant grading can take place, the teacher or professor must grade 100 essays to “teach” the software, which can then grade any number of subsequent papers and give general feedback – for example, if the answer was on topic or not. 

EdX is a nonprofit online course provider founded by Harvard and MIT, and it plans to make the grading software available free on the Internet for any educational institution. “There is huge value in learning with instant feedback,” says Anant Agarwal, the president of EdX. “Students are telling us they learn much better with instant feedback.” He says the software is getting close to what a human grader can do: “We found that the quality of the grading is similar to the variation you find from instructor to instructor.”

Skepticism abounds. “My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” says MIT researcher Les Perelman. In the past, he has submitted nonsense essays and tricked software grading programs into giving high marks. He is part of a group called Professionals Against Machine Scoring of Student Essays in High-Stakes Assessment, which has collected nearly 2,000 signatures. The group’s brochure reads in part, “Computers cannot ‘read.’ They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others.”
Supporters of essay-grading software admit that it’s not perfect but still make the case. Referring to the skeptics, Mark Shermis (University of Akron in Ohio) says, “Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could. There seems to be a lack of appreciation of what is actually going on in the real world.” 

“Software Seen Giving Grades on Essay Tests” by John Markoff in The New York Times, Apr. 5, 2013 (p. 1, A11), http://nyti.ms/10FgUkF 

From the Marshall Memo #480