A Status Report on Computer Scoring of Writing

In this article in Education Week Technology Counts, Caralee Adams reports on the progress being made with artificial-intelligence scoring of students’ written work. Georgia teacher Jeff Pence says that grading 140 essays from his seventh-grade English students used to be a two-week project, and that kept him from giving frequent writing assignments. This year he’s been experimenting with Pearson’s WriteToLearn program and has been pleasantly surprised. “It doesn’t tell them what to do, but it points out where issues may exist,” says Pence. “I feel it’s pretty accurate. Is it perfect? No. But when I reach the 67th essay, I’m not real accurate, either. As a team, we are pretty good.” Now he’s asking students to write a weekly essay and is able to give them helpful feedback.

Computer scoring of writing was first developed in the 1970s, and encountered heavy criticism for doing little more than counting words and word length. The technology took another step in the 1990s with the advent of the Internet and improved computer storage capacity. And in the last few years, programs have developed the capacity to evaluate language, grammar, mechanics, and style, detect plagiarism, and give students quantitative and qualitative feedback. In 2013, the Hewlett Foundation sponsored an open-source competition to spur innovation in the field, and commercial publishers and teams of scientists took part and shared ideas. “It’s a hot topic,” says David Williamson of Educational Testing Service. “There are a lot of researchers and academia and industry looking into this, and that’s a good thing.” The biggest challenges are coming up with a universal definition of good writing and programming computers to evaluate a writer’s “voice.” 

These are some of the companies, universities, and states using computer scoring of writing:

  • ETS has a program to score writing in the GMAT (Graduate Management Admission Test).
  • ETS has developed the Criterion Online Writing Evaluation System for grades 4-12.
  • West Virginia is using a CTB/McGraw-Hill program to score statewide reading language arts assessments in grades 3-11 (a comparability study showed that the software and a trained teacher performed better than two trained teachers).
  • ACT has developed Compass to score exams for community college placement.
  • Pearson has the WriteToLearn program.
  • Pearson also has a program to score GED tests for high-school diplomas.
  • Lightside has developed an open-source system that can find the strongest and weakest sections of writing and give students specific feedback. It is being piloted in schools in New York and Pennsylvania.
  • EdX has developed automated software to grade higher-education open-response questions. 

But a number of major testing initiatives are taking a wait-and-see approach. The College Board has yet to embrace computer scoring for SAT writing sections, and the ACT is also holding off for its college-entrance exams. Both the PARCC and Smarter Balanced consortia (which are developing tests for the Common Core) are assessing computer scoring of student writing to see if it meets their criteria. 

One outspoken skeptic is Les Perelman, the retired director of MIT’s Writing Across the Curriculum program. He believes the programs still rely too much on mechanical algorithms and can’t compete with human judgment. Perelman has proved his point by having students submit gibberish to computer scoring programs and get high scores. “The real danger of this is that it can really dumb down education,” he said. “It will make teachers teach students to write long, meaningless sentences and not care that much about actual content.”

“Automating Writing Evaluations” by Caralee Adams in Education Week Technology Counts, Mar. 13, 2014 (Vol. 33, #25, p. 13, 15), www.edweek.org 

From the Marshall Memo #528

 

Views: 71

Reply to This

FOLLOW SL 2.0

JOIN SL 2.0

SUBSCRIBE TO

SCHOOL LEADERSHIP 2.0

NOTE:  DURING THE MONTH OF MAY, AS HAS BEEN OUR TRADITION, 100% OF ALL NEW MEMBERSHIP FEES COLLECTED WILL BE DONATED TO CHARITY.

School Leadership 2.0 is the premier virtual learning community for school leaders from around the globe.  Our community is a subscription based paid service ($19.95/year)  which will provide school leaders with outstanding resources. Learn more about membership to this service by clicking one our links below.

 

Click HERE to subscribe as an individual.

 

Click HERE to learn about group membership (i.e. association, leadership teams)

CREATE AN EMPLOYER PROFILE AND GET JOB ALERTS AT 

SCHOOLLEADERSHIPJOBS.COM

SCHOOL LEADERSHIP 2.0 EVENTS

Amazon/SL 2.0 Book Store

School Leadership 2.0

© 2019   Created by William Brennan and Michael Keany   Powered by

Badges  |  Report an Issue  |  Terms of Service