Assessing the quality of feedback from humans and ChatGPT on students’ writing

By Feifei Wang, The Chinese University of Hong Kong

Automated writing evaluation (AWE) tools aid educators by quickly evaluating student writing, but often at the expense of accuracy and clarity, and setting them up requires substantial effort and resources. In contrast, generative AI like ChatGPT provides timely, targeted, and adaptive feedback that can enhance student writing. A recent study compared the quality of feedback from ChatGPT and human raters, with potential implications for process-based writing evaluation.

The study involved 200 students randomly sampled across grades 6-12 from two school districts in Southern California. Over two 50-minute class periods, students were tasked with responding to one of two writing prompts. Each prompt demanded that they read four primary and secondary sources and craft an argument analyzing historical interpretations using evidence, reasoning, and required writing skills. Sixteen experienced secondary educators from various disciplines were recruited and trained to provide formative feedback. This feedback was then compared to ChatGPT’s feedback, which was prompted to reflect the same context used by the educators. The researchers coded and analyzed the feedback for comparison.

The findings revealed  that well-trained, paid, and relatively time-rich human raters offered higher quality feedback in four of five key areas: clarity of improvement directions, accuracy, prioritization of essential features, and use of a supportive tone (the fifth area: criteria-based feedback). However,  ChatGPT’s writing feedback closely matched  human feedbac..., and the differences in quality between ChatGPT and human feedback  were modest, considering both overall quality and time savings. The researchers concluded that generative AI could be a useful tool in specific situations, especially for formative drafts or when highly-trained educators are unavailable.