Kids who use ChatGPT as a study assistant do worse on tests

Researchers compare math progress of almost 1,000 high school students

by Jill Barshay

Hechinger Report

September 2, 2024

Summary of "Kids Who Use ChatGPT as a Study Assistant Do Worse on Tests" by Jill Barshay

A recent study conducted by researchers at the University of Pennsylvania presents a cautionary tale about the use of AI tools like ChatGPT in education. The study focused on Turkish high school students and their performance in math when using ChatGPT as a study assistant. The findings suggest that while students who had access to ChatGPT solved more practice problems correctly, they performed worse on subsequent tests compared to students who did not use the AI tool. This raises concerns about the potential negative impact of relying on AI for learning.

The Experiment

The study involved nearly 1,000 students from grades nine through eleven. The students were divided into three groups, each practicing math in a different way: with ChatGPT, with an AI tutor version of ChatGPT, or without any AI assistance. After practicing, the students were tested on their understanding of the material. The experiment was repeated across four different math topics to assess the effectiveness of AI in learning.

Findings and Implications

Students who used ChatGPT as a study tool solved 48% more practice problems correctly than those who did not use AI. However, they scored 17% lower on the math tests. This paradoxical outcome suggests that while ChatGPT might help students get the right answers during practice, it does not necessarily enhance their understanding or retention of the material.

A third group of students used a version of ChatGPT designed to function as a tutor, offering hints rather than direct answers. These students performed even better on practice problems, solving 127% more correctly. Despite this, their test performance did not improve compared to students who practiced without AI. This suggests that even when AI is used in a more educationally supportive manner, it may not translate to better learning outcomes.

Understanding the Problem

The researchers believe that the main issue lies in students using ChatGPT as a "crutch." Instead of developing problem-solving skills, students relied on the AI to provide answers, which inhibited their learning. This over-reliance on AI may also have led to overconfidence. Surveys conducted alongside the experiment revealed that students did not perceive ChatGPT as detrimental to their learning, even though their test scores indicated otherwise.

Another factor contributing to the poor test performance could be ChatGPT's inaccuracies. The chatbot answered math problems correctly only 50% of the time, with 8% of its arithmetic computations being incorrect and 42% of its step-by-step problem-solving methods being flawed. The AI tutor version of ChatGPT was fed correct solutions to minimize errors, but even this did not lead to improved test results.

Broader Implications

The study’s findings suggest that, like other technologies in education, ChatGPT presents a tradeoff. While it can help students get correct answers in the short term, it may not contribute to long-term learning and understanding. The researchers likened the issue to the overreliance on autopilot in aviation, where pilots need to maintain their skills even when technology is available. In the context of education, students need to develop their problem-solving abilities rather than relying on AI to do the work for them.

This research highlights the importance of cautious integration of AI tools in education, ensuring that they support rather than hinder learning.

For the full article, visit https://hechingerreport.org.

Original Article

------------------------------

Prepared with the assistance of AI software

OpenAI. (2024). ChatGPT (4) [Large language model]. https://chat.openai.com