Chatbot customization matters for skill retention

By Zia Hassan, School of Education, Johns Hopkins University

Schools across the nation are responding to emerging AI tools. Proponents of integrating generative AI (GenAI) tools like ChatGPT into the classroom argue that GenAI software can fine-tune its instruction to personalize learning, meeting each student where they are and building skills at the pace necessary for progress. Enthusiasm around AI’s potential for personalizing learning has led  schools in Texas and Florida to implement a program where each student must work with a GenAI tutor for two hours daily.

Researchers in Turkey conducted a randomized controlled trial with approximately 1,000 participants to evaluate how effectively GenAI supports the development of  independent skills. After a shared math lesson,  students practiced skills in one of three conditions: using a standard ChatGPT interface that was capable of providing answers (GPT base), using a pre-trained GPT tutor that was programmed to scaffold skills (GPT tutor),  or relying solely on class notes.

 The results showed that AI tools dramatically improved performance during  practice sessions and that students performed the best with the AI Tutor, which scaffolded skills without giving the correct answer (127% increase in skills for the GPT Tutor group vs. 47% increase for the GPT base group when compared to the control). However, when researchers gave the assessment and technology was removed, the GPT base group fared far worse than the control, receiving a reduction of 17%, while the GPT tutor group’s performance on the assessment was about the same as the control. 

These results indicate that if schools are to integrate generative AI tools into curricula, careful thought must be given to customizing general-purpose bots such as ChatGPT. If schools become too permissive with access to GenAI tools at critical learning moments, they could unintentionally cause a dramatic drop in transferrable skills.