Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent
Saved in:
| Title: | Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent |
|---|---|
| Language: | English |
| Authors: | Teresa M. Ober, Shan Zhang, Diego Zapata-Rivera, Noah L. Schroeder, Anthony F. Botelho |
| Source: | Journal of Educational Data Mining. 2026 18(1):208-243. |
| Availability: | International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM |
| Peer Reviewed: | Y |
| Page Count: | 36 |
| Publication Date: | 2026 |
| Sponsoring Agency: | National Science Foundation (NSF) Institute of Education Sciences (ED) |
| Contract Number: | 2229612 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Junior High Schools Middle Schools Secondary Education |
| Descriptors: | Artificial Intelligence, Natural Language Processing, Academic Persistence, Dialogs (Language), Technology Uses in Education, Middle School Students, Middle School Mathematics, English Learners, Student Interests, Self Efficacy, Prior Learning |
| ISSN: | 2157-2100 |
| Abstract: | Conversational learning systems offer new opportunities to examine learning processes through chat log data. Constructs such as persistence, self-efficacy, interest, perceived challenge, and prior knowledge are known predictors of student performance but are challenging to detect at scale using traditional methods. This study explores the use of Large Language Models (LLMs) to automatically code indicators of these constructs from student chat logs collected through a conversation-based assessment (CBA) for middle school mathematics. Indicators included observable behaviors such as students' expressions of challenge, help-seeking, goal-setting, and self-regulatory strategies evident in their conversational interactions within the CBA. We evaluated multiple configurations of ChatGPT4o, varying temperature settings (0, 0.3, 0.7, 1) and model types (mini vs. regular), against human expert coders. The dataset comprised over 10,000 student turns collected from 107 middle school students classified as English learners as they interact with the CBA. Reliability was assessed within and between LLM configurations and humans. Results reveal systematic patterns: constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings. Self-efficacy showed the highest human-LLM alignment. These findings illustrate the challenges of measuring complex psychological constructs and highlight the promise of human-LLM collaboration to enhance qualitative coding efficiency and validity in educational research. Supplemental materials are available online here: https://doi.org/10.17605/osf.io/s85ck. |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2026 |
| Accession Number: | EJ1506384 |
| Database: | ERIC |
Be the first to leave a comment!
