View in EDS

Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent

Saved in:
Bibliographic Details
Title: Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent
Language: English
Authors: Teresa M. Ober, Shan Zhang, Diego Zapata-Rivera, Noah L. Schroeder, Anthony F. Botelho
Source: Journal of Educational Data Mining. 2026 18(1):208-243.
Availability: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
Peer Reviewed: Y
Page Count: 36
Publication Date: 2026
Sponsoring Agency: National Science Foundation (NSF)
Institute of Education Sciences (ED)
Contract Number: 2229612
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Artificial Intelligence, Natural Language Processing, Academic Persistence, Dialogs (Language), Technology Uses in Education, Middle School Students, Middle School Mathematics, English Learners, Student Interests, Self Efficacy, Prior Learning
ISSN: 2157-2100
Abstract: Conversational learning systems offer new opportunities to examine learning processes through chat log data. Constructs such as persistence, self-efficacy, interest, perceived challenge, and prior knowledge are known predictors of student performance but are challenging to detect at scale using traditional methods. This study explores the use of Large Language Models (LLMs) to automatically code indicators of these constructs from student chat logs collected through a conversation-based assessment (CBA) for middle school mathematics. Indicators included observable behaviors such as students' expressions of challenge, help-seeking, goal-setting, and self-regulatory strategies evident in their conversational interactions within the CBA. We evaluated multiple configurations of ChatGPT4o, varying temperature settings (0, 0.3, 0.7, 1) and model types (mini vs. regular), against human expert coders. The dataset comprised over 10,000 student turns collected from 107 middle school students classified as English learners as they interact with the CBA. Reliability was assessed within and between LLM configurations and humans. Results reveal systematic patterns: constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings. Self-efficacy showed the highest human-LLM alignment. These findings illustrate the challenges of measuring complex psychological constructs and highlight the promise of human-LLM collaboration to enhance qualitative coding efficiency and validity in educational research. Supplemental materials are available online here: https://doi.org/10.17605/osf.io/s85ck.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2026
Accession Number: EJ1506384
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1506384
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1506384
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Teresa+M%2E+Ober%22">Teresa M. Ober</searchLink><br /><searchLink fieldCode="AR" term="%22Shan+Zhang%22">Shan Zhang</searchLink><br /><searchLink fieldCode="AR" term="%22Diego+Zapata-Rivera%22">Diego Zapata-Rivera</searchLink><br /><searchLink fieldCode="AR" term="%22Noah+L%2E+Schroeder%22">Noah L. Schroeder</searchLink><br /><searchLink fieldCode="AR" term="%22Anthony+F%2E+Botelho%22">Anthony F. Botelho</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Data+Mining%22"><i>Journal of Educational Data Mining</i></searchLink>. 2026 18(1):208-243.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 36
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)<br />Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2229612
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Persistence%22">Academic Persistence</searchLink><br /><searchLink fieldCode="DE" term="%22Dialogs+%28Language%29%22">Dialogs (Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Mathematics%22">Middle School Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22English+Learners%22">English Learners</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Interests%22">Student Interests</searchLink><br /><searchLink fieldCode="DE" term="%22Self+Efficacy%22">Self Efficacy</searchLink><br /><searchLink fieldCode="DE" term="%22Prior+Learning%22">Prior Learning</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2157-2100
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Conversational learning systems offer new opportunities to examine learning processes through chat log data. Constructs such as persistence, self-efficacy, interest, perceived challenge, and prior knowledge are known predictors of student performance but are challenging to detect at scale using traditional methods. This study explores the use of Large Language Models (LLMs) to automatically code indicators of these constructs from student chat logs collected through a conversation-based assessment (CBA) for middle school mathematics. Indicators included observable behaviors such as students' expressions of challenge, help-seeking, goal-setting, and self-regulatory strategies evident in their conversational interactions within the CBA. We evaluated multiple configurations of ChatGPT4o, varying temperature settings (0, 0.3, 0.7, 1) and model types (mini vs. regular), against human expert coders. The dataset comprised over 10,000 student turns collected from 107 middle school students classified as English learners as they interact with the CBA. Reliability was assessed within and between LLM configurations and humans. Results reveal systematic patterns: constructs with moderate theoretical coherence benefited from higher temperatures, while well-defined constructs required deterministic settings. Self-efficacy showed the highest human-LLM alignment. These findings illustrate the challenges of measuring complex psychological constructs and highlight the promise of human-LLM collaboration to enhance qualitative coding efficiency and validity in educational research. Supplemental materials are available online here: https://doi.org/10.17605/osf.io/s85ck.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1506384
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1506384
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 36
        StartPage: 208
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Academic Persistence
        Type: general
      – SubjectFull: Dialogs (Language)
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Middle School Mathematics
        Type: general
      – SubjectFull: English Learners
        Type: general
      – SubjectFull: Student Interests
        Type: general
      – SubjectFull: Self Efficacy
        Type: general
      – SubjectFull: Prior Learning
        Type: general
    Titles:
      – TitleFull: Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Teresa M. Ober
      – PersonEntity:
          Name:
            NameFull: Shan Zhang
      – PersonEntity:
          Name:
            NameFull: Diego Zapata-Rivera
      – PersonEntity:
          Name:
            NameFull: Noah L. Schroeder
      – PersonEntity:
          Name:
            NameFull: Anthony F. Botelho
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2157-2100
          Numbering:
            – Type: volume
              Value: 18
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Data Mining
              Type: main
ResultId 1