View in EDS

How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-Ended Responses

Saved in:
Bibliographic Details
Title: How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-Ended Responses
Language: English
Authors: Jionghao Lin, Eason Chen, Zifei Han, Ashish Gurung, Danielle R. Thomas, Wei Tan, Ngoc Dang Nguyen, Kenneth R. Koedinger
Source: International Educational Data Mining Society. 2024.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Peer Reviewed: Y
Page Count: 15
Publication Date: 2024
Sponsoring Agency: Richard King Mellon Foundation
Learning Engineering Virtual Institute
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: Artificial Intelligence, Man Machine Systems, Natural Language Processing, Feedback (Response), Tutoring, Electronic Learning, Positive Reinforcement, Prediction, Accuracy
Abstract: Automated explanatory feedback systems play a crucial role in facilitating learning for a large cohort of learners by offering feedback that incorporates explanations, significantly enhancing the learning process. However, delivering such explanatory feedback in real-time poses challenges, particularly when high classification accuracy for domain-specific, nuanced responses is essential. Our study leverages the capabilities of large language models, specifically Generative Pre-Trained Transformers (GPT), to explore a sequence labeling approach focused on identifying components of desired and undesired praise for providing explanatory feedback within a tutor training dataset. Our aim is to equip tutors with actionable, explanatory feedback during online training lessons. To investigate the potential of GPT models for providing the explanatory feedback, we employed two commonly-used approaches: "prompting" and "fine-tuning." To quantify the quality of highlighted praise components identified by GPT models, we introduced a Modified Intersection over Union (M-IoU) score. Our findings demonstrate that: (1) the M-IoU score effectively correlates with human judgment in evaluating sequence quality; (2) using two-shot prompting on GPT-3.5 resulted in decent performance in recognizing effort-based (M-IoU of 0.46) and outcome-based praise (M-IoU of 0.68); and (3) our optimally fine-tuned GPT-3.5 model achieved M-IoU scores of 0.64 for effort-based praise and 0.84 for outcome-based praise, aligning with the satisfaction levels evaluated by human coders. Our results show promise for using GPT models to provide feedback that focuses on specific elements in their open-ended responses that are desirable or could use improvement. [For the complete proceedings, see ED675485.]
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED675545
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED675545
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED675545
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-Ended Responses
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Jionghao+Lin%22">Jionghao Lin</searchLink><br /><searchLink fieldCode="AR" term="%22Eason+Chen%22">Eason Chen</searchLink><br /><searchLink fieldCode="AR" term="%22Zifei+Han%22">Zifei Han</searchLink><br /><searchLink fieldCode="AR" term="%22Ashish+Gurung%22">Ashish Gurung</searchLink><br /><searchLink fieldCode="AR" term="%22Danielle+R%2E+Thomas%22">Danielle R. Thomas</searchLink><br /><searchLink fieldCode="AR" term="%22Wei+Tan%22">Wei Tan</searchLink><br /><searchLink fieldCode="AR" term="%22Ngoc+Dang+Nguyen%22">Ngoc Dang Nguyen</searchLink><br /><searchLink fieldCode="AR" term="%22Kenneth+R%2E+Koedinger%22">Kenneth R. Koedinger</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Richard King Mellon Foundation<br />Learning Engineering Virtual Institute
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Man+Machine+Systems%22">Man Machine Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Tutoring%22">Tutoring</searchLink><br /><searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Positive+Reinforcement%22">Positive Reinforcement</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Automated explanatory feedback systems play a crucial role in facilitating learning for a large cohort of learners by offering feedback that incorporates explanations, significantly enhancing the learning process. However, delivering such explanatory feedback in real-time poses challenges, particularly when high classification accuracy for domain-specific, nuanced responses is essential. Our study leverages the capabilities of large language models, specifically Generative Pre-Trained Transformers (GPT), to explore a sequence labeling approach focused on identifying components of desired and undesired praise for providing explanatory feedback within a tutor training dataset. Our aim is to equip tutors with actionable, explanatory feedback during online training lessons. To investigate the potential of GPT models for providing the explanatory feedback, we employed two commonly-used approaches: "prompting" and "fine-tuning." To quantify the quality of highlighted praise components identified by GPT models, we introduced a Modified Intersection over Union (M-IoU) score. Our findings demonstrate that: (1) the M-IoU score effectively correlates with human judgment in evaluating sequence quality; (2) using two-shot prompting on GPT-3.5 resulted in decent performance in recognizing effort-based (M-IoU of 0.46) and outcome-based praise (M-IoU of 0.68); and (3) our optimally fine-tuned GPT-3.5 model achieved M-IoU scores of 0.64 for effort-based praise and 0.84 for outcome-based praise, aligning with the satisfaction levels evaluated by human coders. Our results show promise for using GPT models to provide feedback that focuses on specific elements in their open-ended responses that are desirable or could use improvement. [For the complete proceedings, see ED675485.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED675545
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED675545
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Man Machine Systems
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Tutoring
        Type: general
      – SubjectFull: Electronic Learning
        Type: general
      – SubjectFull: Positive Reinforcement
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Accuracy
        Type: general
    Titles:
      – TitleFull: How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-Ended Responses
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Jionghao Lin
      – PersonEntity:
          Name:
            NameFull: Eason Chen
      – PersonEntity:
          Name:
            NameFull: Zifei Han
      – PersonEntity:
          Name:
            NameFull: Ashish Gurung
      – PersonEntity:
          Name:
            NameFull: Danielle R. Thomas
      – PersonEntity:
          Name:
            NameFull: Wei Tan
      – PersonEntity:
          Name:
            NameFull: Ngoc Dang Nguyen
      – PersonEntity:
          Name:
            NameFull: Kenneth R. Koedinger
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1