View in EDS

Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models

Saved in:
Bibliographic Details
Title: Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models
Language: English
Authors: Wesley Morris (ORCID 0000-0001-6316-6479), Scott Crossley, Langdon Holmes, Chaohua Ou, Mihai Dascalu, Danielle McNamara
Source: International Journal of Artificial Intelligence in Education. 2025 35(3):1022-1043.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 22
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
High Schools
Secondary Education
Descriptors: Formative Evaluation, Feedback (Response), Textbooks, Artificial Intelligence, Natural Language Processing, Computer Uses in Education, Automation, Writing Evaluation, Reading Comprehension, Writing (Composition), College Students, High School Students
DOI: 10.1007/s40593-024-00395-0
ISSN: 1560-4292
1560-4306
Abstract: As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need to make them more interactive arises. An alternative is to ask students to generate knowledge in response to textbook content and provide feedback about the produced knowledge. This study develops Natural Language Processing models to automatically provide feedback to students about the quality of summaries written at the end of intelligent textbook sections. The study builds on the work of Botarleanu et al. (2022), who used a Longformer Large Language Model (LLM) to develop a summary grading model. Their model explained around 55% of holistic summary score variance as assigned by human raters. This study uses a principal component analysis to distill summary scores from an analytic rubric into two principal components -- content and wording. This study uses two encoder-only classification large language models finetuned from Longformer on the summaries and the source texts using these principal components explained 82% and 70% of the score variance for content and wording, respectively. On a dataset of summaries collected on the crowd-sourcing site Prolific, the content model was shown to be robust although the accuracy of the wording model was reduced compared to the training set. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real time. The models can also be used for other summarization applications in learning systems.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1488267
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1488267
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wesley+Morris%22">Wesley Morris</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-6316-6479">0000-0001-6316-6479</externalLink>)<br /><searchLink fieldCode="AR" term="%22Scott+Crossley%22">Scott Crossley</searchLink><br /><searchLink fieldCode="AR" term="%22Langdon+Holmes%22">Langdon Holmes</searchLink><br /><searchLink fieldCode="AR" term="%22Chaohua+Ou%22">Chaohua Ou</searchLink><br /><searchLink fieldCode="AR" term="%22Mihai+Dascalu%22">Mihai Dascalu</searchLink><br /><searchLink fieldCode="AR" term="%22Danielle+McNamara%22">Danielle McNamara</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Artificial+Intelligence+in+Education%22"><i>International Journal of Artificial Intelligence in Education</i></searchLink>. 2025 35(3):1022-1043.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Formative+Evaluation%22">Formative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Textbooks%22">Textbooks</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Uses+in+Education%22">Computer Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Comprehension%22">Reading Comprehension</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+%28Composition%29%22">Writing (Composition)</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s40593-024-00395-0
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1560-4292<br />1560-4306
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need to make them more interactive arises. An alternative is to ask students to generate knowledge in response to textbook content and provide feedback about the produced knowledge. This study develops Natural Language Processing models to automatically provide feedback to students about the quality of summaries written at the end of intelligent textbook sections. The study builds on the work of Botarleanu et al. (2022), who used a Longformer Large Language Model (LLM) to develop a summary grading model. Their model explained around 55% of holistic summary score variance as assigned by human raters. This study uses a principal component analysis to distill summary scores from an analytic rubric into two principal components -- content and wording. This study uses two encoder-only classification large language models finetuned from Longformer on the summaries and the source texts using these principal components explained 82% and 70% of the score variance for content and wording, respectively. On a dataset of summaries collected on the crowd-sourcing site Prolific, the content model was shown to be robust although the accuracy of the wording model was reduced compared to the training set. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real time. The models can also be used for other summarization applications in learning systems.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1488267
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1488267
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s40593-024-00395-0
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 1022
    Subjects:
      – SubjectFull: Formative Evaluation
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Textbooks
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Computer Uses in Education
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Reading Comprehension
        Type: general
      – SubjectFull: Writing (Composition)
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: High School Students
        Type: general
    Titles:
      – TitleFull: Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wesley Morris
      – PersonEntity:
          Name:
            NameFull: Scott Crossley
      – PersonEntity:
          Name:
            NameFull: Langdon Holmes
      – PersonEntity:
          Name:
            NameFull: Chaohua Ou
      – PersonEntity:
          Name:
            NameFull: Mihai Dascalu
      – PersonEntity:
          Name:
            NameFull: Danielle McNamara
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1560-4292
            – Type: issn-electronic
              Value: 1560-4306
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: International Journal of Artificial Intelligence in Education
              Type: main
ResultId 1