View in EDS

One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs

Saved in:
Bibliographic Details
Title: One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs
Language: English
Authors: Andreea Dutulescu, Stefan Ruseti, Mihai Dascalu, Danielle McNamara
Source: International Educational Data Mining Society. 2025.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Peer Reviewed: Y
Page Count: 7
Publication Date: 2025
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305T240035
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: Scoring, Computational Linguistics, Computer Software, Artificial Intelligence, Generalization, Scoring Rubrics, Student Evaluation, Prediction, Evaluation Criteria, Learning Strategies, Feedback (Response), Evaluation Methods, Writing Evaluation
Abstract: The assessment of student responses to learning-strategy prompts, such as self-explanation, summarization, and paraphrasing, is essential for evaluating cognitive engagement and comprehension. However, manual scoring is resource-intensive, limiting its scalability in educational settings. This study investigates the use of Large Language Models for automating the evaluation of student responses based on expert-defined rubrics. We fine-tune open-source LLMs on annotated datasets to predict expert ratings across multiple scoring rubrics, ensuring consistency and efficiency in assessment. Our findings indicate that multi-task fine-tuning, which involves training a single model across multiple scoring tasks, consistently outperforms single-task training by enhancing generalization and mitigating overfitting. This advantage is particularly noticeable in recent architectures, where multi-task training enables robust performance across diverse evaluation criteria. Notably, our Llama 3.2 3B model achieved high performance, outperforming a 20x larger zero-shot model while maintaining feasibility for deployment on consumer-grade hardware, emphasizing the potential for scalable AI-driven assessment solutions. This research contributes to open education by fine-tuning open-source models and publicly releasing trained models, training scripts, and evaluation frameworks. The proposed approach supports automated, reproducible, and scalable assessment of learning strategies, facilitating timely feedback for students and reducing the burden on educators. [For the complete proceedings, see ED675583.]
Abstractor: As Provided
Notes: https://github.com/upb-nlp/EDM-LLM-Scoring
IES Funded: Yes
Entry Date: 2025
Accession Number: ED675648
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED675648
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED675648
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Andreea+Dutulescu%22">Andreea Dutulescu</searchLink><br /><searchLink fieldCode="AR" term="%22Stefan+Ruseti%22">Stefan Ruseti</searchLink><br /><searchLink fieldCode="AR" term="%22Mihai+Dascalu%22">Mihai Dascalu</searchLink><br /><searchLink fieldCode="AR" term="%22Danielle+McNamara%22">Danielle McNamara</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305T240035
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Computational+Linguistics%22">Computational Linguistics</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software%22">Computer Software</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Generalization%22">Generalization</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring+Rubrics%22">Scoring Rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Criteria%22">Evaluation Criteria</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Strategies%22">Learning Strategies</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The assessment of student responses to learning-strategy prompts, such as self-explanation, summarization, and paraphrasing, is essential for evaluating cognitive engagement and comprehension. However, manual scoring is resource-intensive, limiting its scalability in educational settings. This study investigates the use of Large Language Models for automating the evaluation of student responses based on expert-defined rubrics. We fine-tune open-source LLMs on annotated datasets to predict expert ratings across multiple scoring rubrics, ensuring consistency and efficiency in assessment. Our findings indicate that multi-task fine-tuning, which involves training a single model across multiple scoring tasks, consistently outperforms single-task training by enhancing generalization and mitigating overfitting. This advantage is particularly noticeable in recent architectures, where multi-task training enables robust performance across diverse evaluation criteria. Notably, our Llama 3.2 3B model achieved high performance, outperforming a 20x larger zero-shot model while maintaining feasibility for deployment on consumer-grade hardware, emphasizing the potential for scalable AI-driven assessment solutions. This research contributes to open education by fine-tuning open-source models and publicly releasing trained models, training scripts, and evaluation frameworks. The proposed approach supports automated, reproducible, and scalable assessment of learning strategies, facilitating timely feedback for students and reducing the burden on educators. [For the complete proceedings, see ED675583.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://github.com/upb-nlp/EDM-LLM-Scoring
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED675648
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED675648
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
    Subjects:
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Computational Linguistics
        Type: general
      – SubjectFull: Computer Software
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Generalization
        Type: general
      – SubjectFull: Scoring Rubrics
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Evaluation Criteria
        Type: general
      – SubjectFull: Learning Strategies
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
    Titles:
      – TitleFull: One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Andreea Dutulescu
      – PersonEntity:
          Name:
            NameFull: Stefan Ruseti
      – PersonEntity:
          Name:
            NameFull: Mihai Dascalu
      – PersonEntity:
          Name:
            NameFull: Danielle McNamara
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1