View in EDS

ChatGPT vs. Machine Learning: Assessing the Efficacy and Accuracy of Large Language Models for Automated Essay Scoring. EdWorkingPaper No. 25-1335

Saved in:
Bibliographic Details
Title: ChatGPT vs. Machine Learning: Assessing the Efficacy and Accuracy of Large Language Models for Automated Essay Scoring. EdWorkingPaper No. 25-1335
Language: English
Authors: Youngwon Kim, Reagan Mozer, Shireen Al-Adeimi, Luke Miratrix, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 28
Publication Date: 2025
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D220032
Document Type: Reports - Research
Education Level: Elementary Education
Grade 4
Intermediate Grades
Grade 5
Middle Schools
Grade 6
Grade 7
Junior High Schools
Secondary Education
Grade 10
High Schools
Descriptors: Automation, Essays, Writing Evaluation, Scoring, Artificial Intelligence, Natural Language Processing, Algorithms, Grade 4, Grade 5, Grade 6, Grade 7, Persuasive Discourse, Grade 10, Classification, Prediction, Performance, Expository Writing, Evaluation Methods, Technology Uses in Education, Writing Skills
Abstract: Automated Essay Scoring (AES) is a critical tool in education that aims to enhance the efficiency and objectivity of educational assessments. Recent advancements in Large Language Models (LLMs), such as ChatGPT, have sparked interest in their potential for AES. However, comprehensive comparisons of LLM-based methods with traditional machine learning (ML) methods across different assessment contexts remain limited. This study compares the efficacy of LLMs with supervised ML algorithms in assessing both categorical essay opinions and continuous writing quality scores. Using two distinct datasets--argumentative essays from 4th-7th graders about iPad usage in schools, and persuasive essays from 10th graders on censorship in libraries--we systematically assess the performance of ChatGPT compared to four tree-based ML algorithms trained on extensive statistical text features. Our findings show that while LLMs perform well in essay classification tasks, ML methods consistently outperform LLMs in predicting writing quality. We highlight the importance of prompting and fine tuning techniques in LLM-based scoring, along with the strengths and limitations of both approaches. We also discuss the potential of LLMs to enhance AES in educational settings while underscoring the continued importance of human oversight in evaluating complex writing skills. Overall, this study demonstrates the complementary strengths of different approaches to AES, providing guidance for researchers and educators interested in leveraging LLMs in educational assessment.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2026
Accession Number: ED678295
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678295
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678295
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: ChatGPT vs. Machine Learning: Assessing the Efficacy and Accuracy of Large Language Models for Automated Essay Scoring. EdWorkingPaper No. 25-1335
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Youngwon+Kim%22">Youngwon Kim</searchLink><br /><searchLink fieldCode="AR" term="%22Reagan+Mozer%22">Reagan Mozer</searchLink><br /><searchLink fieldCode="AR" term="%22Shireen+Al-Adeimi%22">Shireen Al-Adeimi</searchLink><br /><searchLink fieldCode="AR" term="%22Luke+Miratrix%22">Luke Miratrix</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305D220032
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="EL" term="%22Intermediate+Grades%22">Intermediate Grades</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+10%22">Grade 10</searchLink><br /><searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="DE" term="%22Persuasive+Discourse%22">Persuasive Discourse</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+10%22">Grade 10</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Performance%22">Performance</searchLink><br /><searchLink fieldCode="DE" term="%22Expository+Writing%22">Expository Writing</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Skills%22">Writing Skills</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Automated Essay Scoring (AES) is a critical tool in education that aims to enhance the efficiency and objectivity of educational assessments. Recent advancements in Large Language Models (LLMs), such as ChatGPT, have sparked interest in their potential for AES. However, comprehensive comparisons of LLM-based methods with traditional machine learning (ML) methods across different assessment contexts remain limited. This study compares the efficacy of LLMs with supervised ML algorithms in assessing both categorical essay opinions and continuous writing quality scores. Using two distinct datasets--argumentative essays from 4th-7th graders about iPad usage in schools, and persuasive essays from 10th graders on censorship in libraries--we systematically assess the performance of ChatGPT compared to four tree-based ML algorithms trained on extensive statistical text features. Our findings show that while LLMs perform well in essay classification tasks, ML methods consistently outperform LLMs in predicting writing quality. We highlight the importance of prompting and fine tuning techniques in LLM-based scoring, along with the strengths and limitations of both approaches. We also discuss the potential of LLMs to enhance AES in educational settings while underscoring the continued importance of human oversight in evaluating complex writing skills. Overall, this study demonstrates the complementary strengths of different approaches to AES, providing guidance for researchers and educators interested in leveraging LLMs in educational assessment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678295
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678295
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
    Subjects:
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Algorithms
        Type: general
      – SubjectFull: Grade 4
        Type: general
      – SubjectFull: Grade 5
        Type: general
      – SubjectFull: Grade 6
        Type: general
      – SubjectFull: Grade 7
        Type: general
      – SubjectFull: Persuasive Discourse
        Type: general
      – SubjectFull: Grade 10
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Performance
        Type: general
      – SubjectFull: Expository Writing
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Writing Skills
        Type: general
    Titles:
      – TitleFull: ChatGPT vs. Machine Learning: Assessing the Efficacy and Accuracy of Large Language Models for Automated Essay Scoring. EdWorkingPaper No. 25-1335
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Youngwon Kim
      – PersonEntity:
          Name:
            NameFull: Reagan Mozer
      – PersonEntity:
          Name:
            NameFull: Shireen Al-Adeimi
      – PersonEntity:
          Name:
            NameFull: Luke Miratrix
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 11
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1