View in EDS

Can AI Grade Like a Human? Validity, Reliability, and Fairness in University Coursework Assessment

Saved in:
Bibliographic Details
Title: Can AI Grade Like a Human? Validity, Reliability, and Fairness in University Coursework Assessment
Language: English
Authors: Georgios Zacharis (ORCID 0000-0003-1158-9175), Stamatios Papadakis (ORCID 0000-0003-3184-1147)
Source: Educational Process: International Journal. Article e2025591 2025 19.
Availability: UNIVERSITEPARK Limited. iTOWER Plaza (No61, 9th floor) Merkez Mh Akar Cd No3, Sisli, Istanbul, Turkey 34382. e-mail: editor@edupij.com; Web site: http://www.edupij.com/
Peer Reviewed: Y
Page Count: 22
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Artificial Intelligence, Technology Uses in Education, Computer Assisted Testing, Grading, Student Evaluation, College Students, Evaluators, Scoring, Foreign Countries, Interrater Reliability, High Stakes Tests, Construct Validity, Test Reliability, Formative Evaluation, Feedback (Response), Test Bias
Geographic Terms: Greece
ISSN: 2147-0901
2564-8020
Abstract: Background/purpose: Generative artificial intelligence (GenAI) is often promoted as a transformative tool for assessment, yet evidence of its validity compared to human raters remains limited. This study examined whether an AI-based rater could be used interchangeably with trained faculty in scoring complex coursework. Materials/methods: Ninety-one essays from teacher education courses at two Greek universities were independently evaluated by two human raters and an AI system, using a common rubric. Results: Human inter-rater reliability was excellent (ICC(2,1) = 0.884; ICC(2,k) k=2 = 0.938). In contrast, AI-human agreement was substantially weaker (AI vs Human-Z: ICC(2,1) = 0.406; ICC(2,k) = 0.578; AI vs Human-S: ICC(2,1) = 0.279; ICC(2,k) = 0.436). The AI consistently inflated scores by 2.71-3.32 points and compressed distributions, limiting its ability to discriminate across performance levels. Bland-Altman analyses confirmed systematic proportional bias, with overscoring of weaker work and under-scoring of stronger work. Results revealed significant inconsistency in AI performance: while the model failed to align with Human-S ([kappa] = 0.017), it demonstrated statistically significant, moderate agreement with Human-Z ([kappa] = 0.367). This discrepancy highlights the lack of standardization in human grading and the sensitivity of algorithms to divergent interpretive frameworks. A principal component analysis suggested that AI captured a narrower construct of quality than human raters. Conclusion: These findings indicate that current GenAI tools are not suitable for high-stakes assessment in higher education, where fairness and construct validity are essential. They may, however, offer value in formative feedback or administrative support if used transparently and under human oversight.
Abstractor: As Provided
Notes: https://osf.io/7ay9b/?view_only=dbbceae688584cd3ad2d93ec3b6a7405
Entry Date: 2025
Accession Number: EJ1491083
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1491083
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1491083
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Can AI Grade Like a Human? Validity, Reliability, and Fairness in University Coursework Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Georgios+Zacharis%22">Georgios Zacharis</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-1158-9175">0000-0003-1158-9175</externalLink>)<br /><searchLink fieldCode="AR" term="%22Stamatios+Papadakis%22">Stamatios Papadakis</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-3184-1147">0000-0003-3184-1147</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Process%3A+International+Journal%22"><i>Educational Process: International Journal</i></searchLink>. Article e2025591 2025 19.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: UNIVERSITEPARK Limited. iTOWER Plaza (No61, 9th floor) Merkez Mh Akar Cd No3, Sisli, Istanbul, Turkey 34382. e-mail: editor@edupij.com; Web site: http://www.edupij.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Grading%22">Grading</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22High+Stakes+Tests%22">High Stakes Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Construct+Validity%22">Construct Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Formative+Evaluation%22">Formative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Greece%22">Greece</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2147-0901<br />2564-8020
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background/purpose: Generative artificial intelligence (GenAI) is often promoted as a transformative tool for assessment, yet evidence of its validity compared to human raters remains limited. This study examined whether an AI-based rater could be used interchangeably with trained faculty in scoring complex coursework. Materials/methods: Ninety-one essays from teacher education courses at two Greek universities were independently evaluated by two human raters and an AI system, using a common rubric. Results: Human inter-rater reliability was excellent (ICC(2,1) = 0.884; ICC(2,k) k=2 = 0.938). In contrast, AI-human agreement was substantially weaker (AI vs Human-Z: ICC(2,1) = 0.406; ICC(2,k) = 0.578; AI vs Human-S: ICC(2,1) = 0.279; ICC(2,k) = 0.436). The AI consistently inflated scores by 2.71-3.32 points and compressed distributions, limiting its ability to discriminate across performance levels. Bland-Altman analyses confirmed systematic proportional bias, with overscoring of weaker work and under-scoring of stronger work. Results revealed significant inconsistency in AI performance: while the model failed to align with Human-S ([kappa] = 0.017), it demonstrated statistically significant, moderate agreement with Human-Z ([kappa] = 0.367). This discrepancy highlights the lack of standardization in human grading and the sensitivity of algorithms to divergent interpretive frameworks. A principal component analysis suggested that AI captured a narrower construct of quality than human raters. Conclusion: These findings indicate that current GenAI tools are not suitable for high-stakes assessment in higher education, where fairness and construct validity are essential. They may, however, offer value in formative feedback or administrative support if used transparently and under human oversight.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/7ay9b/?view_only=dbbceae688584cd3ad2d93ec3b6a7405
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1491083
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1491083
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Grading
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: High Stakes Tests
        Type: general
      – SubjectFull: Construct Validity
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Formative Evaluation
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Greece
        Type: general
    Titles:
      – TitleFull: Can AI Grade Like a Human? Validity, Reliability, and Fairness in University Coursework Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Georgios Zacharis
      – PersonEntity:
          Name:
            NameFull: Stamatios Papadakis
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 2147-0901
            – Type: issn-electronic
              Value: 2564-8020
          Numbering:
            – Type: volume
              Value: 19
          Titles:
            – TitleFull: Educational Process: International Journal
              Type: main
ResultId 1