Can AI Assess Writing Skills Like a Human? A Reliability Analysis
Saved in:
| Title: | Can AI Assess Writing Skills Like a Human? A Reliability Analysis |
|---|---|
| Language: | English |
| Authors: | Hüseyin Ataseven (ORCID |
| Source: | Journal of Theoretical Educational Science. 2025 18(4):736-754. |
| Availability: | Afyon Kocatepe University. ANS Kampusu, Egitim Fakultesi, Merkez, Afyonkarahisar 03200, Turkey. Tel: +90-272-2181740; Fax: +90-272-2281418; e-mail: editorkebd@gmail.com; Web site: https://dergipark.org.tr/en/pub/akukeg |
| Peer Reviewed: | Y |
| Page Count: | 19 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Artificial Intelligence, Technology Uses in Education, Writing Skills, Student Evaluation, Scoring, English (Second Language), Interrater Reliability, College Students, Foreign Countries |
| Geographic Terms: | Turkey |
| ISSN: | 1308-1659 |
| Abstract: | This study investigates the reliability and consistency of a custom GPT-based scoring system in comparison to trained human raters, focusing on B1-level opinion paragraphs written by English preparatory students. Addressing the limited evidence on how AI scoring systems align with human evaluations in foreign language contexts, the study provides insights into both strengths and limitations of automated writing assessment. A total of 175 student writings were evaluated twice by human raters and twice by the AI system using analytic rubric. Findings indicate excellent agreement among human raters and high consistency across AI-generated scores, but only moderate alignment between human and AI evaluations, with the AI showing a tendency to assign higher scores and overlook off-topic content. These results suggest that while AI scoring systems offer efficiency and consistency, they still lack the interpretive depth of human judgment. The study highlights the potential of AI as a complementary tool in writing assessment, with practical implications for language testing policy and classroom pedagogy. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1487685 |
| Database: | ERIC |
| Abstract: | This study investigates the reliability and consistency of a custom GPT-based scoring system in comparison to trained human raters, focusing on B1-level opinion paragraphs written by English preparatory students. Addressing the limited evidence on how AI scoring systems align with human evaluations in foreign language contexts, the study provides insights into both strengths and limitations of automated writing assessment. A total of 175 student writings were evaluated twice by human raters and twice by the AI system using analytic rubric. Findings indicate excellent agreement among human raters and high consistency across AI-generated scores, but only moderate alignment between human and AI evaluations, with the AI showing a tendency to assign higher scores and overlook off-topic content. These results suggest that while AI scoring systems offer efficiency and consistency, they still lack the interpretive depth of human judgment. The study highlights the potential of AI as a complementary tool in writing assessment, with practical implications for language testing policy and classroom pedagogy. |
|---|---|
| ISSN: | 1308-1659 |
