View in EDS

Evaluating the applicability of the transformer-based grammatical error correction system for assessing language accuracy.

Saved in:
Bibliographic Details
Title: Evaluating the applicability of the transformer-based grammatical error correction system for assessing language accuracy.
Authors: Hwang, Haerim1 haerimhwang@cuhk.edu.hk
Source: Language Testing in Asia. 12/4/2025, Vol. 15 Issue 1, p1-11. 11p.
Subject Terms: *Fluency (Language learning), *Sentences (Grammar), *Mixed methods research, Linguistic errors, Korean language
Abstract: The current study focuses on the use of Grammatical Error Correction (GEC) technology for assessing language accuracy, which has received relatively less attention than complexity and fluency in the context of automated assessment. Adopting a technology-enhanced approach to language assessment, rather than a technology-driven approach, we critically assessed the suitability of the state-of-the-art GEC system for assessing language accuracy in Korean, an understudied language in this regard. We analyzed how reliable this system is quantitatively and what types of error can be generated by this system qualitatively. Also, we used out-of-domain, inclusive data from heritage speakers of Korean, which has never been considered in the development of GEC. Our accuracy analyses show that the system achieves a fairly high accuracy in differentiating between correct and incorrect sentences on our data (F0.5 = 0.819). However, the system exhibits a tendency to make unnecessary corrections, such as inserting topics/adverbials or correcting particles, while failing to correct ungrammatical ones in some cases. These findings from our mixed-method analyses suggest that language evaluators should recognize the potential for inaccurate assessments when using a GEC system, as its output may be incorrect at this moment, thus highlighting the critical need for digital language assessment literacy. [ABSTRACT FROM AUTHOR]
Copyright of Language Testing in Asia is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
Description
Abstract:The current study focuses on the use of Grammatical Error Correction (GEC) technology for assessing language accuracy, which has received relatively less attention than complexity and fluency in the context of automated assessment. Adopting a technology-enhanced approach to language assessment, rather than a technology-driven approach, we critically assessed the suitability of the state-of-the-art GEC system for assessing language accuracy in Korean, an understudied language in this regard. We analyzed how reliable this system is quantitatively and what types of error can be generated by this system qualitatively. Also, we used out-of-domain, inclusive data from heritage speakers of Korean, which has never been considered in the development of GEC. Our accuracy analyses show that the system achieves a fairly high accuracy in differentiating between correct and incorrect sentences on our data (F0.5 = 0.819). However, the system exhibits a tendency to make unnecessary corrections, such as inserting topics/adverbials or correcting particles, while failing to correct ungrammatical ones in some cases. These findings from our mixed-method analyses suggest that language evaluators should recognize the potential for inaccurate assessments when using a GEC system, as its output may be incorrect at this moment, thus highlighting the critical need for digital language assessment literacy. [ABSTRACT FROM AUTHOR]
ISSN:22290443
DOI:10.1186/s40468-025-00415-3