Limitations of QWK in Evaluating Automated and Human Scoring Systems
Saved in:
| Title: | Limitations of QWK in Evaluating Automated and Human Scoring Systems |
|---|---|
| Language: | English |
| Authors: | Jennifer Lewis (ORCID |
| Source: | Educational Measurement: Issues and Practice. 2026 45(1). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 14 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Interrater Reliability, Evaluation Methods, Measurement Techniques, Automation, Artificial Intelligence, Scores, Responses |
| DOI: | 10.1111/emip.70017 |
| ISSN: | 0731-1745 1745-3992 |
| Abstract: | To assess the interrater reliability of human ratings of constructed responses (CR), or the accuracy of scores given by automated scoring engines, concordance metrics quantify agreement between measures. This article examines the quadratic weighted kappa (QWK) in these contexts and highlights its practical limitations compared to other metrics. Both empirical and simulation study results reveal how different factors including the shape of the marginal distributions and score scale length may impact the estimates and how we can adjust for these properties of the contingency table. The results highlight the QWK's sensitivities and suggest that additional caution should be taken before decisions about whether to keep a CR item on a test form are made. If using QWK without the proper interpretive supports, such decisions may be misinformed. Consequently, we make suggestions for best practices to promote responsible evaluation of agreement in the context of CR scoring in educational testing. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1498710 |
| Database: | ERIC |
| Abstract: | To assess the interrater reliability of human ratings of constructed responses (CR), or the accuracy of scores given by automated scoring engines, concordance metrics quantify agreement between measures. This article examines the quadratic weighted kappa (QWK) in these contexts and highlights its practical limitations compared to other metrics. Both empirical and simulation study results reveal how different factors including the shape of the marginal distributions and score scale length may impact the estimates and how we can adjust for these properties of the contingency table. The results highlight the QWK's sensitivities and suggest that additional caution should be taken before decisions about whether to keep a CR item on a test form are made. If using QWK without the proper interpretive supports, such decisions may be misinformed. Consequently, we make suggestions for best practices to promote responsible evaluation of agreement in the context of CR scoring in educational testing. |
|---|---|
| ISSN: | 0731-1745 1745-3992 |
| DOI: | 10.1111/emip.70017 |
