Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
Saved in:
| Title: | Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems |
|---|---|
| Language: | English |
| Authors: | Alexander Kwako (ORCID |
| Source: | Educational Measurement: Issues and Practice. 2026 45(2). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 14 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Models, Confidence Testing, Scores, Scoring, Computer Assisted Testing, Man Machine Systems, Automation, Item Response Theory |
| DOI: | 10.1111/emip.70019 |
| ISSN: | 0731-1745 1745-3992 |
| Abstract: | In large-scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult-to-score responses for human review. We utilize "hybrid performance curves" to visualize the impact of routing on performance. Additionally, we propose several "hybrid scoring policies" for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human- and automated-only systems. Moreover, the superior performance of the hybrid system is less expensive than a human-only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1506976 |
| Database: | ERIC |
| Abstract: | In large-scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult-to-score responses for human review. We utilize "hybrid performance curves" to visualize the impact of routing on performance. Additionally, we propose several "hybrid scoring policies" for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human- and automated-only systems. Moreover, the superior performance of the hybrid system is less expensive than a human-only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole. |
|---|---|
| ISSN: | 0731-1745 1745-3992 |
| DOI: | 10.1111/emip.70019 |
