Optimizing Inter-Rater Reliability in Foreign Language Constructed-Response Assessments
Saved in:
| Title: | Optimizing Inter-Rater Reliability in Foreign Language Constructed-Response Assessments |
|---|---|
| Language: | English |
| Authors: | Jose Fabian Elizondo-Gonzalez (ORCID |
| Source: | Research in Pedagogy. 2025 15(2):471-482. |
| Availability: | Preschool Teacher Training College "Mihailo Palov" and Serbian Academy of Education in Belgrade. Omladinski Trg 1, Vrsac, 26300 Serbia. Tel: +381-832517; Fax: +381-832517; Web site: http://research.rs |
| Peer Reviewed: | Y |
| Page Count: | 12 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Interrater Reliability, Second Language Learning, English (Second Language), Language Tests, Language Proficiency, Foreign Countries, Test Reliability, Error Patterns, Scoring, Prompting, Responses, Test Construction |
| Geographic Terms: | Costa Rica |
| Assessment and Survey Identifiers: | Test of English as a Foreign Language |
| ISSN: | 2217-7337 2406-2006 |
| Abstract: | This study examines inter-rater reliability in a constructed-response English proficiency test developed by the Foreign Language Assessment Program (PELEx) in Costa Rica. Thirty university instructors completed three writing tasks aligned with A2, B2, and C1 CEFR bands, each scored by two trained raters. Inter-rater reliability was estimated using percent agreement, Cohen's weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory). While traditional estimators suggested good to excellent reliability, G-Theory revealed additional sources of error not accounted for by kappa or ICC, particularly prompt-related variability. For example, in the C1 task, person-by-prompt interaction accounted for over 20% of total variance. These findings suggest that while rater training remains important, prompt-related variability must also be addressed to ensure fairness and score comparability. Incorporating calibrated prompts and structured scoring protocols across proficiency levels may strengthen the reliability of constructed-response tasks, especially in high-stakes settings. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1498173 |
| Database: | ERIC |
| Abstract: | This study examines inter-rater reliability in a constructed-response English proficiency test developed by the Foreign Language Assessment Program (PELEx) in Costa Rica. Thirty university instructors completed three writing tasks aligned with A2, B2, and C1 CEFR bands, each scored by two trained raters. Inter-rater reliability was estimated using percent agreement, Cohen's weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory). While traditional estimators suggested good to excellent reliability, G-Theory revealed additional sources of error not accounted for by kappa or ICC, particularly prompt-related variability. For example, in the C1 task, person-by-prompt interaction accounted for over 20% of total variance. These findings suggest that while rater training remains important, prompt-related variability must also be addressed to ensure fairness and score comparability. Incorporating calibrated prompts and structured scoring protocols across proficiency levels may strengthen the reliability of constructed-response tasks, especially in high-stakes settings. |
|---|---|
| ISSN: | 2217-7337 2406-2006 |
