Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test
Saved in:
| Title: | Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test |
|---|---|
| Language: | English |
| Authors: | Zhiqiang Yang (ORCID |
| Source: | European Journal of Education. 2026 61(2). |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 12 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Language Tests, High Stakes Tests, Translation, Test Bias, Item Response Theory, Protocol Analysis, Evaluators, English, Chinese |
| DOI: | 10.1111/ejed.70638 |
| ISSN: | 0141-8211 1465-3435 |
| Abstract: | Human ratings introduce rating bias, thereby undermining the reliability and fairness of the test. While existing studies have identified rating bias as a construct-irrelevant factor, the sources of rating bias are underexplored. Therefore, this study explored the sources of rating bias of the translation task of a high-stakes English language test. Based on the performance of 25 students on the translation test task, this study invited nine raters from three universities to rate these responses and collected their rating performance data on the first and last days of CET-4 rating. Many-facet Rasch measurement model and think-aloud were employed to examine the rating bias and explore its sources. The findings revealed that raters' reliability was acceptable but decreased on the last day, and the sources of rating bias were mainly derived from three areas, including rating administration, raters' cognition and construct-irrelevance factors. Implications on balancing practicality and validity regarding human rating are discussed based on the results. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1507192 |
| Database: | ERIC |
| Abstract: | Human ratings introduce rating bias, thereby undermining the reliability and fairness of the test. While existing studies have identified rating bias as a construct-irrelevant factor, the sources of rating bias are underexplored. Therefore, this study explored the sources of rating bias of the translation task of a high-stakes English language test. Based on the performance of 25 students on the translation test task, this study invited nine raters from three universities to rate these responses and collected their rating performance data on the first and last days of CET-4 rating. Many-facet Rasch measurement model and think-aloud were employed to examine the rating bias and explore its sources. The findings revealed that raters' reliability was acceptable but decreased on the last day, and the sources of rating bias were mainly derived from three areas, including rating administration, raters' cognition and construct-irrelevance factors. Implications on balancing practicality and validity regarding human rating are discussed based on the results. |
|---|---|
| ISSN: | 0141-8211 1465-3435 |
| DOI: | 10.1111/ejed.70638 |
