View in EDS

Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test

Saved in:
Bibliographic Details
Title: Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test
Language: English
Authors: Zhiqiang Yang (ORCID 0000-0001-9833-8242), Qing Li (ORCID 0000-0003-0654-2022), Tao Zhang (ORCID 0000-0002-7279-7130), Baohua Dong (ORCID 0000-0002-3219-6541)
Source: European Journal of Education. 2026 61(2).
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 12
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Descriptors: Language Tests, High Stakes Tests, Translation, Test Bias, Item Response Theory, Protocol Analysis, Evaluators, English, Chinese
DOI: 10.1111/ejed.70638
ISSN: 0141-8211
1465-3435
Abstract: Human ratings introduce rating bias, thereby undermining the reliability and fairness of the test. While existing studies have identified rating bias as a construct-irrelevant factor, the sources of rating bias are underexplored. Therefore, this study explored the sources of rating bias of the translation task of a high-stakes English language test. Based on the performance of 25 students on the translation test task, this study invited nine raters from three universities to rate these responses and collected their rating performance data on the first and last days of CET-4 rating. Many-facet Rasch measurement model and think-aloud were employed to examine the rating bias and explore its sources. The findings revealed that raters' reliability was acceptable but decreased on the last day, and the sources of rating bias were mainly derived from three areas, including rating administration, raters' cognition and construct-irrelevance factors. Implications on balancing practicality and validity regarding human rating are discussed based on the results.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1507192
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1507192
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Zhiqiang+Yang%22">Zhiqiang Yang</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9833-8242">0000-0001-9833-8242</externalLink>)<br /><searchLink fieldCode="AR" term="%22Qing+Li%22">Qing Li</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0654-2022">0000-0003-0654-2022</externalLink>)<br /><searchLink fieldCode="AR" term="%22Tao+Zhang%22">Tao Zhang</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7279-7130">0000-0002-7279-7130</externalLink>)<br /><searchLink fieldCode="AR" term="%22Baohua+Dong%22">Baohua Dong</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3219-6541">0000-0002-3219-6541</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22European+Journal+of+Education%22"><i>European Journal of Education</i></searchLink>. 2026 61(2).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 12
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22High+Stakes+Tests%22">High Stakes Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Translation%22">Translation</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Protocol+Analysis%22">Protocol Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22English%22">English</searchLink><br /><searchLink fieldCode="DE" term="%22Chinese%22">Chinese</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/ejed.70638
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0141-8211<br />1465-3435
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Human ratings introduce rating bias, thereby undermining the reliability and fairness of the test. While existing studies have identified rating bias as a construct-irrelevant factor, the sources of rating bias are underexplored. Therefore, this study explored the sources of rating bias of the translation task of a high-stakes English language test. Based on the performance of 25 students on the translation test task, this study invited nine raters from three universities to rate these responses and collected their rating performance data on the first and last days of CET-4 rating. Many-facet Rasch measurement model and think-aloud were employed to examine the rating bias and explore its sources. The findings revealed that raters' reliability was acceptable but decreased on the last day, and the sources of rating bias were mainly derived from three areas, including rating administration, raters' cognition and construct-irrelevance factors. Implications on balancing practicality and validity regarding human rating are discussed based on the results.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1507192
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1507192
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/ejed.70638
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
    Subjects:
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: High Stakes Tests
        Type: general
      – SubjectFull: Translation
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Protocol Analysis
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: English
        Type: general
      – SubjectFull: Chinese
        Type: general
    Titles:
      – TitleFull: Tracing the Origin of Rating Bias in a High-Stakes Chinese-English Translation Test
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Zhiqiang Yang
      – PersonEntity:
          Name:
            NameFull: Qing Li
      – PersonEntity:
          Name:
            NameFull: Tao Zhang
      – PersonEntity:
          Name:
            NameFull: Baohua Dong
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0141-8211
            – Type: issn-electronic
              Value: 1465-3435
          Numbering:
            – Type: volume
              Value: 61
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: European Journal of Education
              Type: main
ResultId 1