View in EDS

'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program

Saved in:
Bibliographic Details
Title: 'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program
Language: English
Authors: Yan, Xun (ORCID 0000-0002-4544-4938), Chuang, Ping-Lin (ORCID 0000-0003-2907-5324)
Source: Language Testing. Jan 2023 40(1):153-179.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 27
Publication Date: 2023
Document Type: Journal Articles
Reports - Research
Descriptors: Evaluators, Interrater Reliability, Item Response Theory, Certification, Training, Second Language Learning, English (Second Language), Writing Tests, Placement Tests, Essay Tests, Scaling, Learning Processes, Program Effectiveness, Writing Evaluation
DOI: 10.1177/02655322221074913
ISSN: 0265-5322
1477-0946
Abstract: This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1362771
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1362771
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: 'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Yan%2C+Xun%22">Yan, Xun</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4544-4938">0000-0002-4544-4938</externalLink>)<br /><searchLink fieldCode="AR" term="%22Chuang%2C+Ping-Lin%22">Chuang, Ping-Lin</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2907-5324">0000-0003-2907-5324</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Language+Testing%22"><i>Language Testing</i></searchLink>. Jan 2023 40(1):153-179.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 27
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Certification%22">Certification</searchLink><br /><searchLink fieldCode="DE" term="%22Training%22">Training</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Tests%22">Writing Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Placement+Tests%22">Placement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Essay+Tests%22">Essay Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Scaling%22">Scaling</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Processes%22">Learning Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/02655322221074913
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0265-5322<br />1477-0946
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1362771
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1362771
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/02655322221074913
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 27
        StartPage: 153
    Subjects:
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Certification
        Type: general
      – SubjectFull: Training
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Writing Tests
        Type: general
      – SubjectFull: Placement Tests
        Type: general
      – SubjectFull: Essay Tests
        Type: general
      – SubjectFull: Scaling
        Type: general
      – SubjectFull: Learning Processes
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
    Titles:
      – TitleFull: 'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Yan, Xun
      – PersonEntity:
          Name:
            NameFull: Chuang, Ping-Lin
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 0265-5322
            – Type: issn-electronic
              Value: 1477-0946
          Numbering:
            – Type: volume
              Value: 40
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Language Testing
              Type: main
ResultId 1