'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program
Saved in:
| Title: | 'How Do Raters Learn to Rate?' Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program |
|---|---|
| Language: | English |
| Authors: | Yan, Xun (ORCID |
| Source: | Language Testing. Jan 2023 40(1):153-179. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 27 |
| Publication Date: | 2023 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Evaluators, Interrater Reliability, Item Response Theory, Certification, Training, Second Language Learning, English (Second Language), Writing Tests, Placement Tests, Essay Tests, Scaling, Learning Processes, Program Effectiveness, Writing Evaluation |
| DOI: | 10.1177/02655322221074913 |
| ISSN: | 0265-5322 1477-0946 |
| Abstract: | This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments. |
| Abstractor: | As Provided |
| Entry Date: | 2023 |
| Accession Number: | EJ1362771 |
| Database: | ERIC |
Be the first to leave a comment!
