View in EDS

The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations.

Saved in:
Bibliographic Details
Title: The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations.
Authors: Lamprianou, Iasonas1, Tsagari, Dina2, Kyriakou, Nansia1 kyriakou.nansia@ucy.ac.cy
Source: Language Testing. Apr2021, Vol. 38 Issue 2, p273-301. 29p.
Subject Terms: *Examination answer sheets, *Examinations, *Adaptive testing, *Educational tests & measurements, Forecasting
Abstract: This longitudinal study (2002–2014) investigates the stability of rating characteristics of a large group of raters over time in the context of the writing paper of a national high-stakes examination. The study uses one measure of rater severity and two measures of rater consistency. The results suggest that the rating characteristics of individual raters are not stable. Thus, predictions from one administration to the next are difficult, although not impossible. In fact, as the membership of the group of raters changes from year to year, past data on rating characteristics become less useful. When the membership of the group of raters is retained, the community of raters develops more stable characteristics. However, "cultural shocks" (low retention of raters and large numbers of newcomers) destabilize the rating characteristics of the community and predictions become more difficult. We propose practical measures to increase the stability of rating across time and offer methodological suggestions for more efficient rater effect-related research designs and analyses. [ABSTRACT FROM AUTHOR]
Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
Description
Abstract:This longitudinal study (2002–2014) investigates the stability of rating characteristics of a large group of raters over time in the context of the writing paper of a national high-stakes examination. The study uses one measure of rater severity and two measures of rater consistency. The results suggest that the rating characteristics of individual raters are not stable. Thus, predictions from one administration to the next are difficult, although not impossible. In fact, as the membership of the group of raters changes from year to year, past data on rating characteristics become less useful. When the membership of the group of raters is retained, the community of raters develops more stable characteristics. However, "cultural shocks" (low retention of raters and large numbers of newcomers) destabilize the rating characteristics of the community and predictions become more difficult. We propose practical measures to increase the stability of rating across time and offer methodological suggestions for more efficient rater effect-related research designs and analyses. [ABSTRACT FROM AUTHOR]
ISSN:02655322
DOI:10.1177/0265532220940960