The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations.
Saved in:
| Title: | The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations. |
|---|---|
| Authors: | Lamprianou, Iasonas1, Tsagari, Dina2, Kyriakou, Nansia1 kyriakou.nansia@ucy.ac.cy |
| Source: | Language Testing. Apr2021, Vol. 38 Issue 2, p273-301. 29p. |
| Subject Terms: | *Examination answer sheets, *Examinations, *Adaptive testing, *Educational tests & measurements, Forecasting |
| Abstract: | This longitudinal study (2002–2014) investigates the stability of rating characteristics of a large group of raters over time in the context of the writing paper of a national high-stakes examination. The study uses one measure of rater severity and two measures of rater consistency. The results suggest that the rating characteristics of individual raters are not stable. Thus, predictions from one administration to the next are difficult, although not impossible. In fact, as the membership of the group of raters changes from year to year, past data on rating characteristics become less useful. When the membership of the group of raters is retained, the community of raters develops more stable characteristics. However, "cultural shocks" (low retention of raters and large numbers of newcomers) destabilize the rating characteristics of the community and predictions become more difficult. We propose practical measures to increase the stability of rating across time and offer methodological suggestions for more efficient rater effect-related research designs and analyses. [ABSTRACT FROM AUTHOR] |
| Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 149907239 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Lamprianou%2C+Iasonas%22">Lamprianou, Iasonas</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Tsagari%2C+Dina%22">Tsagari, Dina</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Kyriakou%2C+Nansia%22">Kyriakou, Nansia</searchLink><relatesTo>1</relatesTo><i> kyriakou.nansia@ucy.ac.cy</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Language+Testing%22">Language Testing</searchLink>. Apr2021, Vol. 38 Issue 2, p273-301. 29p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Examination+answer+sheets%22">Examination answer sheets</searchLink><br />*<searchLink fieldCode="DE" term="%22Examinations%22">Examinations</searchLink><br />*<searchLink fieldCode="DE" term="%22Adaptive+testing%22">Adaptive testing</searchLink><br />*<searchLink fieldCode="DE" term="%22Educational+tests+%26+measurements%22">Educational tests & measurements</searchLink><br /><searchLink fieldCode="DE" term="%22Forecasting%22">Forecasting</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This longitudinal study (2002–2014) investigates the stability of rating characteristics of a large group of raters over time in the context of the writing paper of a national high-stakes examination. The study uses one measure of rater severity and two measures of rater consistency. The results suggest that the rating characteristics of individual raters are not stable. Thus, predictions from one administration to the next are difficult, although not impossible. In fact, as the membership of the group of raters changes from year to year, past data on rating characteristics become less useful. When the membership of the group of raters is retained, the community of raters develops more stable characteristics. However, "cultural shocks" (low retention of raters and large numbers of newcomers) destabilize the rating characteristics of the community and predictions become more difficult. We propose practical measures to increase the stability of rating across time and offer methodological suggestions for more efficient rater effect-related research designs and analyses. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=149907239 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1177/0265532220940960 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 29 StartPage: 273 Subjects: – SubjectFull: Examination answer sheets Type: general – SubjectFull: Examinations Type: general – SubjectFull: Adaptive testing Type: general – SubjectFull: Educational tests & measurements Type: general – SubjectFull: Forecasting Type: general Titles: – TitleFull: The longitudinal stability of rating characteristics in an EFL examination: Methodological and substantive considerations. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Lamprianou, Iasonas – PersonEntity: Name: NameFull: Tsagari, Dina – PersonEntity: Name: NameFull: Kyriakou, Nansia IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 04 Text: Apr2021 Type: published Y: 2021 Identifiers: – Type: issn-print Value: 02655322 Numbering: – Type: volume Value: 38 – Type: issue Value: 2 Titles: – TitleFull: Language Testing Type: main |
| ResultId | 1 |
