Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators.
Saved in:
| Title: | Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators. |
|---|---|
| Authors: | Wind, Stefanie A. (AUTHOR), Hooper, Alison (AUTHOR), Hallam, Rena (AUTHOR) |
| Source: | Applied Measurement in Education. Apr-Jun2025, Vol. 38 Issue 2, p118-137. 20p. |
| Subjects: | Sample size (Statistics), Rasch models, Evaluators, Simulation methods & models, Authentic assessment |
| Abstract: | Although research on rater-mediated assessments includes considerations related to identifying rater effects in a variety of performance assessment contexts, researchers have not specifically focused on contexts with relatively small examinee sample sizes (N ≤ 100). Inspired by performance assessment contexts in which small examinee sample sizes may be common, such as childcare evaluation systems, we explored rater effect indicators in these conditions. We used a real data illustration from an assessment of home-based childcare providers to provide context for our study. Then, we used a simulation study to consider the performance of rater effect indicators in a wider range of conditions that reflect important aspects of these assessment systems. Overall, our results support the use of rater effect indicators from the Many-Facet Rasch Model to identify rater severity and rater range restriction effects in conditions with small examinee sample sizes. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR] |
| Copyright of Applied Measurement in Education is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Psychology and Behavioral Sciences Collection |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Although research on rater-mediated assessments includes considerations related to identifying rater effects in a variety of performance assessment contexts, researchers have not specifically focused on contexts with relatively small examinee sample sizes (N ≤ 100). Inspired by performance assessment contexts in which small examinee sample sizes may be common, such as childcare evaluation systems, we explored rater effect indicators in these conditions. We used a real data illustration from an assessment of home-based childcare providers to provide context for our study. Then, we used a simulation study to consider the performance of rater effect indicators in a wider range of conditions that reflect important aspects of these assessment systems. Overall, our results support the use of rater effect indicators from the Many-Facet Rasch Model to identify rater severity and rater range restriction effects in conditions with small examinee sample sizes. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 08957347 |
| DOI: | 10.1080/08957347.2025.2573290 |
