Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments
Saved in:
| Title: | Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments |
|---|---|
| Language: | English |
| Authors: | Chunling Niu (ORCID |
| Source: | Practical Assessment, Research & Evaluation. 2025 30(1). |
| Availability: | University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/ |
| Peer Reviewed: | Y |
| Page Count: | 37 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Evaluation Methods, Interrater Reliability, Test Bias, Research Problems, Performance Based Assessment, Scoring Rubrics, Factor Analysis, Error of Measurement, Evaluators |
| ISSN: | 1531-7714 |
| Abstract: | Rater-mediated performance assessments (RMPAs) involve third-party raters evaluating individual performance and are increasingly used across educational, organizational, and research contexts. However, challenges persist in accounting for rater bias and measurement errors, as well as addressing concerns around equity and fairness, especially for historically marginalized populations. This paper addresses these challenges by first discussing the methodological limitations of widely used RMPA evaluation techniques based on classical test theory (CTT), including factor analysis, Cronbach's alpha, and interrater reliability analysis. An alternative approach using Many-Facet Rasch Modeling (MFRM) is then introduced. The two frameworks are systematically compared from both theoretical and empirical perspectives. An empirical example using AI safety evaluation data from the DICES dataset demonstrates how MFRM yields enhanced diagnostic insights (including rater severity differences, rating scale functioning issues, and construct dimensionality) that CTT approaches may not readily provide. Finally, commonly used MFRM-based analytical techniques are introduced for typical RMPA evaluation studies. This paper not only aims to enhance the methodological rigor of RMPAs but also seeks to contribute to the ongoing dialogues on creating more equitable and fair performance assessment practices. |
| Abstractor: | As Provided |
| Notes: | https://github.com/google-research-datasets/dices-dataset |
| Entry Date: | 2026 |
| Accession Number: | EJ1495672 |
| Database: | ERIC |
| Abstract: | Rater-mediated performance assessments (RMPAs) involve third-party raters evaluating individual performance and are increasingly used across educational, organizational, and research contexts. However, challenges persist in accounting for rater bias and measurement errors, as well as addressing concerns around equity and fairness, especially for historically marginalized populations. This paper addresses these challenges by first discussing the methodological limitations of widely used RMPA evaluation techniques based on classical test theory (CTT), including factor analysis, Cronbach's alpha, and interrater reliability analysis. An alternative approach using Many-Facet Rasch Modeling (MFRM) is then introduced. The two frameworks are systematically compared from both theoretical and empirical perspectives. An empirical example using AI safety evaluation data from the DICES dataset demonstrates how MFRM yields enhanced diagnostic insights (including rater severity differences, rating scale functioning issues, and construct dimensionality) that CTT approaches may not readily provide. Finally, commonly used MFRM-based analytical techniques are introduced for typical RMPA evaluation studies. This paper not only aims to enhance the methodological rigor of RMPAs but also seeks to contribute to the ongoing dialogues on creating more equitable and fair performance assessment practices. |
|---|---|
| ISSN: | 1531-7714 |
