View in EDS

Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments

Saved in:
Bibliographic Details
Title: Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments
Language: English
Authors: Chunling Niu (ORCID 0000-0002-9106-0417), Kelly Bradley (ORCID 0000-0002-4682-8212), Rui Jin (ORCID 0000-0001-5363-454X), Ashley Love (ORCID 0000-0002-4024-796X)
Source: Practical Assessment, Research & Evaluation. 2025 30(1).
Availability: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
Peer Reviewed: Y
Page Count: 37
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Evaluation Methods, Interrater Reliability, Test Bias, Research Problems, Performance Based Assessment, Scoring Rubrics, Factor Analysis, Error of Measurement, Evaluators
ISSN: 1531-7714
Abstract: Rater-mediated performance assessments (RMPAs) involve third-party raters evaluating individual performance and are increasingly used across educational, organizational, and research contexts. However, challenges persist in accounting for rater bias and measurement errors, as well as addressing concerns around equity and fairness, especially for historically marginalized populations. This paper addresses these challenges by first discussing the methodological limitations of widely used RMPA evaluation techniques based on classical test theory (CTT), including factor analysis, Cronbach's alpha, and interrater reliability analysis. An alternative approach using Many-Facet Rasch Modeling (MFRM) is then introduced. The two frameworks are systematically compared from both theoretical and empirical perspectives. An empirical example using AI safety evaluation data from the DICES dataset demonstrates how MFRM yields enhanced diagnostic insights (including rater severity differences, rating scale functioning issues, and construct dimensionality) that CTT approaches may not readily provide. Finally, commonly used MFRM-based analytical techniques are introduced for typical RMPA evaluation studies. This paper not only aims to enhance the methodological rigor of RMPAs but also seeks to contribute to the ongoing dialogues on creating more equitable and fair performance assessment practices.
Abstractor: As Provided
Notes: https://github.com/google-research-datasets/dices-dataset
Entry Date: 2026
Accession Number: EJ1495672
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1495672
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1495672
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Chunling+Niu%22">Chunling Niu</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-9106-0417">0000-0002-9106-0417</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kelly+Bradley%22">Kelly Bradley</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4682-8212">0000-0002-4682-8212</externalLink>)<br /><searchLink fieldCode="AR" term="%22Rui+Jin%22">Rui Jin</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5363-454X">0000-0001-5363-454X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Ashley+Love%22">Ashley Love</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4024-796X">0000-0002-4024-796X</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Practical+Assessment%2C+Research+%26+Evaluation%22"><i>Practical Assessment, Research & Evaluation</i></searchLink>. 2025 30(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 37
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Problems%22">Research Problems</searchLink><br /><searchLink fieldCode="DE" term="%22Performance+Based+Assessment%22">Performance Based Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring+Rubrics%22">Scoring Rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Factor+Analysis%22">Factor Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1531-7714
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Rater-mediated performance assessments (RMPAs) involve third-party raters evaluating individual performance and are increasingly used across educational, organizational, and research contexts. However, challenges persist in accounting for rater bias and measurement errors, as well as addressing concerns around equity and fairness, especially for historically marginalized populations. This paper addresses these challenges by first discussing the methodological limitations of widely used RMPA evaluation techniques based on classical test theory (CTT), including factor analysis, Cronbach's alpha, and interrater reliability analysis. An alternative approach using Many-Facet Rasch Modeling (MFRM) is then introduced. The two frameworks are systematically compared from both theoretical and empirical perspectives. An empirical example using AI safety evaluation data from the DICES dataset demonstrates how MFRM yields enhanced diagnostic insights (including rater severity differences, rating scale functioning issues, and construct dimensionality) that CTT approaches may not readily provide. Finally, commonly used MFRM-based analytical techniques are introduced for typical RMPA evaluation studies. This paper not only aims to enhance the methodological rigor of RMPAs but also seeks to contribute to the ongoing dialogues on creating more equitable and fair performance assessment practices.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://github.com/google-research-datasets/dices-dataset
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1495672
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1495672
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 37
    Subjects:
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Research Problems
        Type: general
      – SubjectFull: Performance Based Assessment
        Type: general
      – SubjectFull: Scoring Rubrics
        Type: general
      – SubjectFull: Factor Analysis
        Type: general
      – SubjectFull: Error of Measurement
        Type: general
      – SubjectFull: Evaluators
        Type: general
    Titles:
      – TitleFull: Systematic Comparison of Two Approaches for Evaluating and Using Rater-Mediated Performance Assessments
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Chunling Niu
      – PersonEntity:
          Name:
            NameFull: Kelly Bradley
      – PersonEntity:
          Name:
            NameFull: Rui Jin
      – PersonEntity:
          Name:
            NameFull: Ashley Love
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 1531-7714
          Numbering:
            – Type: volume
              Value: 30
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Practical Assessment, Research & Evaluation
              Type: main
ResultId 1