View in EDS HTML Full Text PDF Full Text

Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators.

Saved in:
Bibliographic Details
Title: Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators.
Authors: Wind, Stefanie A. (AUTHOR), Hooper, Alison (AUTHOR), Hallam, Rena (AUTHOR)
Source: Applied Measurement in Education. Apr-Jun2025, Vol. 38 Issue 2, p118-137. 20p.
Subjects: Sample size (Statistics), Rasch models, Evaluators, Simulation methods & models, Authentic assessment
Abstract: Although research on rater-mediated assessments includes considerations related to identifying rater effects in a variety of performance assessment contexts, researchers have not specifically focused on contexts with relatively small examinee sample sizes (N ≤ 100). Inspired by performance assessment contexts in which small examinee sample sizes may be common, such as childcare evaluation systems, we explored rater effect indicators in these conditions. We used a real data illustration from an assessment of home-based childcare providers to provide context for our study. Then, we used a simulation study to consider the performance of rater effect indicators in a wider range of conditions that reflect important aspects of these assessment systems. Overall, our results support the use of rater effect indicators from the Many-Facet Rasch Model to identify rater severity and rater range restriction effects in conditions with small examinee sample sizes. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR]
Copyright of Applied Measurement in Education is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Psychology and Behavioral Sciences Collection
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: pbh
DbLabel: Psychology and Behavioral Sciences Collection
An: 189686077
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wind%2C+Stefanie+A%2E%22">Wind, Stefanie A.</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Hooper%2C+Alison%22">Hooper, Alison</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Hallam%2C+Rena%22">Hallam, Rena</searchLink> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Applied+Measurement+in+Education%22">Applied Measurement in Education</searchLink>. Apr-Jun2025, Vol. 38 Issue 2, p118-137. 20p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Sample+size+%28Statistics%29%22">Sample size (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Rasch+models%22">Rasch models</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Simulation+methods+%26+models%22">Simulation methods & models</searchLink><br /><searchLink fieldCode="DE" term="%22Authentic+assessment%22">Authentic assessment</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Although research on rater-mediated assessments includes considerations related to identifying rater effects in a variety of performance assessment contexts, researchers have not specifically focused on contexts with relatively small examinee sample sizes (N ≤ 100). Inspired by performance assessment contexts in which small examinee sample sizes may be common, such as childcare evaluation systems, we explored rater effect indicators in these conditions. We used a real data illustration from an assessment of home-based childcare providers to provide context for our study. Then, we used a simulation study to consider the performance of rater effect indicators in a wider range of conditions that reflect important aspects of these assessment systems. Overall, our results support the use of rater effect indicators from the Many-Facet Rasch Model to identify rater severity and rater range restriction effects in conditions with small examinee sample sizes. We discuss implications for research and practice. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Applied Measurement in Education is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=189686077
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/08957347.2025.2573290
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
        StartPage: 118
    Subjects:
      – SubjectFull: Sample size (Statistics)
        Type: general
      – SubjectFull: Rasch models
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Simulation methods & models
        Type: general
      – SubjectFull: Authentic assessment
        Type: general
    Titles:
      – TitleFull: Detecting Rater Effects with Small Examinee Sample Sizes: Examining the Impacts of Rating Design and Item Sample Size on Rater Effect Indicators.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wind, Stefanie A.
      – PersonEntity:
          Name:
            NameFull: Hooper, Alison
      – PersonEntity:
          Name:
            NameFull: Hallam, Rena
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Text: Apr-Jun2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 08957347
          Numbering:
            – Type: volume
              Value: 38
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Applied Measurement in Education
              Type: main
ResultId 1