View in EDS PDF Full Text

Detecting Measurement Disturbances in Rater-Mediated Assessments.

Saved in:
Bibliographic Details
Title: Detecting Measurement Disturbances in Rater-Mediated Assessments.
Authors: Wind, Stefanie A.1 swind@ua.edu, Schumacker, Randall E.1
Source: Educational Measurement: Issues & Practice. Winter2017, Vol. 36 Issue 4, p44-51. 8p.
Subject Terms: *Academic achievement, *Educational tests & measurements, SERVQUAL (Service quality framework), Psychometrics, Psychiatric rating scales
Abstract: The term measurement disturbance has been used to describe systematic conditions that affect a measurement process, resulting in a compromised interpretation of person or item estimates. Measurement disturbances have been discussed in relation to systematic response patterns associated with items and persons, such as start-up, plodding, boredom, or fatigue. An understanding of the different types of measurement disturbances can lead to a more complete understanding of persons or items in terms of the construct being measured. Although measurement disturbances have been explored in several contexts, they have not been explicitly considered in the context of performance assessments. The purpose of this study is to illustrate the use of graphical methods to explore measurement disturbances related to raters within the context of a writing assessment. Graphical displays that illustrate the alignment between expected and empirical rater response functions are considered as they relate to indicators of rating quality based on the Rasch model. Results suggest that graphical displays can be used to identify measurement disturbances for raters related to specific ranges of student achievement that suggest potential rater bias. Further, results highlight the added diagnostic value of graphical displays for detecting measurement disturbances that are not captured using Rasch model-data fit statistics. [ABSTRACT FROM AUTHOR]
Copyright of Educational Measurement: Issues & Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: ehh
DbLabel: Education Research Complete
An: 126564541
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Detecting Measurement Disturbances in Rater-Mediated Assessments.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wind%2C+Stefanie+A%2E%22">Wind, Stefanie A.</searchLink><relatesTo>1</relatesTo><i> swind@ua.edu</i><br /><searchLink fieldCode="AR" term="%22Schumacker%2C+Randall+E%2E%22">Schumacker, Randall E.</searchLink><relatesTo>1</relatesTo>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Educational+Measurement%3A+Issues+%26+Practice%22">Educational Measurement: Issues & Practice</searchLink>. Winter2017, Vol. 36 Issue 4, p44-51. 8p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Academic+achievement%22">Academic achievement</searchLink><br />*<searchLink fieldCode="DE" term="%22Educational+tests+%26+measurements%22">Educational tests & measurements</searchLink><br /><searchLink fieldCode="DE" term="%22SERVQUAL+%28Service+quality+framework%29%22">SERVQUAL (Service quality framework)</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Psychiatric+rating+scales%22">Psychiatric rating scales</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The term measurement disturbance has been used to describe systematic conditions that affect a measurement process, resulting in a compromised interpretation of person or item estimates. Measurement disturbances have been discussed in relation to systematic response patterns associated with items and persons, such as start-up, plodding, boredom, or fatigue. An understanding of the different types of measurement disturbances can lead to a more complete understanding of persons or items in terms of the construct being measured. Although measurement disturbances have been explored in several contexts, they have not been explicitly considered in the context of performance assessments. The purpose of this study is to illustrate the use of graphical methods to explore measurement disturbances related to raters within the context of a writing assessment. Graphical displays that illustrate the alignment between expected and empirical rater response functions are considered as they relate to indicators of rating quality based on the Rasch model. Results suggest that graphical displays can be used to identify measurement disturbances for raters related to specific ranges of student achievement that suggest potential rater bias. Further, results highlight the added diagnostic value of graphical displays for detecting measurement disturbances that are not captured using Rasch model-data fit statistics. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Educational Measurement: Issues & Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=126564541
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.12164
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 8
        StartPage: 44
    Subjects:
      – SubjectFull: Academic achievement
        Type: general
      – SubjectFull: Educational tests & measurements
        Type: general
      – SubjectFull: SERVQUAL (Service quality framework)
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Psychiatric rating scales
        Type: general
    Titles:
      – TitleFull: Detecting Measurement Disturbances in Rater-Mediated Assessments.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wind, Stefanie A.
      – PersonEntity:
          Name:
            NameFull: Schumacker, Randall E.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Text: Winter2017
              Type: published
              Y: 2017
          Identifiers:
            – Type: issn-print
              Value: 07311745
          Numbering:
            – Type: volume
              Value: 36
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Educational Measurement: Issues & Practice
              Type: main
ResultId 1