View in EDS

Rounding in Angoff Ratings

Saved in:
Bibliographic Details
Title: Rounding in Angoff Ratings
Language: English
Authors: Wyse, Adam E.
Source: Practical Assessment, Research & Evaluation. May 2018 23(6).
Availability: Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail: pare@umass.edu; Tel: 413-577-2180; Web site: https://scholarworks.umass.edu/pare
Peer Reviewed: Y
Page Count: 8
Publication Date: 2018
Document Type: Journal Articles
Reports - Research
Descriptors: Interrater Reliability, Evaluation Criteria, Scoring Formulas, Achievement Rating, Rating Scales, Cutting Scores, Standard Setting (Scoring), Medical Education, Correlation, Credentials
ISSN: 1531-7714
Abstract: One common modification to the Angoff standard-setting method is to have panelists round their ratings to the nearest 0.05 or 0.10 instead of 0.01. Several reasons have been offered as to why it may make sense to have panelists round their ratings to the nearest 0.05 or 0.10. In this article, we examine one reason that has been suggested, which is that even if panelists are given the opportunity to provide ratings to the nearest 0.01 they often round their ratings to the nearest 0.05 or 0.10 anyway. Using data from four standard settings, we show that in many cases ratings ended in a 0 or 5 when panelists were given the option of using a scale from 0 to 100 in one-point increments and that only about 9% of all ratings ended in a digit other than a 0 or 5. We also examined the impact of different rounding rules and we found that results were quite similar when using different rounding rules. Additional analyses showed the common phenomenon of panelists giving too high of ratings for hard items and too low of ratings for easy items in comparison to conditional p-values. It is suggested that rounding ratings to the nearest 0.05 or 0.10 represent reasonable alternatives to rounding ratings to the nearest 0.01.
Abstractor: As Provided
Entry Date: 2018
Accession Number: EJ1180138
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1180138
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1180138
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Rounding in Angoff Ratings
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wyse%2C+Adam+E%2E%22">Wyse, Adam E.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Practical+Assessment%2C+Research+%26+Evaluation%22"><i>Practical Assessment, Research & Evaluation</i></searchLink>. May 2018 23(6).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail: pare@umass.edu; Tel: 413-577-2180; Web site: https://scholarworks.umass.edu/pare
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 8
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Criteria%22">Evaluation Criteria</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring+Formulas%22">Scoring Formulas</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Rating%22">Achievement Rating</searchLink><br /><searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Cutting+Scores%22">Cutting Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Standard+Setting+%28Scoring%29%22">Standard Setting (Scoring)</searchLink><br /><searchLink fieldCode="DE" term="%22Medical+Education%22">Medical Education</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Credentials%22">Credentials</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1531-7714
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: One common modification to the Angoff standard-setting method is to have panelists round their ratings to the nearest 0.05 or 0.10 instead of 0.01. Several reasons have been offered as to why it may make sense to have panelists round their ratings to the nearest 0.05 or 0.10. In this article, we examine one reason that has been suggested, which is that even if panelists are given the opportunity to provide ratings to the nearest 0.01 they often round their ratings to the nearest 0.05 or 0.10 anyway. Using data from four standard settings, we show that in many cases ratings ended in a 0 or 5 when panelists were given the option of using a scale from 0 to 100 in one-point increments and that only about 9% of all ratings ended in a digit other than a 0 or 5. We also examined the impact of different rounding rules and we found that results were quite similar when using different rounding rules. Additional analyses showed the common phenomenon of panelists giving too high of ratings for hard items and too low of ratings for easy items in comparison to conditional p-values. It is suggested that rounding ratings to the nearest 0.05 or 0.10 represent reasonable alternatives to rounding ratings to the nearest 0.01.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2018
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1180138
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1180138
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 8
    Subjects:
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Evaluation Criteria
        Type: general
      – SubjectFull: Scoring Formulas
        Type: general
      – SubjectFull: Achievement Rating
        Type: general
      – SubjectFull: Rating Scales
        Type: general
      – SubjectFull: Cutting Scores
        Type: general
      – SubjectFull: Standard Setting (Scoring)
        Type: general
      – SubjectFull: Medical Education
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Credentials
        Type: general
    Titles:
      – TitleFull: Rounding in Angoff Ratings
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wyse, Adam E.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Type: published
              Y: 2018
          Identifiers:
            – Type: issn-electronic
              Value: 1531-7714
          Numbering:
            – Type: volume
              Value: 23
            – Type: issue
              Value: 6
          Titles:
            – TitleFull: Practical Assessment, Research & Evaluation
              Type: main
ResultId 1