Rounding in Angoff Ratings
Saved in:
| Title: | Rounding in Angoff Ratings |
|---|---|
| Language: | English |
| Authors: | Wyse, Adam E. |
| Source: | Practical Assessment, Research & Evaluation. May 2018 23(6). |
| Availability: | Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail: pare@umass.edu; Tel: 413-577-2180; Web site: https://scholarworks.umass.edu/pare |
| Peer Reviewed: | Y |
| Page Count: | 8 |
| Publication Date: | 2018 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Interrater Reliability, Evaluation Criteria, Scoring Formulas, Achievement Rating, Rating Scales, Cutting Scores, Standard Setting (Scoring), Medical Education, Correlation, Credentials |
| ISSN: | 1531-7714 |
| Abstract: | One common modification to the Angoff standard-setting method is to have panelists round their ratings to the nearest 0.05 or 0.10 instead of 0.01. Several reasons have been offered as to why it may make sense to have panelists round their ratings to the nearest 0.05 or 0.10. In this article, we examine one reason that has been suggested, which is that even if panelists are given the opportunity to provide ratings to the nearest 0.01 they often round their ratings to the nearest 0.05 or 0.10 anyway. Using data from four standard settings, we show that in many cases ratings ended in a 0 or 5 when panelists were given the option of using a scale from 0 to 100 in one-point increments and that only about 9% of all ratings ended in a digit other than a 0 or 5. We also examined the impact of different rounding rules and we found that results were quite similar when using different rounding rules. Additional analyses showed the common phenomenon of panelists giving too high of ratings for hard items and too low of ratings for easy items in comparison to conditional p-values. It is suggested that rounding ratings to the nearest 0.05 or 0.10 represent reasonable alternatives to rounding ratings to the nearest 0.01. |
| Abstractor: | As Provided |
| Entry Date: | 2018 |
| Accession Number: | EJ1180138 |
| Database: | ERIC |
Be the first to leave a comment!
