Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions.
Saved in:
| Title: | Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. |
|---|---|
| Authors: | Schauber, Stefan K.1,2 stefanks@uio.no, Olsen, Anne O.3, Werner, Erik L.4, Magelssen, Morten5 |
| Source: | Advances in Health Sciences Education. Nov2024, Vol. 29 Issue 5, p1749-1767. 19p. |
| Subject Terms: | *Pass-fail grading system, *Decision making, *Medical education, *Educational evaluation, Heuristic |
| Abstract: | Introduction: Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts. Methods: We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics. Results: We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs. Discussion: Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices. [ABSTRACT FROM AUTHOR] |
| Copyright of Advances in Health Sciences Education is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 180805933 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Schauber%2C+Stefan+K%2E%22">Schauber, Stefan K.</searchLink><relatesTo>1,2</relatesTo><i> stefanks@uio.no</i><br /><searchLink fieldCode="AR" term="%22Olsen%2C+Anne+O%2E%22">Olsen, Anne O.</searchLink><relatesTo>3</relatesTo><br /><searchLink fieldCode="AR" term="%22Werner%2C+Erik+L%2E%22">Werner, Erik L.</searchLink><relatesTo>4</relatesTo><br /><searchLink fieldCode="AR" term="%22Magelssen%2C+Morten%22">Magelssen, Morten</searchLink><relatesTo>5</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Advances+in+Health+Sciences+Education%22">Advances in Health Sciences Education</searchLink>. Nov2024, Vol. 29 Issue 5, p1749-1767. 19p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Pass-fail+grading+system%22">Pass-fail grading system</searchLink><br />*<searchLink fieldCode="DE" term="%22Decision+making%22">Decision making</searchLink><br />*<searchLink fieldCode="DE" term="%22Medical+education%22">Medical education</searchLink><br />*<searchLink fieldCode="DE" term="%22Educational+evaluation%22">Educational evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Heuristic%22">Heuristic</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Introduction: Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts. Methods: We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics. Results: We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs. Discussion: Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Advances in Health Sciences Education is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=180805933 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10459-024-10328-0 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 19 StartPage: 1749 Subjects: – SubjectFull: Pass-fail grading system Type: general – SubjectFull: Decision making Type: general – SubjectFull: Medical education Type: general – SubjectFull: Educational evaluation Type: general – SubjectFull: Heuristic Type: general Titles: – TitleFull: Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Schauber, Stefan K. – PersonEntity: Name: NameFull: Olsen, Anne O. – PersonEntity: Name: NameFull: Werner, Erik L. – PersonEntity: Name: NameFull: Magelssen, Morten IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 11 Text: Nov2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 13824996 Numbering: – Type: volume Value: 29 – Type: issue Value: 5 Titles: – TitleFull: Advances in Health Sciences Education Type: main |
| ResultId | 1 |
