Checkbox grading of large-scale mathematics exams with multiple assessors: Field study on assessors' inter-rater reliability, time investment and usage experience.
Saved in:
| Title: | Checkbox grading of large-scale mathematics exams with multiple assessors: Field study on assessors' inter-rater reliability, time investment and usage experience. |
|---|---|
| Authors: | Moons, Filip1,2 (AUTHOR) f.moons@uu.nl, Vandervieren, Ellen1,2 (AUTHOR) ellen.vandervieren@uantwerpen.be, Colpaert, Jozef1,2 (AUTHOR) jozef.colpaert@uantwerpen.be |
| Source: | Studies in Educational Evaluation. Jun2025, Vol. 85, pN.PAG-N.PAG. 1p. |
| Subject Terms: | *Mathematics exams, *Field research, Multiples (Mathematics), Appraisers, Ticks |
| Abstract: | Assessing exams with multiple assessors is challenging regarding inter-rater reliability and feedback. This paper presents 'checkbox grading,' a digital method where exam designers have predefined checkboxes with both feedback and associated partial grades. Assessors then tick the checkboxes relevant to a student solution. Dependencies between checkboxes ensure consistency among assessors in following the grading scheme. Moreover, the approach supports 'blind grading' by hiding the grades associated with the checkboxes, thus focusing assessors on the criteria rather than the scores. The approach was studied during a large-scale mathematics state exam. Results show that assessors perceived checkbox grading as very useful. However, compared to traditional grading—where assessors follow a correction scheme and communicate the resulting grade—more time is spent on checkbox grading, while both approaches are equally reliable. Blind grading improved inter-rater reliability for some tasks. Overall, checkbox grading might lead to a smoother process where feedback, not solely grades, is communicated to students. • Checkbox grading uses feedback-linked checkboxes; assessors tick those that apply. • Checkbox dependencies make grading dynamic, forming a flowchart-like structure. • Scores can be shown (visible) or hidden (blind) from assessors during grading. • Blind checkbox grading improves reliability when strict interpretation is needed. • Assessors prefer visible grading for clarity and validation of their judgments. [ABSTRACT FROM AUTHOR] |
| Copyright of Studies in Educational Evaluation is the property of Pergamon Press - An Imprint of Elsevier Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 184947879 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Checkbox grading of large-scale mathematics exams with multiple assessors: Field study on assessors' inter-rater reliability, time investment and usage experience. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Moons%2C+Filip%22">Moons, Filip</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)<i> f.moons@uu.nl</i><br /><searchLink fieldCode="AR" term="%22Vandervieren%2C+Ellen%22">Vandervieren, Ellen</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)<i> ellen.vandervieren@uantwerpen.be</i><br /><searchLink fieldCode="AR" term="%22Colpaert%2C+Jozef%22">Colpaert, Jozef</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)<i> jozef.colpaert@uantwerpen.be</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Studies+in+Educational+Evaluation%22">Studies in Educational Evaluation</searchLink>. Jun2025, Vol. 85, pN.PAG-N.PAG. 1p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Mathematics+exams%22">Mathematics exams</searchLink><br />*<searchLink fieldCode="DE" term="%22Field+research%22">Field research</searchLink><br /><searchLink fieldCode="DE" term="%22Multiples+%28Mathematics%29%22">Multiples (Mathematics)</searchLink><br /><searchLink fieldCode="DE" term="%22Appraisers%22">Appraisers</searchLink><br /><searchLink fieldCode="DE" term="%22Ticks%22">Ticks</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Assessing exams with multiple assessors is challenging regarding inter-rater reliability and feedback. This paper presents 'checkbox grading,' a digital method where exam designers have predefined checkboxes with both feedback and associated partial grades. Assessors then tick the checkboxes relevant to a student solution. Dependencies between checkboxes ensure consistency among assessors in following the grading scheme. Moreover, the approach supports 'blind grading' by hiding the grades associated with the checkboxes, thus focusing assessors on the criteria rather than the scores. The approach was studied during a large-scale mathematics state exam. Results show that assessors perceived checkbox grading as very useful. However, compared to traditional grading—where assessors follow a correction scheme and communicate the resulting grade—more time is spent on checkbox grading, while both approaches are equally reliable. Blind grading improved inter-rater reliability for some tasks. Overall, checkbox grading might lead to a smoother process where feedback, not solely grades, is communicated to students. • Checkbox grading uses feedback-linked checkboxes; assessors tick those that apply. • Checkbox dependencies make grading dynamic, forming a flowchart-like structure. • Scores can be shown (visible) or hidden (blind) from assessors during grading. • Blind checkbox grading improves reliability when strict interpretation is needed. • Assessors prefer visible grading for clarity and validation of their judgments. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Studies in Educational Evaluation is the property of Pergamon Press - An Imprint of Elsevier Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=184947879 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.stueduc.2024.101443 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 1 StartPage: N.PAG Subjects: – SubjectFull: Mathematics exams Type: general – SubjectFull: Field research Type: general – SubjectFull: Multiples (Mathematics) Type: general – SubjectFull: Appraisers Type: general – SubjectFull: Ticks Type: general Titles: – TitleFull: Checkbox grading of large-scale mathematics exams with multiple assessors: Field study on assessors' inter-rater reliability, time investment and usage experience. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Moons, Filip – PersonEntity: Name: NameFull: Vandervieren, Ellen – PersonEntity: Name: NameFull: Colpaert, Jozef IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 06 Text: Jun2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 0191491X Numbering: – Type: volume Value: 85 Titles: – TitleFull: Studies in Educational Evaluation Type: main |
| ResultId | 1 |
