Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help?
Saved in:
| Title: | Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help? |
|---|---|
| Language: | English |
| Authors: | Lanah Stafford, Erin Cousins, Linda Bol, Megan Mize |
| Source: | Research & Practice in Assessment. 2023 18(1):19-32. |
| Availability: | Virginia Assessment Group.; Fax: 504-247-1232; e-mail: editor@rpajournal.com; Web site: http://www.rpajournal.com/ |
| Peer Reviewed: | Y |
| Page Count: | 14 |
| Publication Date: | 2023 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Scoring Rubrics, Reliability, Evaluation Methods, Faculty Development, Group Activities, Training Methods, Norm Referenced Tests, Interdisciplinary Approach, Individual Activities, Comparative Analysis, Interrater Reliability, College Faculty, Undergraduate Study, Portfolio Assessment, Electronic Publishing, Writing Skills, Critical Thinking, Reflection, Synthesis, Employment Potential, Job Skills, Experiential Learning |
| ISSN: | 2161-4210 |
| Abstract: | Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting using a popular norming protocol. Results indicate that group training does not improve interrater reliability, though it does improve rater confidence in their rubric scores. Implications include the need for more research comparing individual and group training as well as investigating the efficacy of current training protocols. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1462671 |
| Database: | ERIC |
| Abstract: | Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting using a popular norming protocol. Results indicate that group training does not improve interrater reliability, though it does improve rater confidence in their rubric scores. Implications include the need for more research comparing individual and group training as well as investigating the efficacy of current training protocols. |
|---|---|
| ISSN: | 2161-4210 |
