View in EDS

Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help?

Saved in:
Bibliographic Details
Title: Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help?
Language: English
Authors: Lanah Stafford, Erin Cousins, Linda Bol, Megan Mize
Source: Research & Practice in Assessment. 2023 18(1):19-32.
Availability: Virginia Assessment Group.; Fax: 504-247-1232; e-mail: editor@rpajournal.com; Web site: http://www.rpajournal.com/
Peer Reviewed: Y
Page Count: 14
Publication Date: 2023
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Scoring Rubrics, Reliability, Evaluation Methods, Faculty Development, Group Activities, Training Methods, Norm Referenced Tests, Interdisciplinary Approach, Individual Activities, Comparative Analysis, Interrater Reliability, College Faculty, Undergraduate Study, Portfolio Assessment, Electronic Publishing, Writing Skills, Critical Thinking, Reflection, Synthesis, Employment Potential, Job Skills, Experiential Learning
ISSN: 2161-4210
Abstract: Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting using a popular norming protocol. Results indicate that group training does not improve interrater reliability, though it does improve rater confidence in their rubric scores. Implications include the need for more research comparing individual and group training as well as investigating the efficacy of current training protocols.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1462671
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1462671
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1462671
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Lanah+Stafford%22">Lanah Stafford</searchLink><br /><searchLink fieldCode="AR" term="%22Erin+Cousins%22">Erin Cousins</searchLink><br /><searchLink fieldCode="AR" term="%22Linda+Bol%22">Linda Bol</searchLink><br /><searchLink fieldCode="AR" term="%22Megan+Mize%22">Megan Mize</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+%26+Practice+in+Assessment%22"><i>Research & Practice in Assessment</i></searchLink>. 2023 18(1):19-32.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Virginia Assessment Group.; Fax: 504-247-1232; e-mail: editor@rpajournal.com; Web site: http://www.rpajournal.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Scoring+Rubrics%22">Scoring Rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Development%22">Faculty Development</searchLink><br /><searchLink fieldCode="DE" term="%22Group+Activities%22">Group Activities</searchLink><br /><searchLink fieldCode="DE" term="%22Training+Methods%22">Training Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Norm+Referenced+Tests%22">Norm Referenced Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Interdisciplinary+Approach%22">Interdisciplinary Approach</searchLink><br /><searchLink fieldCode="DE" term="%22Individual+Activities%22">Individual Activities</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22College+Faculty%22">College Faculty</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Study%22">Undergraduate Study</searchLink><br /><searchLink fieldCode="DE" term="%22Portfolio+Assessment%22">Portfolio Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Electronic+Publishing%22">Electronic Publishing</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Skills%22">Writing Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Critical+Thinking%22">Critical Thinking</searchLink><br /><searchLink fieldCode="DE" term="%22Reflection%22">Reflection</searchLink><br /><searchLink fieldCode="DE" term="%22Synthesis%22">Synthesis</searchLink><br /><searchLink fieldCode="DE" term="%22Employment+Potential%22">Employment Potential</searchLink><br /><searchLink fieldCode="DE" term="%22Job+Skills%22">Job Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Experiential+Learning%22">Experiential Learning</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2161-4210
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Integrative learning is an important outcome for graduates of higher education. Therefore, it should be well-defined and assessed reliably. The American Association of Colleges & Universities has developed a rubric to define and assess integrative learning, but it has low reliability. This pilot study examines whether this rubric's reliability can be improved by training users on how to use the rubric in a group setting rather than individually. Twelve faculty were trained to score undergraduate ePortfolios using the Integrative and Applied Learning VALUE Rubric. Half of the faculty were trained in an individual setting and half in a group setting using a popular norming protocol. Results indicate that group training does not improve interrater reliability, though it does improve rater confidence in their rubric scores. Implications include the need for more research comparing individual and group training as well as investigating the efficacy of current training protocols.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1462671
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1462671
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 19
    Subjects:
      – SubjectFull: Scoring Rubrics
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Faculty Development
        Type: general
      – SubjectFull: Group Activities
        Type: general
      – SubjectFull: Training Methods
        Type: general
      – SubjectFull: Norm Referenced Tests
        Type: general
      – SubjectFull: Interdisciplinary Approach
        Type: general
      – SubjectFull: Individual Activities
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: College Faculty
        Type: general
      – SubjectFull: Undergraduate Study
        Type: general
      – SubjectFull: Portfolio Assessment
        Type: general
      – SubjectFull: Electronic Publishing
        Type: general
      – SubjectFull: Writing Skills
        Type: general
      – SubjectFull: Critical Thinking
        Type: general
      – SubjectFull: Reflection
        Type: general
      – SubjectFull: Synthesis
        Type: general
      – SubjectFull: Employment Potential
        Type: general
      – SubjectFull: Job Skills
        Type: general
      – SubjectFull: Experiential Learning
        Type: general
    Titles:
      – TitleFull: Improving Reliability in Assessing Integrative Learning Using Rubrics: Does Group Norming Help?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lanah Stafford
      – PersonEntity:
          Name:
            NameFull: Erin Cousins
      – PersonEntity:
          Name:
            NameFull: Linda Bol
      – PersonEntity:
          Name:
            NameFull: Megan Mize
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-electronic
              Value: 2161-4210
          Numbering:
            – Type: volume
              Value: 18
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Research & Practice in Assessment
              Type: main
ResultId 1