View in EDS HTML Full Text PDF Full Text

Validating Rubric Scoring Processes: An Application of an Item Response Tree Model.

Saved in:
Bibliographic Details
Title: Validating Rubric Scoring Processes: An Application of an Item Response Tree Model.
Authors: Myers, Aaron J. (AUTHOR), Ames, Allison J. (AUTHOR), Leventhal, Brian C. (AUTHOR), Holzman, Madison A. (AUTHOR)
Source: Applied Measurement in Education. Oct-Dec2020, Vol. 33 Issue 4, p293-308. 16p.
Subjects: Scoring rubrics, Decision trees
Abstract: When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters' scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters' scoring processes, an alternative scoring method was used. With this method, rubric developers' intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed. [ABSTRACT FROM AUTHOR]
Copyright of Applied Measurement in Education is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Psychology and Behavioral Sciences Collection
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: pbh
DbLabel: Psychology and Behavioral Sciences Collection
An: 146514584
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Validating Rubric Scoring Processes: An Application of an Item Response Tree Model.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Myers%2C+Aaron+J%2E%22">Myers, Aaron J.</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ames%2C+Allison+J%2E%22">Ames, Allison J.</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Leventhal%2C+Brian+C%2E%22">Leventhal, Brian C.</searchLink> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Holzman%2C+Madison+A%2E%22">Holzman, Madison A.</searchLink> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Applied+Measurement+in+Education%22">Applied Measurement in Education</searchLink>. Oct-Dec2020, Vol. 33 Issue 4, p293-308. 16p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Scoring+rubrics%22">Scoring rubrics</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+trees%22">Decision trees</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters' scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters' scoring processes, an alternative scoring method was used. With this method, rubric developers' intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Applied Measurement in Education is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=146514584
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/08957347.2020.1789143
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 293
    Subjects:
      – SubjectFull: Scoring rubrics
        Type: general
      – SubjectFull: Decision trees
        Type: general
    Titles:
      – TitleFull: Validating Rubric Scoring Processes: An Application of an Item Response Tree Model.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Myers, Aaron J.
      – PersonEntity:
          Name:
            NameFull: Ames, Allison J.
      – PersonEntity:
          Name:
            NameFull: Leventhal, Brian C.
      – PersonEntity:
          Name:
            NameFull: Holzman, Madison A.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: Oct-Dec2020
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 08957347
          Numbering:
            – Type: volume
              Value: 33
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Applied Measurement in Education
              Type: main
ResultId 1