View in EDS

The Sensitivity of Value-Added Estimates to Test Scoring Decisions. EdWorkingPaper No. 25-1226

Saved in:
Bibliographic Details
Title: The Sensitivity of Value-Added Estimates to Test Scoring Decisions. EdWorkingPaper No. 25-1226
Language: English
Authors: Joshua B. Gilbert (ORCID 0000-0003-3496-2710), James G. Soland (ORCID 0000-0001-8895-2871), Benjamin W. Domingue (ORCID 0000-0002-3894-9049), Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 31
Publication Date: 2025
Document Type: Reports - Research
Descriptors: Value Added Models, Tests, Testing, Scoring, Item Response Theory, Test Reliability, Scores, Statistical Distributions, Error of Measurement, Decision Making, Computation
Abstract: Value-Added Models (VAMs) are both common and controversial in education policy and accountability research. While the sensitivity of VAMs to model specification and covariate selection is well documented, the extent to which test scoring methods (e.g., mean scores vs. IRT-based scores) may affect VA estimates is less studied. We examine the sensitivity of VA estimates to scoring method using empirical item response data from 23 education datasets. We show that VA estimates are frequently highly sensitive to scoring method, holding constant students and items. While the various test scores are highly correlated, on average, different scoring approaches result in VA percentile ranks that vary by over 20 points, and over 50% of teachers or schools ranked in more than one quartile of the VA distribution. Dispersion in VA ranks is reduced with complete item response data and more consistent correlations between baseline and endline scores across scoring methods. We conclude that consideration of both measurement error and model uncertainty is necessary for appropriate interpretation of VAMs.
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED674118
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED674118
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED674118
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Sensitivity of Value-Added Estimates to Test Scoring Decisions. EdWorkingPaper No. 25-1226
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Joshua+B%2E+Gilbert%22">Joshua B. Gilbert</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-3496-2710">0000-0003-3496-2710</externalLink>)<br /><searchLink fieldCode="AR" term="%22James+G%2E+Soland%22">James G. Soland</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8895-2871">0000-0001-8895-2871</externalLink>)<br /><searchLink fieldCode="AR" term="%22Benjamin+W%2E+Domingue%22">Benjamin W. Domingue</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3894-9049">0000-0002-3894-9049</externalLink>)<br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 31
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Value+Added+Models%22">Value Added Models</searchLink><br /><searchLink fieldCode="DE" term="%22Tests%22">Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Testing%22">Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Distributions%22">Statistical Distributions</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Value-Added Models (VAMs) are both common and controversial in education policy and accountability research. While the sensitivity of VAMs to model specification and covariate selection is well documented, the extent to which test scoring methods (e.g., mean scores vs. IRT-based scores) may affect VA estimates is less studied. We examine the sensitivity of VA estimates to scoring method using empirical item response data from 23 education datasets. We show that VA estimates are frequently highly sensitive to scoring method, holding constant students and items. While the various test scores are highly correlated, on average, different scoring approaches result in VA percentile ranks that vary by over 20 points, and over 50% of teachers or schools ranked in more than one quartile of the VA distribution. Dispersion in VA ranks is reduced with complete item response data and more consistent correlations between baseline and endline scores across scoring methods. We conclude that consideration of both measurement error and model uncertainty is necessary for appropriate interpretation of VAMs.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED674118
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED674118
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 31
    Subjects:
      – SubjectFull: Value Added Models
        Type: general
      – SubjectFull: Tests
        Type: general
      – SubjectFull: Testing
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Statistical Distributions
        Type: general
      – SubjectFull: Error of Measurement
        Type: general
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: Computation
        Type: general
    Titles:
      – TitleFull: The Sensitivity of Value-Added Estimates to Test Scoring Decisions. EdWorkingPaper No. 25-1226
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Joshua B. Gilbert
      – PersonEntity:
          Name:
            NameFull: James G. Soland
      – PersonEntity:
          Name:
            NameFull: Benjamin W. Domingue
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 20
              M: 06
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1