View in EDS

Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. EdWorkingPaper No. 25-1173

Saved in:
Bibliographic Details
Title: Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. EdWorkingPaper No. 25-1173
Language: English
Authors: Joshua B. Gilbert (ORCID 0000-0003-3496-2710), Zachary Himmelsbach (ORCID 0000-0002-5444-0648), Luke W. Miratrix (ORCID 0000-0002-0078-1906), Andrew D. Ho (ORCID 0000-0003-1287-9844), Benjamin W. Domingue (ORCID 0000-0002-3894-9049), Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 56
Publication Date: 2025
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D240025
Document Type: Reports - Research
Education Level: Secondary Education
Descriptors: Value Added Models, Reliability, Effect Size, Test Items, Generalizability Theory, Foreign Countries, Secondary School Students, Secondary School Teachers
Geographic Terms: Tanzania
Abstract: Value added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items. Standard VAMs estimate causal effects on the items that are included on the test. Generalizability demands consideration of how estimates would differ had the test included alternative items. We introduce a model that estimates the magnitude of item-by-teacher/school variance accurately, revealing that standard VAMs overstate reliability and overestimate differences between units. Using a case study and 41 measures from 25 studies with item-level outcome data, we show how standard VAMs overstate reliability by an average of 0.12 on the 0-1 reliability scale (median = 0.09, SD = 0.13) and provide standard deviations of teacher/school effects that are on average 22% too large (median = 7%, SD = 41%). We discuss how imprecision due to heterogeneous VA effects across items attenuates effect sizes, obfuscates comparisons across studies, and causes instability over time. Our results suggest that accurate estimation and interpretation of VAMs requires item-level data, including qualitative data about how items represent the content domain.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2025
Accession Number: ED674059
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED674059
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED674059
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. EdWorkingPaper No. 25-1173
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Joshua+B%2E+Gilbert%22">Joshua B. Gilbert</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-3496-2710">0000-0003-3496-2710</externalLink>)<br /><searchLink fieldCode="AR" term="%22Zachary+Himmelsbach%22">Zachary Himmelsbach</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-5444-0648">0000-0002-5444-0648</externalLink>)<br /><searchLink fieldCode="AR" term="%22Luke+W%2E+Miratrix%22">Luke W. Miratrix</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-0078-1906">0000-0002-0078-1906</externalLink>)<br /><searchLink fieldCode="AR" term="%22Andrew+D%2E+Ho%22">Andrew D. Ho</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1287-9844">0000-0003-1287-9844</externalLink>)<br /><searchLink fieldCode="AR" term="%22Benjamin+W%2E+Domingue%22">Benjamin W. Domingue</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3894-9049">0000-0002-3894-9049</externalLink>)<br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 56
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305D240025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Value+Added+Models%22">Value Added Models</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Effect+Size%22">Effect Size</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Generalizability+Theory%22">Generalizability Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Teachers%22">Secondary School Teachers</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Tanzania%22">Tanzania</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Value added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items. Standard VAMs estimate causal effects on the items that are included on the test. Generalizability demands consideration of how estimates would differ had the test included alternative items. We introduce a model that estimates the magnitude of item-by-teacher/school variance accurately, revealing that standard VAMs overstate reliability and overestimate differences between units. Using a case study and 41 measures from 25 studies with item-level outcome data, we show how standard VAMs overstate reliability by an average of 0.12 on the 0-1 reliability scale (median = 0.09, SD = 0.13) and provide standard deviations of teacher/school effects that are on average 22% too large (median = 7%, SD = 41%). We discuss how imprecision due to heterogeneous VA effects across items attenuates effect sizes, obfuscates comparisons across studies, and causes instability over time. Our results suggest that accurate estimation and interpretation of VAMs requires item-level data, including qualitative data about how items represent the content domain.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED674059
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED674059
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 56
    Subjects:
      – SubjectFull: Value Added Models
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Effect Size
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Generalizability Theory
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Secondary School Teachers
        Type: general
      – SubjectFull: Tanzania
        Type: general
    Titles:
      – TitleFull: Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. EdWorkingPaper No. 25-1173
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Joshua B. Gilbert
      – PersonEntity:
          Name:
            NameFull: Zachary Himmelsbach
      – PersonEntity:
          Name:
            NameFull: Luke W. Miratrix
      – PersonEntity:
          Name:
            NameFull: Andrew D. Ho
      – PersonEntity:
          Name:
            NameFull: Benjamin W. Domingue
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 04
              M: 04
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1