View in EDS

Controlling for Measurement Error in Evaluation Models When Treatment Group Assignment Is Based on Noisy Measures: Evaluation of an Achievement Gap-Closing Initiative. EdWorkingPaper No. 25-1291

Saved in:
Bibliographic Details
Title: Controlling for Measurement Error in Evaluation Models When Treatment Group Assignment Is Based on Noisy Measures: Evaluation of an Achievement Gap-Closing Initiative. EdWorkingPaper No. 25-1291
Language: English
Authors: Robert H. Meyer, Michael S. Christian, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 33
Publication Date: 2025
Sponsoring Agency: Region 10 Comprehensive Center
Document Type: Reports - Evaluative
Descriptors: Error of Measurement, Achievement Gap, Achievement Gains, Intervention, Computation, Statistical Bias, Accuracy, Academic Achievement, Correlation, Value Added Models
Abstract: This paper develops new models to evaluate the effects of interventions and intervention-by-site heterogeneity when treatment group assignment is based on a fallible variable and the outcome of interest is determined in part by the corresponding true control variables (measured without error). The specific application concerns a school report card redesign in which school performance is evaluated based on the achievement growth of students in the bottom quartile of prior achievement. We show using Monte Carlo data that the traditional errors-in-variables estimator (EV) produces severely biased estimates of the gap-closing initiative. We develop an augmented EV estimator (AEV) that addresses this bias and is shown to produce highly accurate estimates in Monte Carlo simulations. We also show how AEV can be implemented using regression calibration (RC). Using state data, we find that there are essentially no differences in the average growth in student achievement between students in and not in the lowest quartile. However, the noise-corrected correlation in school growth estimates for the two groups is high (around 0.8), but not perfect. These findings are important given that most (if not all) state accountability systems prioritize reporting of school performance for multiple student sub-groups, including groups with large gaps in prior student achievement.
Abstractor: As Provided
Entry Date: 2026
Accession Number: ED678256
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678256
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678256
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Controlling for Measurement Error in Evaluation Models When Treatment Group Assignment Is Based on Noisy Measures: Evaluation of an Achievement Gap-Closing Initiative. EdWorkingPaper No. 25-1291
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Robert+H%2E+Meyer%22">Robert H. Meyer</searchLink><br /><searchLink fieldCode="AR" term="%22Michael+S%2E+Christian%22">Michael S. Christian</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 33
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Region 10 Comprehensive Center
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Gap%22">Achievement Gap</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Gains%22">Achievement Gains</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Bias%22">Statistical Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Value+Added+Models%22">Value Added Models</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper develops new models to evaluate the effects of interventions and intervention-by-site heterogeneity when treatment group assignment is based on a fallible variable and the outcome of interest is determined in part by the corresponding true control variables (measured without error). The specific application concerns a school report card redesign in which school performance is evaluated based on the achievement growth of students in the bottom quartile of prior achievement. We show using Monte Carlo data that the traditional errors-in-variables estimator (EV) produces severely biased estimates of the gap-closing initiative. We develop an augmented EV estimator (AEV) that addresses this bias and is shown to produce highly accurate estimates in Monte Carlo simulations. We also show how AEV can be implemented using regression calibration (RC). Using state data, we find that there are essentially no differences in the average growth in student achievement between students in and not in the lowest quartile. However, the noise-corrected correlation in school growth estimates for the two groups is high (around 0.8), but not perfect. These findings are important given that most (if not all) state accountability systems prioritize reporting of school performance for multiple student sub-groups, including groups with large gaps in prior student achievement.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678256
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678256
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 33
    Subjects:
      – SubjectFull: Error of Measurement
        Type: general
      – SubjectFull: Achievement Gap
        Type: general
      – SubjectFull: Achievement Gains
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Statistical Bias
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Value Added Models
        Type: general
    Titles:
      – TitleFull: Controlling for Measurement Error in Evaluation Models When Treatment Group Assignment Is Based on Noisy Measures: Evaluation of an Achievement Gap-Closing Initiative. EdWorkingPaper No. 25-1291
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Robert H. Meyer
      – PersonEntity:
          Name:
            NameFull: Michael S. Christian
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1