View in EDS

Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276

Saved in:
Bibliographic Details
Title: Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276
Language: English
Authors: Luke Miratrix, Polina Polskaia, Richard Dorsett, Pei Zhu, Nicholas Commins, J. David Selby, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 67
Publication Date: 2025
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D220028
Document Type: Reports - Research
Descriptors: Artificial Intelligence, Randomized Controlled Trials, Educational Experiments, Sample Size, Computation, Bayesian Statistics, Regression (Statistics), Statistical Analysis
Abstract: This study compares 18 machine learning methods for estimating heterogeneous treatment effects in randomized controlled trials, using simulations calibrated to two large-scale educational experiments. We evaluate performance across continuous and binary outcomes with diverse and realistic treatment effect heterogeneity patterns, varying sample sizes, covariate complexities, and effect magnitudes. Bayesian Additive Regression Trees with S-learner (BART S) outperforms alternatives on average. While no method predicts individual effects with high accuracy, some show promise in identifying who benefits most or least. An empirical application illustrates how ML methods can reveal heterogeneity patterns beyond conventional subgroup analysis. These findings highlight both the potential and the limitations of ML, offering evidence-based practical guidance for analyzing treatment effect variation in experimental evaluations.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2026
Accession Number: ED678238
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678238
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678238
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Luke+Miratrix%22">Luke Miratrix</searchLink><br /><searchLink fieldCode="AR" term="%22Polina+Polskaia%22">Polina Polskaia</searchLink><br /><searchLink fieldCode="AR" term="%22Richard+Dorsett%22">Richard Dorsett</searchLink><br /><searchLink fieldCode="AR" term="%22Pei+Zhu%22">Pei Zhu</searchLink><br /><searchLink fieldCode="AR" term="%22Nicholas+Commins%22">Nicholas Commins</searchLink><br /><searchLink fieldCode="AR" term="%22J%2E+David+Selby%22">J. David Selby</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 67
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305D220028
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Experiments%22">Educational Experiments</searchLink><br /><searchLink fieldCode="DE" term="%22Sample+Size%22">Sample Size</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Bayesian+Statistics%22">Bayesian Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+%28Statistics%29%22">Regression (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study compares 18 machine learning methods for estimating heterogeneous treatment effects in randomized controlled trials, using simulations calibrated to two large-scale educational experiments. We evaluate performance across continuous and binary outcomes with diverse and realistic treatment effect heterogeneity patterns, varying sample sizes, covariate complexities, and effect magnitudes. Bayesian Additive Regression Trees with S-learner (BART S) outperforms alternatives on average. While no method predicts individual effects with high accuracy, some show promise in identifying who benefits most or least. An empirical application illustrates how ML methods can reveal heterogeneity patterns beyond conventional subgroup analysis. These findings highlight both the potential and the limitations of ML, offering evidence-based practical guidance for analyzing treatment effect variation in experimental evaluations.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678238
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678238
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 67
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Educational Experiments
        Type: general
      – SubjectFull: Sample Size
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Bayesian Statistics
        Type: general
      – SubjectFull: Regression (Statistics)
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
    Titles:
      – TitleFull: Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Luke Miratrix
      – PersonEntity:
          Name:
            NameFull: Polina Polskaia
      – PersonEntity:
          Name:
            NameFull: Richard Dorsett
      – PersonEntity:
          Name:
            NameFull: Pei Zhu
      – PersonEntity:
          Name:
            NameFull: Nicholas Commins
      – PersonEntity:
          Name:
            NameFull: J. David Selby
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1