View in EDS

Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276

Saved in:
Bibliographic Details
Title: Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276
Language: English
Authors: Luke Miratrix, Polina Polskaia, Richard Dorsett, Pei Zhu, Nicholas Commins, J. David Selby, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 67
Publication Date: 2025
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305D220028
Document Type: Reports - Research
Descriptors: Artificial Intelligence, Randomized Controlled Trials, Educational Experiments, Sample Size, Computation, Bayesian Statistics, Regression (Statistics), Statistical Analysis
Abstract: This study compares 18 machine learning methods for estimating heterogeneous treatment effects in randomized controlled trials, using simulations calibrated to two large-scale educational experiments. We evaluate performance across continuous and binary outcomes with diverse and realistic treatment effect heterogeneity patterns, varying sample sizes, covariate complexities, and effect magnitudes. Bayesian Additive Regression Trees with S-learner (BART S) outperforms alternatives on average. While no method predicts individual effects with high accuracy, some show promise in identifying who benefits most or least. An empirical application illustrates how ML methods can reveal heterogeneity patterns beyond conventional subgroup analysis. These findings highlight both the potential and the limitations of ML, offering evidence-based practical guidance for analyzing treatment effect variation in experimental evaluations.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2026
Accession Number: ED678238
Database: ERIC
Description
Abstract:This study compares 18 machine learning methods for estimating heterogeneous treatment effects in randomized controlled trials, using simulations calibrated to two large-scale educational experiments. We evaluate performance across continuous and binary outcomes with diverse and realistic treatment effect heterogeneity patterns, varying sample sizes, covariate complexities, and effect magnitudes. Bayesian Additive Regression Trees with S-learner (BART S) outperforms alternatives on average. While no method predicts individual effects with high accuracy, some show promise in identifying who benefits most or least. An empirical application illustrates how ML methods can reveal heterogeneity patterns beyond conventional subgroup analysis. These findings highlight both the potential and the limitations of ML, offering evidence-based practical guidance for analyzing treatment effect variation in experimental evaluations.