Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276
Saved in:
| Title: | Comparing Machine Learning Methods for Estimating Heterogeneous Treatment Effects in Randomized Trials: A Comprehensive Simulation Study. EdWorkingPaper No. 25-1276 |
|---|---|
| Language: | English |
| Authors: | Luke Miratrix, Polina Polskaia, Richard Dorsett, Pei Zhu, Nicholas Commins, J. David Selby, Annenberg Institute for School Reform at Brown University |
| Source: | Annenberg Institute for School Reform at Brown University. 2025. |
| Availability: | Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/ |
| Peer Reviewed: | N |
| Page Count: | 67 |
| Publication Date: | 2025 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305D220028 |
| Document Type: | Reports - Research |
| Descriptors: | Artificial Intelligence, Randomized Controlled Trials, Educational Experiments, Sample Size, Computation, Bayesian Statistics, Regression (Statistics), Statistical Analysis |
| Abstract: | This study compares 18 machine learning methods for estimating heterogeneous treatment effects in randomized controlled trials, using simulations calibrated to two large-scale educational experiments. We evaluate performance across continuous and binary outcomes with diverse and realistic treatment effect heterogeneity patterns, varying sample sizes, covariate complexities, and effect magnitudes. Bayesian Additive Regression Trees with S-learner (BART S) outperforms alternatives on average. While no method predicts individual effects with high accuracy, some show promise in identifying who benefits most or least. An empirical application illustrates how ML methods can reveal heterogeneity patterns beyond conventional subgroup analysis. These findings highlight both the potential and the limitations of ML, offering evidence-based practical guidance for analyzing treatment effect variation in experimental evaluations. |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2026 |
| Accession Number: | ED678238 |
| Database: | ERIC |
Be the first to leave a comment!
