Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?
Saved in:
| Title: | Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions? |
|---|---|
| Language: | English |
| Authors: | Robert B. Olsen, Larry L. Orr, Stephen H. Bell, Elizabeth Petraglia, Elena Badillo-Goicoechea, Atsushi Miyaoka, Elizabeth A. Stuart |
| Source: | Journal of Research on Educational Effectiveness. 2024 17(1):184-210. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 27 |
| Publication Date: | 2024 |
| Sponsoring Agency: | National Institute of Mental Health (NIMH) (DHHS/NIH) |
| Contract Number: | P50MH115842 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Accuracy, Predictor Variables, Randomized Controlled Trials, Regression (Statistics), Bayesian Statistics, Statistics, Models, Influences, Intervention, Program Effectiveness, Measurement Objectives, Guidance Programs, Federal Programs |
| DOI: | 10.1080/19345747.2023.2180464 |
| ISSN: | 1934-5747 1934-5739 |
| Abstract: | Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods--lasso regression and Bayesian Additive Regression Trees (BART)--using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Accession Number: | EJ1408304 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods--lasso regression and Bayesian Additive Regression Trees (BART)--using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites. |
|---|---|
| ISSN: | 1934-5747 1934-5739 |
| DOI: | 10.1080/19345747.2023.2180464 |
