View in EDS HTML Full Text PDF Full Text

Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

Saved in:
Bibliographic Details
Title: Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?
Language: English
Authors: Robert B. Olsen, Larry L. Orr, Stephen H. Bell, Elizabeth Petraglia, Elena Badillo-Goicoechea, Atsushi Miyaoka, Elizabeth A. Stuart
Source: Journal of Research on Educational Effectiveness. 2024 17(1):184-210.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 27
Publication Date: 2024
Sponsoring Agency: National Institute of Mental Health (NIMH) (DHHS/NIH)
Contract Number: P50MH115842
Document Type: Journal Articles
Reports - Research
Descriptors: Accuracy, Predictor Variables, Randomized Controlled Trials, Regression (Statistics), Bayesian Statistics, Statistics, Models, Influences, Intervention, Program Effectiveness, Measurement Objectives, Guidance Programs, Federal Programs
DOI: 10.1080/19345747.2023.2180464
ISSN: 1934-5747
1934-5739
Abstract: Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods--lasso regression and Bayesian Additive Regression Trees (BART)--using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1408304
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHQnjGN8aynPc_mSbg6iEtpAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDofEsq67nKUTVP06wIBEICBm5CsWO5YszhJElQN0kWeYJSPowF_-5TUwzpujGtlziI3egpq9Bf3sDMsSGDVF6FOAfcKLYKEo5TZYx4gO5ISgTqJgyuOMf2GXjGGlHMkdsklqg3inmGY3-_ZFkXvFP7ty0rKSs5rFHUUQZl5Ip0thBk3uzNPXL8qvnBM9rHQBFiBnpE9NPI62J_sI9mpJJ1riLjeKP8ObYwnYZE8
Text:
  Availability: 1
  Value: <anid>AN0174878178;[5ew9]01jan.24;2024Jan22.05:38;v2.2.500</anid> <title id="AN0174878178-1">Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions? </title> <p>Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods—lasso regression and Bayesian Additive Regression Trees (BART)—using a wide range of moderator variables. The main study findings are that: (<reflink idref="bib1" id="ref1">1</reflink>) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (<reflink idref="bib2" id="ref2">2</reflink>) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (<reflink idref="bib3" id="ref3">3</reflink>) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.</p> <p>Keywords: Randomized controlled trials; generalizability; external validity; evidence-based policy; transportability</p> <hd id="AN0174878178-2">Introduction</hd> <p></p> <hd id="AN0174878178-3">Motivation</hd> <p>The goal of evidence-based policy is to use credible evidence to inform policy decisions. A key policy decision faced by policymakers in education and other areas is whether to adopt an intervention—e.g., a new curriculum or an after-school program. Policymakers aiming to improve outcomes for a certain group of individuals should consider whether adopting various interventions would improve outcomes in their context and improve them by enough to warrant the costs. However, the impacts of policy decisions are never known in advance. At best, policymakers have evidence that can be used to predict the impact of their policy decisions. A key question, then, is how accurately policymakers can predict the impact of their decisions—in their local context—in advance and based on the evidence available.</p> <p>To produce evidence of program impact, social policy research has often turned to multi-site, randomized controlled trials (RCTs) because they provide internally valid estimates of the effects of interventions. However, while RCTs are designed to produce unbiased estimates of average impact across all participating sites—and whatever population those sites represent—they are not designed to produce unbiased predictions of impact for individual sites. This is an important limitation when policy decisions are made locally. For example, suppose a school principal is trying to decide whether to adopt an educational program. If the impact of the intervention varies across schools, evidence on the average impact in a multi-site trial may not accurately predict the impact of adopting the intervention in the principal's school.</p> <p>Predicting the impact of an educational intervention for a single school district or school using evidence from a multi-site RCT is particularly challenging when the impact of the intervention varies across sites in the population. And such variation is not uncommon. Weiss et al. ([<reflink idref="bib25" id="ref4">25</reflink>]) provides evidence of cross-site impact variation from 16 RCTs of 16 different interventions, finding large variation for some interventions and little or no variation for others. These results suggest that for at least some interventions, variation in impacts could challenge efforts to predict the impacts in individual sites using evidence from multi-site RCTs.</p> <p>In principle, researchers can address these challenges by modeling the variation in impacts across sites and using those models to predict the impact of the intervention in any site. However, in practice, it is often challenging to build a model that makes accurate predictions. Importantly, the correct model is never known: Researchers are probably unaware of some key variables that moderate the impact of the intervention—or may be aware but the variables not available in the data—and the relationship between these moderators and impacts is never known. Facing imperfect knowledge and substantial data collection costs, researchers conducting RCTs may fail to collect and include some important site-level impact moderators from their models. In addition, building a model that makes accurate out-of-sample predictions is challenging because the relatively small number of sites included in most multi-site RCTs restricts the number of site-level moderator variables that can be included in any one model. Moreover, multi-site RCTs are generally powered to only detect an overall average effect and may have limited ability to estimate moderation effects with precision. Selecting the most important moderators to include in the model may also be challenging without strong prior evidence, which researchers conducting multi-site RCTs often lack. Finally, suppose that all these challenges can be overcome, and researchers are able to build a rich model that produces accurate predictions of impact for individual sites, conditional on site-level moderators collected as part of the study. This model would still be unusable for out-of-sample prediction unless data on the same moderators are available for sites outside of the study sample. In summary, there are serious challenges to building models that are useful for predicting the impact of interventions in individual sites outside the RCT sample—and limited empirical evidence on how well this can be done in practice.</p> <p>In recent work, Orr et al. ([<reflink idref="bib21" id="ref5">21</reflink>]) found that the estimated average impact in the sample from an RCT generally produced inaccurate impact predictions for individual sites, and that basic models developed to account for variation in impacts across sites did not improve the accuracy of those predictions. Specifically, they developed and tested two-level hierarchical linear models of students nested within sites, where the impact for an individual site was specified to be a linear function of site-level moderators—in particular, moderators that were typically publicly available to ensure that they would be known to local policymakers and thus usable for predicting local impacts. These models were used to predict site-level impacts in three RCTs of interventions targeted at children and youth: (<reflink idref="bib1" id="ref6">1</reflink>) educational technology, (<reflink idref="bib2" id="ref7">2</reflink>) charter schools, and (<reflink idref="bib3" id="ref8">3</reflink>) Head Start. The paper demonstrated that using the estimated average impact in the sample as the predicted impact for individual sites generated large prediction errors: The Root Mean Squared Prediction Error ranged from 0.07 to 0.35 standard deviations (measured in the same effect size units as the study-reported impacts). These estimates likely reflect variation in impacts across sites, with large prediction errors when the cross-site impact variance is high and small prediction errors when the cross-site impact variance is low. In addition, the paper found that models with one, two, or five moderators did not yield impact predictions for individual sites that were more accurate than simply using the estimated average impact.</p> <p>However, a few limitations of Orr et al. ([<reflink idref="bib21" id="ref9">21</reflink>]) suggest that additional research is warranted. First, the study examined only three RCTs; evidence from additional studies is needed to build a broader evidence base on this topic. Second, the simple linear models that it used did not take advantage of recent advances in predictive models, especially for contexts with limited sample sizes and large numbers of predictors. Third, Orr et al. ([<reflink idref="bib21" id="ref10">21</reflink>]) focused exclusively on publicly available characteristics of schools when building models, based on the argument that these variables would be known to local policymakers when considering whether to implement the intervention, and excluded other types of moderators that may explain impact variation (Weiss et al., [<reflink idref="bib24" id="ref11">24</reflink>]). However, in some circumstances, local policymakers may be able to predict these other moderators in advance, such as the characteristics of the students likely to participate in the intervention and features of the counterfactual condition faced by students locally. If so, excluding these variables could have led to overly pessimistic conclusions about the potential for using multi-site RCTs to make accurate out-of-sample impact predictions for individual sites.</p> <hd id="AN0174878178-4">Contribution of the Paper</hd> <p>This study extends prior research in three ways. First, it expands the scope and likely generalizability of the evidence base by analyzing a larger number of multi-site RCTs. In addition to reanalyzing data from the evaluation of charter schools examined in Orr et al. ([<reflink idref="bib21" id="ref12">21</reflink>]) (Gleason et al. [<reflink idref="bib12" id="ref13">12</reflink>]), it analyzes data from evaluations of Teach for America (Clark et al., [<reflink idref="bib4" id="ref14">4</reflink>]), the Teaching Fellows Program (Clark et al., [<reflink idref="bib4" id="ref15">4</reflink>]), the U.S. Department of Education's Student Mentoring Program (Bernstein et al., [<reflink idref="bib1" id="ref16">1</reflink>]), the Gang Resistance Education and Training Program (G.R.E.A.T., Esbensen et al., [<reflink idref="bib9" id="ref17">9</reflink>]), and a summer reading program (Wilkins et al., [<reflink idref="bib26" id="ref18">26</reflink>]).</p> <p>Second, the paper improves on the analysis in Orr et al. ([<reflink idref="bib21" id="ref19">21</reflink>]) by testing two modern statistical methods for making out-of-sample predictions: (<reflink idref="bib1" id="ref20">1</reflink>) lasso regression (Tibshirani, [<reflink idref="bib23" id="ref21">23</reflink>]) and (<reflink idref="bib2" id="ref22">2</reflink>) Bayesian Additive Regression Trees (BART; Chipman et al., [<reflink idref="bib3" id="ref23">3</reflink>]). Orr et al. ([<reflink idref="bib21" id="ref24">21</reflink>]) used standard linear regression models with moderator variables selected based on <emph>p</emph> values. However, this approach risks producing models that are overfit to the sample data and yield poor out-of-sample predictions. In contrast, lasso and BART are designed for making out-of-sample predictions and thus may be better suited to informing local policy decisions in sites not included in the evaluation. Lasso's use of cross-validation to select variables and shrink some coefficients toward zero suggests that it may help to avoid overfitting that can lead to inaccurate out-of-sample predictions.[<reflink idref="bib1" id="ref25">1</reflink>] In addition, lasso has been shown to work well in relatively small sample sizes (as few as 20), with more possible parameters (<emph>p</emph>) than observations (<emph>n</emph>) (e.g., Donoho & Stodden, [<reflink idref="bib5" id="ref26">5</reflink>]; Eberlin et al., [<reflink idref="bib8" id="ref27">8</reflink>]; Finch & Finch, [<reflink idref="bib10" id="ref28">10</reflink>]). BART's main strength is its ability to uncover patterns in the data, including nonlinear relationships and high-order interactions, without having to assume a specific functional form for the model or to pre-specify which interactions to include. Kern et al. ([<reflink idref="bib16" id="ref29">16</reflink>]) found that BART performed better than most other methods in a variety of scenarios when generalizing impact estimates to target populations.</p> <p>Third, this study uses a more comprehensive range of site-level moderators to predict the impact in individual sites. In particular, it examines the extent to which data from multi-site trials on four different types of effect moderators—individually or together—can improve the ability of statistical models to predict the impact of interventions in individual sites. In addition to publicly available information on schools and districts used by Orr et al. ([<reflink idref="bib21" id="ref30">21</reflink>]) (e.g., size, student composition, resources, and location), this study uses data collected by the original study teams on three different types of moderators: (<reflink idref="bib1" id="ref31">1</reflink>) characteristics of students participating in the study, (<reflink idref="bib2" id="ref32">2</reflink>) features of the intervention or how it was implemented, and (<reflink idref="bib3" id="ref33">3</reflink>) features of the counterfactual condition faced by individuals assigned to the control group. We assess whether models that include all four types of moderators yield more accurate predictions than models that are restricted to just publicly available information on schools and districts.</p> <p>Fourth, this paper takes a new approach to estimating the magnitude of the likely prediction errors. This approach, detailed below, uses random effects meta-analysis methods to estimate the Root Mean Squared Prediction Error as a function of the bias, sampling error in the predicted impacts, and cross-site heterogeneity in impacts.</p> <p>We focus on the magnitude of prediction errors because local policymakers are in a better position to make policy decisions that will improve student outcomes if informed by impact predictions with small errors than by impact predictions with large errors. While Orr et al. ([<reflink idref="bib21" id="ref34">21</reflink>]) translates prediction errors into estimates of their consequences for decision-making—specifically, on whether policymakers will make the "correct" decision about whether to adopt an intervention based on the evidence—we focus here exclusively on the magnitude of the prediction errors and save the policy consequences of these errors for future research.</p> <hd id="AN0174878178-5">Theory</hd> <p>The goal of this study is to test different methods for predicting the true impact of an intervention in an individual site, and to assess the expected accuracy of those predictions from a randomly selected site from the population. Let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow></msub></math> </ephtml> be the estimand of interest—the true impact in site</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi><mo>,</mo></math> </ephtml> for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi><mo>=</mo><mn>1</mn><mo>,</mo><mo>...</mo><mo>,</mo><mi>N</mi><mo>.</mo></math> </ephtml> For a particular prediction method, let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup></math> </ephtml> be expected value of the site-specific impact prediction—that is, the parameter to which the predicted impact for site</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math> </ephtml> converges as the number of sites in the multi-site RCT used to estimate the prediction model converges to infinity. Under these assumptions, the bias of the prediction method for site</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math> </ephtml> equals</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>≡</mo><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>−</mo><msub><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>.</mo></math> </ephtml> We would in general expect the prediction models to be biased unless the variance of the true impact across sites–</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi>τ</mi></mrow><mrow><mn>2</mn></mrow></msup></math> </ephtml> —equal zero or the assumptions of the model explaining variation in impacts strictly hold.</p> <p>The accuracy of the predicted impact for an individual site depends on both the bias and the variance of those prediction. Since the predictions come from a multi-site RCT, the estimated parameters of the prediction model will contain sampling error that depends on the number of sites included in the RCT. Let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup></math> </ephtml> be the predicted impact from a finite sample and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>ε</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup></math> </ephtml> be the random error in the model's prediction for site</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi><mo>.</mo></math> </ephtml> We model this predicted impact with the equation below:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>=</mo><msub><mrow><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>ε</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>=</mo><mi>δ</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>+</mo><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>+</mo><msubsup><mrow><mi>ε</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>,</mo></math> </ephtml> (<reflink idref="bib1" id="ref35">1</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>∼</mo><mo stretchy="true">(</mo><mrow><mi>δ</mi><mo>,</mo><msup><mrow><mi>τ</mi></mrow><mrow><mn>2</mn></mrow></msup></mrow><mo stretchy="true">)</mo><mo>,</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>∼</mo><mo stretchy="true">(</mo><mrow><mi>b</mi><mo>,</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mrow><mo stretchy="true">)</mo><mo>,</mo></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>ε</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>∼</mo><mo stretchy="true">(</mo><mrow><mn>0</mn><mo>,</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>p</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mrow><mo stretchy="true">)</mo><mo>.</mo></math> </ephtml> </p> <p>These assumptions allow us to define a measure of the accuracy of the prediction method applied to a multi-site RCT with a finite number of sites for a randomly selected site in the population. The Root Mean Squared Prediction Error (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">RMSPE</mi></math> </ephtml> ) can be expressed as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">RMSPE</mi><mo>=</mo><msqrt><msup><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>p</mi></mrow><mrow><mn>2</mn></mrow></msubsup></msqrt></math> </ephtml> (<reflink idref="bib2" id="ref36">2</reflink>)</p> <p>In words, the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">RMSPE</mi></math> </ephtml> is the square root of the sum of the squared average bias of the prediction method (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msup></math> </ephtml> ), the variance of the bias across sites (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> ), and the error variance of the prediction error from estimating the prediction model in a finite sample (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>p</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> ). This formulation will allow us to estimate the RMSPE, as described later, by separately estimating the average bias in the predictions (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math> </ephtml> ) and the sum of the variances of the remaining two components of the prediction error (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>b</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>p</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math> </ephtml> ).</p> <hd id="AN0174878178-6">Data</hd> <p>We used data from six large-scale multi-site RCTs of interventions focused on improving youth outcomes (Table 1). The six RCTs can be thought of a convenience sample because they were chosen largely based on data availability. These RCTs fall within a larger set of studies that are highly visible to policymakers through government reports and evidence clearinghouses. Each of these studies offers an opportunity to test how well we can use the RCT data to make accurate impact predictions for individual sites.</p> <p>Table 1. RCTs included in the analysis.</p> <p> <ephtml> <table><thead><tr><td>Intervention</td><td>Outcome domains</td><td>Study design</td><td>Data source</td></tr><tr><td>Unit of assignment</td><td>Number of sites</td></tr></thead><tbody valign="top"><tr><td>Gang Resistance Education and Training Program (G.R.E.A.T.)</td><td>Behavioral</td><td>Classroom (<italic>N</italic> = 195)</td><td>31 schools</td><td>ICPSR</td></tr><tr><td>Teach for America</td><td>Achievement</td><td>Student (<italic>N</italic> = 4,570)</td><td>45 schools</td><td>NCES</td></tr><tr><td>The Teaching Fellows Program</td><td>Achievement</td><td>Student (<italic>N</italic> = 4,120)</td><td>44 schools</td><td>NCES</td></tr><tr><td>The Student Mentoring Program</td><td>Behavioral, achievement, performance in school</td><td>Student (<italic>N</italic> = 1,970)</td><td>32 program grantees</td><td>NCES</td></tr><tr><td>Charter Middle Schools</td><td>Achievement</td><td>Student (<italic>N</italic> = 2,150)</td><td>36 schools</td><td>NCES</td></tr><tr><td>Summer Reading</td><td>Achievement</td><td>Student (<italic>N</italic> = 1,520)</td><td>112 schools</td><td>NCES</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>. Student sample sizes are rounded to the nearest 10 for compliance with non-disclosure procedures for NCES restricted use data.</p> <p>Student- and site-level data from these studies were used to build site-level analysis files that contain site-level impact estimates for student outcomes and site-level moderator variables that may predict the size and direction of the site-level impacts. Consolidating the student-level data into site-level datasets simplified the application of lasso regression and BART for making site-specific impact predictions without losing any information helpful in predicting the average impact in individual sites.[<reflink idref="bib2" id="ref37">2</reflink>]</p> <p>Each of the six RCTs included multiple sites in the sample and conducted random assignment within sites, allowing us to produce unbiased impact estimates for each site. From a methodological perspective, we defined a "site" as the randomization block in which students (or in one case, classrooms) were randomly assigned. For all but two of the studies, the site was defined as the public school in which the intervention was implemented. For the evaluation of charter middle schools, we defined the site as the collection of public schools from which students applied to a single charter school–or a pair of nearby charter schools with multiple students applying to both—and were either admitted to a charter school or not admitted based on a random lottery. For the evaluation of the Student Mentoring Program, we defined the site as the grantee organization—often a school or school district—that supported the program locally and within which students were randomly assigned.[<reflink idref="bib3" id="ref38">3</reflink>]</p> <p>For each RCT, we used data on one or more student outcomes for which the original study authors estimated and reported the impacts of the intervention. For measures of student achievement based on standardized tests, the student-level outcome variable (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></math> </ephtml> ) was standardized by the original study authors to have a mean of zero and standard deviation of 1 in a defined norming population.[<reflink idref="bib4" id="ref39">4</reflink>] For other continuous or ordinal outcome variables (e.g., the count or number of incidents of a particular type), we created a binary outcome variable equal to 1 or "yes" (e.g., the student had more than one incident) or 0 or "no" (e.g., the student had zero incidents) if the vast majority of students in the sample had no such incidents; otherwise, we standardized the outcome by subtracting off the mean and dividing by the standard deviation for all treatment and control students included in the analysis. Binary outcomes from the original study data (e.g., scored above the state threshold for proficiency in math or reading) were left in their natural scale (0 for no or 1 for yes). See Table 2 for a description of the student outcome variables from the original studies that we used in the analysis along with summary statistics and sample sizes.[<reflink idref="bib5" id="ref40">5</reflink>]</p> <p>Table 2. Student outcome variables included in the analysis.</p> <p> <ephtml> <table><thead><tr><td>Outcome variable</td><td>Description and scale</td><td>Mean</td><td><italic>SD</italic></td><td><italic>N</italic></td></tr></thead><tbody valign="top"><tr><td>Charter Middle Schools</td></tr><tr><td> Math achievement, year 1</td><td>Standardized score on state test</td><td char=".">0.38</td><td char=".">1.01</td><td char=".">2,130</td></tr><tr><td> Math achievement, year 2</td><td>Standardized score on state test</td><td char=".">0.44</td><td char=".">1.14</td><td char=".">2,040</td></tr><tr><td> Reading achievement, year 1</td><td>Standardized score on state test</td><td char=".">0.43</td><td char=".">0.96</td><td char=".">2,150</td></tr><tr><td> Reading achievement, year 2</td><td>Standardized score on state test</td><td char=".">0.39</td><td char=".">0.96</td><td char=".">2,040</td></tr><tr><td>Gang Resistance Education and Training Program (G.R.E.A.T.)</td></tr><tr><td> Delinquency (frequency)</td><td>Number of incidents of delinquency</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">3,564</td></tr><tr><td> Delinquency (variety)</td><td>Number of different types of delinquency</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">3,564</td></tr><tr><td> Violence (any)</td><td>=1 if any violent incidents, =0 otherwise</td><td char=".">0.12</td><td char=".">0.32</td><td char=".">3,564</td></tr><tr><td>Student Mentoring Program</td></tr><tr><td> Absentee rate</td><td>Ratio of absences to days enrolled</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">2,160</td></tr><tr><td> Truancy rate</td><td>Ratio of unexcused absences to days enrolled</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">1,440</td></tr><tr><td> Math grades</td><td>=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">2,000</td></tr><tr><td> English grades</td><td>=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">2,010</td></tr><tr><td> Science grades</td><td>=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">1,950</td></tr><tr><td> Social studies grades</td><td>=1 (F), 2 (D), 3, (C), 4 (B), or 5 (A)</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">1,890</td></tr><tr><td> Math proficiency</td><td>=1 if proficient based on state test, = 0 otherwise</td><td char=".">0.46</td><td char=".">0.5</td><td char=".">1,990</td></tr><tr><td> Reading proficiency</td><td>=1 if proficient based on state test, = 0 otherwise</td><td char=".">0.49</td><td char=".">0.5</td><td char=".">2,030</td></tr><tr><td> Misconduct (any)</td><td>=1 if at least one incident of misconduct, =0 otherwise</td><td char=".">0.26</td><td char=".">0.44</td><td char=".">1,730</td></tr><tr><td> Misconduct (repeated)</td><td>=1 if multiple incidents of misconduct, =0 otherwise</td><td char=".">0.16</td><td char=".">0.37</td><td char=".">1,730</td></tr><tr><td> Delinquency (any)</td><td>=1 if at least one incident of delinquency, =0 otherwise</td><td char=".">0.20</td><td char=".">0.40</td><td char=".">1,780</td></tr><tr><td> Delinquency (repeated)</td><td>=1 if multiple incidents of delinquency, =0 otherwise</td><td char=".">0.11</td><td char=".">0.31</td><td char=".">1,570</td></tr><tr><td>Summer Reading Program</td></tr><tr><td> Reading achievement</td><td>Standardized score on study-administered test (Scholastic Reading Inventory)</td><td char=".">0.00</td><td char=".">1.00</td><td char=".">1,570</td></tr><tr><td>Teach for America</td></tr><tr><td> Math achievement</td><td>Standardized score on state test or study-administered test (NWEA)</td><td char=".">−0.58</td><td char=".">0.93</td><td char=".">4,570</td></tr><tr><td>Teaching Fellows Program</td></tr><tr><td> Math achievement</td><td>Standardized score on state test or study-administered test (NWEA)</td><td char=".">−0.38</td><td char=".">1.07</td><td char=".">4,120</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Notes</emph>. Standardized scores on state tests were standardized by the original study authors as <emph>z</emph>-scores, subtracting off the mean and dividing by the standard deviation for all tested students in the state in the same subject and grade level. Standardized scores on study-administered tests were standardized (<reflink idref="bib1" id="ref41">1</reflink>) relative to a national norming population for Teach for America and the Teaching Fellows Program and (<reflink idref="bib2" id="ref42">2</reflink>) relative to the pooled sample of treatment and control students for the summer reading program. Student sample sizes for all studies except G.R.E.A.T. are rounded to the nearest 10 for compliance with NCES non-disclosure procedures.</p> <p>Finally, we used data on four types of potential impact moderators that were available in the restricted use files for the six multi-site RCTs. Some of these variables come from public sources; others were collected by the original study teams:</p> <p></p> <ulist> <item> <emph>Publicly available information on the characteristics of each site</emph>. The Common Core of Data (CCD) provides publicly available information on all public schools nationwide. For the four studies that assembled CCD data on participating schools, we used data on size, student demographic composition, location, and resources to capture the context in which educational interventions are implemented.</item> <p></p> <item> <emph>Information collected on the subset of students who participated in the study</emph>. Most impact evaluations in education collect data on the characteristics of the students who participated in the study, especially their demographic characteristics, to conduct subgroup analyses. For all six studies, we created site-level averages of the participant characteristics that were collected (e.g., disability status, race/ethnicity, English learner status, and eligibility for free or reduced-price meals) to capture the types of students who would receive the intervention in each site if the intervention were implemented locally.</item> <p></p> <item> <emph>Features of the intervention or how it was implemented</emph>. Many RCTs collect measures related to the intervention itself. For the four studies that collected such measures, we used study-collected measures of intervention features or how the intervention was implemented. These included characteristics of the charter schools attended by students who were admitted by lottery, the characteristics of Teach for America and Teaching Fellows Program teachers for students randomly assigned to their classes, and the characteristics of the local mentoring programs to which students were randomly assigned.</item> <p></p> <item> <emph>Features of the counterfactual condition.</emph> Since impacts result from the contrast between the intervention condition and the counterfactual condition, some RCTs also collect site-level measures of the counterfactual condition experienced by students in the control group. We used study-collected measures of the counterfactual from the four studies that constructed such measures. These include measures of academic programming in the regular public schools attended by students who "lost the lottery" to attend a charter school, characteristics of teachers who taught in the same schools and grade levels as teachers from Teach for America or the Teaching Fellows Program, and participation in mentoring programs by students who were assigned the control group in the evaluation of the U.S. Department of Education's Student Mentoring Program.</item> </ulist> <p>The number of moderator variables included in the analysis for each RCT is provided in Table 3. The definition of each of these moderators is provided in the Online Appendix.</p> <p>Table 3. Number of moderator variables included for each intervention.</p> <p> <ephtml> <table><thead><tr><td>Intervention</td><td>Site characteristics</td><td>Student characteristics (site-level averages)</td><td>Intervention features</td><td>Counterfactual features</td></tr></thead><tbody valign="top"><tr><td>Charter Middle Schools</td><td>21</td><td>21</td><td>90</td><td>90</td></tr><tr><td>G.R.E.A.T.</td><td>0</td><td>67</td><td>0</td><td>0</td></tr><tr><td>Student Mentoring Program</td><td>0</td><td>261</td><td>422</td><td>5</td></tr><tr><td>Summer Reading Program</td><td>0</td><td>14</td><td>0</td><td>0</td></tr><tr><td>Teach for America</td><td>23</td><td>45</td><td>202</td><td>190</td></tr><tr><td>Teaching Fellows Program</td><td>23</td><td>45</td><td>202</td><td>190</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note</emph>. Zero variables were included of a particular type when none were available in the original study data.</p> <hd id="AN0174878178-7">Methods</hd> <p>This study used three types of methods to assess the potential for multi-site RCTs to accurately predict impacts for individual sites outside the study sample: (<reflink idref="bib1" id="ref43">1</reflink>) methods for estimating the true impact in a single site, to serve as a benchmark for assessing different prediction methods, (<reflink idref="bib2" id="ref44">2</reflink>) methods for predicting the impact in that site using only data from other sites, and (<reflink idref="bib3" id="ref45">3</reflink>) methods for assessing the accuracy of those predictions. All three types of methods are described below.</p> <hd id="AN0174878178-8">Methods for Estimating the True Impact in a Single Site</hd> <p>For each of the six multi-site RCTs, we took advantage of its experimental design to estimate the following model and produce an unbiased estimate of the impact of the intervention in each site <emph>j</emph> on student outcome</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>:</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mi>α</mi><mo>+</mo><mi>β</mi><msub><mrow><mi>x</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>+</mo><msub><mrow><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mi>T</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>+</mo><msub><mrow><mi>e</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>,</mo><mi /><msub><mrow><mi>e</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>∼</mo><mi>N</mi><mo stretchy="true">(</mo><mrow><mn>0</mn><mo>,</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mrow><mo stretchy="true">)</mo></math> </ephtml> (<reflink idref="bib3" id="ref46">3</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></math> </ephtml> is the outcome for student <emph>i</emph> in site <emph>j</emph>;</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>x</mi></mrow><mrow><mi>i</mi></mrow></msub></math> </ephtml> is a pre-intervention measure of the outcome for student <emph>i</emph> in site <emph>j</emph>;</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>T</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></math> </ephtml> is the treatment indicator that equals 1 if student <emph>i</emph> in site <emph>j</emph> was assigned to the treatment group and 0 if this student was assigned to the control group; and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>e</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></math> </ephtml> is a random error term that varies across the students in site <emph>j</emph>.[<reflink idref="bib6" id="ref47">6</reflink>] Using the full study sample allowed the original study authors to include a large set of covariates in the models. However, we estimated separate impact regression models for each site, based on a small fraction of the total sample, necessitating a more parsimonious model. In estimating impacts separately by site, we included a single covariate—a pre-intervention value of the outcome—given prior evidence that pretest measures are more predictive of student outcomes than other potential covariates (Bloom et al., [<reflink idref="bib2" id="ref48">2</reflink>]).</p> <p>The parameter</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> reflects the intervention's impact in site <emph>j</emph>, where the "w" superscript denotes that the impact is the impact on students <emph>within</emph> site <emph>j</emph>. We estimated this parameter using (<reflink idref="bib1" id="ref49">1</reflink>) ordinary least squares if the original study authors did not construct and use weights in the impact analysis or (<reflink idref="bib2" id="ref50">2</reflink>) weighted least squares when the study authors did construct and use weights in the impact analysis (typically to address nonresponse bias, variation in the probability of assignment to the treatment, or both).[<reflink idref="bib7" id="ref51">7</reflink>]</p> <hd id="AN0174878178-9">Methods for Predicting the Impact for a Single Site</hd> <p>In broad terms, we tested three methods for predicting the impact of an educational intervention for a site outside of the study sample but that is similar to those in the study sample: (<reflink idref="bib1" id="ref52">1</reflink>) the simple average of the site-level impact estimates for sites in the study sample; (<reflink idref="bib2" id="ref53">2</reflink>) lasso regression; and (<reflink idref="bib3" id="ref54">3</reflink>) BART. The first approach does not use data on site-level moderators; the second and third approaches use site-level moderator data to produce customized impact predictions for individual sites. With all three methods, we excluded a single site, estimated the prediction models in the remaining <emph>J</emph>-1 sites, and used the estimated prediction model—with the site moderator values in the excluded site, for lasso and BART—to predict the impact in the excluded site. The analysis plan was initially published on the Open Science Framework before any analysis was conducted (Olsen et al., [<reflink idref="bib20" id="ref55">20</reflink>]).[<reflink idref="bib8" id="ref56">8</reflink>]</p> <p>For the simple average approach, we took the unweighted average of the unbiased site-level impact estimates and used this average as the predicted impact in the excluded site (referred to below as the "focal" site).[<reflink idref="bib9" id="ref57">9</reflink>] This approach is motivated by evidence-based policymaking that identifies interventions that "work" and promotes them for local adoption.</p> <p>We used lasso regression to fit a linear regression model of the intervention's impact as a function of site-level moderators, and we used the model to predict the impact of the intervention in the excluded site. The lasso was used to penalize the model for large regression coefficients that might improve the model fit in the sample at the expense of lower out-of-sample prediction accuracy. Specifically, we used cross-validation to set the penalty term (also called the "tuning parameter") to minimize the mean squared prediction error. The resulting linear model was used to predict the impact of the intervention in the excluded site, conditional on the site-level moderators in the model.</p> <p>Our third prediction approach, BART, fits a series of regression trees of the intervention's impact as a function of site-level moderators and then uses the set of trees to predict the impact of the intervention in the excluded site. Regression trees sequentially divide the sample into categories, based on binary and continuous variables, to explain variation in the variable being predicted—in this case, the impact of the intervention in individual sites. These trees end in multiple "leaves," or categories defined by multiple variables (e.g., continuous variable X1 > C and binary variable X2 = 0). In our application, the predicted impact for an individual site was set equal to the average impact estimate among the collection of sites in the same category (i.e., at the same leaf). We chose to test a regression tree-based approach to prediction because, unlike standard linear regression models, they are designed for scenarios where the variables used for classification (here, impact moderators) may have non-linear or interacting relationships with the outcome (here, the estimated impact of an education intervention).</p> <p>While many approaches to regression trees exist, we used BART because it is well designed to address the overfitting problem of some random forest approaches that can lead to poor predictions. To avoid producing overly complex trees that overfit the sample data, BART constructs a series of small trees and combines them into an "ensemble," fit using a Bayesian framework with default prior distributions on key parameters (James et al., [<reflink idref="bib14" id="ref58">14</reflink>]). For more details on lasso and BART, and our approach to implementing them, see the Appendix.</p> <p>For both lasso and BART, we implemented a pre-modeling variable selection step, restricting the models to moderators with a correlation of at least 0.20 with the estimated impact. This step was taken to focus the algorithms on variables likely to moderate the impact of the intervention. Our original plan was to offer lasso and BART the full set of potential moderators, regardless of their correlation with the impact and without any prior judgment by us about their relative importance—leaving it entirely to the lasso and BART algorithms to choose the variables that contribute most to producing accurate impact predictions in individual sites. However, we discovered, in testing our approach to the lasso in charter school data without the 0.20 correlation screen, the lasso did not select any predictor variables for the model.[<reflink idref="bib10" id="ref59">10</reflink>] In contrast, setting a minimum correlation of 0.20 led the lasso to select some predictor variables and allowed us to test whether models with predictors selected by the lasso method yielded more accurate predictions than "models" with no predictor variables, as reflected in the simple average of the site-level impact estimates.</p> <hd id="AN0174878178-10">Methods for Assessing the Accuracy of the Predicted Impact for a Single Site</hd> <p>To assess the accuracy with which each of the three methods described above predict the impact of each intervention in a single site, we used and adapted the analytic approach developed in Orr et al. ([<reflink idref="bib21" id="ref60">21</reflink>]):</p> <p></p> <ulist> <item> <emph>Step 1: Designate one site from the RCT as the focal (or "excluded") site.</emph> We pretend that this site—site <emph>j</emph>—did not participate in the evaluation and is the site for which a policy decision is being made, and thus for which an impact prediction is needed.</item> <p></p> <item> <emph>Step 2: Produce an unbiased estimate of the impact in the focal site</emph> (</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> ). Using only the data from the focal site and methods described earlier, we estimated Equation (<reflink idref="bib1" id="ref61">1</reflink>). This produced an unbiased estimate of the true impact in site <emph>j</emph>.</p> <p></p> <ulist> <item> <emph>Step 3: Predict the impact in the focal site using data from the other J-1 sites (</emph> </item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup></math> </ephtml> <emph>).</emph> We fit each of the three prediction methods described earlier using the data from the other <emph>J</emph>-1 sites and then used that model, and moderator values from site <emph>j</emph>, to predict the impact in site <emph>j</emph>.</p> <p></p> <ulist> <item> <emph>Step 4: Estimate the prediction error.</emph> The true prediction error for site <emph>j</emph> equals</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><msub><mrow><mi>P</mi><mi>E</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>=</mo><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>−</mo><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mo>,</mo></math> </ephtml> where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> is the true impact in that site. We estimated the prediction error for site <emph>j</emph>,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mrow><mover accent="true"><mrow><mi>P</mi><mi>E</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow></msub><mo>,</mo></math> </ephtml> by taking the difference between the predicted impact,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>,</mo></math> </ephtml> and the unbiased estimate of the impact in site <emph>j</emph>,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mo>:</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><msub><mrow><mrow><mover accent="true"><mrow><mi>P</mi><mi>E</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow></msub><mo>=</mo><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>−</mo><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mo>.</mo></math> </ephtml> </p> <p></p> <ulist> <item> <emph>Step 5: Repeat steps 1–3 for each of the remaining sites</emph>. This treats each site in the sample as the focal site for a local policy decision and estimates the prediction error for that site.</item> <p></p> <item> <emph>Step 6: Estimate the RMSPE, adjusted for sampling error in the focal site, across all sites in the RCT</emph>. We calculated the RMSPE instead of the mean prediction error to ensure that positive errors do not offset negative errors. To calculate the RMSPE, we conducted a random effects meta-analysis. This meta-analysis accounts for sampling error in the unbiased impact estimate for each site.</item> </ulist> <p>Figure 1 provides an illustration of the site-level estimates used to estimate the RMSPE for different combinations of study, outcome variable, prediction method, and sets of moderator variables. For the Teach for America study, this figure displays the site-level unbiased impact estimates and the predicted impacts generated using BART with all four types of moderator variables.</p> <p>Graph: Figure 1. Comparison of predicted impact to unbiased impact estimate for individual sites in the Teach for America Study: BART predictions with all four types of moderators.</p> <p>The meta-analysis builds on the model developed earlier in the <emph>Theory</emph> section of the paper. To conduct the meta-analysis, we regressed the estimated prediction error (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mrow><mover accent="true"><mrow><mi>P</mi><mi>E</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow></msub></math> </ephtml> ) on an intercept term. The random effects meta-analysis estimated the cross-site variance of the prediction error, excluding the within-site variance of the unbiased impact estimate (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> ).[<reflink idref="bib11" id="ref62">11</reflink>] To estimate the RMPSE, we took the square root of the sum of two terms: (<reflink idref="bib1" id="ref63">1</reflink>) the square of the estimated intercept, which can be interpreted as the average bias in the prediction method, and (<reflink idref="bib2" id="ref64">2</reflink>) the estimated cross-site variance from the meta-analysis, which can be interpreted as the sum of the variance of the prediction bias across sites and the variance of the predicted impacts that results from estimating prediction models in finite samples. The estimated RMSPE provides an estimate of the prediction error that can be expected for a randomly selected site from the population represented by sites that participated in the RCT.</p> <p>To assess the precision of the RMSPE estimates and the differences between RMSPEs from different models, we used the bootstrap method. In particular, 1,000 bootstrap samples of <emph>J</emph> sites were selected with replacement—where <emph>J</emph> equals the number of sites included in the RCT—estimated the RMSPE for each bootstrap sample, and we concluded that prediction method A yielded a lower RMSPE than prediction method B if the estimated RMSPE was lower for A than for B in 950 or more of the 1,000 bootstrap samples. Similarly, we concluded that prediction method B yielded a lower RMSPE than prediction method A if the estimated RMSPE was lower for B than for A in 950 or more of the 1,000 bootstrap samples. In effect, we used bootstrapping to test for significant differences at the 90 percent level with a two-tailed test.</p> <p>Finally, we assessed the magnitude of the RMSPE estimates by comparing them to external estimates of the average impacts of educational interventions. This comparison was made to determine whether the prediction errors were large enough to make an intervention with zero impact appear to have an average impact—one of average magnitude—or an intervention with an average impact appear to have a zero impact. For the four studies that reported impacts on achievement outcomes—the evaluations of charter schools, Teach for America, the Teaching Fellows Program, and the summer reading program—we compared the RMSPE estimates to a threshold value of 0.04 standard deviations, the average impact across 1,352 estimates from randomized trials of the effects of educational interventions on broad tests of student achievement in mathematics or English language arts (Kraft, [<reflink idref="bib17" id="ref65">17</reflink>]).[<reflink idref="bib12" id="ref66">12</reflink>] For the evaluation of G.R.E.A.T., we compared the RMSPE estimates to a threshold value of 0.09, the average effect size across 68 estimates of the effects of school-based programs for preventing problem behaviors in middle school (Wilson et al., [<reflink idref="bib27" id="ref67">27</reflink>]).[<reflink idref="bib13" id="ref68">13</reflink>] For the evaluation of the Student Mentoring Program, we compared the RMSPE estimates to a threshold value of 0.21, the average effect size across 83 estimates of the effects of mentoring programs for youth from randomized trials and non-randomized comparison group designs (DuBois et al., [<reflink idref="bib7" id="ref69">7</reflink>]).[<reflink idref="bib14" id="ref70">14</reflink>] Lastly, the evaluations of G.R.E.A.T. and the Student Mentoring Program reported impacts for some binary outcomes. Because the published evidence from meta-analyses generally focused on continuous outcomes, we lack evidence on the average effects of educational interventions on binary outcomes. For this reason, we simply report the RMSPEs on these outcomes and leave it to readers to make their own assessment of whether prediction errors implied by these estimates should be considered large or small.</p> <hd id="AN0174878178-11">Findings</hd> <p>To provide context for the main findings, Table 4 provides estimates of the unconditional cross-site standard deviation of the impact for each study and outcome variable. These estimates indicate the challenge that prediction methods face in producing accurate impact estimates. When the unconditional cross-site standard deviation of impact is large, the only way to produce accurate impact predictions for individual sites is through a model that explains a sizable share of the impact variation across sites.</p> <p>Table 4. Cross-site standard deviation of impact.</p> <p> <ephtml> <table><thead><tr><td>Outcome variable</td><td>Scale at the student level</td><td>Cross-site standard deviation of impact</td></tr></thead><tbody valign="top"><tr><td>Charter Middle Schools</td></tr><tr><td> Math achievement, year 1</td><td>Continuous (standard deviations)</td><td>0.161</td></tr><tr><td> Math achievement, year 2</td><td>Continuous (standard deviations)</td><td>0.311</td></tr><tr><td> Reading achievement, year 1</td><td>Continuous (standard deviations)</td><td>0.184</td></tr><tr><td> Reading achievement, year 2</td><td>Continuous (standard deviations)</td><td>0.192</td></tr><tr><td>Gang Resistance Education and Training Program (G.R.E.A.T.)</td></tr><tr><td> Delinquency (frequency)</td><td>Ordinal (standard deviations)</td><td>0.000</td></tr><tr><td> Delinquency (variety)</td><td>Ordinal (standard deviations)</td><td>0.000</td></tr><tr><td> Violence (any)</td><td>Binary (0 or 1)</td><td>0.045</td></tr><tr><td>Student Mentoring Program</td></tr><tr><td> Absentee rate</td><td>Continuous (0–1)</td><td>0.001</td></tr><tr><td> Truancy rate</td><td>Continuous (0–1)</td><td>0.001</td></tr><tr><td> Math grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.000</td></tr><tr><td> English grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.000</td></tr><tr><td> Science grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.038</td></tr><tr><td> Social studies grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.043</td></tr><tr><td> Math proficiency</td><td>Binary (0 or 1)</td><td>0.000</td></tr><tr><td> Reading proficiency</td><td>Binary (0 or 1)</td><td>0.000</td></tr><tr><td> Misconduct (any)</td><td>Binary (0 or 1)</td><td>0.019</td></tr><tr><td> Misconduct (repeated)</td><td>Binary (0 or 1)</td><td>0.008</td></tr><tr><td> Delinquency (any)</td><td>Binary (0 or 1)</td><td>0.000</td></tr><tr><td> Delinquency (repeated)</td><td>Binary (0 or 1)</td><td>0.000</td></tr><tr><td>Summer Reading Program</td></tr><tr><td> Reading achievement</td><td>Continuous (standard deviations)</td><td>0.231</td></tr><tr><td>Teach for America</td></tr><tr><td> Math achievement</td><td>Continuous (standard deviations)</td><td>0.174</td></tr><tr><td>Teaching Fellows Program</td></tr><tr><td> Math achievement</td><td>Continuous (standard deviations)</td><td>0.321</td></tr></tbody></table> </ephtml> </p> <p>Table 4 shows that the six studies cover the range of the cross-site impact variation from prior studies. Weiss et al. ([<reflink idref="bib25" id="ref71">25</reflink>]) report estimates of cross-site standard deviation of impacts ranging from 0 for an afterschool math program to 0.35 for math achievement in year 2 from the same charter school evaluation analyzed in this study. Similarly, the cross-site standard deviation reported in Table 4 ranged from 0 for several outcomes in the evaluations of G.R.E.A.T. and the Student Mentoring Program to 0.31 for math achievement in year 2 for the evaluation of charter schools.</p> <p>The first question we addressed is whether a study's estimate of the average impact produces accurate predictions of impact for individual sites. Table 5 presents the RMSPE estimates for the prediction method that uses the simple average of the impact estimates in the other sites. For the four studies that focused on impacts on student achievement measures—studies of charter middle schools, a summer reading program, Teach for America, and the New Teachers Program—the RMSPE estimates are indicative of large prediction errors, ranging from 0.13 for the impact of Teach for America on math achievement to 0.33 standard deviations for the impact of charter middle schools on math achievement in year 2. These RMSPE estimates are all greater than 0.04, the average impact of educational interventions cited earlier (Kraft, [<reflink idref="bib17" id="ref72">17</reflink>]). This suggest that using the simple average of the impact estimates in other sites yields predictions with errors that are large enough to mislead local policymakers into reaching the wrong conclusion about whether the program would be effective locally.</p> <p>Table 5. RMSPE estimates from the simple average of the impact estimates in the other sites.</p> <p> <ephtml> <table><thead><tr><td>Outcome variable</td><td>Scale at the student level</td><td>RMSPE</td></tr></thead><tbody valign="top"><tr><td>Charter Middle Schools</td></tr><tr><td> Math achievement, year 1</td><td>Continuous (standard deviations)</td><td>0.17</td></tr><tr><td> Math achievement, year 2</td><td>Continuous (standard deviations)</td><td>0.32</td></tr><tr><td> Reading achievement, year 1</td><td>Continuous (standard deviations)</td><td>0.19</td></tr><tr><td> Reading achievement, year 2</td><td>Continuous (standard deviations)</td><td>0.20</td></tr><tr><td>Gang Resistance Education and Training Program (G.R.E.A.T.)</td></tr><tr><td> Delinquency (frequency)</td><td>Ordinal (standard deviations)</td><td>0.02</td></tr><tr><td> Delinquency (variety)</td><td>Ordinal (standard deviations)</td><td>0.02</td></tr><tr><td> Violence (any)</td><td>Binary (0 or 1)</td><td>0.05</td></tr><tr><td>Student Mentoring Program</td></tr><tr><td> Absentee rate</td><td>Continuous (0–1)</td><td>0.03</td></tr><tr><td> Truancy rate</td><td>Continuous (0–1)</td><td>0.02</td></tr><tr><td> Math grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.04</td></tr><tr><td> English grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.02</td></tr><tr><td> Science grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.04</td></tr><tr><td> Social studies grades</td><td>Ordinal (1, 2, 3, 4, or 5)</td><td>0.04</td></tr><tr><td> Math proficiency</td><td>Binary (0 or 1)</td><td>0.01</td></tr><tr><td> Reading proficiency</td><td>Binary (0 or 1)</td><td>0.02</td></tr><tr><td> Truancy rate</td><td>Continuous (0–1)</td><td>0.02</td></tr><tr><td> Misconduct (any)</td><td>Binary (0 or 1)</td><td>0.03</td></tr><tr><td> Misconduct (repeated)</td><td>Binary (0 or 1)</td><td>0.01</td></tr><tr><td> Delinquency (any)</td><td>Binary (0 or 1)</td><td>0.02</td></tr><tr><td> Delinquency (repeated)</td><td>Binary (0 or 1)</td><td>0.03</td></tr><tr><td>Summer Reading Program</td></tr><tr><td> Reading achievement</td><td>Continuous (standard deviations)</td><td>0.26</td></tr><tr><td>Teach for America</td></tr><tr><td> Math achievement</td><td>Continuous (standard deviations)</td><td>0.12</td></tr><tr><td>Teaching Fellows Program</td></tr><tr><td> Math achievement</td><td>Continuous (standard deviations)</td><td>0.15</td></tr></tbody></table> </ephtml> </p> <p>In contrast, for the other two studies, the findings suggest that the impact of the intervention in a site can be predicted accurately by taking the simple average of the impact estimates in the other sites. For the evaluation of G.R.E.A.T., the RMSPE estimates in Table 5 for outcomes measured in standard deviation units were close to zero and smaller than the average effect size of 0.09 found for other middle school-based prevention programs (Wilson et al., [<reflink idref="bib27" id="ref73">27</reflink>]). For the evaluation of a federal Student Mentoring Program, the RMSPE estimates for outcomes measured in standard deviation units were generally close to zero and uniformly much smaller than the average effect size of 0.21 found for similar programs (DuBois et al., [<reflink idref="bib7" id="ref74">7</reflink>]). Together, the estimates from these two studies reflect prediction errors that are too small to mislead local policymakers into reaching the wrong conclusion about the likely impacts on these outcomes locally.[<reflink idref="bib15" id="ref75">15</reflink>]</p> <p>The second question we addressed is whether the advanced statistical methods we tested, lasso and BART, combined with different sets of potential moderators, produced more accurate impact predictions than the simple average of the impact estimates in the other sites, and predictions accurate enough to support local policymaking. To address this question, we analyzed data from the four studies that estimated the impact of the intervention on measures of student achievement, and where the simple average consistently yielded inaccurate impact estimates, as evidenced by RMSPE estimates of greater than the threshold that we established earlier: (<reflink idref="bib1" id="ref76">1</reflink>) charter middle schools, (<reflink idref="bib2" id="ref77">2</reflink>) Teach for America, (<reflink idref="bib3" id="ref78">3</reflink>) the Teaching Fellows Program, and (<reflink idref="bib4" id="ref79">4</reflink>) the summer reading program. However, most analyses were restricted to the three studies that collected data on all four types of moderators: (<reflink idref="bib1" id="ref80">1</reflink>) charter middle schools, (<reflink idref="bib2" id="ref81">2</reflink>) Teach for America, and (<reflink idref="bib3" id="ref82">3</reflink>) the Teaching Fellows Program.[<reflink idref="bib16" id="ref83">16</reflink>] We tested the performance of lasso and BART relative to the simple average approach and relative to each other, separately for each of the four types of moderator variables described earlier as well as for all four moderator types together.</p> <p>The results (Figures 2–6) provide little evidence that lasso regression produces more accurate impact predictions for individual sites than the simple average of the impact estimates in the other sites. The method did not produce consistently smaller RMSPE estimates than the simple average of the impact estimates in the other sites for either the four types of moderator variables individually (Figures 2–5) or for all four types together (Figure 6). Lasso regression increased the RMSPE for the Teaching Fellows Program for all tested combinations of moderator variables[<reflink idref="bib17" id="ref84">17</reflink>]; for the other two studies, it increased the RMSPE for some outcomes and types of moderators and reduced it for others.[<reflink idref="bib18" id="ref85">18</reflink>]</p> <p>Graph: Figure 2. RMSPE for different prediction methods using site characteristics as moderators. Note. "*" identifies scenarios in which no moderator variables were offered to lasso or BART because none of them had a correlation of at least 0.20 with the outcome.</p> <p>Graph: Figure 3. RMSPE for different prediction methods using aggregate study participant (student) characteristics as moderators. Note. "*" identifies scenarios in which no moderator variables were offered to lasso or BART because none of them had a correlation of at least 0.20 with the outcome.</p> <p>Graph: Figure 4. RMSPE for different prediction methods using intervention-related characteristics as moderators.</p> <p>Graph: Figure 5. RMSPE for different prediction methods using counterfactual-related characteristics as moderators.</p> <p>Graph: Figure 6. RMSPE for different prediction methods using all four types of moderators.</p> <p>The performance of BART is more mixed, generally reducing the RMSPEs for the charter middle school study but not for the other three studies. For charter middle schools, BART produced predictions with lower RMSPE than the simple average prediction method for 13 of the 15 predictions across the different outcomes and sets of moderator variables. Of these 13 differences, 12 were statistically significant. In contrast, for the other three studies—Teach for America, the Teaching Fellows Program, and the summer reading program—BART produced lower RMSPE for some predictions and higher RMSPE for others. These differences were statistically significant for only one of the nine BART predictions—roughly as we should expect if the differences were purely due to random chance instead of systematic differences in the performance of different methods.</p> <p>However, even with BART, the prediction errors were large enough to potentially make effective interventions appear ineffective and vice versa. For the four studies in which the simple average prediction method produced RMSPE estimates of greater than 0.04 standard deviations—the average impact of educational interventions on broad measures of student achievement—BART was never able to reduce the RMSPE below 0.04 standard deviations. The smallest RMSPE generated by BART was 0.09 standard deviations for predictions of the impact of Teach for America on math achievement using all four types of moderators.</p> <p>Figures 7 and 8 show how much the prediction accuracy of lasso and BART models improve with access to data from all four types of moderators versus data from only a single type of moderator. For lasso, Figure 7 shows that prediction accuracy does not typically improve from having access to all four types of moderators: The RMSPE from the models with all four types (as denoted with the vertical bar) is sometimes larger and sometimes smaller than the RMSPE from models with a single type. In contrast, for BART, Figure 8 shows that prediction accuracy often but not always improves from having access to all four types of moderators. For the charter middle school study, the reductions in RMSPE from BART were greater with all four types of moderators than with just a single type. These differences were statistically significant for 8 of 11 combinations of outcome variable and type of moderator variable. This suggests that for this one study, BART predictions benefit from having access to a wider range of moderator types. This pattern generally held for the study of Teach for America: The estimated RMSPE was lower with all four moderator types than for any single type of moderator, but only one of those four differences were statistically significant. For the study of the Teaching Fellows Program, the results were mixed: The estimated RMSPE was lower with all four moderator types than the RMSPEs with two of the four moderator types but higher than the RMSPEs with the other two types of moderators. Finally, this analysis could not be completed for the summer reading study because the data only contained one type of moderator (characteristics of the students who participated in the study).</p> <p>Graph: Figure 7. RMSPE with all four types of moderators: lasso. Note. "*" identifies scenarios in which no moderator variables were offered to lasso because none of them had a correlation of at least 0.20 with the outcome.</p> <p>Graph: Figure 8. RMSPE with all four types of moderators: BART. Note. "*" identifies scenarios in which no moderator variables were offered to BART because none of them had a correlation of at least 0.20 with the outcome.</p> <hd id="AN0174878178-12">Discussion</hd> <p>The findings from this study show that how accurately multi-site RCTs can predict impacts in individual sites depends on the variation in impacts across sites. For the four RCTs that examined the impact of educational interventions on mathematics or reading achievement, the average impact estimate provided an inaccurate prediction of the intervention's impact for individual sites. In contrast, for the RCTs that focused on behavioral outcomes—the G.R.E.A.T. and student mentoring studies—the average impact estimate did provide fairly accurate impact predictions for individual sites. It appears that the key factor influencing the accuracy of these predictions is the extent to which the impact varies across sites: This variation was substantial for the four studies focused on mathematics and reading achievement but close to zero for the two studies focused on behavioral outcomes. Multi-site RCTs should report evidence on the cross-site impact variation to help readers assess the likelihood that the average impact reported by the RCT provides reasonably accurate impact predictions for individual sites.</p> <p>When the intervention's impact varies substantially across sites, additional findings cast doubt on whether accurate site-level impact predictions can be produced through statistical modeling. While one of the two statistical methods tested—BART—was often able to produce more accurate site-level impact predictions than the simple average of the impact estimates in the other sites, the prediction errors were still large enough to make an effective intervention appear to be ineffective or an ineffective intervention appear to be effective. The prediction errors from BART and lasso were large even with data on four types of potential impact moderators. These types of moderators are not even always available in all RCTs. And even when they are, local policymakers may not be able to predict what the values of those moderators will be locally before the intervention is implemented, making models that include them of questionable usefulness to local policymakers trying to decide whether to implement the intervention. The study's findings suggest that even with moderators that probably could not practically be used to make local impact predictions, predictions from lasso and BART models produced impact predictions that were too inaccurate to inform local policy decisions. In addition, this study examined settings where impacts were being predicted for sites that were randomly—not systematically—different from the sites in the hypothetical studies. The ability to predict site-specific impacts would likely decline even further in scenarios where the sites in a study are not representative of the population of interest for predicting site-specific impacts.</p> <p>However, we believe it is too early in the development of this literature to reach firm, negative conclusions about the ability to use multi-site RCTs to make accurate predictions of impact in individual sites. A more optimistic interpretation of the findings could be justified by the fact that the highest performing approach to modeling impacts—BART with rich data on impact moderators—improved the accuracy of site-level impact predictions for some of the RCTs. Furthermore, while these models are only useful to local policymakers that know or can predict the local values of those moderators in advance, there may be circumstances where local knowledge of key moderators—including the characteristics of the students who would participate and of the counterfactual condition—would allow local policymakers to benefit from these models. Assessing the practical value of these models would require research evidence on what local policymakers know or can predict in advance when deciding whether to adopt an intervention locally.</p> <p>We acknowledge that the results here may be sensitive to the moderator variables that are available for use. Further theoretical or empirical research on the factors that moderate treatment effects could guide randomized trials to collect data on stronger impact moderators, and ultimately lead to better models of impact moderation that can be used to predict impacts in individual sites. Randomized trials are generally not designed to estimate site-specific impacts, and our exploration of their use for this purpose may be pushing them beyond their original goals. However, it is crucial that randomized trials do inform practice, and methods for designing trials to help facilitate their use for predicting impacts for individual sites should be encouraged.</p> <p>Finally, estimates of the magnitude of prediction errors for individual sites should not be the last word on the performance of different prediction methods. The RMSPE estimates produced by this study do not account for the effects that those errors may have on policy decisions about whether to adopt interventions, or the welfare consequences of adopting ineffective interventions—or foregoing effective ones—due to errors in predicting their effects. While Orr et al. ([<reflink idref="bib21" id="ref86">21</reflink>]) takes an initial step in that direction, additional research is needed to identify the conditions under which predictions errors of different magnitudes have serious consequences for decision-making and social welfare.</p> <hd id="AN0174878178-13">Open Research Statements</hd> <p>This manuscript was not required to disclose open research practices, as it was initially submitted prior to JREE mandating open research statements in April 2022.</p> <hd id="AN0174878178-14">Appendix: Lasso and BART for Predicting Impacts in Individual Sites</hd> <p>This appendix describes our approach to estimating regression models and trees using lasso and BART for use in predicting the impacts in individual sites.</p> <hd id="AN0174878178-15">Lasso</hd> <p>Lasso regression is a widely used technique for variable selection in data science and predictive modeling applications (Tibshirani, 1996). We used lasso regression with cross-validation to fit the following prediction model:</p> <p>(A1)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mo>=</mo><mi>α</mi><mo>+</mo><mrow><msubsup><mo stretchy="false">∑</mo><mrow><mi>m</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>M</mi></mrow></msubsup><mrow><msub><mrow><msub><mrow><mi>γ</mi></mrow><mrow><mi>m</mi></mrow></msub><mi>Z</mi></mrow><mrow><mi>m</mi><mi>j</mi></mrow></msub></mrow></mrow><mo>+</mo><msub><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>,</mo><mi /><msub><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>∼</mo><mi>N</mi><mo stretchy="true">(</mo><mrow><mn>0</mn><mo>,</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><mi>u</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mrow><mo stretchy="true">)</mo><mo>,</mo></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>δ</mi></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> is the unbiased estimate of the impact in site <emph>j</emph> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>Z</mi></mrow><mrow><mi>j</mi></mrow></msub></math> </ephtml> is a vector of site-level characteristics which may moderate the intervention's impact.</p> <p>Lasso regression was used to solve for the coefficients that minimize the following function, conditional on the value of the "tuning parameter,"</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>λ</mi></math> </ephtml> :</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>G</mi><mo stretchy="true">(</mo><mrow><mi>α</mi><mo>,</mo><msub><mrow><mi>γ</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mi mathvariant="normal" /><mo>...</mo><mo>,</mo><mi mathvariant="normal" /><msub><mrow><mi>γ</mi></mrow><mrow><mi>m</mi></mrow></msub><mo>|</mo><mi>λ</mi></mrow><mo stretchy="true">)</mo><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>J</mi></mrow></mfrac><mrow><msubsup><mo stretchy="false">∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>J</mi></mrow></msubsup><mrow><msup><mrow><mo stretchy="true">(</mo><mrow><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup><mo>−</mo><mi>α</mi><mo>−</mo><mrow><msubsup><mo stretchy="false">∑</mo><mrow><mi>m</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>M</mi></mrow></msubsup><mrow><msub><mrow><msub><mrow><mi>γ</mi></mrow><mrow><mi>m</mi></mrow></msub><mi>Z</mi></mrow><mrow><mi>m</mi><mi>j</mi></mrow></msub></mrow></mrow></mrow><mo stretchy="true">)</mo></mrow><mrow><mn>2</mn></mrow></msup></mrow></mrow><mo>+</mo><mi>λ</mi><mrow><msubsup><mo stretchy="false">∑</mo><mrow><mi>m</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>M</mi></mrow></msubsup><mrow><mo stretchy="true">|</mo><mrow><msub><mrow><mi>γ</mi></mrow><mrow><mi>m</mi></mrow></msub></mrow><mo stretchy="true">|</mo></mrow></mrow></math> </ephtml> . The tuning parameter imposes a penalty for large coefficient estimates, shrinking many of them to zero to avoid overfitting. Cross-validation was used to set the tuning parameter to the value that minimized the mean squared prediction error. We divided the sample into "folds," which are disjoint subsets of the whole sample. For each fold, we estimated a model of site impacts using "training data" from the other folds pooled together, and we estimated the mean squared prediction error in the "validation data" from the selected fold. Finally, we choose the value of the tuning parameter that minimized the mean squared prediction error across folds. To implement lasso with cross-validation, we used the R package called "glmnet."</p> <p>More specifically, we used the following steps to fit the lasso model and predict the impact on each site:</p> <p></p> <ulist> <item> <bold> Randomly allocate the J sites in the study to K folds. </bold> Our default approach was to divide the sample into 10 folds (e.g., a sample with 30 sites would be divided into 10 folds of 3 sites each).</item> <p></p> </ulist> <hd1 id="AN0174878178-16"> • Set the tuning parameter to an initial value ( </hd1> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">λ</mi><mo>=</mo><msup><mrow><mi mathvariant="bold-italic">λ</mi></mrow><mrow><mi mathvariant="bold-italic">′</mi></mrow></msup></math> </ephtml> <bold>).</bold> Following Friedman et al. (2010), glmnet by default considers 100 possible values, in equal-sized increments.[<reflink idref="bib19" id="ref87">19</reflink>] Let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>λ</mi></mrow><mrow><mi mathvariant="normal">max</mi></mrow></msub></math> </ephtml> equal the value at which the penalized parameters are set to zero. Initially, we set the tuning parameter to</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.001</mn><mo>·</mo><msub><mrow><mi>λ</mi></mrow><mrow><mi mathvariant="normal">max</mi></mrow></msub><mo>.</mo></math> </ephtml> </p> <p></p> <hd1 id="AN0174878178-17"> • Calculate the mean-squared prediction error separately for each fold ( </hd1> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold-italic">MSPE</mi></mrow><mrow><mi mathvariant="bold-italic">k</mi></mrow></msub></math> </ephtml> <bold>).</bold> For each fold, we:</p> <p></p> <ulist> <item> Divided the data into a validation sample, which included the sites in the selected fold, and a training sample, which included the sites in the other <emph>K</emph> − 1 folds;</item> <p></p> <item> Calculated the parameters (</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi><mo>,</mo><msub><mrow><mi>γ</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mi /><mo>...</mo><mo>,</mo><mi /><msub><mrow><mi>γ</mi></mrow><mrow><mi>M</mi></mrow></msub></math> </ephtml> ) that minimized</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>G</mi><mo stretchy="true">(</mo><mrow><mi>α</mi><mo>,</mo><msub><mrow><mi>γ</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mi /><mo>...</mo><mo>,</mo><mi /><msub><mrow><mi>γ</mi></mrow><mrow><mi>M</mi></mrow></msub><mo>|</mo><msup><mrow><mi>λ</mi></mrow><mrow><mo>′</mo></mrow></msup></mrow><mo stretchy="true">)</mo></math> </ephtml> in the training sample;</p> <p></p> <ulist> <item> Used these parameters to predict the impact for each site in the validation sample;</item> <p></p> <item> Calculated the difference between the prediction for each site in the validation sample and the unbiased estimate of the site's impact (the site-level prediction error, or</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>w</mi></mrow></msubsup></math> </ephtml> );</p> <p></p> <ulist> <item> Calculated the mean squared prediction error for the</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><mi>J</mi></mrow><mo>/</mo><mrow><mi>K</mi></mrow></mrow></math> </ephtml> sites in the validation sample (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">MSPE</mi></mrow><mrow><mi>k</mi></mrow></msub></math> </ephtml> ).</p> <p></p> <ulist> <item> <bold> Calculate the average MSPE across folds. </bold> In particular, we took the simple average of</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">MSPE</mi></mrow><mrow><mi>k</mi></mrow></msub></math> </ephtml> for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi><mo>=</mo><mn>1</mn><mo>,</mo><mo>...</mo><mo>,</mo><mi>K</mi></math> </ephtml> .</p> <p></p> <ulist> <item> <bold> Repeat steps 3 and 4 for up to 99 other possible values of the tuning parameter. </bold> This allowed us to compare MSPE for different values of the tuning parameter, using glmnet's default sequence of 100 equally-spaced values between</item> </ulist> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.001</mn><mo>·</mo><msub><mrow><mi mathvariant="normal">λ</mi></mrow><mrow><mi mathvariant="normal">max</mi></mrow></msub></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mn>1.000</mn><mo>·</mo><mi>λ</mi></mrow><mrow><mi mathvariant="normal">max</mi></mrow></msub></math> </ephtml> , stopping once the MSPE shows only negligible changes.</p> <p></p> <hd1 id="AN0174878178-18"> • Choose the optimal value of the tuning parameter ( </hd1> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi mathvariant="bold-italic">λ</mi></mrow><mrow><mi mathvariant="bold-italic">*</mi></mrow></msup></math> </ephtml> <bold>).</bold> We chose the value of the tuning parameter that produced the smallest MSPE.[<reflink idref="bib20" id="ref88">20</reflink>]</p> <p></p> <hd1 id="AN0174878178-19"> • Calculate the optimal values of the model parameters given </hd1> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi mathvariant="bold-italic">λ</mi></mrow><mrow><mi mathvariant="bold-italic">*</mi></mrow></msup></math> </ephtml> <bold>(</bold> </p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi mathvariant="bold-italic">α</mi></mrow><mrow><mi mathvariant="bold-italic">*</mi></mrow></msup><mo>,</mo><mi mathvariant="bold-italic" /><msubsup><mrow><mi mathvariant="bold-italic">γ</mi></mrow><mrow><mn>1</mn></mrow><mrow><mi mathvariant="bold-italic">*</mi></mrow></msubsup><mo>,</mo><mi mathvariant="bold-italic" /><mo>...</mo><mo>,</mo><msubsup><mrow><mi mathvariant="bold-italic">γ</mi></mrow><mrow><mi mathvariant="bold-italic">M</mi></mrow><mrow><mi mathvariant="bold-italic">*</mi></mrow></msubsup><mi mathvariant="bold-italic" /></math> </ephtml> <bold>).</bold> In particular, we minimized</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>G</mi><mo stretchy="true">(</mo><mrow><mi>α</mi><mo>,</mo><msub><mrow><mi>γ</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mi /><mo>...</mo><mo>,</mo><mi /><msub><mrow><mi>γ</mi></mrow><mrow><mi>M</mi></mrow></msub><mo>|</mo><msup><mrow><mi>λ</mi></mrow><mrow><mi>*</mi></mrow></msup></mrow><mo stretchy="true">)</mo></math> </ephtml> with respect to the parameters (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi><mo>,</mo><msub><mrow><mi>γ</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mi /><mo>...</mo><mo>,</mo><mi /><msub><mrow><mi>γ</mi></mrow><mrow><mi>M</mi></mrow></msub></math> </ephtml> ).</p> <p>With the parameters produced in step 7 above, we predicted the impact for each site <emph>j</emph>:</p> <p>(A2)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mrow><mover accent="true"><mrow><mi>δ</mi></mrow><mo>̂</mo></mover></mrow></mrow><mrow><mi>j</mi></mrow><mrow><mi>p</mi></mrow></msubsup><mo>=</mo><msup><mrow><mi>α</mi></mrow><mrow><mi>*</mi></mrow></msup><mo>+</mo><mrow><msubsup><mo stretchy="false">∑</mo><mrow><mi>m</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>M</mi></mrow></msubsup><mrow><msub><mrow><msubsup><mrow><mi>γ</mi></mrow><mrow><mi>m</mi></mrow><mrow><mi>*</mi></mrow></msubsup><mi>Z</mi></mrow><mrow><mi>m</mi><mi>j</mi></mrow></msub></mrow></mrow></math> </ephtml> </p> <hd id="AN0174878178-20">BART</hd> <p>BART is an increasingly popular nonparametric Bayesian approach to model continuous, binary, categorical, or time-to-event outcomes, initially developed by Chipman et al. (2010). BART has been shown to have excellent performance for causal inference purposes (Dorie et al., [<reflink idref="bib6" id="ref89">6</reflink>]; Hill, 2011; Kern et al., 2016). BART can be thought of as a Bayesian implementation of some recent machine learning approaches that build on the basic ideas of classification and regression trees (CART), but it is implemented in a way that reduces some of the limitations of CART-based approaches, in particular their tendency to overfit to the data and thus not perform well for out-of-sample prediction. Like regression trees, BART has the benefit of allowing flexible models that identify non-linear relationships between predictors and the outcome of interest in an automated way. Tree-based approaches, including BART, do this by splitting the sample into groups based on the predictors in a way that minimizes the within-group variance in the outcome of interest. Boosted regression tree approaches including BART address the overfitting problems of traditional CART approaches by essentially fitting a series of small trees and combining them into an "ensemble." BART improves on other boosted regression tree approaches by embedding the tree in a likelihood structure and setting prior distributions on parameters that control the size of the trees, the group means, and the error term in the underlying model. This combination of prior and likelihood in a Bayesian framework provides coherent uncertainty intervals – that reflect the additional uncertainty when predicting impacts outside the range of the data – and a principled statistical underpinning for the boosted tree approach. Estimation of the BART approach is done using a Gibbs sampler Markov Chain Monte Carlo approach (Chipman et al., 2010). Convergence of the Gibbs sampler is assessed using Geweke's convergence diagnostic, which compares the first 10 percent and the last 50 percent of a Markov chain (Geweke, 1992).</p> <p>We used BART to predict impacts as a function of site-level characteristics, as was done using lasso. Differences from lasso include 1) the flexible non-linear nature of the underlying trees and thus the ability to pick up non-linear relationships and interactions, and 2) the Bayesian framework for model estimation. We implemented BART using the dbarts package in R (Dorie et al., 2020). The prior distributions that determine the structure of the decision trees underlying the resulting BART model are governed by the following arguments in the dbarts::barts()function call: sigest, sigdf, base, and power. The values for these arguments can be specified by the user or left at default settings. The argument sigest is an initial estimate of the error variance,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi>σ</mi></mrow><mrow><mn>2</mn></mrow></msup></math> </ephtml> , and is used to calibrate an inverse-chi-squared prior distribution for that parameter. If not specified, sigest is set to be the residual variance from an OLS regression of the outcome as a function of the covariates. The argument sigdf specifies the degrees of freedom of the inverse-chi-squared prior distribution of the error variance. The arguments base and power together determine the prior on the tree structures, specified as the probability of a new node split at branch B, at a given tree depth t. Its calculation follows from the geometric distribution: P[B|<emph>t</emph>] = base*(1 + <emph>t</emph>)<sups>−power</sups>. Lower values for base will tend to yield smaller trees, effectively penalizing tree growth and thus preventing overfitting. Default values are those recommended in Chipman et al (2010), base = 0.95 and power = 2, which yields prior probabilities of 0.05, 0.55, 0.28, 0.09 and 0.03, for a tree to have 1, 2, 3, 4 and ≥5 terminal nodes, respectively.</p> <ref id="AN0174878178-21"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Cross-validation is a technique used to protect against overfitting in a predictive model, particularly in a case where the amount of data may be limited. In cross-validation, the dataset is divided into several subsets, typically referred to as "folds." A given model is fit multiple times, excluding a single fold from each model run, and then using the model to generate predictions for the remaining folds. This process is repeated while varying model hyperparameters, most often hyperparameters that reduce or increase the number of variables included in the model. The "best" model is typically identified as the one with the lowest average out-of-sample prediction error. The use of multiple folds generally decreases the model variance since the model is less likely to be influenced by outliers or unlucky selection of a particular fold to which the model is fit.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> For some purposes, aggregating student-level data to the site level would limit the analysis (e.g., remove the ability to test for subgroup differences within site). However, for our purposes, only site-level variation is helpful in predicting site-level impacts—that is, the average impact in a site. Individual-level covariates cannot explain any additional variation across sites beyond what is explained by the site-level aggregates.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Ten of the grantees enrolled students into the study in two successive years. Unlike the original study authors, we treated both cohorts of students for a single grantee as belonging to the same site.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref14" type="bt">4</bibl> <bibtext> For state tests, the study authors standardized the scores relative to all test takers in the same state and grade level. For nationally normed tests, the study authors standardized the scores relative to all test takers in the national norming population.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref26" type="bt">5</bibl> <bibtext> These summary statistics include those students who contributed to this study. In constructing impact estimates for a particular outcome, we excluded sites with missing values for all students in that site.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref47" type="bt">6</bibl> <bibtext> Missing data in the pre-intervention measure of the outcome was addressed using the dummy variable method (e.g., Jones, [15]; Puma et al., [22]). Specifically, the model included an indicator variable that equaled 1 if the pre-intervention measure of the outcome was missing and equaled 0 otherwise. When the pre-intervention measure of the outcome was missing, it was set to 0 for the analysis.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref51" type="bt">7</bibl> <bibtext> If our goal were to obtain the most accurate estimate of the impact in each site using data from all sites, we would have used Empirical Bayes methods to estimate site-specific impacts. However, our goal was to use OLS to produce unbiased impact estimates for the meta-analysis described later in the paper.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref27" type="bt">8</bibl> <bibtext> The original analysis plan was posted on June 12, 2020 (see https://osf.io/vbs36). Minor revisions were made to the analysis plan during the conduct of the analysis; a revised analysis plan was posted on November 29, 2021 (see https://osf.io/yqfzt). Finally, the paper deviated from the analysis plan to address feedback during the peer review process.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref17" type="bt">9</bibl> <bibtext> At earlier stages in the analysis, using the approach to estimating the RMSPE from Orr et al. ([21]), we also tested precision-weighted averages of the site-specific impact estimates. In that analysis, we found little difference in RMSPE between the simple average of the site-specific impact estimates and the precision-weighted average of those estimates.</bibtext> </blist> <blist> <bibtext> This could be due to either the lack of any real signal in the data for predicting site-specific impacts or due to lasso's inability to distinguish signal from noise when confronting a large number of potential moderators, many of which may be weak moderators at best.</bibtext> </blist> <blist> <bibtext> The meta-analysis can decompose the error variance into the cross-site variance and the within-site variance because evidence on the within-site variance was available based on the estimated standard errors of the unbiased impact estimates for each site. While the small sample sizes in each site could potentially lead to biased random effects meta-analysis estimates through correlations between the estimated effects and their standard errors, a simulation study on this issue found the bias to be small for difference-in-means and standardized mean difference estimators (Lin, [18]).</bibtext> </blist> <blist> <bibtext> Kraft ([17]) reported the unweighted mean, the weighted mean—weighted by the inverse of the variance of the impact estimates—and the median. We used the weighted mean for consistency with the other meta-analyses, which reported only weighted means.</bibtext> </blist> <blist> <bibtext> While the meta-analysis included estimates from studies that the authors acknowledged were based on weak study designs, their analysis suggests that the inclusion of these estimates did not substantially bias their estimates of average effects (Wilson et al., [27], pp. 262–263).</bibtext> </blist> <blist> <bibtext> While this meta-analysis reported separate estimates by outcome domain—estimates we could have matched to the different outcome domains explored in our study—the meta-analysis found little variation in impacts by outcome domain (DuBois et al., [7], pp. 67–68).</bibtext> </blist> <blist> <bibtext> Ongoing research by the authors of this paper directly estimates the probability that site-level predictions from all six studies would lead local policymakers to the wrong conclusions about the effectiveness of the interventions.</bibtext> </blist> <blist> <bibtext> The study of summer reading could only be included in analyses focused on moderators based on the characteristics of participating students: The other types of moderator variables were not collected by the original study.</bibtext> </blist> <blist> <bibtext> These differences were typically statistically significant at the 90% level using two-tailed tests. For example, when lasso regression was applied to moderator variables on intervention features, lasso yielded a larger RMSPE than the simple average of the impacts in the other sites for 994 out of 1,000 bootstrap samples.</bibtext> </blist> <blist> <bibtext> While the differences were typically not statistically significant, for reading achievement in year 1 of the charter middle school study, lasso with site characteristics, counterfactual characteristics, and all four types of moderators yielded larger RMSPEs than the simple average of the impacts in the other sites for at least 950 out of 1,000 bootstrap samples.</bibtext> </blist> <blist> <bibtext> Some sources suggest refining the lambda sequence near the apparent minimum MSPE, to test whether it is possible to achieve an even lower minimum. Given the number of models we were running, any slight gains in prediction due to this refinement were far outweighed by the operational challenges of implementing this testing.</bibtext> </blist> <blist> <bibtext> Some sources recommend selecting the model with MSPE one standard error greater than the minimum, to select a simpler model that is not statistically different from the "best" model. Since our goal here was optimal prediction and we are not concerned with model simplicity or interpretability, we selected the model with minimum average MSPE.</bibtext> </blist> <blist> <bibtext> Supplemental data for this article is available online at https://doi.org/10.1080/19345747.2023.2180464.</bibtext> </blist> </ref> <ref id="AN0174878178-22"> <title> References </title> <blist> <bibtext> Bernstein, L., Rappaport, C. D., Olsho, L., Hunt, D., & Levin, M. (2009). Impact evaluation of the US Department of Education's student mentoring program. Final report. NCEE 2009-4047. National Center for Education Evaluation and Regional Assistance.</bibtext> </blist> <blist> <bibtext> Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29 (1), 30 – 59. https://doi.org/10.3102/0162373707299550</bibtext> </blist> <blist> <bibtext> Chipman, H. A., George, E. I., & McCulloch, R. E. (2010). BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4 (1), 266 – 298. https://doi.org/10.1214/09-AOAS285</bibtext> </blist> <blist> <bibtext> Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M. (2013). The effectiveness of secondary math teachers from teach for America and the teaching fellows programs. NCEE 2013-4015. National Center for Education Evaluation and Regional Assistance.</bibtext> </blist> <blist> <bibtext> Donoho, D., & Stodden, V. (2006, July). Breakdown point of model selection when the number of variables exceeds the number of observations. In The 2006 IEEE International Joint Conference on Neural Network Proceedings (pp. 1916–1921). IEEE. https://doi.org/10.1109/IJCNN.2006.246934</bibtext> </blist> <blist> <bibtext> Dorie, V., Hill, J., Shalit, U., Scott, M., & Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34 (1), 43–68. https://doi.org/10.1214/18-STS667</bibtext> </blist> <blist> <bibtext> DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest : A Journal of the American Psychological Society, 12 (2), 57 – 91.</bibtext> </blist> <blist> <bibtext> Eberlin, L. S., Tibshirani, R. J., Zhang, J., Longacre, T. A., Berry, G. J., Bingham, D. B., Norton, J. A., Zare, R. N., & Poultsides, G. A. (2014). Molecular assessment of surgical-resection margins of gastric cancer by mass-spectrometric imaging. Proceedings of the National Academy of Sciences of the United States of America, 111 (7), 2436 – 2441.</bibtext> </blist> <blist> <bibtext> Esbensen, F. A., Peterson, D., Taylor, T. J., & Osgood, D. W. (2012). Results from a multi-site evaluation of the GREAT program. Justice Quarterly, 29 (1), 125 – 151. https://doi.org/10.1080/07418825.2011.585995</bibtext> </blist> <blist> <bibtext> Finch, W. H., & Finch, M. E. H. (2016). Regularization methods for fitting linear models with small sample sizes: Fitting the Lasso estimator using R. Practical Assessment, Research, and Evaluation, 21 (1), 7.</bibtext> </blist> <blist> <bibtext> Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33 (1), 1 – 22. https://doi.org/10.18637/jss.v033.i01</bibtext> </blist> <blist> <bibtext> Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final report. NCEE 2010–4029. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.</bibtext> </blist> <blist> <bibtext> Hill, J. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20 (1), 217 – 240. https://doi.org/10.1198/jcgs.2010.08162</bibtext> </blist> <blist> <bibtext> James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, p. 18). Springer.</bibtext> </blist> <blist> <bibtext> Jones, M. P. (1996). Indicator and stratification methods for missing explanatory variables in multiple linear regression. Journal of the American Statistical Association, 91 (433), 222 – 230. https://doi.org/10.1080/01621459.1996.10476680</bibtext> </blist> <blist> <bibtext> Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 9 (1), 103 – 127.</bibtext> </blist> <blist> <bibtext> Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49 (4), 241 – 253. https://doi.org/10.3102/0013189X20912798</bibtext> </blist> <blist> <bibtext> Lin, L. (2018). Bias caused by sampling error in meta-analysis with small sample sizes. PloS One, 13 (9), e0204056.</bibtext> </blist> <blist> <bibtext> Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. National Center for Special Education Research.</bibtext> </blist> <blist> <bibtext> Olsen, R., Stuart, E. A., Orr, L., & Bell, S. (2020). How much can evidence from national studies improve local policy decisions that affect youth? - Analysis plan. The Open Science Framework. https://osf.io/vbs36/</bibtext> </blist> <blist> <bibtext> Orr, L. L., Olsen, R. B., Bell, S. H., Schmid, I., Shivji, A., & Stuart, E. A. (2019). Using the results from rigorous multisite evaluations to inform local policy decisions. Journal of Policy Analysis and Management, 38 (4), 978 – 1003. https://doi.org/10.1002/pam.22154</bibtext> </blist> <blist> <bibtext> Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to do when data are missing in group randomized controlled trials. NCEE 2009-0049. National Center for Education Evaluation and Regional Assistance.</bibtext> </blist> <blist> <bibtext> Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58 (1), 267 – 288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x</bibtext> </blist> <blist> <bibtext> Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33 (3), 778 – 808. https://doi.org/10.1002/pam.21760</bibtext> </blist> <blist> <bibtext> Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10 (4), 843 – 876. https://doi.org/10.1080/19345747.2017.1300719</bibtext> </blist> <blist> <bibtext> Wilkins, C., Gersten, R., Decker, L. E., Grunden, L., Brasiel, S., Brunnert, K., & Jayanthi, M. (2012). Does a summer reading program based on lexiles affect reading comprehension? Final report. NCEE 2012-4006. National Center for Education Evaluation and Regional Assistance.</bibtext> </blist> <blist> <bibtext> Wilson, D. B., Gottfredson, D. C., & Najaka, S. S. (2001). School-based prevention of problem behaviors: A meta-analysis. Journal of Quantitative Criminology, 17 (3), 247–272. https://doi.org/10.1023/A:1011050217296</bibtext> </blist> </ref> <aug> <p>By Robert B. Olsen; Larry L. Orr; Stephen H. Bell; Elizabeth Petraglia; Elena Badillo-Goicoechea; Atsushi Miyaoka and Elizabeth A. Stuart</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib25" firstref="ref4"></nolink> <nolink nlid="nl2" bibid="bib21" firstref="ref5"></nolink> <nolink nlid="nl3" bibid="bib24" firstref="ref11"></nolink> <nolink nlid="nl4" bibid="bib12" firstref="ref13"></nolink> <nolink nlid="nl5" bibid="bib26" firstref="ref18"></nolink> <nolink nlid="nl6" bibid="bib23" firstref="ref21"></nolink> <nolink nlid="nl7" bibid="bib10" firstref="ref28"></nolink> <nolink nlid="nl8" bibid="bib16" firstref="ref29"></nolink> <nolink nlid="nl9" bibid="bib20" firstref="ref55"></nolink> <nolink nlid="nl10" bibid="bib14" firstref="ref58"></nolink> <nolink nlid="nl11" bibid="bib11" firstref="ref62"></nolink> <nolink nlid="nl12" bibid="bib17" firstref="ref65"></nolink> <nolink nlid="nl13" bibid="bib27" firstref="ref67"></nolink> <nolink nlid="nl14" bibid="bib13" firstref="ref68"></nolink> <nolink nlid="nl15" bibid="bib15" firstref="ref75"></nolink> <nolink nlid="nl16" bibid="bib18" firstref="ref85"></nolink> <nolink nlid="nl17" bibid="bib19" firstref="ref87"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1408304
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Robert+B%2E+Olsen%22">Robert B. Olsen</searchLink><br /><searchLink fieldCode="AR" term="%22Larry+L%2E+Orr%22">Larry L. Orr</searchLink><br /><searchLink fieldCode="AR" term="%22Stephen+H%2E+Bell%22">Stephen H. Bell</searchLink><br /><searchLink fieldCode="AR" term="%22Elizabeth+Petraglia%22">Elizabeth Petraglia</searchLink><br /><searchLink fieldCode="AR" term="%22Elena+Badillo-Goicoechea%22">Elena Badillo-Goicoechea</searchLink><br /><searchLink fieldCode="AR" term="%22Atsushi+Miyaoka%22">Atsushi Miyaoka</searchLink><br /><searchLink fieldCode="AR" term="%22Elizabeth+A%2E+Stuart%22">Elizabeth A. Stuart</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+on+Educational+Effectiveness%22"><i>Journal of Research on Educational Effectiveness</i></searchLink>. 2024 17(1):184-210.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 27
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Institute of Mental Health (NIMH) (DHHS/NIH)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: P50MH115842
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Predictor+Variables%22">Predictor Variables</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+%28Statistics%29%22">Regression (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Bayesian+Statistics%22">Bayesian Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Statistics%22">Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Influences%22">Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement+Objectives%22">Measurement Objectives</searchLink><br /><searchLink fieldCode="DE" term="%22Guidance+Programs%22">Guidance Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Federal+Programs%22">Federal Programs</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2023.2180464
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1934-5747<br />1934-5739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods--lasso regression and Bayesian Additive Regression Trees (BART)--using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1408304
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1408304
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2023.2180464
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 27
        StartPage: 184
    Subjects:
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Predictor Variables
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Regression (Statistics)
        Type: general
      – SubjectFull: Bayesian Statistics
        Type: general
      – SubjectFull: Statistics
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Influences
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Measurement Objectives
        Type: general
      – SubjectFull: Guidance Programs
        Type: general
      – SubjectFull: Federal Programs
        Type: general
    Titles:
      – TitleFull: Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Robert B. Olsen
      – PersonEntity:
          Name:
            NameFull: Larry L. Orr
      – PersonEntity:
          Name:
            NameFull: Stephen H. Bell
      – PersonEntity:
          Name:
            NameFull: Elizabeth Petraglia
      – PersonEntity:
          Name:
            NameFull: Elena Badillo-Goicoechea
      – PersonEntity:
          Name:
            NameFull: Atsushi Miyaoka
      – PersonEntity:
          Name:
            NameFull: Elizabeth A. Stuart
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1934-5747
            – Type: issn-electronic
              Value: 1934-5739
          Numbering:
            – Type: volume
              Value: 17
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1