View in EDS HTML Full Text PDF Full Text

The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence from Four New Within-Study Comparisons

Saved in:
Bibliographic Details
Title: The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence from Four New Within-Study Comparisons
Language: English
Authors: Sims, Sam (ORCID 0000-0002-5585-8202), Anders, Jake (ORCID 0000-0003-0930-2884), Zieger, Laura
Source: Journal of Research on Educational Effectiveness. 2022 15(4):876-897.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 22
Publication Date: 2022
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Secondary Education
Descriptors: Validity, Comparative Analysis, Statistical Analysis, Intervention, Evaluation Methods, Institutional Characteristics, Outcomes of Education, Elementary Secondary Education, Foreign Countries, Databases, Student Records, Teaching Methods, Mathematics Achievement
Geographic Terms: United Kingdom (England)
DOI: 10.1080/19345747.2022.2051652
ISSN: 1934-5747
1934-5739
Abstract: Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1367853
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEHmNVjxpAyvKyoEgF21hfsAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOKV31jFg2nd_YMjsQIBEICBm0TWGhTZq_UoLlyLQ5FwfwxCOz63vCsHNdFcnsDFHp_FqWlkK8Yo4po48dK6vqTDK7_fhJPZqBPkdTA5zBDGFIRM9wZNE1E1fPODtHw7tXlogoA6SEW0uBBmDZfFZq06Hjww88ADdHP2Vna-rV5LZgK-Ooy6I4djeMv1JAIflB5A6jWCRGtHU6xGMpGLKdLCXXiHcbRIREOaha7q
Text:
  Availability: 1
  Value: <anid>AN0159448677;[5ew9]01oct.22;2022Oct04.09:25;v2.2.500</anid> <title id="AN0159448677-1">The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence From Four New Within-Study Comparisons </title> <p>Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.</p> <p>Keywords: Comparative interrupted time series; randomised controlled trials; within-study comparisons; internal validity</p> <hd id="AN0159448677-2">Introduction</hd> <p>In education, many interventions of interest to school leaders and policymakers are implemented at the school-level, without random allocation. Consequently, researchers are required to adopt a non-experimental evaluation method. Comparative interrupted time series (CITS) (Cook & Campbell, [<reflink idref="bib11" id="ref1">11</reflink>]) methods constitute one such design, premised on projecting outcomes from the pre- into the post-treatment period. Perhaps the best-known variant of the CITS is the conventional two-period difference-in-difference (Card & Krueger, [<reflink idref="bib7" id="ref2">7</reflink>]), in which the change in the outcome in the comparison group is used to project the pretreatment level of the outcome in the treatment group into the post-treatment period. Under the parallel trends assumption, this acts as a proxy for the untreated outcomes among the treated units. However, the family of CITS methods also includes variants that focus on quantifying the relative deviation from trends, rather than levels, of the outcome. These alternative approaches are valuable in settings where selection into treatment occurs based on pretreatment values of the outcome variable, rendering the common trends assumption implausible (St. Clair & Cook, [<reflink idref="bib49" id="ref3">49</reflink>]). The deviation from pretreatment trends in the comparison group is then used to adjust the projection of the pretreatment trend in the treatment group into the post-treatment period, providing the necessary proxy for untreated outcomes among the treated units.</p> <p>The various CITS specifications were first deployed to evaluate a school improvement programme (Bloom et al., [<reflink idref="bib5" id="ref4">5</reflink>]). Since then, CITS have been used in evaluations of No Child Left Behind (Dee et al., [<reflink idref="bib17" id="ref5">17</reflink>]; Grissom et al., [<reflink idref="bib25" id="ref6">25</reflink>]; Lee & Reeves, [<reflink idref="bib31" id="ref7">31</reflink>]; Wong et al., [<reflink idref="bib55" id="ref8">55</reflink>]), curricular reforms (Bell et al., [<reflink idref="bib1" id="ref9">1</reflink>]; Jacob et al., [<reflink idref="bib26" id="ref10">26</reflink>]), school wraparound programmes (Gandhi et al., [<reflink idref="bib23" id="ref11">23</reflink>]), and school turnaround programmes (Henry et al., [<reflink idref="bib37" id="ref12">37</reflink>]; Strunk et al., [<reflink idref="bib47" id="ref13">47</reflink>]). Indeed, the approach is in many ways well-suited to education settings. Conducting a CITS evaluation requires a well-measured outcome variable recorded consistently across multiple periods. In many countries, high-stakes school examination results are recorded and made publicly available for each academic year, meaning the data requirements for CITS are often easy to fulfill for education researchers. Moreover, CITS' emphasis on modeling and then projecting the outcome variable, rather than focusing on adjusting for or matching on covariates, means that the method can often be implemented without access to sensitive individual-level data on pupil characteristics (Jacob et al., [<reflink idref="bib27" id="ref14">27</reflink>]). In addition, the key identifying assumption of CITS-that unobserved time-varying confounders are shared between the treatment and comparison groups-are often more plausible in education, where schools are nested within common regional and national administrative units.</p> <p>Nevertheless, any violation of the CITS identifying assumptions could undermine internal validity, yielding biased impact estimates. Since the identifying assumption relates to a counterfactual, it cannot be directly empirically tested in any given evaluation. However, an alternative method for assessing bias is available in the form of within-study comparisons (WSC). These involve comparing the impact estimates from a benchmark study—often a randomized controlled trial (RCT)—with those of an alternative evaluation of the same intervention using the same outcomes, but employing a separate, non-experimental comparison group (LaLonde, [<reflink idref="bib30" id="ref15">30</reflink>]; Wong et al., [<reflink idref="bib56" id="ref16">56</reflink>]). Thus, between-study comparisons quantify differences in effect size estimates for a given intervention. RCTs in education tend to provide noisy estimates of the true impact (Lortie-Forgues & Inglis, [<reflink idref="bib32" id="ref17">32</reflink>]; Spybrook et al., [<reflink idref="bib48" id="ref18">48</reflink>]) which means that comparing the effect sizes from a single within-study comparison provides a noisy estimate of bias. However, the accumulation of multiple WSCs for a given non-experimental design does allow bias to be estimated across WSCs (Glazerman et al., [<reflink idref="bib33" id="ref19">33</reflink>]).</p> <hd id="AN0159448677-3">Within-Study Comparisons of the CITS Design</hd> <p>Wong et al. ([<reflink idref="bib56" id="ref20">56</reflink>]) identified twenty WSCs of interrupted time series (ITS) or CITS designs with experimental benchmarks across environmental policy (Ferraro & Miranda, [<reflink idref="bib21" id="ref21">21</reflink>]); epidemiology (Fretheim et al., [<reflink idref="bib22" id="ref22">22</reflink>]); health (Schneeweiss et al., [<reflink idref="bib43" id="ref23">43</reflink>]); welfare-to-work (Bloom et al., [<reflink idref="bib6" id="ref24">6</reflink>]) and immigration (McKenzie et al., [<reflink idref="bib38" id="ref25">38</reflink>]). Coopersmith et al. ([<reflink idref="bib13" id="ref26">13</reflink>]) provide the most recent meta-analytic summary, identifying 12 unique CITS WSCs. The average deviation of the CITS estimates from the benchmark estimates was 0.03 SD. At the level of individual WSCs, all but one of the twelve deviated from the benchmark estimate by less than 0.1 SD.</p> <p>Within education, we identified six WSCs that contrasted CITS impact estimates with those from RCTs. St. Clair et al. ([<reflink idref="bib50" id="ref27">50</reflink>]) report WSCs from three whole-school interventions: two in Indiana and one in Florida. In the two trials in which the pre-trends were stable and thus easier to model, the non-experimental estimates came within 0.02 school-level SD of the experimental benchmark. In the third trial, where the pre-trends were highly unstable and thus harder to model, the non-experimental estimates deviated from the experimental estimates substantially. This highlights the importance of the stability of pre-trends for the validity of the CITS design (Kim & Steiner, [<reflink idref="bib28" id="ref28">28</reflink>]). Hallberg et al. ([<reflink idref="bib36" id="ref29">36</reflink>]) report WSCs from a further three whole-school interventions—a college readiness programme, an early literacy programme, and an online mathematics programme—and find an average absolute bias of between 0.05 and 0.08 school-level SD in the immediate post-intervention period. However, the bias was found to be strongly influenced by the specific modeling approach adopted and the authors recommend deciding which of the specific CITS specifications to adopt based on careful inspection of the pre-trends.</p> <hd id="AN0159448677-4">Aims and Scope</hd> <p>The current paper aims to substantially increase the number of such WSCs in the education literature by conducting four additional WSCs of CITS. As previously discussed, contrasts of impact estimates from single WSCs provide limited information about bias because the typical RCT impact estimate in education is noisy. Accumulating four new WSCs is therefore particularly valuable in that it allows estimates of bias across studies, netting out the noise from the individual RCTs. By synthesizing the results across four new WSCs, we aim to contribute to the empirical evidence on the validity of a widely used evaluation design in the education literature. The next section describes the set of interventions that we reevaluate and the school-level data that we employ in this analysis. The 'Methods' section then describes our approach to conducting each of the CITS evaluations and the techniques by which these are subsequently synthesized. The 'Results' section presents our findings for each of the individual WSCs, followed by the within-study comparisons, and the 'Discussion' section concludes by reflecting on the limitations and implications of these findings.</p> <hd id="AN0159448677-5">Sample</hd> <p></p> <hd id="AN0159448677-6">Interventions</hd> <p>We set out to reevaluate interventions previously tested using RCTs by the Education Endowment Foundation (EEF). The EEF is a large research funder in England, which typically commissions RCTs of interventions judged to have previously shown evidence of promise. EEF was established in 2010 and has since commissioned over 100 RCTs, the results of which are routinely published on their website (Dawson et al., [<reflink idref="bib15" id="ref30">15</reflink>]). These interventions can be reevaluated using school-level CITS if five conditions hold: (<reflink idref="bib1" id="ref31">1</reflink>) the intervention was originally randomized at school-level, (<reflink idref="bib2" id="ref32">2</reflink>) the trial used all pupils in the treated cohorts' results in standardized national tests at age 11 or age 16 as one of the outcome measures, (<reflink idref="bib3" id="ref33">3</reflink>) the trial is complete and results have been reported, (<reflink idref="bib4" id="ref34">4</reflink>) the time series necessary to conduct the CITS does not include discontinuities in how the outcome variables were recorded, and (<reflink idref="bib5" id="ref35">5</reflink>) it is possible to calculate effect sizes from the original RCT that are comparable to those generated by our CITS.</p> <p>The EEF supplied us with a list of all of their trials/interventions that met criteria 1–3 as of December 2019, which amounted to 19 in total.[<reflink idref="bib1" id="ref36">1</reflink>] Among these, eight did not meet criterion 4) due to the relevant section of time series straddling major reforms of either age-11 examinations (implemented in 2015/16) or age 16 examinations (implemented in 2016/17).[<reflink idref="bib2" id="ref37">2</reflink>] Similarly, three interventions had to be dropped because the relevant time series included a year in which we had missing data due to e.g., a the boycott of the age-11 examinations.[<reflink idref="bib3" id="ref38">3</reflink>] A further three interventions had to be dropped because the original evaluation report did not include the information necessary to calculate an effect size measure that would be comparable with our school-level CITS evaluations.[<reflink idref="bib4" id="ref39">4</reflink>] Finally, one intervention had to be dropped because of discrepancies in the number of observations in the EEF data archive and the published evaluation report, which could not be resolved by the EEF's data scientists.</p> <p>This left four interventions: two in primary schools and two in secondary schools (see Table 1). The interventions are diverse in nature, ranging from teacher professional development (PD) to the use of educational technology to adapt instruction. All four trials measured outcomes for all pupils in the relevant cohorts in participating schools and conducted analysis on an intention-to-treat basis. Attrition at both the school and pupil level was below the What Works Clearinghouse conservative standards across all four trials (What Works Clearinghouse, [<reflink idref="bib52" id="ref40">52</reflink>]). The estimated effect sizes reported in the original evaluation reports range from 0.00–0.26 school-level standard deviations (SD) or 0.00–0.09 pupil-level SD. None of these estimates were statistically significant at conventional levels, which is perhaps unsurprising given the lack of statistical power in many educational trials (Lortie-Forgues & Inglis, [<reflink idref="bib32" id="ref41">32</reflink>]; Spybrook et al., [<reflink idref="bib48" id="ref42">48</reflink>]). In line with EEF analysis guidelines, all four trials used pupil prior achievement as the only covariate (besides strata and cluster variables) in their main analytical models.</p> <p>Table 1. The interventions.</p> <p> <ephtml> <table><thead><tr><td>Reference</td><td>Phase</td><td>RCT results in pupil-level SD</td><td>RCT results in school-level SD</td><td>Sample size</td><td>Attrition</td><td>Brief description</td></tr></thead><tbody valign="top"><tr><td>Flipped learning (FLIP)</td><td>Rudd et al. (<xref ref-type="bibr" rid="bibr42">2017</xref>)</td><td>Prim.</td><td>ES = 0.09 SE = 0.19 <italic>p</italic> = 0.62</td><td>ES = 0.26 SE = 0.55 <italic>p</italic> = 0.62</td><td>Schools: 24 Pupils: 1,214</td><td>Schools: 0% Pupils: 7%</td><td>Pupils study new math at home, then consolidate in class time</td></tr><tr><td>Learner response (LERS)<sup>a</sup></td><td>Wiggins et al. (<xref ref-type="bibr" rid="bibr54">2017</xref>)</td><td>Prim.</td><td>ES = 0.00<sup>b</sup> SE = 0.07 <italic>p</italic> = 0.96</td><td>ES = 0.00 SE = 0.84 <italic>p</italic> = 0.96</td><td>Schools: 97 Pupils: 3,213</td><td>Schools: 3.1% Pupils: 11.7%</td><td>Immediate in-class pupil feedback from teachers via electronic devices</td></tr><tr><td>The RISE project (RISE)<sup>a</sup></td><td>Wiggins et al. (<xref ref-type="bibr" rid="bibr53">2019</xref>)</td><td>Sec.</td><td>ES = 0.09 SE = 0.06 <italic>p</italic> = 0.15</td><td>ES = 0.19 SE = 0.13 <italic>p</italic> = 0.15</td><td>Schools: 40 Pupils: 7,468</td><td>Schools: 2.5% Pupils: 8.8%</td><td>Teacher "research leads" supporting teachers' use of research</td></tr><tr><td>Teacher observation (TEOB)<sup>a</sup></td><td>Worth et al. (<xref ref-type="bibr" rid="bibr58">2017</xref>)</td><td>Sec.</td><td>ES = 0.06 SE = 0.04 <italic>p</italic> = 0.13</td><td>ES = 0.15 SE = 0.10 <italic>p</italic> = 0.13</td><td>Schools: 82 Pupils: 7,366</td><td>Schools: 0% Pupils: 1.4%</td><td>Teacher PD based on programme of structured peer lesson observations</td></tr></tbody></table> </ephtml> </p> <p>1 Prim: primary; Sec: secondary; PD: professional development; ES: effect size derived from intention-to-treat analysis based on all pupils in the relevant cohorts within the schools. SE: standard error; All standard errors are derived by dividing the widths of the 95% confidence intervals reported in the EEF evaluation reports by 3.92. Sample size is the number of schools/pupils initially randomized to generate the effect size in the "RCT results" column.</p> <ulist> <item>2 <sups>a</sups>For these trials we can only use the first treatment cohort because the relevant time series for the second treatment cohort straddles a discontinuity in the time series.</item> <item>3 <sups>b</sups>The results was reported as "0.00" in the evaluation report, rather than to a given number of significant figures.</item> </ulist> <hd id="AN0159448677-7">Data</hd> <p>We utilize school-level data derived from the UK Department for Education's National Pupil Database, an administrative census containing information on all publicly funded schools in England since 1991/92.[<reflink idref="bib5" id="ref43">5</reflink>] Each school in the data has a unique identifier, which allows them to be linked across years. However, a large proportion of schools have either converted to academy status or undergone other changes that cause their identifier to change between years. Accordingly, we utilize an additional database of consistent school identifiers to ensure that we maintain the integrity of the time series when linking schools across waves. Since these school-level data are technically averages of repeated cross-sections of pupils, it is important that we control for compositional changes (Hallberg et al., [<reflink idref="bib35" id="ref44">35</reflink>]; Pohl et al., [<reflink idref="bib39" id="ref45">39</reflink>]). We do this using the following covariates: the percentage of pupils that are eligible for free school meals (FSM), the percentage of pupils with English as an additional language (EAL), percentage of pupils with special educational needs (SEN), school size (number of pupils) and school type.[<reflink idref="bib6" id="ref46">6</reflink>] Summary statistics are available in Table 2.</p> <p>Table 2. Characteristics of the RCT treatment group, RCT control group, and CITS comparison group averaged across the four pretreatment years.</p> <p> <ephtml> <table><thead><tr><td>Intervention</td><td>Covariate</td><td>RCT treat</td><td>RCT cont</td><td>CITS comp</td><td>Test of difference on RCT cont and CITS comp</td></tr></thead><tbody valign="top"><tr><td>–</td><td># Schools</td><td>12</td><td>12</td><td>4169</td><td>–</td></tr><tr><td>FLIP</td><td>%FSM</td><td>36.1</td><td>40.5</td><td>23.9</td><td>**</td></tr><tr><td>%EAL</td><td>31.2</td><td>41.7</td><td>9.8</td><td>**</td></tr><tr><td>%SEN</td><td>9.9</td><td>9.5</td><td>10.2</td><td>–</td></tr><tr><td>Prior ach.</td><td>14.7</td><td>14.4</td><td>15.3</td><td>**</td></tr><tr><td>–</td><td># Schools</td><td>48</td><td>45</td><td>7228</td><td>–</td></tr><tr><td>LERS</td><td>%FSM</td><td>58.3</td><td>59.2</td><td>28.9</td><td>**</td></tr><tr><td>%EAL</td><td>9.8</td><td>18.8</td><td>17.7</td><td>–</td></tr><tr><td>%SEN</td><td>16.2</td><td>16.4</td><td>10.6</td><td>**</td></tr><tr><td>Prior ach.</td><td>13.7</td><td>13.7</td><td>15.2</td><td>**</td></tr><tr><td>–</td><td># Schools</td><td>20</td><td>19</td><td>2269</td><td>–</td></tr><tr><td>RISE</td><td>%FSM</td><td>23.6</td><td>27.6</td><td>27.9</td><td>–</td></tr><tr><td>%EAL</td><td>4.9</td><td>15.9</td><td>14.4</td><td>–</td></tr><tr><td>%SEN</td><td>6.9</td><td>6.1</td><td>7.0</td><td>–</td></tr><tr><td>%PLO</td><td>14.4</td><td>16.0</td><td>15.7</td><td>–</td></tr><tr><td>Prior ach.</td><td>27.6</td><td>27.3</td><td>27.4</td><td>–</td></tr><tr><td>–</td><td># Schools</td><td>41</td><td>40</td><td>2877</td><td>–</td></tr><tr><td>TEOB</td><td>%FSM</td><td>33.3</td><td>27.5</td><td>23.5</td><td>*</td></tr><tr><td>%EAL</td><td>16.3</td><td>11.6</td><td>12.8</td><td>–</td></tr><tr><td>%SEN</td><td>9.7</td><td>9.0</td><td>8.5</td><td>–</td></tr><tr><td>%PLO</td><td>19.3</td><td>19.0</td><td>16.0</td><td>*</td></tr><tr><td>Prior ach.</td><td>26.6</td><td>26.7</td><td>27.3</td><td>*</td></tr></tbody></table> </ephtml> </p> <ulist> <item>4 See Table 1 for intervention acronyms. FSM: free school meals; EAL: English as an additional language; SEN: special educational needs; Prior Ach: prior achievement; For the primary school interventions (FLIP and LERS) prior achievement is the average Key Stage 1 (age 7) SATs score of the cohort. For the secondary school interventions (RISE and TEOB) prior achievement is the average Key Stage 2 (age 11) SATs score. PLO: pupils with low prior attainment at end of KS2; Variables in brackets: information not available for all years (not included in CITS modeling and conditional plots).</item> <item>5 **<emph>p</emph> < 0.01. *<emph>p</emph> < 0.05.</item> </ulist> <p>For both of our primary school interventions, the original RCTs used as their outcome measure pupils' scores in SATs (Statutory Attainment Tests), which are standardized, high-stakes tests taken by all pupils in state schools in England at age 11. For both of our secondary interventions, the original RCTs used as their outcome measure pupils' scores in their math GCSE (General Certificate of Secondary Education), which are standardized, high-stakes tests taken by all pupils in state schools in England at age 16. In line with this, we use school average math SATs scores (primary) and school average math GCSE score (secondary) as our outcome measure in the within-study comparisons. For each of our four interventions, we then create a time-series of the relevant outcome measure, spanning the four years prior to the intervention being introduced (<emph>t-4</emph>, <emph>t-3</emph>, <emph>t-2</emph> and <emph>t-1</emph>) and a single period after the intervention (<emph>t + 1</emph>), which is always the same year in which the original RCT estimated the impact.[<reflink idref="bib7" id="ref47">7</reflink>] We also utilize prior achievement measures: end of Key Stage 1 (age 7) SATs math scores for the primary school interventions, and end of Key Stage 2 (age 11) math SATs scores for secondary.</p> <hd id="AN0159448677-8">Methods</hd> <p></p> <hd id="AN0159448677-9">Models for the within-Study Comparisons</hd> <p>The conventional two-period difference-in-difference model (also referred to as the "baseline mean" model) can be written as:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Maths</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>=</mo><mi mathvariant="normal" /><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn></mrow></msub><mo>+</mo><mi mathvariant="normal" /><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub><msub><mrow><mi>X</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>u</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>ϵ</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> </p> <p>Where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Maths</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> is the average math score in school <emph>s</emph> in year <emph>y;</emph></p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub></math> </ephtml> is a binary variable indicating whether school <emph>s</emph> is in the treatment group or not<emph>;</emph></p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub></math> </ephtml> is a binary variable capturing whether the treatment is "on" or "off" in year <emph>y</emph> (our models only include one post-treatment period);</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>X</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> is a vector of school-level covariates (FSM, EAL, SEN and cohort prior achievement);</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>u</mi></mrow><mrow><mi>s</mi></mrow></msub></math> </ephtml> is a school-specific random error term and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>ϵ</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> is a year-specific random error term.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub></math> </ephtml> captures the difference in the level of the outcome variables between the treatment and comparison groups in the pretreatment period and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub></math> </ephtml> captures the level change between the pre- and post-treatment periods in the comparison group.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub></math> </ephtml> is the coefficient of interest since it captures the level change between the pre- and post-treatment period in the treatment group, over and above that in the comparison group. The key threats to the validity of the baseline mean model in our setting are changes in the composition of students attending the schools from one year to the next in a way that is related to the outcome, or a lack of common trends in treatment and comparison schools (Cunningham, [<reflink idref="bib14" id="ref48">14</reflink>]).</p> <p>In the absence of common trends, the baseline-linear trends model (Bloom, [<reflink idref="bib4" id="ref49">4</reflink>]) can be adopted instead:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Maths</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>=</mo><mi mathvariant="normal" /><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow></msub><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>5</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>6</mn></mrow></msub><msub><mrow><mi>X</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>u</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>ϵ</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> </p> <p>Where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow></msub></math> </ephtml> is a variable capturing the periods <emph>t-4</emph>, ..., <emph>t + 1</emph> and all other variables are defined as in the previous model.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal" /><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn></mrow></msub></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub><mi mathvariant="normal" /></math> </ephtml> now capture the intercepts for the comparison and treatment groups, respectively.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub></math> </ephtml> now captures the gradient of the comparison group time series in the pretreatment period,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub></math> </ephtml> does the same for the treatment group, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub></math> </ephtml> captures the deviation from trend in the post treatment period for the comparison group.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>5</mn></mrow></msub></math> </ephtml> is now the coefficient of interest since it captures the deviation from trend in the treatment group, over and above that in the comparison group. The key threats to the validity of the baseline linear-trends model in our setting are compositional changes, time-varying confounders that cause the treatment group to deviate from trend in the post-treatment period differently to those in the comparison group, or misspecification of the model capturing the pretreatment trends such that they would be incorrectly projected into the post-treatment period (Hallberg et al., [<reflink idref="bib35" id="ref50">35</reflink>]; St. Clair & Cook, [<reflink idref="bib49" id="ref51">49</reflink>]).</p> <p>Misspecification of the model due to non-linear pretreatment trends showing a single turning point (quadratic functional form) can be accommodated through the addition of quadratic year terms in the baseline nonlinear-trends model:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="italic">Maths</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>=</mo><mi mathvariant="normal" /><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub><msubsup><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>s</mi></mrow></msub><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>5</mn></mrow></msub><msubsup><mrow><mi mathvariant="italic">Year</mi></mrow><mrow><mi>y</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>6</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>7</mn></mrow></msub><msub><mrow><mi mathvariant="italic">Post</mi></mrow><mrow><mi>y</mi></mrow></msub><mi mathvariant="normal">*</mi><msub><mrow><mi mathvariant="italic">Treat</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>β</mi></mrow><mrow><mn>8</mn></mrow></msub><msub><mrow><mi>X</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub><mo>+</mo><msub><mrow><mi>u</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>+</mo><msub><mrow><mi>ϵ</mi></mrow><mrow><mi>s</mi><mi>y</mi></mrow></msub></math> </ephtml> </p> <p>In this model,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub><mi mathvariant="normal" /></math> </ephtml> capture the linear and quadratic components of the pretreatment trends in the comparison group and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>5</mn></mrow></msub></math> </ephtml> captures the linear and quadratic components of the way in which the pretreatment trend in the treatment group deviates from that of the comparison group.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>β</mi></mrow><mrow><mn>7</mn></mrow></msub></math> </ephtml> is the coefficient of interest, since it captures the deviation from trend in the treatment group, over and above that in the comparison group.</p> <p>The choice between the various CITS specifications should be based on the functional form of the pretreatment trends (St. Clair et al., [<reflink idref="bib50" id="ref52">50</reflink>]) and it is therefore common practice for CITS papers to begin by graphing the time series. Technically, however, the relevant consideration is the pretreatment trends conditional on observables. We therefore use the binsreg command (Cattaneo et al., [<reflink idref="bib8" id="ref53">8</reflink>]) to plot the mean values of the outcome variables for each year and each intervention, conditional on the covariates. These can be found in Appendix Figure A1. Based on visual inspection of these figures, we categorized each of the interventions as showing either: parallel trends, in which case we analyzed it using the baseline mean model; non-parallel linear trends, in which case we analyzed it using the linear baseline-trends model; non-parallel trends displaying one turning point, in which case we analyzed it using the non-linear baseline-trend model (Hallberg et al., [<reflink idref="bib36" id="ref54">36</reflink>]).</p> <p>A final methodological decision relates to which schools to include in our comparison time series. In line with guidance based on previous within-study comparisons, we chose to use only geographically local schools, on the grounds that they are more likely be similar in terms of unobservable characteristics (Bifulco, [<reflink idref="bib3" id="ref55">3</reflink>]; Cook et al., [<reflink idref="bib12" id="ref56">12</reflink>]; Hallberg et al., [<reflink idref="bib34" id="ref57">34</reflink>]; Wong et al., [<reflink idref="bib57" id="ref58">57</reflink>]). More specifically, for each intervention we include in our comparison group all state-funded schools in the same phase (primary/secondary) in regions of England that contain at least one treatment school.[<reflink idref="bib8" id="ref59">8</reflink>] Comparisons of the characteristics of schools in the RCT treatment, RCT control and CITS comparisons groups can be found in Table 2. Comparisons of the outcomes across the same three groups, after conditioning on covariates, can be found in Figure A1 in the appendix. Appendix Table A3 provides a summary of the design of all four within-study comparisons.</p> <hd id="AN0159448677-10">Estimating Overall Bias</hd> <p>Once we have used the relevant models to estimate impact relative to this comparison group, we are in a position to compare the CITS and RCT results. Within-study comparisons generally compare both the impact estimates (in effect size units) and the precisions or <emph>p</emph> values from the two evaluations. The latter often involves comparing the <emph>p</emph> values of the benchmark (experimental) impact estimate and the <emph>p</emph> values of the comparison (non-experimental) study, which yields a simple binary metric for whether the results of the two studies agree. We avoid this approach for two reasons. First, <emph>p</emph> values have very different interpretations in experimental and observational research, calling into questions their comparability (Berk, [<reflink idref="bib2" id="ref60">2</reflink>]; Rosenbaum, [<reflink idref="bib41" id="ref61">41</reflink>]). Second, the difference between one statistically significant impact estimate and another not statistically significant impact estimate may not itself be statistically significant (Gelman & Stern, [<reflink idref="bib24" id="ref62">24</reflink>]).</p> <p>Instead, we focus exclusively on comparisons of the impact estimates expressed as the standardized mean difference effect sizes, calculated by subtracting the mean math outcome in the untreated schools (<emph>T</emph> = 0) from that in the treated schools (<emph>T</emph> = 1) and dividing by the SD across both <emph>T</emph> = 0 and <emph>T</emph> = 1 schools pooled together. This is also known as the <emph>standardized effect difference</emph> (Steiner & Wong, [<reflink idref="bib46" id="ref63">46</reflink>]). The school-level data that we employ for our CITS evaluations leads us to employ the school (s) level SD of the outcome measures in the denominator. Where effect sizes in the original RCT reports were calculated using the pupil level SD, we recalculate these using the school-level SD in order to ensure comparability (see Appendix Table A1 for details). Using school-level SD is also consistent with existing CITS WSC, which makes our findings comparable with the existing literature (Hallberg et al., [<reflink idref="bib36" id="ref64">36</reflink>]; St. Clair et al., [<reflink idref="bib50" id="ref65">50</reflink>]). However, because most analysts are used to pupil-level effect sizes, which are much smaller, we also report these wherever possible (Kraft, [<reflink idref="bib29" id="ref66">29</reflink>]).</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">Effect</mi><mi mathvariant="normal" /><mi mathvariant="italic">Size</mi><mo>=</mo><mi mathvariant="normal" /><mfrac><mrow><mo>(</mo><msub><mrow><mover accent="true"><mrow><mi mathvariant="italic">Maths</mi></mrow><mo>¯</mo></mover></mrow><mrow><mi>s</mi><mo>,</mo><mi mathvariant="normal" /><mi>T</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>−</mo><mi mathvariant="normal" /><msub><mrow><mover accent="true"><mrow><mi mathvariant="italic">Maths</mi></mrow><mo>¯</mo></mover></mrow><mrow><mi>s</mi><mo>,</mo><mi mathvariant="normal" /><mi>T</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>)</mo></mrow><mrow><msub><mrow><mi>S</mi><mi>D</mi></mrow><mrow><mi>s</mi><mo>,</mo><mi mathvariant="normal" /><mi mathvariant="italic">pooled</mi></mrow></msub></mrow></mfrac></math> </ephtml> </p> <hd id="AN0159448677-11">Summary Measures of Bias</hd> <p>Well-implemented RCTs provide an unbiased estimator for the causal impacts of interventions. In an individual RCT with finite sample size, however, there will be imprecision due to sampling error or so-called "realized confounding" (Deaton & Cartwright, [<reflink idref="bib16" id="ref67">16</reflink>]). Indeed, in education, RCT impact estimates tend to be very imprecise—the mean width of 95% confidence intervals in EEF trials, for example, is 0.34 pupil-level SD (Lortie-Forgues & Inglis, [<reflink idref="bib32" id="ref68">32</reflink>]). For a given intervention <emph>i,</emph> differences in treatment effect estimates between an RCT (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup><mo stretchy="false">)</mo></math> </ephtml> and a CITS evaluation (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo stretchy="false">)</mo><mi mathvariant="normal" /></math> </ephtml> can therefore reflect either selection bias from the CITS</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">(</mo><msubsup><mrow><mi>θ</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo stretchy="false">)</mo><mi mathvariant="normal" /></math> </ephtml> or random sampling error from the RCT (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> ) or the CITS (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> ), which have mean 0 and variance</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>v</mi><mo>:</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup><mi mathvariant="normal" /><mo>−</mo><mi mathvariant="normal" /><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>θ</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mi mathvariant="normal" /><mo>+</mo><mi mathvariant="normal" /><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup><mo>+</mo><mi mathvariant="normal" /><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup><mo>∼</mo><mi>N</mi><mo>(</mo><mn>0</mn><mo>,</mo><mi mathvariant="normal" /><msubsup><mrow><mi>v</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup><mo>)</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo>∼</mo><mi>N</mi><mo>(</mo><mn>0</mn><mo>,</mo><mi mathvariant="normal" /><msubsup><mrow><mi>v</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo>)</mo></math> </ephtml> </p> <p>Given that the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow></msub></math> </ephtml> are unknown, we need some method for isolating</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>θ</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mo>.</mo></math> </ephtml> A simple way to address this is to take the mean of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> −</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mi mathvariant="normal" /></math> </ephtml> across interventions <emph>i</emph>. Since</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>e</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> are zero in expectation, across many WSC we would expect these terms to fall out of the equation above. An important corollary of this is that we should not consider</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> −</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> for a given intervention to provide an unbiased estimate of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>θ</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> for that intervention. For a given intervention, we have no way of knowing how much of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> −</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mi mathvariant="normal" /></math> </ephtml> is composed of bias</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mi>θ</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup><mi mathvariant="normal" /></math> </ephtml> and how much is composed of noise. We therefore only interpret the mean of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">RCT</mi></mrow></msubsup></math> </ephtml> −</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mrow><mover accent="true"><mrow><mi mathvariant="italic">ATT</mi></mrow><mo>̂</mo></mover></mrow><mrow><mi>i</mi></mrow><mrow><mi mathvariant="italic">CITS</mi></mrow></msubsup></math> </ephtml> across our four interventions as an estimate of bias.</p> <p>A more sophisticated approach to combining estimates would use random effects meta-analysis in order to account for both the noise from RCT estimate within each WSC and the variation in selection bias across the WSCs (Chaplin et al., [<reflink idref="bib10" id="ref69">10</reflink>]; Weidmann & Miratrix, [<reflink idref="bib51" id="ref70">51</reflink>]). However, to our knowledge, there is no established method for ascertaining the variance surrounding the <emph>difference</emph> in effect size estimates without access to micro-data. Our school-level data does not permit this. However, in Appendix Table A4, we re-estimate the impact of the original RCTs using our school-level data, bootstrap the standard errors for the effect size difference (Steiner & Wong, [<reflink idref="bib46" id="ref71">46</reflink>]), and then meta-analyse the results. This approach ignores the units (pupils) within each cluster (school) and the resulting standard errors should therefore be considered as an upper bound on the true standard errors. However, this allows us to provide an approximate precision-weighted meta-analytic estimate of the bias across studies, as well as providing information that may be useful for subsequent meta-analysts.</p> <hd id="AN0159448677-12">Results</hd> <p>Table 3 shows the results of the CITS regression models for each of the four interventions, with all coefficients expressed in terms of school-level SD. The coefficient of interest can be found in bold in the first row (Treat × Post), which shows the change in the outcome in the treatment group in the post-treatment period, relative to that in the comparison group. The LERS and RISE models use the baseline linear trends specification. The FLIP and TEOB models use the non-linear baseline trends model, which also include quadratic terms for year interacted with treatment status, which allows for divergent non-linear trends in the treatment and comparison groups. The coefficients of interest vary from −0.03 to + 0.22 SD. It should be kept in mind that these coefficients are expressed in terms of school-level SD, which are typically 1.5–3 times larger than those expressed using pupil-level SD (Kraft, [<reflink idref="bib29" id="ref72">29</reflink>]). For transparency, we also report the results for each intervention using all three CITS specifications in Table A2 in the Appendix.</p> <p>Table 3. Comparative interrupted time series regressions.</p> <p> <ephtml> <table><thead><tr><td /><td>FLIP</td><td>LERS</td><td>RISE</td><td>TEOB</td></tr></thead><tbody valign="top"><tr><td>Treat x post</td><td>0.13 (0.51)</td><td>−0.03 (0.10)</td><td>0.16* (0.08)</td><td>0.22 (0.15)</td></tr><tr><td>Treat</td><td>−0.05</td><td>0.06</td><td>−0.05</td><td>−0.35*</td></tr><tr><td>(0.50)</td><td>(0.09)</td><td>(0.09)</td><td>(0.15)</td></tr><tr><td>Post</td><td>0.09**</td><td>0.15**</td><td>−0.07**</td><td>0.16**</td></tr><tr><td>(0.03)</td><td>(0.01)</td><td>(0.01)</td><td>(0.02)</td></tr><tr><td>Year</td><td>0.70**</td><td>–</td><td>–</td><td>0.43**</td></tr><tr><td>(0.02)</td><td>–</td><td>–</td><td>(0.02)</td></tr><tr><td>Year x treat</td><td>0.13</td><td>–</td><td>–</td><td>0.24</td></tr><tr><td>(0.44)</td><td>–</td><td>–</td><td>(0.13)</td></tr><tr><td>Year<sup>2</sup></td><td>–0.09**</td><td>–</td><td>–</td><td>–0.08**</td></tr><tr><td>(0.01)</td><td>–</td><td>–</td><td>(0.01)</td></tr><tr><td>Year<sup>2</sup> × treat</td><td>–0.01</td><td>–</td><td>–</td><td>–0.04</td></tr><tr><td>(0.09)</td><td>–</td><td>–</td><td>(0.02)</td></tr><tr><td>Marginal <italic>R</italic><sup>2</sup></td><td>0.442</td><td>0.338</td><td>0.707</td><td>0.738</td></tr><tr><td>Conditional <italic>R</italic><sup>2</sup></td><td>0.660</td><td>0.607</td><td>0.881</td><td>0.892</td></tr><tr><td><italic>N</italic></td><td>4,067</td><td>7,117</td><td>2,124</td><td>2,694</td></tr><tr><td>Parallel trends</td><td>–</td><td>✓</td><td>✓</td><td>–</td></tr><tr><td>Model type</td><td>Baseline non-linear trend</td><td>Baseline mean</td><td>Baseline mean</td><td>Baseline non-linear trend</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 Each column is a separate OLS regression. Coefficients and standard errors are expressed in school-level standard deviations. Numbers in parentheses are standard errors.</item> <item>7 **<emph>p</emph> < 0.01. *<emph>p</emph> < 0.05.</item> </ulist> <p>Our primary interest is in comparing the impact estimates from Table 3 with those from prior RCT evaluations of the same interventions. Table 4 shows the results of these four within-study comparisons. The effect size columns report the impact estimates from the original RCT and new CITS evaluations and the final two columns shows the RCT effect size minus the CITS effect size. The differences in effect sizes in the four WSCs vary from 0.03 to 0.13 school-level SD (0–0.04 pupil-level SD) in absolute magnitude. The mean absolute difference is 0.07 school-level SD (0.02 pupil-level SD).</p> <p>Table 4. Within-study comparison results.</p> <p> <ephtml> <table><thead><tr><td /><td>Specification</td><td>ES and (SE) in school-level SD</td><td>ES in pupil-level SD*</td></tr><tr><td>RCT</td><td>CITS</td><td>RCT-CITS</td><td>RCT-CITS</td></tr></thead><tbody valign="top"><tr><td>FLIP</td><td>Non-parallel trends, one turning point: Baseline non-linear trends</td><td char=".">0.26</td><td char=".">0.13</td><td char=".">+0.13</td><td char=".">+0.04</td></tr><tr><td char=".">(0.55)</td><td char=".">(0.51)</td></tr><tr><td>LERS</td><td>Parallel trends: Baseline mean model</td><td char=".">0.00</td><td char=".">−0.03</td><td char=".">+0.03</td><td char=".">0.00</td></tr><tr><td char=".">(0.84)</td><td char=".">(0.10)</td></tr><tr><td>RISE</td><td>Parallel trends: Baseline mean model</td><td char=".">0.19</td><td char=".">0.16</td><td char=".">+0.03</td><td char=".">+0.01</td></tr><tr><td char=".">(0.13)</td><td char=".">(0.08)</td></tr><tr><td>TEOB</td><td>Non-parallel trends, one turning point: Baseline non-linear trends</td><td char=".">0.15</td><td char=".">0.22</td><td char=".">−0.07</td><td char=".">−0.03</td></tr><tr><td char=".">(0.10)</td><td char=".">(0.15)</td></tr><tr><td>–</td><td>–</td><td>Mean difference:</td><td char=".">0.03</td><td char=".">0.01</td></tr><tr><td>–</td><td>–</td><td>Mean absolute difference:</td><td char=".">0.07</td><td char=".">0.02</td></tr></tbody></table> </ephtml> </p> <p>8 See Table 1 for intervention acronyms. ES: effect size; RCT: randomized controlled trial; CITS: comparative interrupted time series; RCT-CITS: RCT effect size minus the CITS effect size; To calculate the RCT-CITS effect size difference in terms of pupil-level SD we carry out a conversion using the pupil-and school-level SD appropriate to each trial, as reported in Appendix Table A1. We multiply the estimated difference by the school-level SD and divide by the pupil-level SD. This is a direct reversal of the process used to convert the RCT-estimated effect size into one expressed in terms of the school-level SD.</p> <p>The two interventions that showed parallel trends in the pretreatment period (LERS and RISE) showed the smallest absolute differences between the RCT and CITS impact estimates: 0.03 school-level SD or ≤ 0.01 pupil-level SD. By contrast, the interventions showing non-linear, non-parallel trends (TEOB and FLIP) showed slightly larger absolute differences: 0.07–0.13 school-level SD or 0.03–0.04 pupil-level SD. While all of these results rely on estimates from single within-study comparisons, it is notable that the more complicated the pretreatment trends are, the further the CITS impact estimate tends to be from the RCT impact estimate. Taking the mean of the signed (rather than absolute) differences helps net out the (mean zero) noise across the RCT impact estimates and gives us our key result on the bias across the four CITS studies in our setting: 0.03 school-level SD or 0.01 pupil-level SD.</p> <p>As discussed in the methods section, we cannot calculate correct standard errors for our effect size differences (RCT-CITS) using our school-level data. However, in Appendix Table A4, we calculate upper-bound estimates, which allows us to conduct a random-effects meta-analysis across the four effect size differences. Using this method, we estimate mean bias of 0.11 school-level SD, with an upper-bound standard error of 0.1 school-level SD (Appendix Figure A2). Scaling this down by the ratio of our school-level and pupil-level SD estimates in Table 4, this is equivalent to 0.04 pupil-level SD, with a standard error of 0.03 pupil-level SD. This precision-weighted average is slightly larger than our estimate in Table 4 (0.01 pupil-level SD) but is not statistically distinguishable from zero (<emph>p</emph> = 0.25).</p> <hd id="AN0159448677-13">Discussion</hd> <p>The CITS is in an increasingly popular approach for studying the impact of education policies, in part because it is well suited to the institutional characteristics and data available in education settings. Yet like all non-experimental methods, the assumptions on which the CITS depend cannot be directly verified. Within-study comparisons provide an alternative approach to evaluating the internal validity of CITS design, by comparison with an experimental benchmark. Using a database of experimental evaluations in England, we have contributed four new within-study comparisons of the school-level CITS design. By summarizing the effect size differences across the WSCs, we were able to get a better estimate of the bias across school-level CITS evaluations, providing important new evidence on the internal validity of the design in this setting.</p> <p>Across the four interventions, we estimate mean bias to be 0.03 school-level SD, or 0.01 pupil-level SD. One way of putting these findings into perspective is to compare it to the effect sizes typically found in the education literature. Median effect sizes for academic achievement in RCTs in education are between 0.07 around 0.1 pupil-level SD (Evans & Yuan, [<reflink idref="bib19" id="ref73">19</reflink>]; Kraft, [<reflink idref="bib29" id="ref74">29</reflink>]). Our estimate of bias thus represents 10–14% of the effect size that researchers might expect to find in similar research. Put another way, a recent review of effect sizes recommended deeming 0–0.05 pupil-level SD as a "small" effect, suggesting that 0.01 is also small (Kraft, [<reflink idref="bib29" id="ref75">29</reflink>]).</p> <p>At the intervention level, the differences between the RCT impact estimates and the CITS impact estimates are all below 0.13 school-level SD or 0.04 pupil-level SD in absolute value. This suggests that our (across intervention) estimate of bias is not the result of large intervention-level bias but with random sign, canceling each other out. Rather, our small across intervention estimate of bias likely reflects similarly small bias at the intervention level.</p> <p>An important caveat on our overall findings is that the choice between the baseline mean, baseline linear trends and baseline non-linear trend models does have an effect on our estimates (see Appendix Table A2). In line with previous research in this literature, this highlights the importance of choosing the correct model specifications through careful inspection of the pre-trends (St. Clair et al., [<reflink idref="bib50" id="ref76">50</reflink>]). However, our findings also serve as reminder that this is not entirely straightforward. Looking across the rows in Table A2 it is clear that we did not always choose the CITS specification that minimized the difference in impact estimates between the CITS and the original EEF RCT.</p> <p>How do our results compare to those from similar education WSCs? St. Clair et al. ([<reflink idref="bib50" id="ref77">50</reflink>]) report results from three within-study comparisons of education interventions. Two of the three interventions showed modellable pretreatment trends and in these cases the CITS impact estimates differed from the RCT impact estimates by 0.01–0.07 school-level SD in the authors' preferred specifications. The third of their three interventions showed pretreatment trends with more than one turning point and, as hypothesized, the CITS impact estimates showed much larger differences with those from the RCT. Across the three CITS within-study comparisons reported by Hallberg et al. ([<reflink idref="bib36" id="ref78">36</reflink>]), the CITS estimates for one year post-intervention differed from the RCT estimates in absolute magnitude by 0.05–0.08 school-level SD, depending on the model specification adopted. Our estimate of 0.03 school-level SD bias is therefore quantitatively similar to those in the existing literature on CITS within-study comparisons.</p> <p>Weidmann and Miratrix ([<reflink idref="bib51" id="ref79">51</reflink>]) report within-study comparisons of 14 interventions but using propensity score matching rather than CITS as their non-experimental estimator. Their meta-analytically derived estimate of mean bias across the interventions is 0.01 pupil-level SD. Our overall estimates of bias are hence also very similar to those from within-study comparisons using propensity score matching. The set of interventions studied by Weidmann and Miratrix are drawn from the same database as ours, and two of these interventions (LERS and FLIP) overlap with those in our study. The CITS impact estimates for the LERS intervention differs from the CITS RCT impact estimate by approximately 0.01 pupil-level SD. The equivalent result in Weidmann & Miratrix is reported graphically rather then numerically but appears very similar in magnitude. Our CITS impact estimate for the FLIP intervention differs from the RCT by approximately 0.05 pupil-level SD, which again appears very similar to the equivalent result in Weidmann and Miratrix. Although based on only two interventions, the results reviewed in this paragraph suggest that CITS and PSM methods, as implemented in these two papers, perform similarly.</p> <hd id="AN0159448677-14">Limitations</hd> <p>These findings should, of course, be interpreted in light of the limitations of this research. In particular, as we have emphasized, selection bias depends upon the setting and characteristics of the specific intervention. The treatments studied here involve school-level interventions deemed by the EEF to have shown evidence of promise in previous evaluations. All schools that participated in these trials volunteered to take part and, as can be seen from Table 2, these volunteer schools tended to have higher proportions of pupils eligible for free school meals than those in the surrounding areas. While volunteer schools appear to be comparable to surrounding schools in terms of their other observable characteristics, they may also differ in terms of unobservable characteristics, such as the extent to which they are open to adopting evidence-based practices. This limits the generalizability of our findings to other settings.</p> <p>Relatedly, our sample of interventions is–by necessity–composed entirely of interventions that have previously been evaluated using RCTs. This highlights an important constraint on the generalization of our findings to settings where an RCT is infeasible for ethical or logistical reasons. This is an important limitation, given that such settings are a prime candidate for the use of non-experimental designs. Having said that, the findings are still informative with respect to CITS designs used to retrospectively evaluate policies in similar settings in which random allocation was feasible but was not in fact used. Indeed, our view is that many of the interventions previously evaluated using CITS could have been randomly allocated but were not (e.g. Bell et al., [<reflink idref="bib1" id="ref80">1</reflink>]; Gandhi et al., [<reflink idref="bib23" id="ref81">23</reflink>]; Henry et al., [<reflink idref="bib37" id="ref82">37</reflink>]; Jacob et al., [<reflink idref="bib26" id="ref83">26</reflink>]; Strunk et al., [<reflink idref="bib47" id="ref84">47</reflink>]). In addition, as we discuss below, the results are also useful in assessing the validity of CITS designs for analysis of interventions in which attrition or noncompliance has raised doubts about the validity of an RCT.</p> <p>A further limitation of the present research is that we only reevaluate four interventions; a larger sample of interventions would do a better job of purging noise from the estimates of bias. Finally, our within-study comparisons exclusively used CITS specifications that projected the counterfactual outcomes one period into the future. Other research has found that CITS that project more periods into the future show larger bias (Hallberg et al., [<reflink idref="bib36" id="ref85">36</reflink>]). Our study is silent on this and therefore does not provide any warrant for using such CITS specifications.</p> <hd id="AN0159448677-15">Implications</hd> <p>Despite these limitations, we believe these findings have implications for the practice of impact evaluation in education. Hierarchies of evidence quality often privilege the use of RCTs over those from observational studies (Clarke et al., [<reflink idref="bib9" id="ref86">9</reflink>]) for the understandable reason that they provide impact estimates that are unbiased in expectation. However, this tends to ignore two important limitations of such studies in education. The first is that many such RCTs are underpowered, yielding findings that are either uninformative with respect to impact (Lortie-Forgues & Inglis, [<reflink idref="bib32" id="ref87">32</reflink>]) or yield apparently positive findings but with exaggerated effect sizes (Sims et al., [<reflink idref="bib45" id="ref88">45</reflink>]). A second limitation of RCTs in education is that they often suffer from high levels of attrition or noncompliance (Edovald & Nevill, [<reflink idref="bib18" id="ref89">18</reflink>]). This limits the internal validity of the trial by undermining the group-level balance initially achieved through randomization.</p> <p>Our results suggest that CITS designs show substantively small bias when used to estimate promising education interventions among volunteer schools. Moreover, in line with theory, differences between the CITS and RCT estimates are particularly small in cases that show stable, easily-modellable and parallel pre-trends. This implies that–if used judiciously–CITS evaluations can be used to complement RCT evaluations in such settings. In particular, school-level CITS designs could be used to corroborate effect size estimates from weakly-powered trials, in order to check that these are not overly influenced by residual imbalance from the RCT. Furthermore, school-level CITS estimates can be used to recover non-experimental impact estimates from "broken" trials in which attrition or contamination has compromised the initial random allocation. In the case of a trial with high attrition, a CITS can help wherever it is possible to obtain the school-level outcome data on the full sample. In the case of trial with treatment contamination, a CITS can help wherever the contamination has been caused by control group schools seeking the treatment <emph>as a result</emph> of being allocated to the control group. The CITS can address this problem by using a comparison group that does not overlap with the RCT control group. Our results suggest that these non-experimental impact estimates are likely to provide a good second-best impact estimate in a situation that would otherwise yield little useful evidence.</p> <p>It should be noted that other non-experimental methods, such as careful propensity score matching designs (Weidmann & Miratrix, [<reflink idref="bib51" id="ref90">51</reflink>]), can also be used to complement RCTs in the same way. Indeed, our results suggest that these two methods result in very similar levels of bias in this setting. The main advantage of the school-level CITS relative to propensity score matching is that it can often be implemented using non-sensitive, publicly available, school-level data (Jacob et al., [<reflink idref="bib27" id="ref91">27</reflink>]). The corresponding downside, as this paper has illustrated, is the need for a consistently recorded time series, which is often not available due to exam reforms or the cancelations of examinations in certain years. The relative benefits of the two non-experimental designs are therefore likely to depend on data availability. Either way, the evidence presented here suggests CITS are a form of non-experimental design that can estimate impact with substantively small bias in similar settings.</p> <hd id="AN0159448677-16">Disclosure Statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <hd id="AN0159448677-17">Appendix</hd> <p>Graph: Figure A1. Treatment and comparison group trends across the four interventions. Notes: FLIP: flipper learning-N = 4,084 schools; LERS: learner response system-N = 7,165 schools; RISE: N = 2,224 schools; TEOB: teacher observation-N = 2,829 schools; All Ns include treatment, control and comparison group schools. Non-trial GOR: schools in the same Government Office Region of England that did not participate in the original RCT; RCT Treatment: schools that were in the treatment group in the original RCT; RCT Control: schools that were in the control group in the original RCT; The years left to the vertical line are used as pretreatment year and the year right to the vertical is the post-treatment year. The mathematics scores were conditioned upon prior achievement, % special educational needs (except for RISE), % free school meals, % English as additional language, % pupil low prior achievement at the end of KS2 (only RISE and TEOB), school type and school size.</p> <p>Graph: Figure A2. Forest plot of the RCT-CITS for the four interventions. Notes: Figure A2 shows a forest plot of the effect size differences taken from Table A4. The meta-analytic (precision weighted) average of these effect size differences is 0.11 school-level standard deviations, with a 95% CI of -0.08–0.3 school-level standard deviations. It should be noted that these SE are upwardly biased (see the notes to Table A4) and should be considered upper bounds on the true SE.</p> <p>Table A1. Calculating the school-level effect sizes for the RCT impact estimates.</p> <p> <ephtml> <table><thead><tr><td>Study</td><td>EEF RCT pupil-level effect size</td><td>EEF pupil-level pooled SD</td><td>Measure correction formula</td><td>School-level pooled SD</td><td>RCT effect size using school-level pooled SD</td><td>Notes</td></tr></thead><tbody valign="top"><tr><td>FLIP</td><td char=".">0.09</td><td char=".">5.28</td><td>None</td><td char=".">1.82</td><td char=".">0.26</td><td>EEF-reported effect size extracted from Table 4 in the FLIP report.</td></tr><tr><td>LERS</td><td char=".">0.00</td><td char=".">22.22</td><td>Unknown but moot due to zero effect</td><td char=".">1.83</td><td char=".">0.00</td><td>EEF-reported effect size for Cohort 1 extracted from Table 10 in the LERS report.</td></tr><tr><td>RISE</td><td char=".">0.09</td><td char=".">10.79</td><td>1/6 * SD(X) = SD(Y) with Y the original scale of RISE and X our scale</td><td char=".">5.06</td><td char=".">0.19</td><td>EEF-reported effect size for cohort A extracted from Table 12 in the RISE report.</td></tr><tr><td>TEOB</td><td char=".">0.06</td><td char=".">11.98</td><td>None</td><td char=".">4.83</td><td char=".">0.15</td><td>EEF-reported effect size for Cohort 1 extracted from Table 13 in the TEOP report.</td></tr></tbody></table> </ephtml> </p> <p>9 Study acronyms can be found in Table 1.</p> <p>Table A2. CITS results using all three specifications.</p> <p> <ephtml> <table><thead><tr><td /><td>EEF RCT School-level effect size</td><td>CITS</td></tr><tr><td>Study</td><td>Baseline mean</td><td>Linear trends</td><td>Non-linear trends</td></tr></thead><tbody valign="top"><tr><td>FLIP</td><td char=".">0.26 </td><td char=".">0.14</td><td char=".">0.04 </td><td char="."><bold>0.13</bold></td></tr><tr><td char=".">(0.55)</td><td char=".">(0.21)</td><td char=".">(0.28)</td><td char="."><bold>(0.51)</bold></td></tr><tr><td>LERS</td><td char=".">0.00</td><td char="."><bold>−0.03</bold></td><td char=".">0.17</td><td char=".">0.40</td></tr><tr><td char=".">(0.84)</td><td char="."><bold>(0.10)</bold></td><td char=".">(0.13)</td><td char=".">(0.25)</td></tr><tr><td>RISE</td><td char=".">0.19</td><td char="."><bold>0.16</bold></td><td char=".">0.06</td><td char=".">0.16 </td></tr><tr><td char=".">(0.13)</td><td char="."><bold>(0.08)</bold></td><td char=".">(0.11)</td><td char=".">(0.20)</td></tr><tr><td>TEOB</td><td char=".">0.15</td><td char=".">0.08</td><td char=".">0.01</td><td char="."><bold>0.22</bold></td></tr><tr><td char=".">(0.10)</td><td char=".">(0.06)</td><td char=".">(0.08)</td><td char="."><bold>(0.15)</bold></td></tr></tbody></table> </ephtml> </p> <p>10 Study acronyms can be found in Table 1. Bolded numbers represent the specification adopted in our preferred estimates. Coefficients and standard errors are shown in school-level SD. Numbers in parentheses are standard errors.</p> <p>Table A3. Summary table for the within-study comparisons.</p> <p> <ephtml> <table><thead><tr><td>Intervention</td><td>Original EEF RCT</td><td>Within-study comparison CITS</td></tr><tr><td>Name</td><td>Brief description</td><td>Estimand</td><td>No. of schools</td><td>Location of schools</td><td>Control variables</td><td>Attrition</td><td>Estimand</td><td>No. of schools</td><td>Periods pre/ post intervention</td><td>Preferred CITS specification</td><td>Control variables</td><td>Location of schools</td></tr></thead><tbody valign="top"><tr><td>FLIP: flipped learning</td><td>Pupils study new math at home, then consolidate through work in class.</td><td>Intention to treat</td><td>24</td><td>SE, SW, WM</td><td>KS1 SATs prior test scores</td><td>Schools: 0% pupils: 7%</td><td>Intention to treat</td><td>4,181</td><td>Pre: 4 Post: 1</td><td>Baseline non-linear trends</td><td>KS1 SATs, %FSM, %EAL, %SEN, school type and size.</td><td>SE, SW, WM</td></tr><tr><td>LERS: learner response system</td><td>Immediate in-class pupil feedback from teachers via electronic devices.</td><td>Intention to treat</td><td>97</td><td>EE, LO, NW, YH</td><td>KS1 SATs prior test scores</td><td>Schools: 3.1% pupils: 11.7%</td><td>Intention to treat</td><td>7,276</td><td>Pre: 4 Post: 1</td><td>Baseline mean</td><td>KS1 SATs, %FSM, %EAL, %SEN, school type and size.</td><td>EE, LO, NW, YH</td></tr><tr><td>RISE: rise programme</td><td>Teacher "Research Leads" supporting teachers' use of research.</td><td>Intention to treat</td><td>40</td><td>LO, NE, NW, SE, SW, WM, YH</td><td>KS2 SATs prior test scores</td><td>Schools: 2.5% pupils: 8.8%</td><td>Intention to treat</td><td>2,289</td><td>Pre: 4 Post: 1</td><td>Baseline mean</td><td>KS2 SATs, %FSM, %EAL, %PLO, school type and size.</td><td>LO, NE, NW, SE, SW, WM, YH</td></tr><tr><td>TEOB: teacher observation</td><td>Teacher PD based on programme of structured peer lesson observations.</td><td>Intention to treat</td><td>82</td><td>All of England</td><td>KS2 SATs prior test scores</td><td>Schools: 0% pupils: 1.4%</td><td>Intention to treat</td><td>2,918</td><td>Pre: 4 Post: 1</td><td>Baseline non-linear trends</td><td>KS2 SATs, %FSM, %EAL, %SEN, %PLO, school type and size.</td><td>All of England</td></tr></tbody></table> </ephtml> </p> <p>11 EEF: education endowment foundation; RCT: randomized controlled trial; CITS: comparative interrupted time series; KS1: key stage 1 (age 5–7); KS2: key stage 2 (age 8–11); SATs: statutory attainment tests; FSM: free school meals; EAL: English as an additional language; SEN: special educational needs; EE: East of England; LO: London; NE: North East; NW: North West; SE: South East; SW: South West; WM: West Midlands; YH: Yorkshire and the Humber.</p> <p>Table A4. Summary table for the within-study comparisons (expressed in school-level SD).</p> <p> <ephtml> <table><thead><tr><td /><td>RCT</td><td>CITS</td><td>RCT-CITS</td></tr><tr><td /><td>ES</td><td>SE</td><td>ES</td><td>SE</td><td>ES</td><td>SE</td></tr></thead><tbody valign="top"><tr><td>FLIP</td><td char=".">0.76</td><td char=".">0.41</td><td char=".">0.13</td><td char=".">0.43</td><td char=".">0.63</td><td char=".">0.52</td></tr><tr><td>LERS</td><td char=".">0.12</td><td char=".">0.19</td><td char=".">−0.02</td><td char=".">0.12</td><td char=".">0.13</td><td char=".">0.18</td></tr><tr><td>RISE</td><td char=".">0.23</td><td char=".">0.16</td><td char=".">0.15</td><td char=".">0.07</td><td char=".">0.08</td><td char=".">0.15</td></tr><tr><td>TEOB</td><td char=".">0.27</td><td char=".">0.17</td><td char=".">0.21</td><td char=".">0.10</td><td char=".">0.07</td><td char=".">0.20</td></tr></tbody></table> </ephtml> </p> <p>12 Table A4 summarizes the inputs to our meta-analysis of the effect size differences across our four within-study comparisons. All effect sizes (ES) and standard errors (SE) are expressed in terms of school-level standard deviations. The two RCT columns show the results of re-estimating the impact of the four interventions using the experimental treatment and control group schools, using school-level data. In each case, the models use the same outcomes, measured at the same timepoint, for the same treatment cohorts, using the same covariates (albeit measured at school-level), as the original EEF RCT. The equivalent results derived from pupil-level data, as reported in the original EEF evaluation reports, can be found in the "RCT" column of Table 4. The ES differ from those in Table 4 because we do not have access to the underlying micro data, meaning that we cannot control for prior attainment at the pupil-level. In addition, the SE are upwardly biased because they do not take account of the units (pupils) within each cluster (schools). The SE should therefore be considered upper bounds on the true SE. The CITS columns contain the results from Table 3. The ES in the RCT-CITS columns are simply the ES from the RCT column minus the ES from the CITS columns. The SE in the RCT-CITS columns are derived using the procedure from footnote 3 of Steiner and Wong ([<reflink idref="bib46" id="ref92">46</reflink>]) and are upwardly biased for the same reason that the SE in the in the RCT columns are upwardly biased. The results in the RCT-CITS columns form the bases for our meta-analysis, as reported in Figure A2.</p> <ref id="AN0159448677-18"> <title> References </title> <blist> <bibl id="bib1" idref="ref9" type="bt">1</bibl> <bibtext> Bell, S. H., Olsen, R. B., Orr, L. L., & Stuart, E. A. (2016). Estimates of external validity bias when impact evaluations select sites nonrandomly. Educational Evaluation and Policy Analysis, 38 (2), 318 – 335. https://doi.org/10.3102/0162373715617549</bibtext> </blist> <blist> <bibl id="bib2" idref="ref32" type="bt">2</bibl> <bibtext> Berk, R. A. (2004). Regression analysis: A constructive critique. SAGE.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref33" type="bt">3</bibl> <bibtext> Bifulco, R. (2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within‐study comparison. Journal of Policy Analysis and Management, 31 (3), 729 – 751. https://doi.org/10.1002/pam.20637</bibtext> </blist> <blist> <bibl id="bib4" idref="ref34" type="bt">4</bibl> <bibtext> Bloom, H. S. (2003). Using "short" interrupted time-series analysis to measure the impacts of whole-school reforms. With applications to a study of accelerated schools. Evaluation Review, 27 (1), 3 – 49. https://doi.org/10.1177/0193841X02239017</bibtext> </blist> <blist> <bibl id="bib5" idref="ref4" type="bt">5</bibl> <bibtext> Bloom, S., Ham, S., Melton, L., & O'Brien, J. (2001). Evaluating the accelerated schools approach: A look at early implementation and impacts in eight elementary schools. Manpower Demonstration Research Corporation.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref24" type="bt">6</bibl> <bibtext> Bloom, H., Michalopoulos, C., Hill, C., & Lei, Y. (2002). Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? (MDRC Working Papers on Research Methodology). https://<ulink href="http://www.mdrc.org/sites/default/files/full%5f43.pdf">www.mdrc.org/sites/default/files/full%5f43.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib7" idref="ref2" type="bt">7</bibl> <bibtext> Card, D., & Krueger, A. B. (1993). Minimum wages and employment: A case study of the fast food industry in New Jersey and Pennsylvania (No. w4509). National Bureau of Economic Research.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref53" type="bt">8</bibl> <bibtext> Cattaneo, M. D., Crump, R. K., Farrell, M. H., & Feng, Y. (2019). On binscatter. arXiv preprint arXiv:1902.09608.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref86" type="bt">9</bibl> <bibtext> Clarke, B., Gillies, D., Illari, P., Russo, F., & Williamson, J. (2014). Mechanisms and the evidence hierarchy. Topoi, 33 (2), 339 – 360. https://doi.org/10.1007/s11245-013-9220-9</bibtext> </blist> <blist> <bibtext> Chaplin, D. D., Cook, T. D., Zurovac, J., Coopersmith, J. S., Finucane, M. M., Vollmer, L. N., & Morris, R. E. (2018). The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study comparisons. Journal of Policy Analysis and Management, 37 (2), 403 – 429. https://doi.org/10.1002/pam.22051</bibtext> </blist> <blist> <bibtext> Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin.</bibtext> </blist> <blist> <bibtext> Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons. Journal of Policy Analysis and Management, 27 (4), 724 – 750. https://doi.org/10.1002/pam.20375</bibtext> </blist> <blist> <bibtext> Coopersmith, J., Cook, T. D., Zurovac, J., Chaplin, D., & Forrow, L. V. (2022). Internal and external validity of the comparative interrupted time‐series design: A meta‐analysis. Journal of Policy Analysis and Management, 41 (1), 252 – 277. https://doi.org/10.1002/pam.22361</bibtext> </blist> <blist> <bibtext> Cunningham, S. (2020). Causal inference: The mixtape (Vol. 18). Tufte-Latex.GoogleCode.com.</bibtext> </blist> <blist> <bibtext> Dawson, A., Yeomans, E., & Brown, E. (2018). Methodological challenges in education RCTs: Reflections from England's education endowment foundation. Educational Research, 60 (3), 292 – 310. https://doi.org/10.1080/00131881.2018.1500079</bibtext> </blist> <blist> <bibtext> Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science and Medicine, 210, 2 – 21. https://doi.org/10.1016/j.socscimed.2017.12.005</bibtext> </blist> <blist> <bibtext> Dee, T. S., Jacob, B., & Schwartz, N. L. (2013). The effects of NCLB on school resources and practices. Educational Evaluation and Policy Analysis, 35 (2), 252 – 279. https://doi.org/10.3102/0162373712467080</bibtext> </blist> <blist> <bibtext> Edovald, T., & Nevill, C. (2020). Working out what works: The case of the education endowment foundation in England. ECNU Review of Education, 4 (1), 46–64.</bibtext> </blist> <blist> <bibtext> Evans, D., & Yuan, F. (2020). How big are effect sizes in international education studies? Centre for Global Development Working Paper 545.</bibtext> </blist> <blist> <bibtext> Eyles, A., & Machin, S. (2019). The introduction of academy schools to England's education. Journal of the European Economic Association, 17 (4), 1107–1146. https://doi.org/10.1093/jeea/jvy021</bibtext> </blist> <blist> <bibtext> Ferraro, P. J., & Miranda, J. J. (2014). The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark. Journal of Economic Behavior and Organization, 107, 344 – 365. https://doi.org/10.1016/j.jebo.2014.03.008</bibtext> </blist> <blist> <bibtext> Fretheim, A., Soumerai, S. B., Zhang, F., Oxman, A. D., & Ross- Degnan, D. (2013). Interrupted time-series analysis yielded an effect estimate concordant with the cluster randomized controlled-trial result. Journal of Clinical Epidemiology, 66 (8), 883 – 887.</bibtext> </blist> <blist> <bibtext> Gandhi, A. G., Slama, R., Park, S. J., Russo, P., Winner, K., Bzura, R., Jones, W., & Williamson, S. (2018). Focusing on the whole student: An evaluation of Massachusetts's wraparound zone initiative. Journal of Research on Educational Effectiveness, 11 (2), 240 – 266. https://doi.org/10.1080/19345747.2017.1413691</bibtext> </blist> <blist> <bibtext> Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60 (4), 328 – 331. https://doi.org/10.1198/000313006X152649</bibtext> </blist> <blist> <bibtext> Grissom, J. A., Nicholson-Crotty, S., & Harrington, J. R. (2014). Estimating the effects of "no child left behind on teachers" work environments and job attitudes. Educational Evaluation and Policy Analysis, 36 (4), 417 – 436. https://doi.org/10.3102/0162373714533817</bibtext> </blist> <blist> <bibtext> Jacob, B., Dynarski, S., Frank, K., & Schneider, B. (2017). Are expectations alone enough? Estimating the effect of a mandatory college-prep curriculum in Michigan. Educational Evaluation and Policy Analysis, 39 (2), 333 – 360. https://doi.org/10.3102/0162373716685823</bibtext> </blist> <blist> <bibtext> Jacob, R. T., Goddard, R. D., & Kim, E. S. (2014). Assessment of the use of aggregate data in the evaluation of school-based interventions: Implications for evaluation research and state policy regarding public-use data. Educational Evaluation and Policy Analysis, 36 (1), 44 – 66. https://doi.org/10.3102/0162373713485814</bibtext> </blist> <blist> <bibtext> Kim, Y., & Steiner, P. (2016). Quasi-experimental designs for causal inference. Educational Psychologist, 51 (3–4), 395 – 405.</bibtext> </blist> <blist> <bibtext> Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49 (4), 241 – 253. https://doi.org/10.3102/0013189X20912798</bibtext> </blist> <blist> <bibtext> LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 76 (4), 604 – 620.</bibtext> </blist> <blist> <bibtext> Lee, J., & Reeves, T. (2012). Revisiting the impact of NCLB high-stakes school accountability, capacity, and resources: State NAEP 1990–2009 reading and math achievement gaps and trends. Educational Evaluation and Policy Analysis, 34 (2), 209 – 231. https://doi.org/10.3102/0162373711431604</bibtext> </blist> <blist> <bibtext> Lortie-Forgues, H., & Inglis, M. (2019). Rigorous large-scale educational RCTs are often uninformative: Should we be concerned? Educational Researcher, 48 (3), 158 – 166. https://doi.org/10.3102/0013189X19832850</bibtext> </blist> <blist> <bibtext> Glazerman, S., Levy, D. M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589 (1), 63 – 93. https://doi.org/10.1177/0002716203254879</bibtext> </blist> <blist> <bibtext> Hallberg, K., Wong, V. C., & Cook, T. D. (2016). Evaluating methods for selecting school-level comparisons in quasi-experimental designs: Results from a within-study comparison. EdPolicyWorks Working Paper Series No. 47.</bibtext> </blist> <blist> <bibtext> Hallberg, K., Williams, R., Swanlund, A., & Eno, J. (2018). Short comparative interrupted time series using aggregate school-level data in education research. Educational Researcher, 47 (5), 295 – 306. https://doi.org/10.3102/0013189X18769302</bibtext> </blist> <blist> <bibtext> Hallberg, K., Williams, R., & Swanlund, A. (2020). Improving the use of aggregate longitudinal data on school performance to assess program effectiveness: Evidence from three within study comparisons. Journal of Research on Educational Effectiveness, 13 (3), 518 – 545. https://doi.org/10.1080/19345747.2019.1698088</bibtext> </blist> <blist> <bibtext> Henry, G. T., Pham, L. D., Kho, A., & Zimmer, R. (2020). Peeking into the black box of school turnaround: A formal test of mediators and suppressors. Educational Evaluation and Policy Analysis, 42 (2), 232 – 256. https://doi.org/10.3102/0162373720908600</bibtext> </blist> <blist> <bibtext> McKenzie, D., Stillman, S., & Gibson, J. (2010). How important is selection? Experimental vs. non-experimental measures of the income gains from migration. Journal of the European Economic Association, 8 (4), 913 – 945. https://doi.org/10.1111/j.1542-4774.2010.tb00544.x</bibtext> </blist> <blist> <bibtext> Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31 (4), 463 – 479. https://doi.org/10.3102/0162373709343964</bibtext> </blist> <blist> <bibtext> Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (Vol. 2, pp. 295 – 316). Russell Sage Foundation.</bibtext> </blist> <blist> <bibtext> Rosenbaum, P. R. (2017). Observation and experiment. Harvard University Press.</bibtext> </blist> <blist> <bibtext> Rudd, P., Aguilera, A., Elliott, L., & Chambers, E. (2017). Mathsflip: Flipped learning. Evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> Schneeweiss, S., Maclure, M., Carleton, B., Glynn, R. J., & Avorn, J. (2004). Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: Direct comparison of randomised and observational evaluations. British Medical Journal, 328 (7439), 560.</bibtext> </blist> <blist> <bibtext> Sibieta, L., Greaves, E., & Sianesi, B. (2016). Increasing pupil motivation: Evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> Sims, S., Anders, J., Inglis, M., & Lortie-Forgue, H. (2021). Quantifying 'promising trials bias' in randomized controlled trials in education. CEPEO Working Paper Series, 20–16. UCL.</bibtext> </blist> <blist> <bibtext> Steiner, P. M., & Wong, V. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42 (2), 214 – 247.</bibtext> </blist> <blist> <bibtext> Strunk, K. O., Marsh, J. A., Hashim, A. K., & Bush-Mecenas, S. (2016). Innovation and a return to the status quo: A mixed-methods study of school reconstitution. Educational Evaluation and Policy Analysis, 38 (3), 549 – 577. https://doi.org/10.3102/0162373716642517</bibtext> </blist> <blist> <bibtext> Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the US institute of education sciences. International Journal of Research and Method in Education, 39 (3), 255 – 267. https://doi.org/10.1080/1743727X.2016.1150454</bibtext> </blist> <blist> <bibtext> St. Clair, T., & Cook, T. D. (2015). Difference-in-differences methods in public finance. National Tax Journal, 68 (2), 319 – 338. https://doi.org/10.17310/ntj.2015.2.04</bibtext> </blist> <blist> <bibtext> St. Clair, T., Hallberg, K., & Cook, T. D. (2016). The validity and precision of the comparative interrupted time-series design: Three within-study comparisons. Journal of Educational and Behavioral Statistics, 41 (3), 269 – 299. https://doi.org/10.3102/1076998616636854</bibtext> </blist> <blist> <bibtext> Weidmann, B., & Miratrix, L. (2021). Lurking inferential monsters: Quantifying selection bias in non-experimental evaluations of school programs. Journal of Policy Analysis and Management, 40 (3), 964 – 986. https://doi.org/10.1002/pam.22236</bibtext> </blist> <blist> <bibtext> What Works Clearinghouse. (2017). Standards handbook version 4.1. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf</bibtext> </blist> <blist> <bibtext> Wiggins, M., Jerrim, J., Tripney, J., Khatwa, M., & Gough, D. (2019). The RISE project: Evidence-informed school improvement. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> Wiggins, M., Sawtell, M., & Jerrim, J. (2017). Learner response system: Evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> Wong, M., Cook, T. D., & Steiner, P. M. (2015). Adding design elements to improve time series designs: No child left behind as an example of causal pattern-matching. Journal of Research on Educational Effectiveness, 8 (2), 245 – 279. https://doi.org/10.1080/19345747.2013.878011</bibtext> </blist> <blist> <bibtext> Wong, V. C., Steiner, P. M., & Anglin, K. L. (2018). What can be learned from empirical evaluations of nonexperimental methods? Evaluation Review, 42 (2), 147 – 175.</bibtext> </blist> <blist> <bibtext> Wong, V. C., Valentine, J. C., & Miller-Bains, K. (2017). Empirical performance of covariates in education observational studies. Journal of Research on Educational Effectiveness, 10 (1), 207 – 236. https://doi.org/10.1080/19345747.2016.1164781</bibtext> </blist> <blist> <bibtext> Worth, J., Sizmur, J., Walker, J., Bradshaw, S., & Styles, B. (2017). Teacher observation. Evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> </ref> <ref id="AN0159448677-19"> <title> Footnotes </title> <blist> <bibtext> EEF originally sent us 21 interventions but two of these (Affordable Maths Tuition and Engage in Education) did not use school-level outcomes.</bibtext> </blist> <blist> <bibtext> Changing Mindsets (regrant), Grammar for Writing (regrant), Lesson Study, Philosophy for Children, Research Learning Communities, Scratch Maths, Affordable Tuition (regrant) and Embedding Formative Assessment.</bibtext> </blist> <blist> <bibtext> Philosophy for Children; Thinking, Doing, Talking Science; Chess in Primary Schools.</bibtext> </blist> <blist> <bibtext> Children's University, Pupil Motivation Financial, Pupil Motivation Non-financial. The first of these did not report any effect size measure. The latter two did report an effect size but the mean difference was divided by the population-level standard deviation, which not included in the report.</bibtext> </blist> <blist> <bibtext> Publicly accessible school-level data can be found here: https://<ulink href="http://www.compare-school-performance.service.gov.uk/download-data">www.compare-school-performance.service.gov.uk/download-data</ulink></bibtext> </blist> <blist> <bibtext> State-funded schools in England operate under a range of legal frameworks (see Eyles & Machin, [20]). We code the school type variables as having five categories: academy (including free schools and city technology colleges), community, foundation, voluntary aided or voluntary controlled.</bibtext> </blist> <blist> <bibtext> We utilise just one period of post-treatment data in order to minimise the number of time series straddling a change in the way that outcomes are recorded and because previous empirical work suggest that adding more post-treatment periods can increase bias (Hallberg et al., [36]).</bibtext> </blist> <blist> <bibtext> There are nine regions in England, each containing between three million and ten million people.</bibtext> </blist> </ref> <aug> <p>By Sam Sims; Jake Anders and Laura Zieger</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib11" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib49" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib17" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib25" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib31" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib55" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib26" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib23" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib37" firstref="ref12"></nolink> <nolink nlid="nl10" bibid="bib47" firstref="ref13"></nolink> <nolink nlid="nl11" bibid="bib27" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib30" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib56" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib32" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib48" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib33" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib21" firstref="ref21"></nolink> <nolink nlid="nl18" bibid="bib22" firstref="ref22"></nolink> <nolink nlid="nl19" bibid="bib43" firstref="ref23"></nolink> <nolink nlid="nl20" bibid="bib38" firstref="ref25"></nolink> <nolink nlid="nl21" bibid="bib13" firstref="ref26"></nolink> <nolink nlid="nl22" bibid="bib50" firstref="ref27"></nolink> <nolink nlid="nl23" bibid="bib28" firstref="ref28"></nolink> <nolink nlid="nl24" bibid="bib36" firstref="ref29"></nolink> <nolink nlid="nl25" bibid="bib15" firstref="ref30"></nolink> <nolink nlid="nl26" bibid="bib52" firstref="ref40"></nolink> <nolink nlid="nl27" bibid="bib35" firstref="ref44"></nolink> <nolink nlid="nl28" bibid="bib39" firstref="ref45"></nolink> <nolink nlid="nl29" bibid="bib14" firstref="ref48"></nolink> <nolink nlid="nl30" bibid="bib12" firstref="ref56"></nolink> <nolink nlid="nl31" bibid="bib34" firstref="ref57"></nolink> <nolink nlid="nl32" bibid="bib57" firstref="ref58"></nolink> <nolink nlid="nl33" bibid="bib41" firstref="ref61"></nolink> <nolink nlid="nl34" bibid="bib24" firstref="ref62"></nolink> <nolink nlid="nl35" bibid="bib46" firstref="ref63"></nolink> <nolink nlid="nl36" bibid="bib29" firstref="ref66"></nolink> <nolink nlid="nl37" bibid="bib16" firstref="ref67"></nolink> <nolink nlid="nl38" bibid="bib10" firstref="ref69"></nolink> <nolink nlid="nl39" bibid="bib51" firstref="ref70"></nolink> <nolink nlid="nl40" bibid="bib19" firstref="ref73"></nolink> <nolink nlid="nl41" bibid="bib45" firstref="ref88"></nolink> <nolink nlid="nl42" bibid="bib18" firstref="ref89"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1367853
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence from Four New Within-Study Comparisons
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sims%2C+Sam%22">Sims, Sam</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-5585-8202">0000-0002-5585-8202</externalLink>)<br /><searchLink fieldCode="AR" term="%22Anders%2C+Jake%22">Anders, Jake</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-0930-2884">0000-0003-0930-2884</externalLink>)<br /><searchLink fieldCode="AR" term="%22Zieger%2C+Laura%22">Zieger, Laura</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+on+Educational+Effectiveness%22"><i>Journal of Research on Educational Effectiveness</i></searchLink>. 2022 15(4):876-897.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Characteristics%22">Institutional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Databases%22">Databases</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Records%22">Student Records</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22United+Kingdom+%28England%29%22">United Kingdom (England)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2022.2051652
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1934-5747<br />1934-5739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1367853
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1367853
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2022.2051652
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 876
    Subjects:
      – SubjectFull: Validity
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Institutional Characteristics
        Type: general
      – SubjectFull: Outcomes of Education
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Databases
        Type: general
      – SubjectFull: Student Records
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: United Kingdom (England)
        Type: general
    Titles:
      – TitleFull: The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence from Four New Within-Study Comparisons
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sims, Sam
      – PersonEntity:
          Name:
            NameFull: Anders, Jake
      – PersonEntity:
          Name:
            NameFull: Zieger, Laura
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 1934-5747
            – Type: issn-electronic
              Value: 1934-5739
          Numbering:
            – Type: volume
              Value: 15
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1