View in EDS HTML Full Text PDF Full Text

Education Experiments in Latin America: Empirical Evidence to Guide Evaluation Design

Saved in:
Bibliographic Details
Title: Education Experiments in Latin America: Empirical Evidence to Guide Evaluation Design
Language: English
Authors: Steven Glazerman, Larissa Campuzano (ORCID 0009-0003-0386-9959), Nancy Murray
Source: Evaluation Review. 2025 49(1):115-146.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 32
Publication Date: 2025
Sponsoring Agency: Agency for International Development (IDCA)
Contract Number: AIDOAAM1200020
Document Type: Journal Articles
Reports - Research
Descriptors: Educational Experiments, Foreign Countries, Educational Assessment, Research Design, Educational Research, Multivariate Analysis, Randomized Controlled Trials, Sample Size, Surveys, Correlation, Data Collection, Equations (Mathematics), Reading Tests, Reading Comprehension, Reading Fluency
Geographic Terms: Latin America, Nicaragua, Honduras, Guatemala, Peru
DOI: 10.1177/0193841X241241354
ISSN: 0193-841X
1552-3926
Abstract: Randomized experiments involving education interventions are typically implemented as cluster randomized trials, with schools serving as clusters. To design such a study, it is critical to understand the degree to which learning outcomes vary between versus within clusters (schools), specifically the intraclass correlation coefficient. It is also helpful to anticipate the benefits, in terms of statistical power, of collecting household data, testing students at baseline, or relying on administrative data on previous cohorts from the same school. We use data from multiple cluster-randomized trials in four Latin American countries to provide information on the intraclass correlations in early grade literacy outcomes. We also describe the proportion of variance explained by different types of covariates. These parameters will help future researchers conduct statistical power analysis, estimate the required sample size, and determine the necessity of collecting different types of baseline data such as child assessments, administrative data at the school level, or household surveys.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1454813
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFOHTsi-sfjhnsDh2gNCHxvAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDAxMSW_jmKE-GC1xigIBEICBmjsILBEfU0eQC4Dl1PRVTDrEbaZrBnMsSoFHQbSqY-PK0okduncjylbpB5Nsn3JJl1jDXBrFI7cbOkXPO3-j6n4yOk195oPVkOekH_XPm0yc3r7d7sahOG4srIkHbT_NrG8x8jddi0VnPovF3G2Voa6PMkNlhOViQyjFmv9THzV40Zp7VOnvMHVWC9-ZZnrnC1H4XWAWkMsW2Uc=
Text:
  Availability: 1
  Value: <anid>AN0181774545;evr01feb.25;2024Dec23.04:14;v2.2.500</anid> <title id="AN0181774545-1">Education Experiments in Latin America: Empirical Evidence to Guide Evaluation Design </title> <p>Randomized experiments involving education interventions are typically implemented as cluster randomized trials, with schools serving as clusters. To design such a study, it is critical to understand the degree to which learning outcomes vary between versus within clusters (schools), specifically the intraclass correlation coefficient. It is also helpful to anticipate the benefits, in terms of statistical power, of collecting household data, testing students at baseline, or relying on administrative data on previous cohorts from the same school. We use data from multiple cluster-randomized trials in four Latin American countries to provide information on the intraclass correlations in early grade literacy outcomes. We also describe the proportion of variance explained by different types of covariates. These parameters will help future researchers conduct statistical power analysis, estimate the required sample size, and determine the necessity of collecting different types of baseline data such as child assessments, administrative data at the school level, or household surveys.</p> <p>Keywords: statistical power; intraclass correlation; experimental design; education evaluation; C8; C9; I2</p> <p>Many educational interventions need to be implemented at the school level; for example, professional development programs are often offered to all teachers in a school to ensure that the teachers' instruction is similar. In these cases, experimental evaluations conduct random assignment at the school level. Because groups of students clustered in schools are the unit of assignment, this design is referred to as a cluster-randomized control trial (RCT). Randomizing schools guarantees that the offer of the intervention is independent of observed and unobserved school and student characteristics, but because students, teachers, and other resources are not randomly allocated into schools, using the school as the unit of assignment creates a new problem: statistical imprecision. Generating precise estimates of model parameters requires a large number of clusters.</p> <p>Recent studies have focused on how to design cluster-randomized studies that provide adequate precision ([<reflink idref="bib28" id="ref1">28</reflink>]; [<reflink idref="bib30" id="ref2">30</reflink>]). Researchers use statistical power calculations ([<reflink idref="bib6" id="ref3">6</reflink>]) to estimate optimal sample sizes and thus can design an efficient experiment ([<reflink idref="bib10" id="ref4">10</reflink>]; [<reflink idref="bib24" id="ref5">24</reflink>]). Power calculations for cluster-randomized trials need information on the degree to which outcomes vary between schools versus within schools. The statistic used to summarize this is the intraclass correlation (ICC). Previous research has provided information on the values of ICCs for education in the United States ([<reflink idref="bib10" id="ref6">10</reflink>], [<reflink idref="bib8" id="ref7">8</reflink>]; [<reflink idref="bib24" id="ref8">24</reflink>]; [<reflink idref="bib26" id="ref9">26</reflink>]), in Germany ([<reflink idref="bib29" id="ref10">29</reflink>]), in sub-Saharan Africa ([<reflink idref="bib13" id="ref11">13</reflink>]; [<reflink idref="bib25" id="ref12">25</reflink>]), and in a sample of 81 countries ([<reflink idref="bib4" id="ref13">4</reflink>]). But this information is rarely available for other regions, so researchers often make assumptions that turn out to be incorrect. In this paper, we provide information on the possible values of ICCs for education interventions based on studies conducted in Latin America.</p> <p>Researchers often collect baseline data or time-invariant background data to use as covariates in their impact estimation models to increase the precision of the study. Including covariates in the statistical model to estimate the impact reduces the error variance and increases the precision of the parameter estimates ([<reflink idref="bib21" id="ref14">21</reflink>]). Deciding which covariates to include in the impact models should account for both precision gains and the cost of measuring (collecting data on) those covariates.</p> <p>Evaluations of education interventions often include baseline assessments as covariates in the impact estimation models to increase the statistical precision. However, collecting these assessments can be costly. For example, child assessments may have to be administered individually. Even group-administered cognitive tests require trained proctors and take valuable time away from instruction. Household surveys may provide information on parents' education or socio-economic information that could help increase the precision of the study. But collecting household information requires extra travel to communities in addition to school visits, and may require respondent payments to ensure sufficiently high response rates. Other forms of data may be less costly to collect, such as administrative data on student or school performance, but such data may not be available at the right level of disaggregation or may require considerable effort to obtain and clean for research purposes.</p> <p>Empirical researchers need detailed information at the planning stage to forecast the degree to which a given type of baseline data will increase the precision of model parameters and to assess the cost-variance tradeoff of each type of data collection. The summary statistic that captures this information is the <emph>R</emph><sups>2</sups> from a regression of the outcome of interest on covariates. Previous research has provided information on the precision gains of including different sets of covariates in the U.S. context ([<reflink idref="bib3" id="ref15">3</reflink>]; [<reflink idref="bib10" id="ref16">10</reflink>]; [<reflink idref="bib12" id="ref17">12</reflink>]; [<reflink idref="bib14" id="ref18">14</reflink>]; [<reflink idref="bib21" id="ref19">21</reflink>]). This paper provides information on values of <emph>R</emph><sups>2</sups> based on evaluations conducted in Honduras, Guatemala, Nicaragua, and Peru.</p> <p>The goal of obtaining these parameters and conducting power analysis is to plan the optimal study design that allocates sample most efficiently between number of clusters (schools) and cluster size (number of students per school) as well as the total sample size. The researcher seeks to minimize costs subject to a constraint of being able to detect a given impact with some acceptable level of statistical power ([<reflink idref="bib20" id="ref20">20</reflink>]; [<reflink idref="bib27" id="ref21">27</reflink>]). Once we set the power level (typically 80 or 90%) and minimum detectable effect size (MDES), and we input the marginal costs of collecting data per student and per school, we can compute the optimal sample size in terms of both students and schools.</p> <p>The ICC and <emph>R</emph><sups>2</sups> are critical inputs to this exercise, but such variance parameters might depend on the context. It helps to think of why the ICC or covariate model <emph>R</emph><sups>2</sups> might be high or low. For example, in some rural areas, schools might be located in ethnic communities of different linguistic and cultural features that influence literacy in the national language. Such locations also might have different pay or working conditions that influence their ability to attract good school teachers or principals. In these cases, variation between schools may be high and so the ICC will be high, which makes it more challenging to conduct an efficient cluster-randomized trial, but a high <emph>R</emph><sups>2</sups> (or stratification of schools on dominant ethnicity or language) might mitigate the problem.</p> <p>Alternatively, an urban setting might have many schools that draw from a similar population of students and prospective teachers and have centrally determined policies like class size and budgets, resulting in low between-school variance, and hence a low ICC. This type of study setting would permit the researcher to conduct the study with fewer schools and therefore lower costs. Whether the study sample is national or regional can also change the ICCs. [<reflink idref="bib8" id="ref22">8</reflink>] compared ICCs for math and reading outcomes from national U.S. samples to samples by state and found that, for elementary grades, states have lower ICCs than national samples.</p> <p>In this paper, we raise the issue of data collection costs to point out that some covariates are more costly to collect than others, so their contribution to statistical precision should be weighed against those costs. However, the discussion is illustrative, and it does not attempt to obtain an optimal sample that minimizes cost and attains the largest statistical power. A framework which can be used for optimal allocation is presented in [<reflink idref="bib27" id="ref23">27</reflink>]; [<reflink idref="bib26" id="ref24">26</reflink>] present an example from an optimal design perspective.</p> <p>Because the required parameters for planning a cluster-RCT may depend on the context, it is important for social scientists to share findings from as many outcomes and study contexts as possible for the benefit of future researchers. Several researchers have documented estimates of these key parameters for education interventions. Most of these published parameters are based on data from the United States ([<reflink idref="bib7" id="ref25">7</reflink>]; [<reflink idref="bib10" id="ref26">10</reflink>], [<reflink idref="bib8" id="ref27">8</reflink>]; [<reflink idref="bib11" id="ref28">11</reflink>]; [<reflink idref="bib24" id="ref29">24</reflink>]). [<reflink idref="bib13" id="ref30">13</reflink>] reported ICC and multilevel <emph>R</emph><sups>2</sups> results from RCTs in 15 sub-Saharan African countries, [<reflink idref="bib29" id="ref31">29</reflink>] reported parameters for Germany, and [<reflink idref="bib4" id="ref32">4</reflink>] reported parameters for a sample of 81 countries. However, we are not aware of published estimates for Latin America. We contribute to the literature by providing estimates of key parameters used for power calculations for literacy outcomes such as fluency and reading comprehension, based on data from four studies which conducted cluster-RCTs in six sites in four countries in Latin America: Guatemala, Honduras, Nicaragua, and Peru.</p> <p>The RCTs used in this study were funded by the United States Agency for International Development (USAID) as part of the Latin America and the Caribbean (LAC) reads initiative (LAC Reads), whose goal was to improve early grade reading in the region. The four studies, described in more detail in Appendix B, tested different promising reading interventions in various contexts. The study populations varied, but all focused on children in early grades (1 through 3), and the duration of the reading interventions, described below, ranged from 12 to 36 months. The outcomes were typically reading comprehension and associated word-level skills such as reading vocabulary, fluency, and other basic literacy skills (such as phonemic awareness and decoding).</p> <p>This paper is organized as follows. The next section describes the statistical model and parameters that drive precision and sample size requirements. We then discus the data used for this study and the context in which those data were collected. The last two sections present results and discus implications for study design and concluding remarks.</p> <hd id="AN0181774545-2">Impact Estimation Model</hd> <p>The RCT design allows a straightforward estimation of average treatment effects. The following model captures the two levels of aggregation for a cluster-randomized trial for an outcome <emph>y</emph><subs><emph>ij</emph></subs> that varies at the student (<emph>i</emph>) and school (<emph>j</emph>) levels: <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>y</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mi>μ</mi><mo>+</mo><msub><mrow><msup><mi>λ</mi><mo>′</mo></msup><mi mathvariant="bold-italic">T</mi></mrow><mi>j</mi></msub><mo>+</mo><msub><mrow><msup><mi>α</mi><mo>′</mo></msup><mi mathvariant="bold-italic">x</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>+</mo><msub><mrow><msup><mi>θ</mi><mo>′</mo></msup><mi mathvariant="bold-italic">X</mi></mrow><mi>j</mi></msub><mo>+</mo><msub><mi>u</mi><mi>j</mi></msub><mo>+</mo><msub><mi>ε</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></math> </ephtml></p> <p>Graph</p> <p>Here <emph>T</emph><subs><emph>j</emph></subs> is a vector of randomly assigned treatment indicators, which are, by construction, independent of all variables and the error term. Two of the studies had two treatment arms in addition to the omitted group (pure control schools, which received no additional intervention). The others had a simple treatment-control design, so <emph>T</emph><subs><emph>j</emph></subs> reduces to a scalar. <bold><emph>x</emph></bold><subs><emph>ij</emph></subs> is a vector of student-level covariates such as baseline test scores, age, gender, or household characteristics, and <bold><emph>X</emph></bold><subs><bold><emph>j</emph></bold></subs> is a vector of school-level covariates such as whether the school has electricity, type of water supply, type of waste disposal, painted walls, separate latrines for boys and girls, and where relevant, school means of the student characteristics such as average test scores. <emph>ε</emph><subs><emph>ij</emph></subs> is the student error term for student <emph>i</emph> in school <emph>j</emph>, which is assumed to be normal and independent and identically distributed (iid), with mean zero and variance <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mrow><mi>W</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup><mo>.</mo></mrow></math> </ephtml><emph>u</emph><subs><emph>j</emph></subs> is the error term for school <emph>j</emph>, which is assumed to be normal iid, with mean zero and variance <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mrow><mi>B</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup><mo>.</mo></mrow></math> </ephtml> We have conducted all of our analyses without centering variables at the grand mean or cluster mean, so readers who use these results should consider this when specifying similar models in the design phase.</p> <hd id="AN0181774545-3">Variance Parameters</hd> <p>The parameters of interest for this paper are the ICC and the cluster- and individual-level <emph>R</emph><sups>2</sups> terms. The ICC, as noted above, is the proportion of the total variance between schools. <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>I</mi><mi>C</mi><mi>C</mi><mo>=</mo><mfrac><msubsup><mi>σ</mi><mi>B</mi><mn>2</mn></msubsup><msubsup><mrow><msubsup><mi>σ</mi><mi>B</mi><mn>2</mn></msubsup><mo>+</mo><mi>σ</mi></mrow><mi>W</mi><mn>2</mn></msubsup></mfrac></mrow></math> </ephtml></p> <p>Graph</p> <p>where <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>B</mi><mn>2</mn></msubsup></mrow></math> </ephtml> is the between-school variance and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>W</mi><mn>2</mn></msubsup></mrow></math> </ephtml> is the within-school variance. When the between- and within-school variances are obtained from a model described in equation (<reflink idref="bib1" id="ref33">1</reflink>) without covariates, we refer to the ICC described in equation (<reflink idref="bib2" id="ref34">2</reflink>) as unconditional. The ICC could also be calculated with between-school and within-school variances obtained from a model described in equation (<reflink idref="bib1" id="ref35">1</reflink>) with covariates ( <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mrow><mi>B</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mrow><mi>W</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> )—in this case we refer to the ICC as conditional.</p> <p>The standard error of the ICC can be estimated following follow [<reflink idref="bib13" id="ref36">13</reflink>]: <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>S</mi><mi>E</mi><mrow><mo>(</mo><mover accent="true"><mrow><mi>I</mi><mi>C</mi><mi>C</mi></mrow><mo>^</mo></mover><mo>)</mo></mrow><mo>=</mo><msqrt><mfrac><mrow><mn>2</mn><msup><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><mover accent="true"><mrow><mi>I</mi><mi>C</mi><mi>C</mi></mrow><mo>^</mo></mover></mrow><mo>)</mo></mrow><mn>2</mn></msup><msup><mrow><mo>(</mo><mrow><mn>1</mn><mo>+</mo><mrow><mo>(</mo><mrow><mi>N</mi><mo>−</mo><mn>1</mn></mrow><mo>)</mo></mrow><mover accent="true"><mrow><mi>I</mi><mi>C</mi><mi>C</mi></mrow><mo>^</mo></mover></mrow><mo>)</mo></mrow><mn>2</mn></msup></mrow><mrow><mi>N</mi><mo /><mrow><mo>(</mo><mrow><mi>N</mi><mo>−</mo><mn>1</mn></mrow><mo>)</mo></mrow><mo /><mi>J</mi></mrow></mfrac></msqrt></mrow></math> </ephtml></p> <p>Graph</p> <p>where N is the total number of students in the sample and J is the total number of schools.</p> <p>The <emph>R</emph><sups>2</sups> terms at the cluster and individual levels can be calculated as follows: <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtable columnalign="left"><mtr><mtd><mrow><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mo>=</mo><mfrac><mrow><msubsup><mi>σ</mi><mi>B</mi><mn>2</mn></msubsup><mo>−</mo><msubsup><mi>σ</mi><mrow><mi>B</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup></mrow><msubsup><mi>σ</mi><mi>B</mi><mn>2</mn></msubsup></mfrac></mrow></mtd></mtr><mtr><mtd><mrow><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow><mn>2</mn></msubsup><mo>=</mo><mfrac><mrow><msubsup><mi>σ</mi><mi>W</mi><mn>2</mn></msubsup><mo>−</mo><msubsup><mi>σ</mi><mrow><mi>W</mi><mo>_</mo><mi>a</mi><mi>d</mi><mi>j</mi></mrow><mn>2</mn></msubsup></mrow><msubsup><mi>σ</mi><mi>W</mi><mn>2</mn></msubsup></mfrac></mrow></mtd></mtr></mtable></mrow></math> </ephtml></p> <p>Graph</p> <p>where <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> is the proportion of variance at the school level explained by the covariates, and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> is the proportion of variance at the student level explained by the covariates. These quantities will vary with the explanatory power of the covariates included in the model as well as the context. Therefore, in this study we present results for different sets of covariates (different models) and for different contexts (targeted regions in four countries).</p> <p>To assess a study's ability to detect policy-relevant effects, researchers calculate the minimum detectable impact (MDI), which is the smallest true impact for which there is a high probability of detection. The smaller the MDI, the greater the study's ability to detect an impact of policy-relevant magnitude. Both the ICC and <emph>R</emph><sups>2</sups> affect the MDI of a cluster-randomized trial. Typically, researchers fix the calculation at 80% power (probability of rejecting the null, given that the null is false) and a two-tailed test with 5% significance level. To facilitate comparison across studies, researchers often report results in terms of effect sizes—impact divided by the standard deviation of the outcome. We can calculate minimum detectable <emph>effect size</emph> (MDES) by dividing the MDI by the standard deviation of the outcome. As [<reflink idref="bib2" id="ref37">2</reflink>] has shown, the MDES can be written as follows: <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>M</mi><mi>D</mi><mi>E</mi><mi>S</mi><mo>=</mo><msub><mi>M</mi><mrow><mi>J</mi><mo>−</mo><mi>g</mi><mo>−</mo><mn>2</mn></mrow></msub><msqrt><mrow><mfrac><mrow><mi>I</mi><mi>C</mi><mi>C</mi><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow><mn>2</mn></msubsup></mrow><mo>)</mo></mrow></mrow><mrow><mi>P</mi><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><mi>P</mi></mrow><mo>)</mo></mrow><mi>J</mi></mrow></mfrac><mo>+</mo><mfrac><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><mi>I</mi><mi>C</mi><mi>C</mi><mrow><mo>)</mo></mrow><mrow><mo>(</mo></mrow><mn>1</mn><mo>−</mo><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow><mn>2</mn></msubsup></mrow><mo>)</mo></mrow><mrow><mi>P</mi><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><mi>P</mi></mrow><mo>)</mo></mrow><mi>n</mi><mi>J</mi></mrow></mfrac></mrow></msqrt></mrow></math> </ephtml></p> <p>Graph</p> <p>where <emph>P</emph> is the proportion of groups assigned to the treatment group, <emph>J</emph> is the number of groups randomly assigned, <emph>n</emph> is the average number of individuals in a group, <emph>g</emph> is the number of group-level covariates included, and <emph>M</emph><subs><emph>J</emph>-<emph>g</emph>-2</subs> is a factor that depends on the statistical power level, the level of confidence of the test, and the degrees of freedom given by <emph>J-g-2.</emph> The other parameters have been defined above.</p> <p>As is evident from equation (<reflink idref="bib5" id="ref38">5</reflink>), for a given experimental design, a higher ICC increases the MDES and therefore makes it harder to detect impacts. A larger <emph>R</emph><sups>2</sups> at either level decreases the MDES, and therefore makes it easier to detect impacts. As the ICC gets larger, the second term under the radical becomes less important to the standard error (and the MDES), so the role of cluster size and the explanatory power of individual-level covariates become relatively less important for improving the design. Abstracting away from the role of covariates, the design effect—that is, the penalty associated with the clustering relative to an individual RCT—is proportional to <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msqrt><mrow><mtext>ICC</mtext><mo>+</mo><mfrac><mrow><mo>(</mo><mrow><mn>1</mn><mo>−</mo><mtext>ICC</mtext></mrow><mo>)</mo></mrow><mi>n</mi></mfrac></mrow></msqrt></mrow></math> </ephtml> , which makes apparent the critical role of the ICC in planning clustered RCT study designs. Despite the central role of the ICC, the stakes for data collection, including potential costs, are high enough that the explanatory power of covariates deserves considerable attention also. While some other research has focused on selecting sample sizes to optimize cost given different costs of data collection at the group or individual level [<reflink idref="bib15" id="ref39">15</reflink>], this paper focuses on the tradeoff between statistical power gains and the cost of collecting different types of data, for example, data that can be collected in schools versus data that require household visits.</p> <hd id="AN0181774545-4">Models Tested</hd> <p>We used data from cluster-RCTs conducted in Latin America to estimate key parameters for a cluster-level design. We estimated ICCs for cluster-RCTs conducted in six sites in four countries. We also estimated <emph>R</emph><sups>2</sups> for different sets of covariates. Calculating <emph>R</emph><sups>2</sups>s for different sets of covariates allows us to assess how much the MDES would change if we collected different types of data.</p> <p>The models simulate various combinations of having or not having a pretest at baseline, a baseline household survey, a school observation, or school administrative data. We take advantage of the fact that the studies collected different combinations of data. For example, two of the studies collected household data because the study focus included community engagement. Other studies did not collect household data but collected school-level achievement for more than one year before the intervention started. These different data sets allow us to assess how much outcome variance would be explained by different combinations of covariates.</p> <p>We first estimated the model with only the treatment indicator as a covariate. This is what we refer to as the unadjusted model and used to obtain estimates for the ICC. We then added different sets of student-level variables and/or school-level variables to assess how much variance was explained by each set of covariates, and we obtained the other two key parameters, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> . Table 1 summarizes the covariates included in each model. For example, models 3 and 5 require a baseline household survey. Models 7, 8, and 9 add school administrative data. In the case of models 8 and 9, we are testing whether such administrative data can substitute for a baseline child assessment, which could save considerable resources if such a strategy yielded adequate statistical precision. We assume that student age and gender are always known. We made this assumption because all the studies discussed collected student age and gender. However, these two variables make a small contribution to explaining the variance, so the results for models that do not include these two covariates would not differ much from the results presented here.</p> <p>Table 1. Covariates Included in Each Model.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">Characteristic</th><th align="center">Model 1</th><th align="center">Model 2</th><th align="center">Model 3</th><th align="center">Model 4</th><th align="center">Model 5</th><th align="center">Model 6</th><th align="center">Model 7</th><th align="center">Model 8</th><th align="center">Model 9</th></tr></thead><tbody valign="top"><tr><td align="left">Treatment indicator(s)</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Student pretest</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left">Student age, gender</td><td align="left" /><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Student household characteristics</td><td align="left" /><td align="left" /><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left">School means of pretest from previous cohort</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">School infrastructure</td><td align="left" /><td align="left" /><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="center">X</td></tr></tbody></table> </ephtml> </p> <hd id="AN0181774545-5">Data</hd> <p>The data used in this study were obtained from studies conducted as part of the LAC Reads initiative. Below we describe the data and sample for each study, but it is worth noting what they had in common:</p> <p></p> <ulist> <item> • They were all clustered-RCTs with schools or communities as the unit of randomization. Outcomes were measured at the student level.</item> <p></p> <item> • They focused on early grade reading as the outcome, so the population is students in grades 1 to 3.</item> <p></p> <item> • They are school-level interventions. The interventions differed but all aimed to improve reading.</item> <p></p> <item> • They were implemented roughly from 2014 to 2017, with intervention durations of 12–36 months.</item> <p></p> <item> • The evaluations were funded by USAID, with evaluation design and implementation conducted by Mathematica in partnership with local research and data collection partners.</item> </ulist> <hd id="AN0181774545-6">Interventions and Contexts</hd> <p>Each of the studies examined a different intervention and had slightly different characteristics, as shown in Table 2. The interventions included teacher supports (training, coaching, and mentoring), curriculum materials, afterschool (or before-school) enrichment programs, and community-level supports for literacy-promoting activities such as mobile book banks, reading fairs, and classes for parents. The interventions were implemented in diverse contexts, including urban, rural, highlands, and coastal/island communities. They were also linguistically diverse, including predominantly Spanish-speaking communities as well as those where indigenous languages were commonly spoken at home. Appendix B provides more detailed information on the interventions and contexts.</p> <p>Table 2. Description of the Six Sites in Which Cluster-RCTs Were Conducted.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="center" rowspan="2">Characteristic</th><th align="center" colspan="2">Amazon Reads</th><th align="center" colspan="2"><italic>Leer Juntos</italic></th><th align="center">PRI</th><th align="center"><italic>Espacios para Crecer</italic></th></tr><tr><th align="center">Peru site 1</th><th align="center">Peru site 2</th><th align="center">Guatemala</th><th align="center">Peru</th><th align="center">Honduras</th><th align="center">Nicaragua</th></tr></thead><tbody valign="top"><tr><td align="left">Intervention name</td><td align="left" colspan="2"><italic>Amazonía Lee</italic></td><td align="left" colspan="2"><italic>Leer Juntos, Aprender Juntos</italic></td><td align="left"><italic>EducAcción</italic> promising reading intervention</td><td align="left"><italic>Espacios para Crecer</italic></td></tr><tr><td align="left">Intervention description</td><td align="left" colspan="2">Teacher training, coaching, and curriculum materials</td><td align="left" colspan="2">Teacher training, coaching, community reading activities</td><td align="left">Student assessment data support</td><td align="left">Afterschool enrichment</td></tr><tr><td align="left">Geography</td><td align="left" colspan="2">Rural and peri-urban, Amazon rain forest region</td><td align="left" colspan="2">Rural, mountainous region</td><td align="left">Large city, small city, small town, and rural</td><td align="left">Urban and rural coastal and island</td></tr><tr><td align="left">Grade of cohort at endline</td><td align="center" colspan="2">2nd</td><td align="center" colspan="2">3rd</td><td align="center">3rd</td><td align="center">1st to 5th</td></tr><tr><td align="left">Language of instruction</td><td align="left">Spanish</td><td align="left">Spanish</td><td align="left">Mixed Spanish-K'iche'</td><td align="left">Mixed Spanish Quechua</td><td align="center">Spanish</td><td align="left">Mostly Spanish, some Kriol, Miskitu, Ulwa</td></tr><tr><td align="left">Number of clusters (schools except for Nicaragua)</td><td align="center">200</td><td align="center">75</td><td align="center">150</td><td align="center">147</td><td align="center">180</td><td align="center">139</td></tr><tr><td align="left">Number of students in the data collection sample</td><td align="center">8</td><td align="center">10</td><td align="center">10</td><td align="center">10</td><td align="center">35</td><td align="center">8</td></tr><tr><td align="left">Source of data</td><td align="left" colspan="2"><xref ref-type="bibr" rid="bibr5">Campuzano et al. (2018)</xref></td><td align="left"><xref ref-type="bibr" rid="bibr18">Lugo Gil et al. (2019a)</xref></td><td align="left"><xref ref-type="bibr" rid="bibr19">Lugo Gil et al. (2019b)</xref></td><td align="left"><xref ref-type="bibr" rid="bibr17">Liuzzi et al. (2019)</xref></td><td align="left"><xref ref-type="bibr" rid="bibr1">Bagby et al. (2019)</xref></td></tr></tbody></table> </ephtml> </p> <p>1 PRI, Promising Reading Intervention; RCT, randomized controlled trial.</p> <hd id="AN0181774545-7">Outcomes</hd> <p>For three of the four studies, the outcomes were measured using the widely used Early Grade Reading Assessment (EGRA). The EGRA captured reading comprehension, phonemic awareness, familiar word reading, decoding, and oral reading fluency ([<reflink idref="bib23" id="ref40">23</reflink>]). Reading comprehension was measured as the percentage of questions answered correctly after reading a passage. Decoding accuracy was measured as the number of pseudo-words read correctly in 1 minute. Fluency was measured as the number of real words read correctly in 1 minute.</p> <p>In this paper, we focus on two outcomes: reading comprehension and reading fluency. For Nicaragua, the EGRA was adapted because of the age range of the children, to include test components from Dynamic Indicators of Reading Success, and to focus on only decoding and reading comprehension. The children in the Nicaragua study were also assessed on oral comprehension (for children who could not decode) and oral reading fluency.</p> <p>The three studies that used the EGRA administered it at both baseline and endline, although the baseline versions were modified to adapt to children's lower expected level of literacy skills before being exposed to any intervention (including more than a year of school itself in most cases). In the case of the <emph>Leer Juntos, Aprender Juntos</emph> study in Peru and Guatemala, a screener was used to determine which language to use in administering the baseline assessment, since it was translated into Quechua and K'iche', the predominant indigenous languages in those two study sites, respectively.</p> <p>The Honduras study did not administer the EGRA. Instead, the outcome was a nationally standardized Spanish language reading test developed to align with the National Basic Curriculum. It included an overall reading score and a subscore for reading comprehension. Baseline administrative records were not available at the student level in Honduras, so the study used school-level data. We present the Honduras results separately from the three studies that used the EGRA.</p> <hd id="AN0181774545-8">Covariates</hd> <p>Household surveys were administered for three of the study sites, in Guatemala, Peru, and Nicaragua, because those evaluations included at least one main outcome that required household data. Two of the studies, one in Peru and the Honduras study, did not include a household survey, so they are excluded from some of the <emph>R</emph><sups>2</sups> calculations below. The household characteristics in Nicaragua had almost no variation (98% of the households had the same household characteristics); therefore, we also excluded Nicaragua from those comparisons. Some of the variables collected from household surveys were whether the student attended preschool or kindergarten before first grade, the number of people in the household, parents' education, family income, and so on. Table A3 in Appendix A lists the household variables included in each model.</p> <p>School infrastructure data were collected from a principal interview, which recorded school characteristics in Honduras. The other studies included school visits with data collected on school infrastructure characteristics such as classroom materials, water supply, availability of working latrines, separate latrines for boys and girls, library, kitchen, outdoor space, computers, printers, music room, health center, and physical hazards at schools. We tried to include similar infrastructure characteristics in all the models, but not all the studies had the same infrastructure data. Table A2 in the Appendix presents a list of the infrastructure variables included in each model. Other school- or teacher-level data were not collected consistently across the projects; therefore, only infrastructure data were included as covariates. Furthermore, school infrastructure may be correlated with other school resources and/or other factors such as the capacity of the principal or the involvement of the community, which could affect student achievement. Hence, we assume school infrastructure serves as a proxy for school resources and other factors that could affect student achievement.</p> <hd id="AN0181774545-9">Results</hd> <p>Below we present results for ICCs and <emph>R</emph><sups>2</sups> at the student and school levels for reading comprehension and reading fluency.</p> <hd id="AN0181774545-10">ICC Results</hd> <p>The ICC analysis illustrates how homogeneous or heterogeneous the schools were in each study site. ICCs ranged from 0.04 to 0.28, depending on the setting and the outcome. In Table 3, we present the ICC for each setting for the two main outcomes of interest: reading fluency and reading comprehension. We also show the number of schools and students and the grade at which the outcome was measured in each study.</p> <p>Table 3. ICC and Standard Error (SE) for Fluency and Reading Comprehension, by Study Site.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="2" /><th align="center" colspan="4">Amazon Reads</th><th align="center" colspan="4">Leer Juntos</th><th align="center" colspan="2"><italic>PRI</italic></th><th align="center" colspan="2">Espacios para Crecer</th></tr><tr><th align="left" colspan="2">Peru site 1</th><th align="center" colspan="2">Peru site 2</th><th align="center" colspan="2">Guatemala</th><th align="center" colspan="2">Peru</th><th align="center" colspan="2">Honduras</th><th align="center" colspan="2">Nicaragua</th></tr></thead><tbody valign="top"><tr><td align="left" /><td align="left">ICC</td><td align="left">SE</td><td align="left">ICC</td><td align="left">SE</td><td align="left">ICC</td><td align="left">SE</td><td align="left">ICC</td><td align="left">SE</td><td align="left">ICC</td><td align="left">SE</td><td align="left">ICC</td><td align="left">SE</td></tr><tr><td align="left">Fluency</td><td align="left">0.15</td><td align="left">0.01</td><td align="left">0.25</td><td align="left">0.03</td><td align="left">0.28</td><td align="left">0.02</td><td align="left">0.18</td><td align="left">0.02</td><td align="left">n.a.</td><td align="left">n.a.</td><td align="left">0.06</td><td align="left">0.01</td></tr><tr><td align="left">Reading comprehension</td><td align="left">0.08</td><td align="left">0.01</td><td align="left">0.17</td><td align="left">0.02</td><td align="left">0.28</td><td align="left">0.02</td><td align="left">0.20</td><td align="left">0.02</td><td align="left">0.21</td><td align="left">0.02</td><td align="left">0.04</td><td align="left">0.00</td></tr><tr><td align="left">Number of schools<xref ref-type="table-fn" rid="tfn7">a</xref></td><td align="center" colspan="2">200</td><td align="center" colspan="2">70</td><td align="center" colspan="2">150</td><td align="center" colspan="2">147</td><td align="center" colspan="2">180</td><td align="center" colspan="2">139</td></tr><tr><td align="left">Number of students in the analysis sample</td><td align="center" colspan="2">1642</td><td align="center" colspan="2">740</td><td align="center" colspan="2">1338</td><td align="center" colspan="2">1022</td><td align="center" colspan="2">6375</td><td align="center" colspan="2">1106</td></tr><tr><td align="left">Grade</td><td align="center" colspan="2">2nd</td><td align="center" colspan="2">2nd</td><td align="center" colspan="2">3rd</td><td align="center" colspan="2">3rd</td><td align="center" colspan="2">3rd</td><td align="center" colspan="2">1st–5th</td></tr></tbody></table> </ephtml> </p> <ulist> <item>2 ICC, intraclass correlation; n.a, not available; SE, standard error.</item> <item>7 <sups>a</sups>In Nicaragua, clusters are communities not schools.</item> </ulist> <p>Most of the variance in these outcomes occurs within schools rather than between them. This is good news for planning cluster-randomized trials, suggesting that the number of clusters does not have to be as large as it would be if the ICC were higher. Nevertheless, there is considerable variation. Nicaragua had the lowest ICCs, possibly because the cluster was a community instead of a school and it included students from several grades. Two regions of Peru, the Leer Juntos site (Andahuaylas) and site 1 of the Peru Amazon Reads study (San Martín), had lower ICCs for reading fluency (0.18 and 0.15, respectively) compared to Guatemala and site 2 of Peru's Amazon Reads study (Department of Ucayali), where we found larger ICCs (0.28 and 0.25, respectively). The two sites with ICCs higher than 0.2 are in rural areas, with schools relatively far apart, and in the Guatemala study site different indigenous communities attend different schools.</p> <p>In this study, ICCs for reading ranged from 0.04 to 0.28. [<reflink idref="bib13" id="ref41">13</reflink>] found ICCs between 0.08 and 0.60 for sub-Saharan Africa using national samples in grade 6. One explanation of why ICCs are lower in this study is that we used local samples instead of national samples and previous studies have found that local ICCs are lower than national ICCs ([<reflink idref="bib8" id="ref42">8</reflink>]). This study's estimates are similar to findings from [<reflink idref="bib8" id="ref43">8</reflink>], where ICCs ranged from 0.06 to 0.22 for grade 3 state level samples. Our estimates are also similar to the ranges found in [<reflink idref="bib24" id="ref44">24</reflink>].</p> <hd id="AN0181774545-11">R 2 Results</hd> <p>As discussed above, including covariates in the regression model can reduce the residual variance of the outcome and increase statistical precision. Covariates with stronger linear association with the outcome will be better at increasing statistical precision. [<reflink idref="bib3" id="ref45">3</reflink>] and [<reflink idref="bib10" id="ref46">10</reflink>] provided information on the precision gains of including covariates such as pretests or demographic information in the U.S. context, and [<reflink idref="bib29" id="ref47">29</reflink>] did so for the German context. In this section, we present the estimates of the amount of variance explained at both the group (school) and individual (student) level for different sets of covariates, according to the models presented above for the Latin American contexts reflected in our studies. Table 4 presents the estimated <emph>R</emph><sups>2</sups> at the group level and the individual level, using reading fluency as the outcome for the different models and in each region.[<reflink idref="bib3" id="ref48">3</reflink>]</p> <p>Table 4. R 2 at School Level, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> , and Student Level, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> , for Reading Fluency, by Study Site.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="2" /><th align="center" colspan="3" rowspan="2">Student-level covariates</th><th align="center" colspan="2" rowspan="2">School-level covariates</th><th align="center" colspan="4">Amazon Reads</th><th align="center" colspan="4"><italic>Leer Juntos</italic></th><th align="center" colspan="2"><italic>Espacios para Crecer (EPC)</italic></th></tr><tr><th align="left" colspan="2">Peru site 1</th><th align="center" colspan="2">Peru site 2</th><th align="center" colspan="2">Guatemala</th><th align="center" colspan="2">Peru</th><th align="center" colspan="2">Nicaragua</th></tr><tr><td align="left">Model</td><td align="center">Pretest</td><td align="center">Age and sex</td><td align="center">Household characteristics</td><td align="center">Previous cohort pretest</td><td align="center">Infrastructure</td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td><td align="center"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow><mn mathvariant="bold">2</mn></msubsup></mrow></math></p></td></tr></thead><tbody valign="top"><tr><td align="center">2</td><td align="center">X</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">0.40</td><td align="center">0.37</td><td align="center">0.76</td><td align="center">0.39</td><td align="center">0.56</td><td align="center">0.29</td><td align="center">0.37</td><td align="center">0.32</td><td align="center">0.64</td><td align="center">0.61</td></tr><tr><td align="center">3</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">0.59</td><td align="center">0.33</td><td align="center">0.40</td><td align="center">0.35</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">4</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="center">X</td><td align="left">0.66</td><td align="left">0.37</td><td align="left">0.93</td><td align="left">0.39</td><td align="left">0.67</td><td align="left">0.32</td><td align="left">0.46</td><td align="left">0.32</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">5</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="left">0.69</td><td align="left">0.33</td><td align="left">0.50</td><td align="left">0.35</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">6</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="left">0.72</td><td align="left">0.37</td><td align="left">0.95</td><td align="left">0.39</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">7</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="left">0.51</td><td align="left">0.37</td><td align="left">0.78</td><td align="left">0.39</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">8</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="left">0.36</td><td align="left">0.02</td><td align="left">0.35</td><td align="left">0.02</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="center">9</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="left">0.59</td><td align="left">0.02</td><td align="left">0.77</td><td align="left">0.02</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 Note. Individual-level variables are not centered at the group mean level.</item> <item>4 EPC, <emph>Espacios para Crecer;</emph> n.a, not available.</item> </ulist> <hd id="AN0181774545-12">Student Pretest Explains More Variance Than Other Data Collected</hd> <p>Student pretests explain 37% of the variance at the group level and 32% of the variance at the individual level for reading fluency in the Peru Leer Juntos site. In contrast, in the Amazon Reads site 1, the pretest explains 76% of the variance at the group level and 39% of the variance at the individual level. The percentage of variance explained in all other regions lies between these two cases. (We excluded Honduras from this discussion because there were no student-level baseline data, no household survey, and the test measured general reading, not fluency; see Table A1 in the Appendix). This result is consistent with previous research, which found that including a pretest as a covariate can explain two-thirds of between-group variance and up to half of within-group variance ([<reflink idref="bib3" id="ref49">3</reflink>]; [<reflink idref="bib10" id="ref50">10</reflink>]).</p> <hd id="AN0181774545-13">Including Household Characteristics in Regression Models Adds Little to Variance Explained Wh...</hd> <p>Model 3 includes household information in addition to the pretest. In the Peru Leer Juntos site, with student pretest, age, gender, and household variables we can explain 40% of the variance at the school level and 35 at the student level, which is similar to what can be explained with only a pretest (37% and 32%). Similarly, in Guatemala, adding household characteristics to the regression that already has pretest data did not increase the <emph>R</emph><sups>2</sups> by much; the school-level <emph>R</emph><sups>2</sups> changed from 56 to 59% and the student-level <emph>R</emph><sups>2</sups> changed from 29 to 33%. This result is similar to previous research, which found that demographic information does little to increase power once a pretest is accounted for ([<reflink idref="bib10" id="ref51">10</reflink>]). Collecting household data is not common in U.S. education studies, but some studies in international contexts collect household data so it is important to assess how much power will be gained by collecting these data. Given the high cost of collecting household data, doing so to provide contextual information may not be worthwhile, and studies should prioritize collecting pretest information.</p> <hd id="AN0181774545-14">School Infrastructure Information Explained a Substantial Portion of School-Level Variance</hd> <p>Model 4 shows that, if we account for infrastructure information in addition to the pretest, we can explain around 10 percentage points more in Guatemala (school level <emph>R</emph><sups>2</sups> goes from 0.56 to 0.67) and in Peru (from 0.37 to 0.46) and even more in the other regions in Peru (school level <emph>R</emph><sups>2</sups> goes from 0.40 to 0.72 in Amazon Reads site 1 and from 0.76 to 0.93 in site 2). Given the low cost of collecting these additional data when visiting schools to collect student pretest data, it is worth collecting infrastructure information and including it in our regressions as covariates. In the U.S. context, school infrastructure may not have the same explanatory power as in the international context, where there may be more variation among schools.</p> <hd id="AN0181774545-15">School-Level Pretest Data Explain Less Variance Than Student Pretests</hd> <p>Often a student-level pretest is not feasible to collect. One possible solution is to use school-level average pretest scores for a previous cohort at the same school. For example, an intervention may target first-grade students, but randomization occurs before students actually enroll, so there is no time to test them before they learn of their treatment status. In these cases, researchers may wish to use the school average scores of first graders from the end of the school year before the experiment began. These average scores would absorb some of the chance differences in school-level baseline skills, but would not be measured for the actual students in the study. Therefore, at best, they would capture school factors or factors associated with attending the school, such as motivation and support of the parents. This is appropriate to the extent that these factors are stable over time, or the students in the older cohort are siblings of the students in the study. In the U.S. context, administrative data are widely available so using school-level data may be more feasible than collecting pretest data for the study. But in international settings, administrative data may not be available so the study may need to collect its own data.</p> <p>Using pretest information at the school level reduced outcome variance but not as much as using student-level pretest data. Model 7 presents the case where school average scores for a cohort of first-grade students who attended the schools one year before the intervention started are used as covariates in addition to a student-level pretest. In Amazon Reads site 1, the school mean pretests on the previous cohort added very little to what the student pretest already explained (the school-level <emph>R</emph><sups>2</sups> went from 0.76 to 0.78). In contrast, in Amazon Reads site 2 the school-level <emph>R</emph><sups>2</sups> went from 0.40 to 0.51. If we had collected only pretest data from the previous cohort (model 8), the percentage of variance we could have explained at the school level would be 0.36 in Amazon Reads site 1 and 0.35 in Amazon Reads site 2. This is less than half of what we can explain with a student pretest (model 2), so the precision gains were larger with a student pretest than with a school pretest on the previous cohort.</p> <p>These results differ from [<reflink idref="bib3" id="ref52">3</reflink>]. In their samples, school-level pretests are as effective as student-level pretests. However, in the Peru study, the student-level pretest corresponds to scores of the students attending the prior year (first grade) and the school-level pretest corresponds to the scores a previous cohort attained the prior year (first grade). In contrast, in [<reflink idref="bib3" id="ref53">3</reflink>], the school-level pretest corresponds to the scores a previous cohort attained in the same grade as when the post-test was measured (second grade). Hence, it is not surprising that our school-level data (previous grade rather than outcome from a previous cohort) is not as effective in reducing the outcome variance as the [<reflink idref="bib3" id="ref54">3</reflink>] data (same grade as outcome from a previous cohort).</p> <p>Furthermore, as we can see in model 8, age and gender explain only 2% of the variance at the student level. This is similar to findings from previous research ([<reflink idref="bib3" id="ref55">3</reflink>]; [<reflink idref="bib10" id="ref56">10</reflink>]).</p> <hd id="AN0181774545-16">School-Level Pretest Data on Previous Cohort in Same Grade Reduced Variance Slightly More Tha...</hd> <p>In Honduras, we collected administrative data at the school level, which allowed us to compare models where only school-level data were available. Table A1 in the Appendix presents the <emph>R</emph><sups>2</sups>s and MDES for these cases. Test scores for third graders at the school level were used as the outcome in all models. In model 2, school average first-grade test scores for the same cohort were used as baseline data. In model 7, school average third-grade scores for a previous cohort that was not exposed to the intervention were used as baseline information. The <emph>R</emph><sups>2</sups> is 0.17 for the model with school average first-grade pretest and 0.23 for the model with school average third-grade pretest. Hence, it seems that a school-level pretest on the same cohort in previous grades increases power slightly less than a pretest on a previous cohort but in the same grade as the outcome. We might have expected more precision gains using the pretest for the same cohort, but perhaps because there is large variation on reading in first grade this pretest was not more useful than the third-grade pretest on a different cohort. [<reflink idref="bib3" id="ref57">3</reflink>] also found that precision gains reduced as the time lag between the pretest and the post-test increases.</p> <hd id="AN0181774545-17">Translating Results Into MDES</hd> <p>One way to understand the role of the parameters presented above for ICC and <emph>R</emph><sups>2</sups> for the school and student levels is to calculate the MDES that would result from each set of variance parameters. We used a sample size of 45 schools in the treatment group and 45 schools in the control group, and a cluster size of 10 students per school. We chose a sample size (90 schools) that was within the range of the studies we used for this exercise (70–200). The actual sample size is illustrative to show the relationships among the MDES and the sets of covariates used in each specification. Those relationships would be similar for different numbers of clusters used in the design. Similarly, the ICC would affect these exact numbers, but for realism we used the realized ICCs from each of the RCTs conducted. The MDES are shown in Table 5.</p> <p>Table 5. MDES for Reading Fluency, by Site and Model.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="3" /><th align="center" colspan="3" rowspan="2">Student-level covariates</th><th align="center" colspan="2" rowspan="2">School-level covariates</th><th align="center" colspan="5">MDES</th></tr><tr><th align="left" colspan="2">Amazon Reads</th><th align="center" colspan="2">Leer Juntos</th><th align="center">EPC</th></tr><tr><th align="left">Pre-test</th><th align="center">Age and sex</th><th align="center">Household characteristics</th><th align="center">Previous cohort pretest</th><th align="center">Infrastructure characteristics</th><th align="center">Peru site 1</th><th align="center">Peru site 2</th><th align="center">Guatemala</th><th align="center">Peru</th><th align="center">Nicaragua</th></tr></thead><tbody valign="top"><tr><td align="left">1</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">0.29</td><td align="center">0.34</td><td align="center">0.35</td><td align="center">0.30</td><td align="center">0.20</td></tr><tr><td align="left">2</td><td align="center">X</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">0.22</td><td align="center">0.19</td><td align="center">0.25</td><td align="center">0.24</td><td align="center">0.12</td></tr><tr><td align="left">3</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">0.24</td><td align="center">0.24</td><td align="center">n.a.</td></tr><tr><td align="left">4</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="center">X</td><td align="center">0.19</td><td align="center">0.15</td><td align="center">0.22</td><td align="center">0.23</td><td align="center">n.a.</td></tr><tr><td align="left">5</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">0.22</td><td align="center">0.22</td><td align="center">n.a.</td></tr><tr><td align="left">6</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">0.18</td><td align="center">0.14</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="left">7</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">0.21</td><td align="center">0.19</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="left">8</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">0.25</td><td align="center">0.29</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="left">9</td><td align="left" /><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="center">0.22</td><td align="center">0.21</td><td align="center">n.a.</td><td align="center">n.a.</td><td align="center">n.a.</td></tr><tr><td align="left">ICC</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">0.15</td><td align="center">0.25</td><td align="center">0.28</td><td align="center">0.18</td><td align="center">0.06</td></tr></tbody></table> </ephtml> </p> <ulist> <item>5 Note. We assumed the same sample size for all the sites and models. MDES were calculated for 90 schools (45 in treatment group and 45 in control group) and 900 students (10 students per school).</item> <item>6 EPC, <emph>Espacios para Crecer;</emph> ICC, intraclass correlation; MDES, minimum detectable effect sizes; n.a, not available.</item> </ulist> <p>Without including any covariates (model 1), assuming 90 schools and 900 students our studies can detect effect sizes of around 0.30. The highest MDES is 0.35 for Guatemala and the lowest is 0.29 for Peru, Amazon Reads site 2.</p> <p>Including pretests in our regression models (model 2), allows us to detect smaller effect sizes. For example, we can detect effect sizes of 0.25 in Guatemala and 0.22 in Peru, Amazon Reads site 2. In Peru, Amazon Reads site 1, adding the pretest allows us to detect an effect size of 0.19; this is a reduction of 44%, which is considerable given that without covariates we could detect only a 0.34 effect size. These reductions are consistent with the finding in [<reflink idref="bib3" id="ref58">3</reflink>] and slightly larger in some cases.</p> <p>Including household characteristics in the regression models in addition to the pretest (comparing model 3 to model 2) does not change the MDES in the Leer Juntos Peru site and it decreases it by only 0.01 in Guatemala. Hence, collecting household variables may not be a cost-efficient way of increasing statistical power if a pretest has already been collected.</p> <p>Including school infrastructure characteristics in the regression models (model 4 compared to model 2) reduced the MDES. We found the lowest reduction in the Leer Juntos Peru site, where the MDES went from 0.24 to 0.23, and the largest reduction in Peru Amazon Reads site 2 where the MDES decreased from 0.19 to 0.15. Given the low cost of collecting school infrastructure characteristics once the schools are visited for pretest, this may be a cost-efficient way of increasing statistical power.</p> <p>Including a student pretest in the regression models attains higher statistical power than including a school average pretest on a previous cohort (model 2 compared to model 8). In Peru Amazon Reads site 2, including a school mean pretest, age, and gender (model 8) allowed us to detect an effect size of 0.25, and including a pretest for the students allowed us to detect an MDES of 0.22. The difference in site 1 was larger, with a decrease from 0.29 to 0.19, which is considerable. Given that administrative data are not widely available for many low- and middle-income countries, collecting pretest data in the sample of students who will be the target of the study and conducting longitudinal analysis may be a better alternative than collecting data on a previous cohort and conducting cross-sectional analysis.</p> <p>The models that include all student- and school-level covariates, (models 5 and 6) attain the lowest MDES. For example, in the case of Peru site 2, we can detect an effect size of 0.14 if we include all covariates, compared to 0.34 without including covariates, which is a 57% reduction.</p> <hd id="AN0181774545-18">Conclusions and Implications for Experimental Design</hd> <p>This paper documents the range over which key parameters for designing education cluster-RCTs vary for several diverse study sites in Latin America and compares them to previous findings from the United States, in particular to findings from [<reflink idref="bib3" id="ref59">3</reflink>] and [<reflink idref="bib10" id="ref60">10</reflink>]. We found that ICCs for reading comprehension and fluency were as low as 0.04 for communities that included students of all elementary grades, and as high as 0.28 for schools in rural areas and those in which different indigenous communities attend different schools. These ranges are similar to what researchers have found in studies in the United States. Our findings can be used to plan future studies by comparing the context of planned study sites to the contexts included in this paper to find the best match. The samples in our studies were in regions smaller than a state. Our data included only public schools and schools located in regions that were performing below the national averages. Future studies to be conducted in rural, low-income regions may find the results for Guatemala Leer Juntos and Peru Amazon Reads site 2 more relevant. Studies in urban and peri-urban areas of low-income regions may find the results from Peru Leer Juntos and Peru Amazon Reads site 1 more relevant. Rural areas seem to have higher ICCs than urban and peri-urban areas in our studies because there is more variation between schools in rural areas.</p> <p>The findings in this paper also inform which type of baseline data provides the greatest gains in terms of statistical power. For each type of covariate, we recommend repeating the power calculation using tools such as Optimal Design ([<reflink idref="bib22" id="ref61">22</reflink>]). Consistent with previous research, student-level pretests increase the statistical power of the study considerably, so it may be worth investing in collecting baseline tests for the study sample. Our findings will allow researchers to use more informed data collection strategies for early grade reading studies, in particular, if the widely adopted early grade reading assessment (EGRA) is used. Once pretest data have been included as covariates, adding household data did little to improve statistical power. Researchers need to consider whether collecting household data is worthwhile, considering the cost of data collection. If an outcome such as time spent reading at home is key for the evaluation, household data collection is necessary, but it should not be prioritized if the only reason to collect it is to improve precision of the impact estimates. Specifically, a researcher planning a prospective study can enter different values for <emph>R</emph><sups>2</sups> for the different types of data collection planned, using the values in Table 4 as a guide.</p> <p>Finally, school infrastructure data may be a cost-efficient way of increasing statistical power given that the cost of collecting it may be marginal in cases where researchers plan to collect student test scores. Researchers can perform the same exercise of changing the percentage of variance explained in their power calculations to see if the lower MDES from including school-level covariates is worth the added cost. A potential explanation of why school infrastructure data may increase precision is that, in low- and middle-income countries, school infrastructure may be a proxy for how involved the community is in the school, or the capacity of the principal, both of which may also affect achievement.</p> <p>The findings we shared about ICCs and the contribution of different types of covariates to the explanatory power of the model can be used to plan efficient experiments. In particular, there is a literature on optimizing sample allocation between levels of a cluster-randomized trial ([<reflink idref="bib2" id="ref62">2</reflink>]; [<reflink idref="bib9" id="ref63">9</reflink>]; [<reflink idref="bib16" id="ref64">16</reflink>]; [<reflink idref="bib20" id="ref65">20</reflink>]), where the ICC and the <emph>R</emph><sups>2</sups> at each level (student and school) are key inputs. As the ratio of data collection costs increases at a given level (e.g., costs of schools relative to students), the optimal sample allocation shifts more toward the other level (in this case towards fewer schools and more students per school, all things equal). The literature on such optimal design frameworks and an expansion of these frameworks to incorporate more perspectives is covered elsewhere ([<reflink idref="bib27" id="ref66">27</reflink>]).</p> <p>It remains for future research to identify the factors that determine why outcomes vary more specifically between schools than within schools and why commonly collected data from child assessments, household surveys, school and classroom observations, and administrative data would do a better or worse job of reducing residual variance. This may require a larger sample of studies. In addition, it is unclear whether these results will hold in high-income countries; for example, school infrastructure may have less explanatory power in those contexts and household data may be more informative. As more researchers publish information on the variance components (specifically ICCs and <emph>R</emph><sups>2</sups>) identified in ongoing and completed RCTs, the field will benefit from better-informed study designs and the pace of scientific knowledge will accelerate.</p> <hd id="AN0181774545-19">Acknowledgments</hd> <p>The authors acknowledge substantive input from colleagues Camila Fernandez, Sarah Liuzzi, Julieta Lugo-Gil, Emilie Bagby, and Virginia Poggio. Ivonne Padilla, Irina Cheban, and Galina Lapadatova provided expert programming assistance. Errors are sole responsibility of the authors.</p> <hd id="AN0181774545-20">Data Availability Statement</hd> <p>The data that support the findings of this study are available from United States Agency for International Development (USAID) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available.</p> <hd id="AN0181774545-21">ORCID iD</hd> <p>Larissa Campuzano https://orcid.org/0009-0003-0386-9959</p> <hd id="AN0181774545-22">Appendix</hd> <p></p> <hd id="AN0181774545-23">Appendix A. Auxiliary Tables</hd> <p>Table A1. R 2 at School Level, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>c</mi><mi>l</mi></mrow>2</msubsup></mrow></math> </ephtml> , and Student Level, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>R</mi><mrow><mi>s</mi><mi>t</mi></mrow>2</msubsup></mrow></math> </ephtml> , and MDES for Overall Reading, Honduras.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" rowspan="2">Model</th><th align="center">Student covariates</th><th align="center" colspan="3">School covariates</th><th align="center" colspan="3">Honduras findings</th></tr><tr><th align="left">Age and gender</th><th align="center">School average of 1st-grade pretest in baseline year</th><th align="center">School average of 3rd-grade reading test in baseline year</th><th align="center">School infrastructure</th><th align="center"><p id="ilm26-0193841X241241354"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">l</mi></mrow>2</msubsup></mrow></math></p></th><th align="center"><p id="ilm27-0193841X241241354"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msubsup><mi mathvariant="bold-italic">R</mi><mrow><mi mathvariant="bold-italic">s</mi><mi mathvariant="bold-italic">t</mi></mrow>2</msubsup></mrow></math></p></th><th align="center">MDES</th></tr></thead><tbody valign="top"><tr><td align="left">2</td><td align="left" /><td align="center">X</td><td align="left" /><td align="left" /><td align="char" char=".">0.17</td><td align="char" char=".">0.01</td><td align="char" char=".">0.19</td></tr><tr><td align="left">3</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /><td align="char" char=".">0.21</td><td align="char" char=".">0.02</td><td align="char" char=".">0.18</td></tr><tr><td align="left">4</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="char" char=".">0.39</td><td align="char" char=".">0.02</td><td align="char" char=".">0.17</td></tr><tr><td align="left">5</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="char" char=".">0.41</td><td align="char" char=".">0.02</td><td align="char" char=".">0.16</td></tr><tr><td align="left">6</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="char" char=".">0.25</td><td align="char" char=".">0.02</td><td align="char" char=".">0.18</td></tr><tr><td align="left">7</td><td align="left" /><td align="left" /><td align="center">X</td><td align="left" /><td align="char" char=".">0.20</td><td align="char" char=".">0.00</td><td align="char" char=".">0.19</td></tr><tr><td align="left">8</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="left" /><td align="char" char=".">0.23</td><td align="char" char=".">0.01</td><td align="char" char=".">0.18</td></tr><tr><td align="left">9</td><td align="center">X</td><td align="left" /><td align="center">X</td><td align="center">X</td><td align="char" char=".">0.39</td><td align="char" char=".">0.01</td><td align="char" char=".">0.16</td></tr></tbody></table> </ephtml> MDES, minimum detectable effect size.</p> <p>Table A2. School Infrastructure Variables Included, by Study.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">School infrastructure variable</th><th align="center">Amazon Reads Peru site 1</th><th align="center">Amazon Reads Peru site 2</th><th align="center">Leer Juntos Guatemala</th><th align="center">Leer Juntos Peru</th><th align="center">Nicaragua</th><th align="center">Honduras</th></tr></thead><tbody valign="top"><tr><td align="left"> Water supply not piped</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Potable drinking water supply</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Permanent building</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left"> Painted exterior walls</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left"> Working restroom facilities for children</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left"> Functional toilets</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left"> Separate facilities for boys and girls</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /></tr><tr><td align="left"> Working waste disposal</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Septic (or similar) disposal</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Plumbing waste disposal</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Other waste disposal</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left"> Kitchen</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Outdoors recreational space</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Gymnasium or sport facilities</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="left" /></tr><tr><td align="left"> Computers for students</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Internet connectivity</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Library or resource room</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Number of physical hazards observed in school</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Number of health hazards observed in school</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="center">X</td><td align="left" /><td align="center">X</td></tr><tr><td align="left"> Classrooms with finished materials (brick)</td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="center">X</td><td align="left" /></tr></tbody></table> </ephtml> </p> <p>Table A3. School Household Variables Included, by Study.</p> <p>Graph</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">School infrastructure variable</th><th align="center">Leer Juntos Guatemala</th><th align="center">Leer Juntos Peru</th></tr></thead><tbody valign="top"><tr><td align="left">Child attended preschool or kindergarten before starting first grade</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Average number of people living in the household</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Number of rooms in the house</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Highest grade attained by mother</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Parent reading skills (knows how to read)</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Mother language K'iche' only</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Family monthly income</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">Land owned by a household member</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with finished floor</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with finished roof</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with electricity</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with phone</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with radio</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with television</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with refrigerator</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with computer</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with bicycle</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with motorcycle</td><td align="center">X</td><td align="center">X</td></tr><tr><td align="left">House with car</td><td align="center">X</td><td align="center">X</td></tr></tbody></table> </ephtml> </p> <hd id="AN0181774545-24">Appendix B. LAC Reads Study Descriptions</hd> <p></p> <hd id="AN0181774545-25">Amazon Reads</hd> <p></p> <hd id="AN0181774545-26">Context</hd> <p>In Peru, Amazon Reads (<emph>Amazonía Lee</emph> in Spanish) was implemented in two sites by the Department of Regional Education Offices (DREs) of Ucayali and San Martín with funding from USAID starting in 2015. Site 1, San Martín lies in the northern part of the Peruvian Amazon forest. Site 2, Ucayali is located south of San Martín, bordering Brazil. These departments are primarily Spanish speaking, with a small percentage of the population speaking other indigenous languages. San Martín is more densely populated than Ucayali. San Martín has approximately 1302 active primary public education institutions, whereas Ucayali has approximately 796. Ucayali and, to a lesser extent, San Martín lag behind the nation on student reading achievement indicators.</p> <hd id="AN0181774545-27">Sample</hd> <p>The DRE in San Martín prepared a list of schools eligible to participate in the intervention. Eligible schools were public schools that had no other support programs and a minimum of 16 second graders in 2012. In San Martín, the final study sample included 200 eligible schools. In Ucayali, the DRE identified schools to participate in the study. The final study sample included 70 schools. In each study school, one first-grade classroom was randomly selected for data collection. Based on the classroom roster for the selected classroom, on average 8 first graders in San Martín and 10 in Ucayali were randomly selected (evenly divided by gender when possible). Students who were identified as having a physical or severe cognitive disability or as no longer attending the school were excluded. At endline, we tracked the study students from the 2015 first-grade cohort to evaluate their reading skills at the end of the second year of the intervention, when they were expected to be in second grade.</p> <hd id="AN0181774545-28">Data</hd> <p>The main data sources for the study are student reading assessments and a school infrastructure survey. The student assessments were collected at baseline (when students were attending first grade) and endline (when students were attending second grade). The endline assessment was a version of the Early Grade Reading Assessment (EGRA) that tested oral comprehension, familiar word reading, decoding, reading fluency, and reading comprehension. We designed and piloted an endline EGRA, in collaboration with a local data collection partner following the guidelines of the Early Grade Reading Assessment (EGRA) Toolkit ([<reflink idref="bib23" id="ref67">23</reflink>]). The reading assessments were administered individually by a trained enumerator. Most students were assessed at school during regular school hours, either in their baseline school or in another school if they had transferred. The final student reading outcomes were as follows: decoding accuracy (correct pseudo-words per minute), reading fluency (correct words per minute), familiar word reading (number of familiar words read correctly), and reading comprehension (number of correct answers).</p> <hd id="AN0181774545-29">Leer Juntos, Aprender Juntos</hd> <p></p> <hd id="AN0181774545-30">Context</hd> <p> <emph>Leer Juntos, Aprender Juntos</emph> was implemented in two sites, Peru and Guatemala. In Peru and Guatemala, the official language is Spanish, but some children speak other languages and have no knowledge of Spanish. <emph>Leer Juntos, Aprender Juntos</emph> is a program funded by USAID that aims to improve early grade reading instruction in communities with linguistically diverse populations. It was developed by Save the Children based on its Literacy Boost model, which includes teacher training and community involvement, and it was implemented in the K'iche'-speaking region of Guatemala and the Quechua-speaking region of Apurímac in Peru. In Guatemala, the study schools were located in five municipalities in the Department of El Quiché. The Department of El Quiché was selected for the study because its population is ethnically and linguistically diverse. In Peru, the study schools were located in the Apurímac region, in the Andes of southern-central Peru. Apurímac was selected because its population is linguistically diverse. The schools were selected from two provinces, Andahuaylas and Chincheros. The implementation of <emph>Leer Juntos, Aprender Juntos</emph> began in May 2013 in Peru and Guatemala, and continued through December 2015 in Peru and March 2016 in Guatemala.</p> <hd id="AN0181774545-31">Sample</hd> <p>The study recruited 147 schools in Peru and 150 schools in Guatemala. In each country, two-thirds of the schools were randomly assigned to an intervention group and one-third was assigned to a control group. Within each school, a group of children were followed from first grade through the end of third grade. The evaluation team randomly selected one first-grade classroom, one teacher, and 10 first-grade students from each school to serve as the analysis sample for the evaluation. The endline follow-up data collection took place three years after the baseline, when most of the children in the study should have progressed to third grade and have had two or more years of potential exposure to the intervention.</p> <hd id="AN0181774545-32">Data</hd> <p>During the baseline data collection, when children were in the first grade, the study assessed children's oral language proficiency and emergent literacy skills (letter identification, emergent writing, emergent reading, phonemic awareness, pseudo-word decoding, and passage comprehension skills) in Spanish and Quechua in Peru and Spanish and K'iche' in Guatemala. At endline, the individual assessment was administered when most of the children in the evaluation attended the third grade and it focused on reading skills in Spanish. The endline assessments were conducted in Spanish because Spanish is the language of instruction in the third-grade classrooms. Children's literacy skills were assessed in three tasks: (<reflink idref="bib1" id="ref68">1</reflink>) pseudo-word reading (decoding), (<reflink idref="bib2" id="ref69">2</reflink>) reading fluency, and (<reflink idref="bib3" id="ref70">3</reflink>) reading comprehension. These three tasks were adapted for Peru and Guatemala following the guidelines of the Early Grade Reading Assessment (EGRA) Toolkit ([<reflink idref="bib23" id="ref71">23</reflink>]). A household survey was conducted at midline when most students were in second grade. This survey was administered in person to the main caregivers (usually the mother or father) of the children in the evaluation at their homes. The purpose of this survey was to learn about the household composition, family socio-economic status, household assets, children's schooling background and routines at home, and children's and families' participation in reading activities offered in their communities.</p> <hd id="AN0181774545-33">Nicaragua</hd> <p></p> <hd id="AN0181774545-34">Context</hd> <p>USAID Nicaragua funded the Community Action for Reading and Security (CARS) program in five municipalities in the Southern Caribbean Atlantic Autonomous Region (Región Autónoma de la Costa Caribe Sur, RACCS) from 2011 through 2017. The RACCS is rural, hard to access, and half the population lives in extreme poverty. The intervention, Spaces to Learn, known by its Spanish name Espacios para Crecer (EpC), is an afterschool program implemented as part of CARS. Eligible communities had to be accessible, have schools that offered first to third grades, and have at least 20 children who were eligible to participate in the intervention. To be eligible to participate in the afterschool program, children had to be 6–16 years old; and they had to be out of school with the equivalent of a third-grade education or less (or enrolled in first, second, or third grade), or enrolled in school and have one risk factor such as having failed a grade, high absenteeism, or a mother tongue different from the language of instruction. Spanish was the primary language of instruction in most communities.</p> <hd id="AN0181774545-35">Sample</hd> <p>The communities included in the study were small, with at least 20 and fewer than 50 eligible children. The evaluation followed two cohorts of students for approximately one-and-a-half years of exposure to the afterschool program. We collected base-year data to measure children's literacy skills but could do so in only one of the cohorts. We collected follow-up data for each cohort (in 2016 for Cohort 1 and in 2017 for Cohort 2). The sample of students was representative of students eligible for the afterschool program: it included children in school with at-risk factors, as well as out-of-school children, within the ages of 6 to 13.</p> <hd id="AN0181774545-36">Data</hd> <p>Literacy skills were measured in the language of instruction in schools in the community. To measure children's literacy skills, we created an assessment based on the EGRA and the Dynamic Indicators of Reading Success (IDEL in Spanish). The four skills measured were (<reflink idref="bib1" id="ref72">1</reflink>) decoding pseudo-words or unfamiliar words, (<reflink idref="bib2" id="ref73">2</reflink>) oral comprehension (administered only to children unable to decode pseudo-words), (<reflink idref="bib3" id="ref74">3</reflink>) oral reading fluency, and (<reflink idref="bib4" id="ref75">4</reflink>) reading comprehension. Local stakeholders reviewed all assessment materials to ensure that the questions were appropriate for the cultural and linguistic context of the RACCS.</p> <hd id="AN0181774545-37">Honduras</hd> <p></p> <hd id="AN0181774545-38">Context</hd> <p>EducAcción was a USAID/Honduras-funded project carried out by AIR, who worked with municipalities, districts, and schools in Honduras since 2011 to promote improved school management techniques, community involvement in schools, and teacher training on Spanish and math instruction. The EducAcción project's school supports included assistance in using formative assessments and end-of-grade (EOG) results to improve instruction. The project was implemented in the two predominantly rural departments of Lempira and Santa Barbara and the two urban areas of Tegucigalpa and La Ceiba.</p> <hd id="AN0181774545-39">Sample</hd> <p>We selected an initial sample of 240 schools that were low-performing, had at least 10 first-grade students in 2013, and EOG test scores in the bottom three quintiles of performance. Schools that were infeasible to visit because they were in areas that were highly prone to violence were excluded. The study followed the cohort of students who were enrolled in second grade as of the last day before the intervention began—May 31, 2015. We measured learning after half a year of support in second grade (2015) and a full year of support in third grade (2016).</p> <hd id="AN0181774545-40">Data</hd> <p>In most years since 2007, the Ministry of Education, with international donor funding from several sources, has supported administration of the EOG tests. Mejorando el Impacto al Desempeño Estudiantil de Honduras (MIDEH) developed and has administered the tests. For this evaluation, we used EOG test data from four academic years: 2013 through 2016. The items in the tests varied from year to year; however, all items were drawn from the same item bank that MIDEH developed and considered to be equally difficult. The study's main outcome measures are third grade EOG test scores in reading for the study cohort in 2016.</p> <ref id="AN0181774545-41"> <title> Note </title> <blist> <bibl id="bib1" idref="ref33" type="bt">1</bibl> <bibtext> The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref34" type="bt">2</bibl> <bibtext> The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was partly supported by the United States Agency for International Development under Task Order AID-OAA-M12-00020.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref15" type="bt">3</bibl> <bibtext> The results for Honduras presented in the appendix are based on a general reading test that includes comprehension, but does not explicitly report results by literacy skill, such as reading fluency.</bibtext> </blist> </ref> <ref id="AN0181774545-42"> <title> References </title> <blist> <bibtext> Bagby E., Torrente C., Glazerman S., Murray N., Padilla I. (2019). Evaluation of Espacios para Crecer (EpC), an afterschool program. In Nicaragua: Final report. Mathematica. Report prepared for USAID. https://pdf.usaid.gov/pdf_docs/PA00XDR4.pdf</bibtext> </blist> <blist> <bibtext> Bloom H. S. (2005). Randomizing groups to evaluate place-based programs. In Bloom H. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). Russell Sage Foundation.</bibtext> </blist> <blist> <bibtext> Bloom H. S., Richburg-Hayes L., Black A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59. https://doi.org/10.3102/0162373707299550</bibtext> </blist> <blist> <bibl id="bib4" idref="ref13" type="bt">4</bibl> <bibtext> Brunner M., Keller U., Wenger M., Fischbach A., Lüdtke O. (2018). Between-school variation in students' achievement, motivation, affect, and learning strategies: Results from 81 countries for planning group-randomized trials in education. Journal of Research on Educational Effectiveness, 11(3), 452–478. https://doi.org/10.1080/19345747.2017.1375584</bibtext> </blist> <blist> <bibl id="bib5" idref="ref38" type="bt">5</bibl> <bibtext> Campuzano L., Fernández C., Lugo-Gil J., Glazerman S., Murray N. (2018). Evaluation of Amazonía Lee reading intervention in Peru: Final report. Mathematica. Report prepared for USAID. https://pdf.usaid.gov/pdf_docs/PA00TCQ1.pdf</bibtext> </blist> <blist> <bibl id="bib6" idref="ref3" type="bt">6</bibl> <bibtext> Duflo E., Glennerster R., Kremer M. (2008). Using randomization in development economics research: A toolkit. Handbook of Development Economics, 4(5), 3895–3962. https://econpapers.repec.org/bookchap/eeedevchp/5-61.htm</bibtext> </blist> <blist> <bibl id="bib7" idref="ref25" type="bt">7</bibl> <bibtext> Hedberg E., Hedges L. (2014). Reference values of within-district intraclass correlations of academic achievement by district characteristics: Results from a meta-analysis of district-specific values. Evaluation Review, 38(6), 546–582. https://doi.org/10.1177/0193841X14554212</bibtext> </blist> <blist> <bibl id="bib8" idref="ref7" type="bt">8</bibl> <bibtext> Hedges L., Hedberg E. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. https://doi.org/10.1177/0193841X14529126</bibtext> </blist> <blist> <bibl id="bib9" idref="ref63" type="bt">9</bibl> <bibtext> Hedges L. V., Borenstein M. (2014). Conditional optimal design in three- and four-Level experiments. Journal of Educational and Behavioral Statistics, 39(4), 257–281. https://doi.org/10.3102/1076998614534897</bibtext> </blist> <blist> <bibtext> Hedges L. V., Hedberg E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706</bibtext> </blist> <blist> <bibtext> Jacob R., Zhu P., Bloom H. (2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198. https://doi.org/10.1080/19345741003592428</bibtext> </blist> <blist> <bibtext> Kelcey B., Phelps G. (2013). Strategies for improving power in school-randomized studies of professional development. Evaluation Review, 37(6), 520–554. https://doi.org/10.1177/0193841X14528906</bibtext> </blist> <blist> <bibtext> Kelcey B., Shen Z., Spybrook J. (2016). Intraclass correlation coefficients for designing cluster-randomized trials in sub-Saharan Africa education. Evaluation Review, 40(6), 500–525. https://doi.org/10.1177/0193841x16660246</bibtext> </blist> <blist> <bibtext> Konstantopoulos S. (2012). The impact of covariates on statistical power in cluster randomized designs: Which level matters more? Multivariate Behavioral Research, 47(3), 392–420. https://doi.org/10.1080/00273171.2012.673898</bibtext> </blist> <blist> <bibtext> Konstantopoulos S. (2013). Optimal design in three-level block randomized designs with two levels of nesting: An ANOVA framework with random effects. Educational and Psychological Measurement, 73(5), 784–802. https://doi.org/10.1177/0013164413485752</bibtext> </blist> <blist> <bibtext> Liu X. (2003). Statistical power and optimum sample allocation ratio for treatment and control having unequal costs per unit of randomization. Journal of Educational and Behavioral Statistics, 28(3), 231–248. https://doi.org/10.3102/10769986028003231</bibtext> </blist> <blist> <bibtext> Liuzzi S., Murray N., Glazerman S., Cheban I. (2019). Data-driven instruction in Honduras: An impact evaluation of the EducAcción-PRI promising reading intervention. Mathematica. Final report. Report prepared for USAID. https://pdf.usaid.gov/pdf_docs/PA00WDW1.pdf</bibtext> </blist> <blist> <bibtext> Lugo-Gil J., Murray N., Glazerman S., Fernández C., Campuzano L. (2019a). Evaluation of Leer Juntos, Aprender Juntos early grade intervention in Guatemala: Final report. Mathematica. Report prepared for USAID. https://<ulink href="http://www.edu-links.org/sites/default/files/media/file/Evaluation/of/Leer/Juntos%2C/Aprender/Juntos/Early/Grade/Reading/Intervention/in/Guatemala/Final/Report.pdf">www.edu-links.org/sites/default/files/media/file/Evaluation/of/Leer/Juntos%2C/Aprender/Juntos/Early/Grade/Reading/Intervention/in/Guatemala/Final/Report.pdf</ulink></bibtext> </blist> <blist> <bibtext> Lugo-Gil J., Murray N., Glazerman S., Fernández C., Campuzano L., Padilla I. (2019b). Evaluation of Leer Juntos, Aprender Juntos early grade intervention in Peru: Final report. Mathematica. Report prepared for USAID. https://pdf.usaid.gov/pdf_docs/PA00XJKM.pdf</bibtext> </blist> <blist> <bibtext> Raudenbush S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://doi.org/10.1037/1082-989X.2.2.173</bibtext> </blist> <blist> <bibtext> Raudenbush S. W., Martinez A., Spybrook J. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5–29. https://doi.org/10.3102/0162373707299460</bibtext> </blist> <blist> <bibtext> Raudenbush S. W., Spybrook J., Congdon R., Liu X., Martinez A., Bloom H., Hill C. (2011). Optimal design plus empirical evidence (Version 3.0). https://<ulink href="http://www.wtgrantfoundation.org/resources/optimal-design">www.wtgrantfoundation.org/resources/optimal-design</ulink></bibtext> </blist> <blist> <bibtext> RTI International. (2016). Early grade reading assessment (EGRA) toolkit (2nd ed.). United States Agency for International Development.</bibtext> </blist> <blist> <bibtext> Schochet P. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87. https://doi.org/10.3102/1076998607302714</bibtext> </blist> <blist> <bibtext> Seidenfeld D., Handa S., de Hoop T., Morey M. (2023). Intraclass correlations values in international development: Evidence across commonly studied domains in sub-Saharan Africa. Evaluation Review, 47(5), 786–819. https://doi.org/10.1177/0193841X231154714</bibtext> </blist> <blist> <bibtext> Shen Z., Curran F. C., You Y., Splett J. W., Zhang H. (2023). Intraclass correlations for evaluating the effects of teacher empowerment programs on student educational outcomes. Educational Evaluation and Policy Analysis, 45(1), 134–156. https://doi.org/10.3102/01623737221111400</bibtext> </blist> <blist> <bibtext> Shen Z., Kelcey B. (2020). Optimal sample allocation under unequal costs in cluster-randomized trials. Journal of Educational and Behavioral Statistics, 45(4), 446–474. https://doi.org/10.3102/1076998620912418</bibtext> </blist> <blist> <bibtext> Spybrook J., Kelcey B. (2016). Introduction to three special issues on design parameter values for planning cluster randomized trials in the social sciences. Evaluation Review, 40(6), 491–499. https://doi.org/10.1177/0193841X16685646</bibtext> </blist> <blist> <bibtext> Stallasch S. E., Lüdtke O., Artelt C., Brunner M. (2021). Multilevel design parameters to plan cluster-randomized intervention studies on student achievement in elementary and secondary school. Journal of Research on Educational Effectiveness, 14(1), 172–206. https://doi.org/10.1080/19345747.2020.1823539</bibtext> </blist> <blist> <bibtext> Westine C. D., Spybrook J., Taylor J. A. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37(6), 490–519. https://doi.org/10.1177/0193841X14531584</bibtext> </blist> </ref> <aug> <p>By Steven Glazerman; Larissa Campuzano and Nancy Murray</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib28" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib30" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib10" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib24" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib26" firstref="ref9"></nolink> <nolink nlid="nl6" bibid="bib29" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib13" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib25" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib21" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib12" firstref="ref17"></nolink> <nolink nlid="nl11" bibid="bib14" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib20" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib27" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib11" firstref="ref28"></nolink> <nolink nlid="nl15" bibid="bib15" firstref="ref39"></nolink> <nolink nlid="nl16" bibid="bib23" firstref="ref40"></nolink> <nolink nlid="nl17" bibid="bib22" firstref="ref61"></nolink> <nolink nlid="nl18" bibid="bib16" firstref="ref64"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1454813
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Education Experiments in Latin America: Empirical Evidence to Guide Evaluation Design
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Steven+Glazerman%22">Steven Glazerman</searchLink><br /><searchLink fieldCode="AR" term="%22Larissa+Campuzano%22">Larissa Campuzano</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0003-0386-9959">0009-0003-0386-9959</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nancy+Murray%22">Nancy Murray</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Evaluation+Review%22"><i>Evaluation Review</i></searchLink>. 2025 49(1):115-146.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 32
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Agency for International Development (IDCA)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: AIDOAAM1200020
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Experiments%22">Educational Experiments</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Multivariate+Analysis%22">Multivariate Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Sample+Size%22">Sample Size</searchLink><br /><searchLink fieldCode="DE" term="%22Surveys%22">Surveys</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Equations+%28Mathematics%29%22">Equations (Mathematics)</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Tests%22">Reading Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Comprehension%22">Reading Comprehension</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Fluency%22">Reading Fluency</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Latin+America%22">Latin America</searchLink><br /><searchLink fieldCode="DE" term="%22Nicaragua%22">Nicaragua</searchLink><br /><searchLink fieldCode="DE" term="%22Honduras%22">Honduras</searchLink><br /><searchLink fieldCode="DE" term="%22Guatemala%22">Guatemala</searchLink><br /><searchLink fieldCode="DE" term="%22Peru%22">Peru</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/0193841X241241354
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0193-841X<br />1552-3926
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized experiments involving education interventions are typically implemented as cluster randomized trials, with schools serving as clusters. To design such a study, it is critical to understand the degree to which learning outcomes vary between versus within clusters (schools), specifically the intraclass correlation coefficient. It is also helpful to anticipate the benefits, in terms of statistical power, of collecting household data, testing students at baseline, or relying on administrative data on previous cohorts from the same school. We use data from multiple cluster-randomized trials in four Latin American countries to provide information on the intraclass correlations in early grade literacy outcomes. We also describe the proportion of variance explained by different types of covariates. These parameters will help future researchers conduct statistical power analysis, estimate the required sample size, and determine the necessity of collecting different types of baseline data such as child assessments, administrative data at the school level, or household surveys.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1454813
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1454813
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/0193841X241241354
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 115
    Subjects:
      – SubjectFull: Educational Experiments
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Research Design
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Multivariate Analysis
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Sample Size
        Type: general
      – SubjectFull: Surveys
        Type: general
      – SubjectFull: Correlation
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: Equations (Mathematics)
        Type: general
      – SubjectFull: Reading Tests
        Type: general
      – SubjectFull: Reading Comprehension
        Type: general
      – SubjectFull: Reading Fluency
        Type: general
      – SubjectFull: Latin America
        Type: general
      – SubjectFull: Nicaragua
        Type: general
      – SubjectFull: Honduras
        Type: general
      – SubjectFull: Guatemala
        Type: general
      – SubjectFull: Peru
        Type: general
    Titles:
      – TitleFull: Education Experiments in Latin America: Empirical Evidence to Guide Evaluation Design
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Steven Glazerman
      – PersonEntity:
          Name:
            NameFull: Larissa Campuzano
      – PersonEntity:
          Name:
            NameFull: Nancy Murray
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0193-841X
            – Type: issn-electronic
              Value: 1552-3926
          Numbering:
            – Type: volume
              Value: 49
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Evaluation Review
              Type: main
ResultId 1