View in EDS HTML Full Text PDF Full Text

Explaining Variation in Treatment Effects: An Impact Evaluation and Mixed-Methods Study of Variation in Early Grade Reading Program Effects

Saved in:
Bibliographic Details
Title: Explaining Variation in Treatment Effects: An Impact Evaluation and Mixed-Methods Study of Variation in Early Grade Reading Program Effects
Language: English
Authors: T. J. D'Agostino, Danice Brown Guzmán, Paul Perrin, Anasthasie Liberiste-Osirus, Kate Schuenke-Lucien
Source: Comparative Education Review. 2024 68(1):85-112.
Availability: University of Chicago Press. Journals Division, P.O. Box 37005, Chicago, IL 60637. Tel: 877-705-1878; Tel: 773-753-3347; Fax: 877-705-1879; Fax: 773-753-0811; e-mail: subscriptions@press.uchicago.edu; Web site: http://www.press.uchicago.edu
Peer Reviewed: Y
Page Count: 28
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Descriptors: Early Reading, Program Effectiveness, Foreign Countries, Reading Comprehension, Literacy, Educational Environment, Attendance, Coaching (Performance), Administrator Attitudes, Geographic Location
Geographic Terms: Haiti
DOI: 10.1086/728393
ISSN: 0010-4086
1545-701X
Abstract: This article contributes to understanding the effectiveness of early grade reading interventions in low-income-country contexts and how and why such interventions vary in their effectiveness across schools. It presents the results from an impact evaluation of an early grade reading intervention in Haiti with a mixed-methods analysis of factors that explain variation in program effects. We study program impact using a randomized controlled trial experimental research design and find significant treatment impact on students' literacy skills, including on students' reading comprehension in Creole. We use a mixed-methods design to explore variation in program effects across school sites. Key factors that explain variation in program effects include the frequency of instructional coaching visits; the dosage and uptake of treatment, which was linked to student and teacher absenteeism and the exclusive use of the treatment curriculum; the behaviors of school leaders; the fluency and quality of lesson delivery; and the remoteness of schools.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1417939
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHYelG2IPxi2fP0_0JYpe-FAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDItODkcNuWZjNFWSlAIBEICBm5bh11YvfFlfLJhnApJhklQuDP0WiVh5u0eMdCP0Zl4V1zQtxeh0gPgC1NQFhLd7nccWzhpQ6TiFwS6ZDSFUNHVViRR1SH63wAIX6L0iyDyDyR2sDsQcA7V_MGI51W8nXoFckSm_JSUvhzlGOJ3LCiSPUlzaqPcD-0ywAgPVB-tFUUkJVEIHf229UwQ1PMclY7NGTV1srqbN4Tsx
Text:
  Availability: 1
  Value: <anid>AN0176222691;cer01feb.24;2024Mar27.06:42;v2.2.500</anid> <title id="AN0176222691-1">Explaining Variation in Treatment Effects: An Impact Evaluation and Mixed-Methods Study of Variation in Early Grade Reading Program Effects </title> <sbt id="AN0176222691-2">Introduction</sbt> <p>This article contributes to understanding the effectiveness of early grade reading interventions in low-income-country contexts and how and why such interventions vary in their effectiveness across schools. It presents the results from an impact evaluation of an early grade reading intervention in Haiti with a mixed-methods analysis of factors that explain variation in program effects. We study program impact using a randomized controlled trial experimental research design and find significant treatment impact on students' literacy skills, including on students' reading comprehension in Creole. We use a mixed-methods design to explore variation in program effects across school sites. Key factors that explain variation in program effects include the frequency of instructional coaching visits; the dosage and uptake of treatment, which was linked to student and teacher absenteeism and the exclusive use of the treatment curriculum; the behaviors of school leaders; the fluency and quality of lesson delivery; and the remoteness of schools.</p> <p>Despite increasing enrollment rates, basic skill acquisition remains low in many low-income countries (World Bank [<reflink idref="bib54" id="ref1">54</reflink>]). Even prior to the COVID-19 pandemic, over half of 10-year-olds in low- and middle-income countries were unable to read or comprehend a simple story (UNICEF [<reflink idref="bib51" id="ref2">51</reflink>]). These challenges were exacerbated by the significant learning loss from school closures during the pandemic (UNICEF [<reflink idref="bib51" id="ref3">51</reflink>]).</p> <p>Some promising solutions have emerged. Recent reviews of evidence point to positive average effects of structured programs in improving students' reading skills (Evans and Popova [<reflink idref="bib22" id="ref4">22</reflink>]; Kim et al. [<reflink idref="bib31" id="ref5">31</reflink>]). For example, Kim et al.'s ([<reflink idref="bib31" id="ref6">31</reflink>]) recent meta-analysis reported an average effect size of.30 from 67 studies across 32 countries.</p> <p>Scholars also note, however, wide variation in treatment effects across contexts, which impedes the efficacy and scalability of programs (Bold et al. [<reflink idref="bib8" id="ref7">8</reflink>]). Kim et al. ([<reflink idref="bib31" id="ref8">31</reflink>]) found negative results in approximately 20 percent of studies in their analysis. Even in early grade reading (EGR) programs that demonstrate large average effects, high proportions of students often fail to make progress. For example, in schools receiving a high-impact reading intervention in Ghana, up to 40 percent of students still registered "zero scores" for the number of words read correctly per minute at the end of grade 2 (USAID [<reflink idref="bib52" id="ref9">52</reflink>]). Large proportions of zero scores were also found in Rwanda, the Philippines (Christina and Vinogradova [<reflink idref="bib15" id="ref10">15</reflink>]), and Haiti (Guzmán et al. [<reflink idref="bib26" id="ref11">26</reflink>]) and appear to be a consistent outcome.</p> <p>This evidence suggests the importance of considering heterogeneity in treatment effects. Scholars have argued for decades that it is important to understand how program effects vary in order to improve the effectiveness of interventions (see Bold et al. [<reflink idref="bib8" id="ref12">8</reflink>]; Bryk [<reflink idref="bib11" id="ref13">11</reflink>]). Yet evaluation research on early grade reading interventions usually measures the average effects with few exceptions (Christina and Vinogradova [<reflink idref="bib15" id="ref14">15</reflink>]; Brunette et al. [<reflink idref="bib9" id="ref15">9</reflink>]).</p> <p>Variation in treatment effects has received greater attention in public health through <emph>implementation research</emph> (Bryk et al. [<reflink idref="bib12" id="ref16">12</reflink>]; Century and Cassata [<reflink idref="bib14" id="ref17">14</reflink>]), which focuses not only on what works but on what works for whom, when, how, why, and under what conditions. Implementation research is concerned with understanding mechanisms that explain variation in program effects so as to identify areas for improvement. Implementation research considers any aspect of implementation, from process to context to user characteristics. Insights can inform intervention design effectiveness and sustainability at scale (Peters et al. [<reflink idref="bib37" id="ref18">37</reflink>]).</p> <p>In this article, we present findings from an impact evaluation of an early grade reading program in Haiti and the results of a mixed-methods study of the factors explaining variation in treatment effects within the treated sample of schools. We begin by reviewing the relevant literature on early grade reading program effectiveness, focusing on the aspects of implementation most relevant to our study, namely, treatment design, implementation and monitoring, and implementing organizations.</p> <hd id="AN0176222691-3">Literature Review</hd> <p></p> <hd id="AN0176222691-4">Treatment design</hd> <p>Most scholarly work has focused on the common design elements of effective EGR programs in low-income contexts. Studies emphasize that effective designs include structured lesson plans, one-to-one student books, in-person and practical teacher training, and on-site instructional coaching with structured rubrics.[<reflink idref="bib2" id="ref19">2</reflink>] Studies have also found that differentiated, small-group or one-on-one instruction customized to students' ability improves learning (Banerjee et al. [<reflink idref="bib3" id="ref20">3</reflink>], [<reflink idref="bib4" id="ref21">4</reflink>]; Christina and Vinogradova [<reflink idref="bib15" id="ref22">15</reflink>]).</p> <hd id="AN0176222691-5">Implementation and monitoring</hd> <p>Research also points to key implementation factors, including the prescriptiveness or adaptability of the implementation, the degree of implementation fidelity and careful monitoring, and the variability in program execution (Quinn and Kim [<reflink idref="bib41" id="ref23">41</reflink>]). Implementation fidelity and execution are frequently cited factors, such as incomplete or poorly attended trainings, variable frequency of coaching visits, teachers reverting to old practices, and books being delayed in delivery or going unused (McKenna et al. [<reflink idref="bib35" id="ref24">35</reflink>]).</p> <p>Most efforts to measure implementation fidelity in EGR programs have focused on the presence, quality, and dosage of onsite coaching of teachers from supervisors or principals as a key monitoring and support function (Stern et al. [<reflink idref="bib49" id="ref25">49</reflink>]). Coaching has been found to explain some variation in treatment effects (Christina and Vinogradova [<reflink idref="bib15" id="ref26">15</reflink>]). In Kenya, student outcomes were higher in classrooms where teachers were supported by coaches responsible for 10 rather than 15 schools, suggesting a coaching dosage effect (Piper and Zuilkowski [<reflink idref="bib39" id="ref27">39</reflink>]).</p> <p>Little scholarly attention, however, has focused on the treatment dosage or fidelity of treatment received by students (the direct treatment on target beneficiaries). Measuring fidelity of treatment to students concerns things like curriculum coverage and the thoroughness or completion of individual lessons. In the rare instances when this has received scholarly attention, findings indicate that both overall pacing and individual lesson fidelity are highly variable (Guzmán et al. [<reflink idref="bib26" id="ref28">26</reflink>]; Fleisch [<reflink idref="bib24" id="ref29">24</reflink>]).</p> <hd id="AN0176222691-6">Implementing organizations</hd> <p>Program effects also vary due to factors related to implementing organizations. The implementing organizations in EGR programs tend to include (<reflink idref="bib1" id="ref30">1</reflink>) NGOs or states, which are responsible for training, coaching, material delivery, and monitoring, and (<reflink idref="bib2" id="ref31">2</reflink>) schools. Variation at both levels may result from differences in organizational culture and the skills, professional capacity, and motivation of staff. For example, Heugh et al. ([<reflink idref="bib28" id="ref32">28</reflink>]) emphasized inconsistent uptake across certain regions of Ethiopia, suggesting differences in implementation at the regional level that explained variation in EGR program effects. Research also points to government buy-in, quality relationships, and effective organizational norms as key characteristics influencing implementing organization effectiveness (Bold et al. [<reflink idref="bib7" id="ref33">7</reflink>]; Piper, Destefano, et al. [<reflink idref="bib38" id="ref34">38</reflink>]).</p> <p>Schools are also key implementing organizations that affect outcomes. Taylor et al. ([<reflink idref="bib50" id="ref35">50</reflink>]) observe that urban and larger schools benefit more from EGR programs than rural and small schools. This may be due to various underlying conditions in rural schools, often linked to students' opportunity to learn and the quality of teachers and leaders. Opportunity to learn is a construct linked primarily to instructional time and content delivery and is affected by both in school and out of school factors. These include school closure rates, distance to school, annual days of instruction, teacher and student absenteeism or tardiness, time on task in class, individual student access to learning materials, and the home literacy environment and community learning environment (Gillies and Quijada [<reflink idref="bib25" id="ref36">25</reflink>]; Dowd et al. [<reflink idref="bib18" id="ref37">18</reflink>]). For example, Bold et al. ([<reflink idref="bib6" id="ref38">6</reflink>]), in a study across seven sub-Saharan African countries, found that students received an average of 2.5 hours of instruction daily, less than half the scheduled time. High student absenteeism, related to the organizational environment and students' socioeconomic status (SES), also predicts lower treatment effects (Dowd and Mabeti [<reflink idref="bib19" id="ref39">19</reflink>]; Dubeck et al. [<reflink idref="bib20" id="ref40">20</reflink>]).</p> <p>A second key characteristic of schools affecting learning is the motivation and skill level of teachers and leaders (Bruns and Luque [<reflink idref="bib10" id="ref41">10</reflink>]; Schleicher [<reflink idref="bib48" id="ref42">48</reflink>]). The effectiveness of teachers can explain up to 1 year's difference in learning gains (Hanushek and Rivkin [<reflink idref="bib27" id="ref43">27</reflink>]), and teacher motivation and effort are particular concerns in low-income countries (World Bank [<reflink idref="bib54" id="ref44">54</reflink>]). Finally, school leadership has been linked indirectly to student outcomes due to leadership influence over teacher selection, organizational culture, and instructional support (Robinson et al. [<reflink idref="bib44" id="ref45">44</reflink>]; Bloom et al. [<reflink idref="bib5" id="ref46">5</reflink>]).</p> <hd id="AN0176222691-7">Summary and contribution</hd> <p>Our review of the relevant research finds limited evidence explaining heterogeneity in program effects. Most research identifies common elements of successful interventions, including their design, the importance of the quality and dosage of coaching, and the key role of implementing organizations, for example, in providing books and training in a timely manner and fostering program uptake. Research has focused on variation across programs, with little attention given to within-program variation across school sites, contexts, and clients.[<reflink idref="bib3" id="ref47">3</reflink>] More of such research will provide greater visibility into program mechanisms and barriers to or enablers of efficacy, providing insights to improve EGR program effectiveness at scale. This study seeks to help fill this research gap.</p> <hd id="AN0176222691-8">Research Questions</hd> <p>Our study combines an impact evaluation of a structured, multilingual EGR intervention in Haiti with a mixed-methods analysis of factors that explain variation within a given program across schools. The mixed-methods design allows for rich exploration of the mechanisms, enablers, and barriers to effectiveness. We focus on the following two research questions: (<reflink idref="bib1" id="ref48">1</reflink>) To what extent did the <emph>Read Haiti</emph> Program impact students' reading skills? And (<reflink idref="bib2" id="ref49">2</reflink>) What programmatic and environmental factors explain variation in student performance across treated schools? We answer the first question using a randomized controlled trial and analyze student performance on the Early Grade Reading Assessment (EGRA) and use a mixed-methods design to address the second question.</p> <hd id="AN0176222691-9">Study Context</hd> <p>Haiti is both the poorest country in the Western Hemisphere and has the poorest-performing education system in the region on most assessed indicators. When students enter third grade, 47 percent of Haitian students are not able to read a word in Creole, and 63 percent cannot read a word in French (Registe [<reflink idref="bib43" id="ref50">43</reflink>]).</p> <p>From 2012 to 2014, RTI International implemented a structured, dual-language EGR intervention in Haiti with support from USAID and the Haitian government. In reports based on a randomized control trial (RCT) evaluation, the project reported strong gains in initial sound identification, substantial gains in the identification of letter sounds, and promising trends in listening comprehension, but no gains in oral reading fluency or reading comprehension (RTI International [<reflink idref="bib46" id="ref51">46</reflink>]). Using this same curriculum, a later project in two regions of Haiti in 50 schools from 2014–16, also using an RCT evaluation, found large positive effects for students at the end of grade 2 in letter sounds, letter identification, and oral reading fluency but not in reading comprehension (Guzmán et al. [<reflink idref="bib26" id="ref52">26</reflink>]). <emph>Read Haiti</emph> (2016–20) is a refined and scaled-up implementation of this project in 350 schools across five regions.</p> <hd id="AN0176222691-10">About the Read Haiti Program</hd> <p> <emph>Read Haiti</emph> is a dual-language, early grade reading intervention that uses structured lesson plans, teacher training, and regular on-site coaching, as well as supplementary literacy activities, as the core program elements. The curriculum and materials use a transitional, early exit, multilingual literacy design. The grade 1 curriculum emphasizes preliteracy, basic phonics, and decoding in Creole and the introduction of oral French. Grade 2 continues oral French and seeks to build reading fluency and comprehension in French and Creole.</p> <p>The <emph>Read Haiti</emph> intervention employed various means of support to foster effective implementation. These included initial in-person teacher and school leader training, periodic refresher teacher training, monthly instructional coaching visits from regional technical supervisors, encouragement of weekly school leader classroom observations, and monthly regional "cluster meetings" of teachers and school leaders from participating schools.</p> <p> <emph>Read Haiti</emph> also sought to emphasize instructional practices that would improve reading comprehension. This involved providing leveled classroom libraries of Creole and French books and supporting teachers to use these during an open reading block on Fridays. Teachers were trained and coached in the following practices: interactive read alouds, facilitation of independent student reading and text selection, and use of guided reading techniques in small groups. Finally, using a formative assessment provided with the classroom libraries, <emph>Read Haiti</emph> experimented with targeted small-group instruction aligned to learners' skill levels after school and during a 3-week summer camp.</p> <hd id="AN0176222691-11">Method</hd> <p>The study had two designs: an impact evaluation and a concurrent mixed-methods design. The mixed-methods design includes an analysis of classroom predictors of treatment effects and a qualitative exploration of mechanisms explaining variation in treatment outcomes, where results are integrated to generate findings (Creswell and Clark [<reflink idref="bib16" id="ref53">16</reflink>]).</p> <hd id="AN0176222691-12">Participant Characteristics</hd> <p>For the first research question, participants are cohorts of first- and second-grade students enrolled in schools treated by the <emph>Read Haiti</emph> program (treatment schools), and students enrolled in other Haitian primary schools (control schools). The impact evaluation was isolated to three of the five regions where <emph>Read Haiti</emph> was implemented and for which an experimental evaluation design was utilized: Centre, Sud, and Grand Anse.[<reflink idref="bib4" id="ref54">4</reflink>]</p> <p>For research question two concerning variation in treatment effects, the quantitative data come from teacher observations in treatment schools in all five regions. The qualitative data are from key staff from two implementing partners, including focus groups with regional technical supervisors who conduct monthly teacher observation and coaching visits and interviews with two program coordinators who manage these supervisors and who each support implementation in a set of 40 schools. Participants were selected for their depth and breadth of knowledge of treatment schools, site conditions, student and staff characteristics and behaviors, and the implementation process.</p> <hd id="AN0176222691-13">Sampling Procedures and Sample Size</hd> <p>Fifteen students were sampled randomly from each grade assessed during the first round of data collection. Follow-up data collection reassessed the same students. To collect data on instruction and classroom characteristics and test their relationship with growth in students' EGRA scores, we utilize secondary data from all five regions from repeated coaching visits and paired these data with students' EGRA scores.</p> <p>Qualitative data were collected from staff from two implementing partners in the Centre region. Two focus groups were conducted with all regional supervisors from each partner organization (a total of 11 participants). Semistructured interviews were conducted with each of the project coordinators responsible for managing the regional supervisors and overseeing implementation (see table 1 for the summary of the qualitative sample).</p> <p>Table 1. Sample for Qualitative Study Component</p> <p> <ephtml> <table><thead><tr><th style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Implementing Partners</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Participants in Focus Group Discussions</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Participants in Semistructured Interviews</th></tr></thead><tbody><tr><td valign="bottom" rowspan="1" colspan="1">BDE Hinche</td><td valign="bottom" rowspan="1" colspan="1">6</td><td valign="bottom" rowspan="1" colspan="1">1</td></tr><tr><td valign="top" rowspan="1" colspan="1">Summits Education</td><td style="border-bottom: solid thin black" valign="bottom" rowspan="1" colspan="1">7</td><td style="border-bottom: solid thin black" valign="bottom" rowspan="1" colspan="1">1</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Total</td><td valign="bottom" rowspan="1" colspan="1">13</td><td valign="bottom" rowspan="1" colspan="1">2</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <hd id="AN0176222691-14">Measures and Covariates</hd> <p>For the impact evaluation, the primary outcomes are measures of reading skills from the EGRA. The EGRA has been widely used in Haiti and in the field (RTI International [<reflink idref="bib45" id="ref55">45</reflink>]). We use the following subsections: initial sound recognition, initial sound identification, letter identification, words read correctly per minute, invented words read per minute, words read correctly from a story per minute, reading comprehension, and listening comprehension.</p> <p>For the investigation of variation in treatment effects, we utilize data from the classroom observation tool, which aims to capture how well the teachers are adhering to the program and delivering lessons. We constructed indices for use in analysis. All indices are averaged by teacher of the relevant grade, and then by school. These include the following:</p> <p></p> <ulist> <item> <emph>Average number of observations per teacher</emph>: This indicator is an average of how many times all teachers were visited by a coach at that school (range of 1–28).</item> <p></p> <item> <emph>Principal observed the class last week</emph>: This indicator is scaled to the range of 0–1, denoting the proportion of times the teacher was observed by the principal in the previous week.</item> <p></p> <item> <emph>Teaching fidelity and fluency index</emph>: Coaches assess the teachers' implementation of the curriculum in terms of consistency, fidelity, and quality of instruction. Items assess teachers' preparedness for the lesson, delivery of all content, adherence to the prescribed activities, adherence to the script/guide, reading fluency, and perceived comfort in lesson delivery. All items use binary (yes/no) responses. The index has been created by averaging all items. Therefore, the possible range is 0–1, with 1 indicating a perfect score and 0 indicating a failure to meet any items. In our sample, the range of this index is 0.14–1 and the Cronbach's alpha is.64, indicating a decent goodness of fit.</item> <p></p> <item> <emph>Number of lessons behind expected pace</emph>. The program calls for teachers to deliver 150 lessons in 1 year. In our dataset, teachers are behind by a range of 0–55 lessons. Because of the wide variance of this variable, we present our analysis by quartiles.</item> </ulist> <hd id="AN0176222691-15">Data Collection</hd> <p>Data were collected between 2018 and 2020.[<reflink idref="bib5" id="ref56">5</reflink>] In three regions (Nord, Artibonite and Centre), students were assessed at the end of first grade (May 2018) and at the end of second grade (May 2019). Students in these cohorts received 2 years of treatment. Due to implementation delays, students in two regions did not receive treatment until fall 2018 (1 year later). In these regions (Sud and Grand Anse), students received 1 year of treatment and were assessed during the first semester of first grade (November 2018) and at the end of first grade (May 2019).</p> <p>Classroom observations were conducted by coaches throughout the academic year 2018–19. Data from observations were collected electronically via tablets and entered into a form using the CommCare platform.</p> <p>Qualitative data collection was conducted in the fall of 2020. Semistructured interviews and focus groups were conducted via Zoom, and sessions were recorded. Recordings were transcribed and translated by a third party.</p> <hd id="AN0176222691-16">Design</hd> <p></p> <hd id="AN0176222691-17">Impact evaluation</hd> <p>The impact evaluation of the <emph>Read Haiti</emph> program follows a stratified clustered randomized controlled experimental design. The unit of treatment is the school, which is a cluster of students. Program staff provided the research team with a list of eligible schools in Centre, Sud, and Grand Anse regions. Eligibility criteria included participation in target school networks. Well-resourced schools were excluded. Researchers randomly assigned schools in each region to treatment or control groups.</p> <p>To compare treatment and control groups, we conducted tests of means at endline and multivariate OLS regressions that control for student baseline scores. For all analysis, robust standard errors are clustered at the school level.</p> <hd id="AN0176222691-18">Study of variation in treatment effects</hd> <p>We investigate variation in treatment effects using a triangulated mixed-methods design. Quantitative analysis tests the factors in the classroom environment predicting variation in student EGRA scores. We merge EGRA data with classroom observation data and conduct multivariate OLS regressions on the literacy outcomes at endline, controlling for classroom level covariates, student gender, and the students' scores at baseline.</p> <p>The qualitative inquiry elicits key stakeholder perspectives about the mechanisms that explain variation in treatment effects. It uses elements of a multicase, positive deviance design, wherein we identified schools as falling in the top or bottom quartile of growth from the treatment among the 40 schools in each partner network. Growth from the treatment was based on EGRA scores using an index variable of four EGRA subskills from midline to endline.[<reflink idref="bib6" id="ref57">6</reflink>] We shared these high- and low-growth lists with subjects and invited them to consider patterns in high- and low-growth schools and factors that might explain variable treatment effects.</p> <p>All focus group and semistructured interviews were recorded and transcribed. Transcripts were read multiple times and a coding scheme was created based on themes in the data. Interviews were hand-coded into a concept cluster matrix in Excel, and the matrix was refined throughout the coding process. In a final step, the quantitative and qualitative analyses were integrated and compared to generate findings.</p> <hd id="AN0176222691-19">Findings</hd> <p></p> <hd id="AN0176222691-20">Participant Flow and Missing Data Analysis</hd> <p>Figure 1 depicts the participant flow. At baseline, 430 schools were visited and 5,500 students were assessed with the EGRA. At follow-up, 2,076 students from 63 schools (37 percent of the sample) were lost to follow-up.[<reflink idref="bib7" id="ref58">7</reflink>] For classroom observations, 246 schools were visited at least once. Since the analytical design of the observation data includes students' baseline literacy scores as a control, the final dataset includes longitudinal data only. In six instances, classroom observation data could not be merged with the EGRA data due to problems with school IDs.</p> <p>Graph: Fig. 1. Sample frame flow chart. The figure depicts the participant flow for this study across three data collections: baseline (May 2018/November 2018), follow-up (May 2019), and ongoing classroom observations from the 2018–19 academic year. The final box provides the sample for the analytical sample, which is used for all analysis.</p> <p>According to the standards of the What Works Clearinghouse, the level of attrition at the level of the cluster (school) presents a tolerable threat of bias under both optimistic and cautious conditions (Institute of Education Sciences [<reflink idref="bib29" id="ref59">29</reflink>], 116). This is because differential attrition between treatment and control groups is low. At the level of the student, this level of attrition is only acceptable under optimistic assumptions.[<reflink idref="bib8" id="ref60">8</reflink>]</p> <hd id="AN0176222691-21">Mean EGRA Scores</hd> <p>Presented in tables 2 and 3 are the mean values of EGRA scores for the three experimental cohorts—first-grade students in Sud and Grande Anse (table 2), and second-grade students in the Centre Department (table 3). Robust standard errors are listed in parentheses.</p> <p>Table 2. Early Grade Reading Assessment Results: First Grade Cohort (Sud/Grand Anse)</p> <p> <ephtml> <table><thead><tr><th style="border-bottom: solid thin black" rowspan="2" scope="col" colspan="1">Variable</th><th align="center" style="border-bottom: solid thin black" colspan="2" scope="colgroup" rowspan="1">Beginning of First Grade</th><th align="center" style="border-bottom: solid thin black" colspan="2" scope="colgroup" rowspan="1">End of First Grade</th></tr><tr><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Treatment</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Control</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Treatment</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Control</th></tr></thead><tbody><tr><td rowspan="1" colspan="1">Letter names read per minute</td><td char="." rowspan="1" colspan="1">9.72</td><td char="." rowspan="1" colspan="1">9.81</td><td char="." rowspan="1" colspan="1">20.02</td><td char="." rowspan="1" colspan="1">18.96</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.05)</td><td char="." rowspan="1" colspan="1">(1.16)</td><td char="." rowspan="1" colspan="1">(1.25)</td><td char="." rowspan="1" colspan="1">(1.61)</td></tr><tr><td rowspan="1" colspan="1">Letter sounds recognized</td><td char="." rowspan="1" colspan="1">1.73</td><td char="." rowspan="1" colspan="1">1.63</td><td char="." rowspan="1" colspan="1">4.33</td><td char="." rowspan="1" colspan="1">2.33</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.18)</td><td char="." rowspan="1" colspan="1">(.23)</td><td char="." rowspan="1" colspan="1">(.25)</td><td char="." rowspan="1" colspan="1">(.25)</td></tr><tr><td rowspan="1" colspan="1">Letter sounds read per minute</td><td char="." rowspan="1" colspan="1">7.98</td><td char="." rowspan="1" colspan="1">7.62</td><td char="." rowspan="1" colspan="1">17.25</td><td char="." rowspan="1" colspan="1">13.05</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.92)</td><td char="." rowspan="1" colspan="1">(.93)</td><td char="." rowspan="1" colspan="1">(1.11)</td><td char="." rowspan="1" colspan="1">(1.08)</td></tr><tr><td rowspan="1" colspan="1">Familiar words read correctly per minute</td><td char="." rowspan="1" colspan="1">2.65</td><td char="." rowspan="1" colspan="1">3.00</td><td char="." rowspan="1" colspan="1">8.76</td><td char="." rowspan="1" colspan="1">7.75</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.48)</td><td char="." rowspan="1" colspan="1">(.51)</td><td char="." rowspan="1" colspan="1">(.73)</td><td char="." rowspan="1" colspan="1">(.97)</td></tr><tr><td rowspan="1" colspan="1">Invented words read correctly per minute</td><td char="." rowspan="1" colspan="1">1.92</td><td char="." rowspan="1" colspan="1">2.38</td><td char="." rowspan="1" colspan="1">6.63</td><td char="." rowspan="1" colspan="1">6.73</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.41)</td><td char="." rowspan="1" colspan="1">(.43)</td><td char="." rowspan="1" colspan="1">(.70)</td><td char="." rowspan="1" colspan="1">(.88)</td></tr><tr><td rowspan="1" colspan="1">Words read from a story correctly per minute</td><td char="." rowspan="1" colspan="1">3.34</td><td char="." rowspan="1" colspan="1">3.03</td><td char="." rowspan="1" colspan="1">9.78</td><td char="." rowspan="1" colspan="1">10.33</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.94)</td><td char="." rowspan="1" colspan="1">(.59)</td><td char="." rowspan="1" colspan="1">(.98)</td><td char="." rowspan="1" colspan="1">(1.41)</td></tr><tr><td rowspan="1" colspan="1">Creole reading comprehension (percent correct)</td><td char="." rowspan="1" colspan="1">4.91</td><td char="." rowspan="1" colspan="1">4.52</td><td char="." rowspan="1" colspan="1">19.99</td><td char="." rowspan="1" colspan="1">17.50</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.11)</td><td char="." rowspan="1" colspan="1">(1.02)</td><td char="." rowspan="1" colspan="1">(2.03)</td><td char="." rowspan="1" colspan="1">(2.29)</td></tr><tr><td rowspan="2" colspan="1">Creole listening comprehension (questions answered correctly)</td><td char="." rowspan="1" colspan="1">1.99</td><td char="." rowspan="1" colspan="1">2.12</td><td char="." rowspan="1" colspan="1">2.85</td><td char="." rowspan="1" colspan="1">2.82</td></tr><tr><td char="." rowspan="1" colspan="1">(.07)</td><td char="." rowspan="1" colspan="1">(.09)</td><td char="." rowspan="1" colspan="1">(.09)</td><td char="." rowspan="1" colspan="1">(.11)</td></tr><tr><td rowspan="2" colspan="1">French listening comprehension (questions answered correctly)</td><td char="." rowspan="1" colspan="1">.62</td><td char="." rowspan="1" colspan="1">.63</td><td char="." rowspan="1" colspan="1">.97</td><td char="." rowspan="1" colspan="1">1.11</td></tr><tr><td char="." rowspan="1" colspan="1">(.06)</td><td char="." rowspan="1" colspan="1">(.08)</td><td char="." rowspan="1" colspan="1">(.08)</td><td char="." rowspan="1" colspan="1">(.14)</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <p>1 Note. Robust standard errors in parentheses.</p> <p>Table 3. Early Grade Reading Assessment Results: Second-Grade Cohort (Centre)</p> <p> <ephtml> <table><thead><tr><th style="border-bottom: solid thin black" rowspan="2" scope="col" colspan="1">Variable</th><th align="center" style="border-bottom: solid thin black" colspan="2" scope="colgroup" rowspan="1">End of First Grade</th><th align="center" style="border-bottom: solid thin black" colspan="2" scope="colgroup" rowspan="1">End of Second Grade</th></tr><tr><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Treatment</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Control</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Treatment</th><th align="center" style="border-bottom: solid thin black" scope="col" rowspan="1" colspan="1">Control</th></tr></thead><tbody><tr><td rowspan="1" colspan="1">Letter names read per minute</td><td char="." rowspan="1" colspan="1">17.24</td><td char="." rowspan="1" colspan="1">12.59</td><td char="." rowspan="1" colspan="1">31.06</td><td char="." rowspan="1" colspan="1">27.37</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.18)</td><td char="." rowspan="1" colspan="1">(1.47)</td><td char="." rowspan="1" colspan="1">(1.69)</td><td char="." rowspan="1" colspan="1">(2.62)</td></tr><tr><td rowspan="1" colspan="1">Letter sounds recognized</td><td char="." rowspan="1" colspan="1">4.45</td><td char="." rowspan="1" colspan="1">2.06</td><td char="." rowspan="1" colspan="1">7.44</td><td char="." rowspan="1" colspan="1">3.52</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.38)</td><td char="." rowspan="1" colspan="1">(.41)</td><td char="." rowspan="1" colspan="1">(.31)</td><td char="." rowspan="1" colspan="1">(.53)</td></tr><tr><td rowspan="1" colspan="1">Letter sounds read per minute</td><td char="." rowspan="1" colspan="1">18.61</td><td char="." rowspan="1" colspan="1">9.00</td><td char="." rowspan="1" colspan="1">39.32</td><td char="." rowspan="1" colspan="1">18.73</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.53)</td><td char="." rowspan="1" colspan="1">(1.14)</td><td char="." rowspan="1" colspan="1">(2.29)</td><td char="." rowspan="1" colspan="1">(2.35)</td></tr><tr><td rowspan="1" colspan="1">Familiar words read per minute</td><td char="." rowspan="1" colspan="1">5.74</td><td char="." rowspan="1" colspan="1">3.18</td><td char="." rowspan="1" colspan="1">21.95</td><td char="." rowspan="1" colspan="1">13.13</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.92)</td><td char="." rowspan="1" colspan="1">(.86)</td><td char="." rowspan="1" colspan="1">(2.13)</td><td char="." rowspan="1" colspan="1">(2.17)</td></tr><tr><td rowspan="1" colspan="1">Invented words read per minute</td><td char="." rowspan="1" colspan="1">3.68</td><td char="." rowspan="1" colspan="1">2.14</td><td char="." rowspan="1" colspan="1">14.50</td><td char="." rowspan="1" colspan="1">9.35</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(.61)</td><td char="." rowspan="1" colspan="1">(.57)</td><td char="." rowspan="1" colspan="1">(1.55)</td><td char="." rowspan="1" colspan="1">(1.60)</td></tr><tr><td rowspan="1" colspan="1">Words read from a story</td><td char="." rowspan="1" colspan="1">5.75</td><td char="." rowspan="1" colspan="1">4.31</td><td char="." rowspan="1" colspan="1">28.31</td><td char="." rowspan="1" colspan="1">16.96</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.01)</td><td char="." rowspan="1" colspan="1">(1.71)</td><td char="." rowspan="1" colspan="1">(2.92)</td><td char="." rowspan="1" colspan="1">(2.80)</td></tr><tr><td rowspan="1" colspan="1">Creole reading comprehension (percent correct)</td><td char="." rowspan="1" colspan="1">9.54</td><td char="." rowspan="1" colspan="1">7.58</td><td char="." rowspan="1" colspan="1">39.84</td><td char="." rowspan="1" colspan="1">24.07</td></tr><tr><td rowspan="1" colspan="1" /><td char="." rowspan="1" colspan="1">(1.66)</td><td char="." rowspan="1" colspan="1">(3.50)</td><td char="." rowspan="1" colspan="1">(3.34)</td><td char="." rowspan="1" colspan="1">(3.83)</td></tr><tr><td rowspan="2" colspan="1">Creole listening comprehension (questions answered correctly)</td><td char="." rowspan="1" colspan="1">2.12</td><td char="." rowspan="1" colspan="1">2.35</td><td char="." rowspan="1" colspan="1">3.01</td><td char="." rowspan="1" colspan="1">3.00</td></tr><tr><td char="." rowspan="1" colspan="1">(.09)</td><td char="." rowspan="1" colspan="1">(.14)</td><td char="." rowspan="1" colspan="1">(.11)</td><td char="." rowspan="1" colspan="1">(.13)</td></tr><tr><td rowspan="2" colspan="1">French listening comprehension (questions answered correctly)</td><td char="." rowspan="1" colspan="1">.52</td><td char="." rowspan="1" colspan="1">.53</td><td char="." rowspan="1" colspan="1">1.39</td><td char="." rowspan="1" colspan="1">.95</td></tr><tr><td char="." rowspan="1" colspan="1">(.07)</td><td char="." rowspan="1" colspan="1">(.14)</td><td char="." rowspan="1" colspan="1">(.13)</td><td char="." rowspan="1" colspan="1">(.15)</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <p>2 Note. Robust standard errors in parentheses.</p> <hd id="AN0176222691-22">Multivariate Analysis of Impact Evaluation Data</hd> <p>In tables 4 and 5 we present results from a multivariate linear regression analysis of each module of the EGRA assessment. Each regression controls for students' baseline score on that module and students' gender. We also include dummy variables for each region and cluster robust standard errors at the school level.</p> <p>Table 4. Summary Statistics for Classroom Observation Data (<emph>N</emph> = 2,191)</p> <p> <ephtml> <table><thead><tr><th style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Variable</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Mean Value</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Robust Standard Error</th><th style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Range</th></tr></thead><tbody><tr><td valign="bottom" rowspan="1" colspan="1">Count of coaching visits per teacher</td><td char="." valign="bottom" rowspan="1" colspan="1">6.40</td><td char="." valign="bottom" rowspan="1" colspan="1">.215</td><td char="." valign="bottom" rowspan="1" colspan="1">[1, 28]</td></tr><tr><td valign="bottom" rowspan="1" colspan="1">Teacher fidelity and fluency index</td><td char="." valign="bottom" rowspan="1" colspan="1">.81</td><td char="." valign="bottom" rowspan="1" colspan="1">.010</td><td valign="bottom" rowspan="1" colspan="1">[.163, 1]</td></tr><tr><td valign="bottom" rowspan="1" colspan="1">Lessons behind</td><td char="." valign="bottom" rowspan="1" colspan="1">8.11</td><td char="." valign="bottom" rowspan="1" colspan="1">.745</td><td char="." valign="bottom" rowspan="1" colspan="1">[0, 55]</td></tr><tr><td valign="bottom" rowspan="1" colspan="1">Principal observed class</td><td char="." valign="bottom" rowspan="1" colspan="1">.45</td><td char="." valign="bottom" rowspan="1" colspan="1">.025</td><td char="." valign="bottom" rowspan="1" colspan="1">[0, 1]</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <p>Table 5. Multivariate Regression Results</p> <p> <ephtml> <table><thead><tr><th align="left" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Variable</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Treatment OLS Regression Coefficient</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">SE</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1"><italic>R</italic><sup>2</sup></th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1"><italic>N</italic></th></tr></thead><tbody><tr><td valign="bottom" rowspan="1" colspan="1">First-grade cohort (Sud and Grande Anse):</td><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter names read per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">1.96</td><td char="." valign="bottom" rowspan="1" colspan="1">−1.82</td><td char="." valign="bottom" rowspan="1" colspan="1">.15</td><td char="." valign="bottom" rowspan="1" colspan="1">1,905</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter sounds recognized</td><td char="." valign="bottom" rowspan="1" colspan="1">2.02<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−.30</td><td char="." valign="bottom" rowspan="1" colspan="1">.21</td><td char="." valign="bottom" rowspan="1" colspan="1">1,905</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter sounds read per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">5.00<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−1.40</td><td char="." valign="bottom" rowspan="1" colspan="1">.13</td><td char="." valign="bottom" rowspan="1" colspan="1">1,903</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Words read correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">2.08<xref ref-type="table-fn" rid="tfn5">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−.94</td><td char="." valign="bottom" rowspan="1" colspan="1">.27</td><td char="." valign="bottom" rowspan="1" colspan="1">1,902</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Invented words read correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">.84</td><td char="." valign="bottom" rowspan="1" colspan="1">−.95</td><td char="." valign="bottom" rowspan="1" colspan="1">.19</td><td char="." valign="bottom" rowspan="1" colspan="1">1,902</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Words read from a story correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">.56</td><td char="." valign="bottom" rowspan="1" colspan="1">−1.54</td><td char="." valign="bottom" rowspan="1" colspan="1">.13</td><td char="." valign="bottom" rowspan="1" colspan="1">1,902</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Reading comprehension questions answered correctly (percent of questions asked)</td><td char="." valign="bottom" rowspan="1" colspan="1">3.70</td><td char="." valign="bottom" rowspan="1" colspan="1">−2.76</td><td char="." valign="bottom" rowspan="1" colspan="1">.07</td><td char="." valign="bottom" rowspan="1" colspan="1">1,869</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Listening comprehension questions answered correctly (Creole), range 0–5</td><td char="." valign="bottom" rowspan="1" colspan="1">.03</td><td char="." valign="bottom" rowspan="1" colspan="1">−.14</td><td char="." valign="bottom" rowspan="1" colspan="1">.05</td><td char="." valign="bottom" rowspan="1" colspan="1">1,905</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Listening comprehension questions answered correctly (French), range 0–5</td><td char="." valign="bottom" rowspan="1" colspan="1">−.18</td><td char="." valign="bottom" rowspan="1" colspan="1">−.16</td><td char="." valign="bottom" rowspan="1" colspan="1">.03</td><td char="." valign="bottom" rowspan="1" colspan="1">1,905</td></tr><tr><td valign="bottom" rowspan="1" colspan="1">Second-grade cohort (Hinche):</td><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter names read per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">−.46</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.05)</td><td char="." valign="bottom" rowspan="1" colspan="1">.45</td><td char="." valign="bottom" rowspan="1" colspan="1">555</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter sounds recognized</td><td char="." valign="bottom" rowspan="1" colspan="1">3.04<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(.53)</td><td char="." valign="bottom" rowspan="1" colspan="1">.32</td><td char="." valign="bottom" rowspan="1" colspan="1">555</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Letter sounds read per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">13.2<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(2.66)</td><td char="." valign="bottom" rowspan="1" colspan="1">.41</td><td char="." valign="bottom" rowspan="1" colspan="1">554</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Words read correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">5.44<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(1.95)</td><td char="." valign="bottom" rowspan="1" colspan="1">.45</td><td char="." valign="bottom" rowspan="1" colspan="1">554</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Invented words read correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">3.16<xref ref-type="table-fn" rid="tfn5">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(1.47)</td><td char="." valign="bottom" rowspan="1" colspan="1">.40</td><td char="." valign="bottom" rowspan="1" colspan="1">554</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Words read from a story correctly per minute</td><td char="." valign="bottom" rowspan="1" colspan="1">9.88<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(3.44)</td><td char="." valign="bottom" rowspan="1" colspan="1">.28</td><td char="." valign="bottom" rowspan="1" colspan="1">554</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Reading comprehension questions answered correctly (%)</td><td char="." valign="bottom" rowspan="1" colspan="1">14.73<xref ref-type="table-fn" rid="tfn6">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(4.36)</td><td char="." valign="bottom" rowspan="1" colspan="1">.09</td><td char="." valign="bottom" rowspan="1" colspan="1">545</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Listening comprehension questions answered correctly (Creole), range 0–5</td><td char="." valign="bottom" rowspan="1" colspan="1">.07</td><td char="." valign="bottom" rowspan="1" colspan="1">(.16)</td><td char="." valign="bottom" rowspan="1" colspan="1">.11</td><td char="." valign="bottom" rowspan="1" colspan="1">555</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Listening comprehension questions answered correctly (French), range 0–5</td><td char="." valign="bottom" rowspan="1" colspan="1">.44<xref ref-type="table-fn" rid="tfn5">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">(.17)</td><td char="." valign="bottom" rowspan="1" colspan="1">.14</td><td char="." valign="bottom" rowspan="1" colspan="1">555</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <ulist> <item>3 Note. Analysis includes covariates for student baseline score, region, and sex. Robust standard errors are in parentheses.</item> <item>4 * <emph>p</emph> <.1.</item> <item>5 ** <emph>p</emph> <.05.</item> <item>6 *** <emph>p</emph> <.01.</item> </ulist> <p>We observe strong and statistically significant effects of the treatment for both first- and second-grade cohorts. Effects of the program are stronger for the second-grade cohort, which received 2 years of treatment. This cohort realized statistically significant effects in seven of the nine EGRA modules, including two not seen in the first-grade cohort (i.e., invented words read correctly per minute and words read correctly from a story per minute).</p> <p>In figures 2 and 3 we present the effect sizes of these impacts for the first- and second-grade cohorts, respectively. The EGRA sections vary in their range—some sections range from 1 to 50 words read, while others range from 0 to 5 questions answered correctly. Effect sizes, which are defined as the treatment coefficient divided by the pooled standard deviation, can be a useful way to standardize the results and understand the relative impact on different outcome variables of interest. In figures 2 and 3 we observe that the effect sizes range from medium (.05–.23) to large (>.23) (Kraft [<reflink idref="bib32" id="ref61">32</reflink>]; Evans and Yuan [<reflink idref="bib23" id="ref62">23</reflink>]), demonstrating strong effects of the <emph>Read Haiti</emph> program, including, notably, in reading comprehension.</p> <p>Graph: Fig. 2. Effect sizes for first-grade cohort (Sud and Grand Anse Departments). The figure displays effect sizes (regression coefficient divided by pooled standard deviation of treatment and control groups) for all sections of the Early Grade Reading Assessment (EGRA), for the first-grade cohort in the study. Effect sizes above.2 are realized for three sections of the EGRA: initial sounds identified, letter sounds read, and words read correctly per minute. Asterisks indicate that coefficients for these sections are also statistically significant at p<.05.</p> <p>Graph: Fig. 3. Effect sizes for second-grade cohort (Centre Department). The figure displays effect sizes (regression coefficient divided by pooled standard deviation of treatment and control groups) for all sections of the Early Grade Reading Assessment (EGRA), for the second-grade cohort in the study. Effect sizes above.2 are realized for seven out of the nine sections of the EGRA: initial sounds identified, letter sounds read, words read correctly, invented words read, words read from a story, reading comprehension, and listening comprehension. Asterisks indicate that coefficients for these sections are also statistically significant at p<.05.</p> <hd id="AN0176222691-23">Explaining Heterogeneity in Treatment Outcomes</hd> <p>Next, we summarize the evidence from the mixed-methods inquiry into the factors that explain variation in treatment effects, as measured by students' EGRA scores and growth in treated schools, integrating quantitative and qualitative analysis to inform our findings. Table 6 presents the outcomes for the first- and second-grade cohort of the factors identified in the quantitative analysis that predict variation in treatment effects. These are the results of multivariate linear regression analysis, which controls for the students' baseline score, gender, and includes fixed effects by region.</p> <p>Table 6. Multivariate Regression of Classroom Observation and Early Grade Reading Assessment Outcomes</p> <p> <ephtml> <table><thead><tr><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1" /><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Letter Names Read per Minute</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Letter Sounds Read per Minute</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Words Read Correctly per Minute</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Words from a Story Read Correctly per Minute</th><th align="center" style="border-bottom: solid thin black" valign="bottom" scope="col" rowspan="1" colspan="1">Creole Reading Comprehension</th></tr></thead><tbody><tr><td valign="bottom" rowspan="1" colspan="1">First-grade cohort (Sud and Grande Anse):</td><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Count of coaching visits per teacher</td><td char="." valign="bottom" rowspan="1" colspan="1">.34</td><td char="." valign="bottom" rowspan="1" colspan="1">.43</td><td char="." valign="bottom" rowspan="1" colspan="1">.35</td><td char="." valign="bottom" rowspan="1" colspan="1">.33</td><td char="." valign="bottom" rowspan="1" colspan="1">1.04</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(.49)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.45)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.37)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.48)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.91)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Teacher fidelity and fluency index</td><td char="." valign="bottom" rowspan="1" colspan="1">13.47<xref ref-type="table-fn" rid="tfn8">*</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">15.06<xref ref-type="table-fn" rid="tfn8">*</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">11.15<xref ref-type="table-fn" rid="tfn9">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">19.09<xref ref-type="table-fn" rid="tfn10">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">22.31<xref ref-type="table-fn" rid="tfn8">*</xref></td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(7.05)</td><td char="." valign="bottom" rowspan="1" colspan="1">(7.76)</td><td char="." valign="bottom" rowspan="1" colspan="1">(5.05)</td><td char="." valign="bottom" rowspan="1" colspan="1">(7.06)</td><td char="." valign="bottom" rowspan="1" colspan="1">(12.70)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Lessons behind:</td><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Second quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">3.83</td><td char="." valign="bottom" rowspan="1" colspan="1">3.26</td><td char="." valign="bottom" rowspan="1" colspan="1">1.41</td><td char="." valign="bottom" rowspan="1" colspan="1">1.04</td><td char="." valign="bottom" rowspan="1" colspan="1">4.57</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(3.52)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.98)</td><td char="." valign="bottom" rowspan="1" colspan="1">(1.85)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.50)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.78)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Third quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">1.44</td><td char="." valign="bottom" rowspan="1" colspan="1">.37</td><td char="." valign="bottom" rowspan="1" colspan="1">.86</td><td char="." valign="bottom" rowspan="1" colspan="1">.67</td><td char="." valign="bottom" rowspan="1" colspan="1">3.30</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(3.28)</td><td char="." valign="bottom" rowspan="1" colspan="1">(3.07)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.12)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.72)</td><td char="." valign="bottom" rowspan="1" colspan="1">(5.36)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Fourth quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">1.26</td><td char="." valign="bottom" rowspan="1" colspan="1">−1.12</td><td char="." valign="bottom" rowspan="1" colspan="1">.92</td><td char="." valign="bottom" rowspan="1" colspan="1">1.72</td><td char="." valign="bottom" rowspan="1" colspan="1">3.54</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(3.40)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.95)</td><td char="." valign="bottom" rowspan="1" colspan="1">(1.88)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.51)</td><td char="." valign="bottom" rowspan="1" colspan="1">(5.45)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Principal observed class</td><td char="." valign="bottom" rowspan="1" colspan="1">.28</td><td char="." valign="bottom" rowspan="1" colspan="1">−3.12</td><td char="." valign="bottom" rowspan="1" colspan="1">−2.18</td><td char="." valign="bottom" rowspan="1" colspan="1">−3.41</td><td char="." valign="bottom" rowspan="1" colspan="1">−9.10<xref ref-type="table-fn" rid="tfn10">*</xref></td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(3.53)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.79)</td><td char="." valign="bottom" rowspan="1" colspan="1">(1.64)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.33)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.67)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"><italic>R</italic><sup>2</sup></td><td char="." valign="bottom" rowspan="1" colspan="1">.14</td><td char="." valign="bottom" rowspan="1" colspan="1">.13</td><td char="." valign="bottom" rowspan="1" colspan="1">.21</td><td char="." valign="bottom" rowspan="1" colspan="1">.12</td><td char="." valign="bottom" rowspan="1" colspan="1">.09</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"><italic>N</italic></td><td valign="bottom" rowspan="1" colspan="1">1,190</td><td valign="bottom" rowspan="1" colspan="1">1,190</td><td valign="bottom" rowspan="1" colspan="1">1,189</td><td valign="bottom" rowspan="1" colspan="1">1,190</td><td valign="bottom" rowspan="1" colspan="1">1,162</td></tr><tr><td valign="bottom" rowspan="1" colspan="1">Second-grade cohort (Nord, Artibonite, and Centre):</td><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /><td rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Count of coaching visits per teacher</td><td char="." valign="bottom" rowspan="1" colspan="1">.50<xref ref-type="table-fn" rid="tfn9">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">.47</td><td char="." valign="bottom" rowspan="1" colspan="1">.29</td><td char="." valign="bottom" rowspan="1" colspan="1">.29</td><td char="." valign="bottom" rowspan="1" colspan="1">1.11<xref ref-type="table-fn" rid="tfn9">**</xref></td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(.23)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.34)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.25)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.46)</td><td char="." valign="bottom" rowspan="1" colspan="1">(.43)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Teacher fidelity and fluency index</td><td char="." valign="bottom" rowspan="1" colspan="1">4.01</td><td char="." valign="bottom" rowspan="1" colspan="1">12.42<xref ref-type="table-fn" rid="tfn9">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">10.20<xref ref-type="table-fn" rid="tfn10">*</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">20.92<xref ref-type="table-fn" rid="tfn9">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">.69</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(4.95)</td><td char="." valign="bottom" rowspan="1" colspan="1">(5.44)</td><td char="." valign="bottom" rowspan="1" colspan="1">(5.71)</td><td char="." valign="bottom" rowspan="1" colspan="1">(9.06)</td><td char="." valign="bottom" rowspan="1" colspan="1">(11.57)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Lessons behind:</td><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /><td valign="bottom" rowspan="1" colspan="1" /></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Second quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">−2.94</td><td char="." valign="bottom" rowspan="1" colspan="1">−5.58<xref ref-type="table-fn" rid="tfn10">*</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−2.25</td><td char="." valign="bottom" rowspan="1" colspan="1">−4.21</td><td char="." valign="bottom" rowspan="1" colspan="1">.47</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(2.21)</td><td char="." valign="bottom" rowspan="1" colspan="1">(3.25)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.70)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.13)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.48)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Third quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">.38</td><td char="." valign="bottom" rowspan="1" colspan="1">−5.62<xref ref-type="table-fn" rid="tfn9">**</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−2.58</td><td char="." valign="bottom" rowspan="1" colspan="1">−4.49</td><td char="." valign="bottom" rowspan="1" colspan="1">−.81</td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(2.26)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.78)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.19)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.03)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.66)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Fourth quartile</td><td char="." valign="bottom" rowspan="1" colspan="1">−6.89<xref ref-type="table-fn" rid="tfn8">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−13.29<xref ref-type="table-fn" rid="tfn8">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−10.82<xref ref-type="table-fn" rid="tfn8">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−14.98<xref ref-type="table-fn" rid="tfn8">***</xref></td><td char="." valign="bottom" rowspan="1" colspan="1">−12.42<xref ref-type="table-fn" rid="tfn9">**</xref></td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(1.99)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.85)</td><td char="." valign="bottom" rowspan="1" colspan="1">(1.99)</td><td char="." valign="bottom" rowspan="1" colspan="1">(3.66)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.96)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"> Principal observed class</td><td char="." valign="bottom" rowspan="1" colspan="1">−.64</td><td char="." valign="bottom" rowspan="1" colspan="1">1.31</td><td char="." valign="bottom" rowspan="1" colspan="1">2.85</td><td char="." valign="bottom" rowspan="1" colspan="1">3.55</td><td char="." valign="bottom" rowspan="1" colspan="1">8.58<xref ref-type="table-fn" rid="tfn9">**</xref></td></tr><tr><td valign="bottom" rowspan="1" colspan="1" /><td char="." valign="bottom" rowspan="1" colspan="1">(1.63)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.47)</td><td char="." valign="bottom" rowspan="1" colspan="1">(2.14)</td><td char="." valign="bottom" rowspan="1" colspan="1">(3.33)</td><td char="." valign="bottom" rowspan="1" colspan="1">(4.18)</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"><italic>R</italic><sup>2</sup></td><td char="." valign="bottom" rowspan="1" colspan="1">.40</td><td char="." valign="bottom" rowspan="1" colspan="1">.41</td><td char="." valign="bottom" rowspan="1" colspan="1">.49</td><td char="." valign="bottom" rowspan="1" colspan="1">.35</td><td char="." valign="bottom" rowspan="1" colspan="1">.13</td></tr><tr><td valign="bottom" rowspan="1" colspan="1"><italic>N</italic></td><td valign="bottom" rowspan="1" colspan="1">1,001</td><td valign="bottom" rowspan="1" colspan="1">1,001</td><td valign="bottom" rowspan="1" colspan="1">1,001</td><td valign="bottom" rowspan="1" colspan="1">1,001</td><td valign="bottom" rowspan="1" colspan="1">977</td></tr></tbody></table> </ephtml> </p> <p>Graph</p> <ulist> <item>7 Note. Covariates not shown: student baseline scores, student gender, and classroom size. Fixed effects for region and robust standard errors clustered by school and included in parentheses.</item> <item>8 * <emph>p</emph> <.1.</item> <item>9 ** <emph>p</emph> <.05.</item> <item>10 *** <emph>p</emph> <.01</item> </ulist> <hd id="AN0176222691-24">Variability Related to Implementation and Monitoring</hd> <p>We found limited evidence of the importance of monitoring and instructional coaching visits as an important implementation function. Higher frequency of supervisor visits predicts higher growth in students' knowledge of letter names ( <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>β</mi><mo>=</mo><mn>.5</mn></mrow></math> </ephtml> , <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">p</mi><mo><</mo><mn>.05</mn></mrow></math> </ephtml> ) and Creole reading comprehension scores ( <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>β</mi><mo>=</mo><mn>1.11</mn></mrow></math> </ephtml> , <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">p</mi><mo><</mo><mn>.05</mn></mrow></math> </ephtml> ) in the second-grade cohort but do not show statistically significant relationships in other EGRA subskills in either the first- or second-grade cohorts. Yet, respondents affirmed the importance of instructional coaches and the regularity of their visits. The coordinators, for example, noted variation in school performance by the dosage of coaching, explaining: "For the low-performing schools, we notice a supervisor who has a lot of schools they are responsible for and with the distance between schools, it becomes difficult." This suggests that a greater supervision load combined with great distances impeded dosage of coaching, which had a perceived effect on treatment impactfulness.</p> <p>Quantitative data also examined the relationship between the regularity of principal observations of classroom instruction and growth in student EGRA scores but found no consistent relationships across most EGRA subskills. The exceptions were a statistically significant positive relationship with Creole reading comprehension in the second-grade cohort ( <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>β</mi><mo>=</mo><mn>8.58</mn></mrow></math> </ephtml> , <ephtml> <math overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">p</mi><mo><</mo><mn>.05</mn></mrow></math> </ephtml> ) and a curious negative relationship in grade 1 reading comprehension, even if only significant at the 10 percent level.</p> <p>There was a relative absence of qualitative evidence supporting the program's theory of change that school leaders' observation of classes would enhance teachers' fidelity of implementation and quality instruction resulting in higher student outcomes. Respondents rarely mentioned school leaders' role in instructional coaching as a key factor. This seems to fit the limited quantitative evidence for the salience of this practice in explaining variation in treatment effects. As explored below, it appears that leaders are important but that different leadership behaviors are more impactful.</p> <hd id="AN0176222691-25">Variability Related to Implementing Organizations: Dosage, Uptake, and Opportunity to Learn</hd> <p>The primary sources of variation in outcomes between implementing organizations identified in our study are at the school level. Both quantitative and qualitative evidence emphasize the dosage of students' exposure to the treatment as a primary factor explaining treatment effects. Quantitative findings point to the number of lessons behind, a dosage measure, as a key predictor of lower growth in EGRA scores. Key factors identified in interviews also focused on dosage of treatment to students, including student absenteeism, teacher absenteeism, school closure, and partial take up of the curriculum. Key factors identified by respondents were related to effective teacher instructional delivery, including the talent level of teachers, their level of commitment, and their preparation of lessons. Finally, qualitative findings point to school leadership as a key factor, particularly behaviors linked to protecting opportunity to learn (e.g., mitigating school closure, teacher and student absenteeism, and time on task) and fostering teachers' commitment to the program. The evidence for each of these factors are summarized below in greater detail.</p> <hd id="AN0176222691-26">Lessons behind</hd> <p>The second-grade cohort had a consistent negative relationship between the number of lessons the teacher is behind and students' performance on all EGRA subskills. These negative relationships were large in magnitude and statistically significant for the bottom quartile, representing schools in which teachers were furthest behind in the proportion of lessons completed. Lessons behind in the second-grade cohort was among the most consistent quantitative predictors of students' growth in all literacy skills.</p> <p>Stakeholder responses confirm and help explain this pacing finding. The most frequently cited issues explaining schools' relatively lower growth relate to barriers to uptake, low opportunity to learn, and lost instructional time, all of which could affect lessons behind and are described below.</p> <hd id="AN0176222691-27">Take-up</hd> <p>A key dosage issue involved whether schools and teachers exclusively adopted the <emph>Read Haiti</emph> literacy materials. Respondents frequently reported that some schools used a mix of program materials and local textbooks that had been previously purchased or with which teachers were familiar. One respondent summarized: "It is always a challenge to have schools adopt the programs they are given, for many reasons ... it is always a fight with the director of the school, the staff, the parents, to convince them of the necessity to exclusively adopt the program." Another participant affirmed this point: "The schools that are rated the lowest ... were resistant to change, resistant to perform differently, to follow the guidelines of the pedagogical advisers."</p> <p>The result of reluctance to change materials was that some schools and teachers completed far fewer <emph>Read Haiti</emph> lessons. A coordinator explained: "Schools use the manuals available on the Haitian market ... which schools are accustomed to. And when you come with the <emph>Read Haiti</emph> program, the schools will continue using the old manuals. They use the <emph>Read Haiti</emph> program three times a week and other manuals two times a week." Here a coordinator explains that some schools alternated use of the new program materials and continued teaching from other textbooks found in the local Haitian market with which teachers were more familiar.</p> <p>Respondents explain that this use of two sets of materials led teachers to fall behind the target of 150 <emph>Read Haiti</emph> lessons, with some completing only 25–40 percent of lessons and respondents regard this phenomenon as a key factor explaining the pacing gaps identified. A coordinator summarized: "If the school is functioning well, the students are attending, the staff is attending, and you are supposed to be in your hundredth lesson or more and you are around the thirtieth or fortieth, it means the other days, you have been doing something else." One coordinator observed that eight of the 10 top-performing schools in the network exclusively used the <emph>Read Haiti</emph> program, while most of the lowest-performing treatment schools used the <emph>Read Haiti</emph> materials alongside another local textbook, supporting the finding that partial uptake of materials explained variation in treatment effects.</p> <hd id="AN0176222691-28">Instructional time and opportunity to learn</hd> <p>Another key dosage issue pertained to instructional time. Students who received less instructional time also received less exposure to the <emph>Read Haiti</emph> program (and instruction in general), and this was regarded by respondents as a major factor explaining variation in treatment effects. A coordinator summarized a number of interrelated issues as follows:</p> <p>For the low-performing schools, we notice the irregularity of students' [attendance]. In some schools, we have a group of students who show and the following week there is another group. We have issues of the financial conditions of parents, students who have to go to the market or gardening with their parents. Some parents do not take seriously that school matters for their children. If the child doesn't go to school, [they say,] "That is not their cup of tea." It makes students miss classes. We also have the distance. There are some students who come from very far away, and when it rains, the roads are not passable for students to come to school.</p> <p>Here the coordinator explains that there are a combination of factors, all related to barriers to opportunity to learn, that explain much of the variation between high- and low-performing schools.</p> <p>Coaches in focus groups often cited high teacher and student absenteeism and inadequate hours of time on task in class as key barriers to opportunity to learn that impeded program effects. They noted, for example, "the regularity of teachers and students in class," "the number of days the students spend in class," "classes given regularly by teachers," and "instruction is given on a regular basis" as key factors.</p> <p>Multiple respondents also reported that the lowest-performing schools in the sample were frequently closed. Respondents indicated that this was linked to teacher financial insecurity, where teachers missed some days to "go to market" to supplement their incomes. One coordinator described visiting the lowest-performing school in the network and learning that the school was closed every Thursday (market day) and Friday. He concluded: "The worst result is explained by the fact that teachers are not motivated, not available, and they do not go to class regularly to teach students."</p> <hd id="AN0176222691-29">Parental engagement and support</hd> <p>Qualitative data indicated that schools varied in the degree to which parents were engaged in and supportive of their children's learning, which respondents perceived as explaining some variation in treatment effects. Parents in some communities were perceived to emphasize the value of education more, to better ensure their children's regular attendance, to provide more learning support at home, and to hold schools accountable for performance. Other communities were known to have high rates of student absenteeism. For example, one coordinator noted that remote schools near the border with the Dominican Republic had lower performance due to low student attendance because parents in these schools frequently took their children out of school for days at a time when traveling across the border for work. Observing this pattern prompted this coordinator to suggest parental engagement to support student attendance as a consideration in program design.</p> <hd id="AN0176222691-30">Instructional Quality</hd> <p>Quality instructional delivery was another source of variation in treatment effects. In the quantitative analysis, we observe a consistent and statistically significant relationship between the teacher fidelity and fluency index and EGRA scores across most skills in both cohorts. A change in teacher fidelity and fluency from 0 to 1 predicts 19–20 more words read correctly from a story among both cohorts. The fidelity and fluency index seeks to measure both the fidelity of implementation during a given lesson and the quality of delivery.</p> <p>The qualitative inquiry similarly pointed to teacher engagement, commitment, and skill as key factors explaining differences between higher and lower-performing treatment schools. Respondents described teachers in high-growth schools as having greater instructional abilities, higher levels of commitment to teaching and to the <emph>Read Haiti</emph> program, better preparation for lessons, and engaging in more collaborative working relationships. For example, in their initial response regarding what separated the highest-performing schools from the lowest, a focus group respondent explained: "There are practices regarding teachers, teachers who are more practical with the program, teachers who are regular and apply the program or the way the course is being dispensed in one classroom, teachers who always prepare their lesson-plans."</p> <p>Another respondent noted that "the training or the background of teachers contribute to the success of students" as another salient factor. The inverse, when skills and training levels were lacking, was also perceived to be a barrier to program effectiveness. Teacher skills were reported to be a gap in more remote schools, which tended to have lower growth. These schools were reported to struggle to recruit and retain qualified teachers. These perceptions affirm and contextualize the relationship observed between the teacher behaviors measured in the fidelity and fluency index and students' EGRA scores.</p> <hd id="AN0176222691-31">School Leadership</hd> <p>A third key area related to school characteristics for which there was mixed quantitative evidence and substantial qualitative evidence is the role of school leaders in supporting program implementation. Respondents frequently cited the commitment and effectiveness of school leaders as a key explanatory factor influencing variation in treatment effects.</p> <p>Both coordinators emphasized the importance of school leadership in their interviews as a key distinguishing factor. One coordinator pointed to leadership as the top explanatory factor distinguishing high from low-performing schools: "These [highest-growth schools] are schools where teachers are highly motivated ... the directors of these schools are involved in activities. For Catholic schools, in addition to the involvement of the directors, we also have the involvement of the priest that makes the difference in those schools." This coordinator went on to offer an example of a very remote school that he was surprised to see on the high-growth list where a highly involved priest appeared to be accounting for its surprisingly strong performance.</p> <p>A coach in a focus group similarly explained the key role of school leaders: "The director in [high-growth school X] supervises, encourages, and motivates the teachers. At [high-growth school Y], the director is always around the teachers, they discuss from time to time, show up during class and when a teacher needs something, he provides." This excerpt underscores the perceived importance of the school leader in fostering close collaborative relationships with teachers and among teachers and in providing supervision, encouragement, and support. Respondents indicated that leaders were particularly influential in garnering the commitment and engagement of teachers to participate fully in the program. They also played key roles in supporting teachers and ensuring their effective and consistent use of the program materials and methods. Responses indicated that leaders set the tone regarding the importance of the program in how they communicated and set staff expectations for engagement.</p> <p>Respondents also described some of the highest-growth schools as "strong institutions," which they attributed largely to leadership quality. They described these schools as having supportive programs for teachers and parents, fostering supportive working and learning environments, and regularly paying teachers. In contrast, respondents indicated that ineffective leadership was sometimes a barrier to implementation. Some schools were described as having leaders and staff who were resistant to change, did not listen to the advice of the supervisors, and struggled in various performance areas. These factors tended to impede program uptake and fidelity, resulting in weaker treatment effects.</p> <p>Additionally, two key leadership behaviors emerged in the qualitative data as associated with higher performance, namely, leaders facilitating structures for student academic support and their efforts to protect opportunity to learn. The program design included some elements of remedial student support via summer camps and after-school programs. Some respondents pointed to school leaders that embraced these norms of extra student support and facilitated additional support mechanisms beyond what was suggested by the program. For example, some schools implemented "recovery classes" during school hours, "pulling out" learners who were falling behind for targeted instruction to address skill gaps. Other principals facilitated make-up lessons for any students who had been absent for multiple days. Finally, respondents noted that leaders in some higher-performing schools would contact parents when students were absent "to motivate them" to support their children to attend school or organize meetings with parents to encourage student attendance. For example, at one high-growth school, a supervisor in one of the focus groups explained that the school previously never had good results, and "the students can read now." At this school, "the director will call parents on the phone when a student misses classes or even go to parents if he has to." The implication is that the commitment of the school director was key in the noteworthy success of this school.</p> <hd id="AN0176222691-32">Remote Schools</hd> <p>A final context-related factor cited by respondents was the degree of rurality or remoteness of schools. Respondents observed that the majority of the highest-growth schools were urban, peri-urban, or relatively close to towns or accessible by roads, while most of the lowest-growth schools were described as "remote," "inaccessible," and "very rural." Rural schools were reported to experience lower rates of teacher and student attendance. They were also more difficult for coaches to access, resulting in fewer monitoring and coaching visits. This suggests that some of the implementation challenges and dosage issues described earlier may be exacerbated in remote school contexts.</p> <p>Schools in rural areas, however, are also likely to have students of lower SES, which might explain some variation in treatment effects as a student background moderator. Finally, rural schools were reported to face difficulty recruiting and retaining qualified teachers, which suggests professional capacity as another possible reason for remote schools' lower performance linked to instructional quality.</p> <hd id="AN0176222691-33">Discussion</hd> <p>This study provides causal evidence of the effectiveness of a multilingual structured reading program with accompanying teacher training, coaching, support, teaching and learning materials, and targeted use of classroom libraries in fostering improved reading outcomes in Haiti, confirming and extending findings from a previous study (Guzmán et al. [<reflink idref="bib26" id="ref63">26</reflink>]). The study demonstrates that such interventions can be effective at a moderate scale (350 schools) and in different regions of Haiti. These findings add to a growing body of evidence documenting the effectiveness of structured multilingual early grade reading approaches in low-income contexts (Evans and Popova [<reflink idref="bib22" id="ref64">22</reflink>]; Kim et al. [<reflink idref="bib30" id="ref65">30</reflink>]). Importantly, this study also demonstrates that such programs can significantly improve students' reading comprehension by the end of second grade, an outcome rarely seen in the literature (Kim et al. [<reflink idref="bib30" id="ref66">30</reflink>]). We hypothesize that these comprehension effects are linked to new program design elements not included in previous versions of the intervention in Haiti (e.g., the additional focus on teaching practices that promote reading comprehension) as well as implementation factors discussed below.</p> <p>The mixed-methods study design offers unique contributions to understanding implementation, organizational, and contextual factors that help explain variation in treatment effects. While much of the extant scholarship has focused on variations in treatment design linked to program efficacy (Piper, Zuilkowski, et al. [<reflink idref="bib40" id="ref67">40</reflink>]; Stern et al. [<reflink idref="bib49" id="ref68">49</reflink>]), our study identifies sources of variation in treatment effects within a given effective treatment design. It points to key aspects of implementation and apparent mechanisms related to schools and contexts. Key factors include the dosage of the treatment that students receive (related to uptake, opportunity to learn, and parental support), implementation fidelity and quality in classroom instruction, key leadership behaviors (including building teacher and community support for the program, remedial student support, and efforts to protect opportunity to learn), and dosage of coaching. More research should explore the role of school leaders as instructional leaders in contexts like Haiti.</p> <p>The strongest evidence points to dosage and implementation fidelity of the instructional program to students, which are related to opportunity to learn and the role of school leaders. A primary quantitative dosage or fidelity indicator is the percentage of target lessons completed, which consistently predicted variation in treatment effects. This supports and extends evidence of the importance of dosage and fidelity in EGR programs (Christina and Vinogradova [<reflink idref="bib15" id="ref69">15</reflink>]). Notably, however, we use a more precise measure of fidelity than is typical in EGR studies, focusing on the fidelity of lesson delivery rather than coaching fidelity, a more precise indicator of treatment fidelity on the treated. The qualitative evidence points to reasons for variability in lesson pacing, namely, barriers to opportunity to learn and the exclusive uptake of the curriculum versus mixed use of new and old books as primary sources of variation. Our pacing measure (percentage of target lessons completed) is a powerful leading indicator that can be used for early feedback to foster program uptake and targeted support to schools and teachers. Overemphasis, however, at the exclusion of quality instruction could incentivize superficial coverage (Fleisch [<reflink idref="bib24" id="ref70">24</reflink>]).</p> <p>Barriers to opportunity to learn also influenced relative levels of treatment exposure and included the frequency of school closure, teacher absenteeism, student absenteeism, and the remoteness of schools. This affirms a wide body of literature on the salience of instructional time and absenteeism in predicting student outcomes (Abadzi [<reflink idref="bib1" id="ref71">1</reflink>]) and a smaller literature on its influence on treatment effects (Christina and Vinogradova [<reflink idref="bib15" id="ref72">15</reflink>]) and the importance of instructional time as a key mechanism in successful EGR programs (Stern et al. [<reflink idref="bib49" id="ref73">49</reflink>]). It suggests that barriers to instructional time, a widespread issue in low- and middle-income countries (Bruns and Luque [<reflink idref="bib10" id="ref74">10</reflink>]; Bold et al. [<reflink idref="bib6" id="ref75">6</reflink>]), impede the effectiveness of EGR programs. The literature on EGR programs emphasizes the importance of ensuring adequate time for reading in the curriculum but has paid little attention to underlying barriers to opportunity to learn (Burchfield et al. [<reflink idref="bib13" id="ref76">13</reflink>]; Raupp et al. [<reflink idref="bib42" id="ref77">42</reflink>]). This is a glaring omission in EGR program design and related scholarship.</p> <p>The issue of remote and rural schools having weaker treatment effects reported by participants has been identified in some studies of similar interventions, such as in Ghana and South Africa (Taylor et al. [<reflink idref="bib50" id="ref78">50</reflink>]; USAID [<reflink idref="bib52" id="ref79">52</reflink>]). Respondents indicated that lower growth in remote schools may be due to instructional time loss, particularly low-student attendance linked to parents not prioritizing or valuing school, low SES of remote communities, and lower human capital of teaching staff. Future research should seek to better understand the factors impeding treatment effects in remote schools to inform program accommodations. For example, communicating to parents about the returns to education in terms of future earnings has significantly and cost effectively boosted student attendance and learning outcomes in similar contexts (World Bank [<reflink idref="bib55" id="ref80">55</reflink>]), and such approaches could be added to the treatment for rural schools to enhance student attendance.</p> <p>Our findings also affirm the importance of teachers faithfully and effectively delivering the program. Respondents explained that variation in instructional fidelity and quality were related to teachers' level of commitment to the program, their degree of preparation prior to delivering the lessons, and their skill levels, for example, in reading clearly and in a manner that engages students in the lessons. The salience of teacher motivation and effort as a source of variation in treatment effects is related to research linking low-teacher effort in many low-income contexts to norms and structures of accountability and the organizational culture in schools (Sabarwal and Abu-Jawdeh [<reflink idref="bib47" id="ref81">47</reflink>]; World Bank [<reflink idref="bib54" id="ref82">54</reflink>]), suggesting another area for further inquiry and program reform.</p> <p>We also found strong qualitative evidence pointing to the role of school leaders, though critically, not always in the ways envisioned by the program's theory of change. These findings point to three primary factors, namely, the role of the school leaders in (<reflink idref="bib1" id="ref83">1</reflink>) facilitating uptake and teacher commitment to the program, (<reflink idref="bib2" id="ref84">2</reflink>) protecting opportunity to learn through parental engagement, and (<reflink idref="bib3" id="ref85">3</reflink>) facilitating differentiated student support.</p> <p>The finding that effective school leadership fosters teacher effort and performance is supported by a growing body of international education research (Bloom et al. [<reflink idref="bib5" id="ref86">5</reflink>]; Leaver et al. [<reflink idref="bib33" id="ref87">33</reflink>]; Adelman and Lemos [<reflink idref="bib2" id="ref88">2</reflink>]). Principals, founders, and pastors were regarded as key actors that influenced teachers' commitment to the program. This suggests possible benefits of engaging local faith leaders in faith-affiliated schools (D'Agostino et al. [<reflink idref="bib17" id="ref89">17</reflink>]) and affirms the role of school leaders in shaping organizational culture and norms (Murphy and Torre [<reflink idref="bib36" id="ref90">36</reflink>]). It also adds to research suggesting a key role for school leaders in explaining variation in treatment effects (WFP [<reflink idref="bib53" id="ref91">53</reflink>]) and suggests the importance of including school leaders in training and key program activities and seeking to identify and spread effective leadership behaviors.</p> <p>For example, the qualitative evidence points to leaders' role in protecting opportunity to learn (e.g., encouraging student and teacher attendance) and providing student support through facilitating make-up lessons for absent students. These findings are consistent with research affirming the benefits of targeted, differentiated student supports (Banerjee et al. [<reflink idref="bib3" id="ref92">3</reflink>], [<reflink idref="bib4" id="ref93">4</reflink>]; World Bank [<reflink idref="bib55" id="ref94">55</reflink>]) and the importance of school management in ensuring performance (Bloom et al. [<reflink idref="bib5" id="ref95">5</reflink>]). These are also replicable behaviors with possible program design implications.</p> <p>Critically, we find an emphasis on the importance of school leadership as a critical factor, but failed to find consistent evidence that school leader's instructional coaching improved most literacy skills, other than a positive relationship with reading comprehension for the grade 2 cohort and limited qualitative evidence. Though research from the United States suggests that instructional leadership is a primary driver of principals' indirect effect on student learning (Robinson et al. [<reflink idref="bib44" id="ref96">44</reflink>]), we found little evidence to support this. This may suggest that Haitian school leaders lack the skills or training to serve as instructional coaches. Or perhaps more plausibly, in contexts like Haiti, where the basic functionality of schools is highly variable, protecting opportunity to learn is a higher-yield focus of school leaders' time and attention than instructional leadership, perhaps suggesting a "hierarchy of needs" for school leaders. These findings suggest the need for more study and programmatic attention on the role of school leaders in EGR programs in LMICs.</p> <p>Finally, there is some evidence about which factors predict higher reading comprehension scores, particularly noteworthy given how rarely reading comprehension effects are found in similar interventions (Kim et al. [<reflink idref="bib30" id="ref97">30</reflink>]). Quantitative findings point to the following factors that predicted higher reading comprehension: in the first-grade cohort, teacher fidelity and fluency and, in the second-grade cohort, the number of lessons behind, the frequency of supervisor visits, and the frequency of principal observation visits. These results suggest the basic efficacy of the curriculum and point to dosage and coaching as primary factors explaining reading comprehension gains. The relationship to coaching may be particularly relevant given the program's emphasis in the second grade on unscripted instructional practices aimed at fostering reading comprehension. These findings provide suggestive evidence that reading comprehension gains are linked to support of teachers in adopting instructional practices, like interactive read-alouds, facilitating independent student reading practice with leveled texts, and engaging students in discussion about texts (Duke and Pearson [<reflink idref="bib21" id="ref98">21</reflink>]; Lennox [<reflink idref="bib34" id="ref99">34</reflink>]). These practices are not easily scripted, require extensive job-embedded coaching, and complement the decoding skills more easily acquired in scripted curricula.</p> <p>There are a number of limitations to note that future research should consider. In our analysis we were not able to link students to particular teachers and instead collapsed the classroom observation data to the respective grade level. A preferred approach may be to nest the data (school → teacher → student) to allow a more precise analysis. Additionally, the level of attrition at the student level is acceptable only under optimistic assumptions. Further analysis demonstrated that student-level attrition relates to region and student gender (girls were more likely to be found at follow-up). This suggests potential bias in our study.</p> <p>We lacked adequate quantitative measures of key instructional practices potentially linked to reading comprehension gains and also lacked robust measures of schools' use of out of school remediation interventions (after-school and summer). This limited our ability to explore these as possible mechanisms.</p> <p>The lack of classroom observation data in the control schools also limited our ability to use moderation analysis to draw causal inferences from the classroom observation data. With the current data, it is not possible to analyze if and how the environmental and behavioral factors associated with variation in student outcomes may have been affected by the treatment. Future research should also seek to include more student background characteristics to further explore student-level mediators and moderators. Finally, given our concurrent design, various findings emerging in the qualitative research could not be tested quantitatively, but future research can and should explore these factors further.</p> <hd id="AN0176222691-34">Conclusion</hd> <p>This study makes a significant contribution to understanding the factors that explain variation in treatment effects in an impactful early grade reading intervention in a fragile and low-income context. It provides a rare, mixed-methods design that combines a randomized controlled trial with a robust exploration of the factors explaining variation in treatment effects.</p> <p>Key findings emphasize the importance of dosage (related primarily to uptake issues and student and teacher absenteeism), challenges in remote schools, the key role of school leaders in protecting opportunity to learn and facilitating differentiated student supports, and affirms the importance of adequate instructional coaching and teacher-support, perhaps especially for teachers' adoption of unscripted instructional practices important for reading comprehension. We strongly encourage more research of this kind focused on variation in program effects to aid efforts to continue to refine and improve the effectiveness of promising education interventions in low- and middle-income countries.</p> <ref id="AN0176222691-35"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref30" type="bt">1</bibl> <bibtext> TJ D'Agostino, is a faculty member of the Pulte Institute for Global Development in the Keough School for Global Affairs at the University of Notre Dame. His scholarship focuses on comparative education policy, early grade literacy, multilingual education, faith-based schools, and systems improvement. Danice Brown Guzmán is associate director of evidence and learning at the Pulte Institute for Global Developmen, at the University of Notre Dame. She holds a master's degree in public policy analysis from Duke University. She has conducted impact evaluations across various sectors and regions of the world. Anasthasie N. Liberiste-Osirus, who specializes in foundational biliteracy, is a training and technical assistance consultant with a 20-year track record in advancing language and literacy education. Her approach combines research, teacher training, and youth and family engagement to empower communities in Haiti, Benin, Botswana, Nigeria, Lesotho, and the United States. Paul Perrin is the director of evidence and learning at the Pulte Institute for Global Development and an associate professor of the practice at the Keough School of Global Affairs, University of Notre Dame. Perrin completed his PhD in International Health at Johns Hopkins University. Kate Schuenke-Lucien is the director for Haiti and senior associate director for strategic planning at the University of Notre Dame's Global Center for the Development of the Whole Child. Her research focuses on improving life outcomes for vulnerable children and families globally.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref19" type="bt">2</bibl> <bibtext> Piper and Zuilkowski ([39]); Christina and Vinogradova ([15]); Piper, Zuilkowski, et al. ([40]); Stern et al. ([49]).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref20" type="bt">3</bibl> <bibtext> Piper and Zuilkowski ([39]); Christina and Vinogradova ([15]); Taylor et al. ([50]); Brunette et al. ([9]).</bibtext> </blist> <blist> <bibl id="bib4" idref="ref21" type="bt">4</bibl> <bibtext> Data from two regions were excluded from the impact evaluation analysis because schools were not randomly assigned in these regions.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref46" type="bt">5</bibl> <bibtext> Given random assignment and the sample size, equivalence was assumed between treatment and control groups and baseline assessment was not conducted at the beginning of first grade.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref38" type="bt">6</bibl> <bibtext> The four subskills were: letter recognition, words read per minute, words read in a story (Creole) and listening comprehension (French). These were selected because they were the indicators where impact was greatest at endline and for the range of skills aligned to program goals. The variables were converted to a "percent correct" indicator, and averaged to create the index.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref33" type="bt">7</bibl> <bibtext> Analysis of missingness of the data demonstrated that student-level attrition relates to region and student gender (girls were more likely to be found at follow-up). See further discussion in the limitations section.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref7" type="bt">8</bibl> <bibtext> Optimistic assumptions imply that the reason for attrition is exogenous to treatment. In our case, loss of students due to follow-up is likely due to students' mobility and irregular school attendance. Qualitative data suggest that these issues are present in treatment and control schools. As such, we may utilize optimistic assumptions for this analysis.</bibtext> </blist> </ref> <ref id="AN0176222691-36"> <title> References </title> <blist> <bibtext> Abadzi, H. 2007. "Absenteeism and Beyond: Instructional Time Loss and Consequences." World Bank Policy Research Working Paper no. 4376, World Bank, Washington, DC.</bibtext> </blist> <blist> <bibtext> Adelman, M., and R. Lemos. 2021. Managing for Learning: Measuring and Strengthening Education Management in Latin America and the Caribbean. International Development in Focus. Washington, DC: World Bank.</bibtext> </blist> <blist> <bibtext> Banerjee, A., R. Banerji, J. Berry, E. Duflo, H. Kannan, S. Mukerji, M. Shotland, and M. Walton. 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of 'Teaching at the Right Level' in India." NBER Working Paper no. 22746, National Bureau of Economic Research, Cambridge, MA.</bibtext> </blist> <blist> <bibtext> Banerjee, A., R. Banerji, J. Berry, E. Duflo, H. Kannan, S. Mukerji, M. Shotland, and M. Walton. 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application." Journal of Economic Perspectives 31 (4): 73–102.</bibtext> </blist> <blist> <bibtext> Bloom, N., R. Lemos, R. Sadun, and J. Van Reenen. 2015. "Does Management Matter in Schools?" Economic Journal 125 (584): 647–74.</bibtext> </blist> <blist> <bibtext> Bold, T., D. Filmer, G. Martin, E. Molina, B. Stacy, C. Rockmore, J. Svensson, and W. Wane. 2017. "Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa." Journal of Economic Perspectives 31 (4): 185–204.</bibtext> </blist> <blist> <bibtext> Bold, T., M. Kimenyi, G. Mwabu, A. Ng'ang'a, and J. Sandefur. 2013. "Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education." Center for Global Development Working Paper no. 321, Center for Global Development, Washington, DC.</bibtext> </blist> <blist> <bibtext> Bold, T., M. Kimenyi, G. Mwabu, and J. Sandefur. 2018. "Experimental Evidence on Scaling Up Education Reforms in Kenya." Journal of Public Economics 168:1–20.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref15" type="bt">9</bibl> <bibtext> Brunette, T., B. Piper, R. Jordan, S. King, and R. Nabacwa. 2019. "The Impact of Mother Tongue Reading Instruction in Twelve Ugandan Languages and the Role of Language Complexity, Socioeconomic Factors, and Program Implementation." Comparative Education Review 63 (4): 591–612.</bibtext> </blist> <blist> <bibtext> Bruns, B., and J. Luque. 2014. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Washington, DC: World Bank.</bibtext> </blist> <blist> <bibtext> Bryk, A. 2021. Improvement in Action: Advancing Quality in America's Schools. Boston: Harvard Education Press.</bibtext> </blist> <blist> <bibtext> Bryk, A., L. Gomez, A. Grunow, and P. LeMahieu. 2015. Learning to Improve: How America's Schools Can Get Better at Getting Better. Boston: Harvard Education Press.</bibtext> </blist> <blist> <bibtext> Burchfield, S., H. Hua, D. Noyes, and W. van de Waal. 2017. "Improving Early Grade Reading Outcomes: Aprender a Ler in Mozambique." New Directions for Child and Adolescent Development 2017 (155): 117–30.</bibtext> </blist> <blist> <bibtext> Century, J., and A. Cassata. 2016. "Implementation Research: Finding Common Ground on What, How, Why, Where, and Who." Review of Research in Education 40 (1): 169–215.</bibtext> </blist> <blist> <bibtext> Christina, R., and E. Vinogradova. 2017. "Differentiation of Effect across Systemic Literacy Programs in Rwanda, the Philippines, and Senegal." New Directions for Child and Adolescent Development 2017 (155): 51–65.</bibtext> </blist> <blist> <bibtext> Creswell, J. W., and V. Clark. 2017. Designing and Conducting Mixed Methods Research. Los Angeles: Sage.</bibtext> </blist> <blist> <bibtext> D'Agostino, T.J., N. D'Sa, and N. Boothby. 2021. "What's Faith Got to Do with It? A Scoping Study on Local Faith Communities Supporting Child Development and Learning." International Journal of Educational Development 81:102325.</bibtext> </blist> <blist> <bibtext> Dowd, A., E. Friedlander, C. Jonason, J. Leer, L. Sorensen, J. Guajardo, N. D'Sa, C. Pava, and L. Pisani. 2017. "Lifewide Learning for Early Reading Development." New Directions for Child and Adolescent Development 2017 (155): 31–49.</bibtext> </blist> <blist> <bibtext> Dowd, A. J., and F. Mabeti. 2011. "Literacy Boost Malawi: Year 2 Report." Save the Children, Fairfield, CT.</bibtext> </blist> <blist> <bibtext> Dubeck, M., M. Jukes, and G. Okello. 2012. "Early Primary Literacy Instruction in Kenya." Comparative Education Review 56 (1): 48–68.</bibtext> </blist> <blist> <bibtext> Duke, N. K., and P. Pearson. 2009. "Effective Practices for Developing Reading Comprehension." Journal of Education 189 (1–2): 107–22.</bibtext> </blist> <blist> <bibtext> Evans, D., and A. Popova. 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews." World Bank Research Observer 31 (2): 242–70.</bibtext> </blist> <blist> <bibtext> Evans, D., and F. Yuan. 2022. "How Big Are Effect Sizes in International Education Studies?" Educational Evaluation and Policy Analysis 44 (3): 532–40.</bibtext> </blist> <blist> <bibtext> Fleisch, B. 2023. "Theory of Change and Theory of Education: Pedagogic and Curriculum Defects in Early Grade Reading Interventions in South Africa." Education as Change 27 (1): 1–14.</bibtext> </blist> <blist> <bibtext> Gillies, J., and J. Quijada. 2008. "Opportunity to Learn: A High Impact Strategy for Improving Educational Outcomes in Developing Countries." Working paper, Academy for Educational Development, Durham.</bibtext> </blist> <blist> <bibtext> Guzmán, J. C., K. Schuenke-Lucien, A.J. D'Agostino, M. Berends, and A. J. Elliot. 2021. "Improving Reading Instruction and Students' Reading Skills in the Early Grades: Evidence from a Randomized Evaluation in Haiti." Reading Research Quarterly 56 (1): 173–93.</bibtext> </blist> <blist> <bibtext> Hanushek, E. A., and S. G. Rivkin. 2010. "Generalizations about Using Value-Added Measures of Teacher Quality." American Economic Review 100 (2): 267–71.</bibtext> </blist> <blist> <bibtext> Heugh, K., C. Benson, B. Bogale, and M. Yohannes. 2007. "Final Report: Study on Medium of Instruction in Primary Schools in Ethiopia." Commissioned by the Ministry of Education, Ethiopia.</bibtext> </blist> <blist> <bibtext> Institute of Education Sciences. 2020. What Works Clearinghouse Standards Handbook, Version 4.1. Washington, DC: US Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/handbooks.</bibtext> </blist> <blist> <bibtext> Kim, Y. S. G., H. Boyle, S. S. Zuilkowski, and P. Nakamura. 2016. Landscape Report on Early Grade Literacy. Washington, DC: USAID.</bibtext> </blist> <blist> <bibtext> Kim, Y. S. G., H. Lee, and S. S. Zuilkowski. 2020. "Impact of Literacy Interventions on Reading Skills in Low- and Middle-Income Countries: A Meta-analysis." Child Development 91 (2): 638–60.</bibtext> </blist> <blist> <bibtext> Kraft, M. A. 2020. "Interpreting Effect Sizes of Education Interventions." Educational Researcher 49 (4): 241–53.</bibtext> </blist> <blist> <bibtext> Leaver, C., R. F. Lemos, and D. Scur. 2019. "Measuring and Explaining Management in Schools: New Approaches Using Public Data." World Bank Policy Research Working Paper no. 9053, World Bank, Washington, DC.</bibtext> </blist> <blist> <bibtext> Lennox, S. 2013. "Interactive Read-Alouds—an Avenue for Enhancing Children's Language for Thinking and Understanding: A Review of Recent Research." Early Childhood Education Journal 41 (5): 381–89.</bibtext> </blist> <blist> <bibtext> McKenna, J. W., A. Flower, and S. Ciullo. 2014. "Measuring Fidelity to Improve Intervention Effectiveness." Intervention in School and Clinic 50 (1): 15–21.</bibtext> </blist> <blist> <bibtext> Murphy, J., and D. Torre. 2014. Creating Productive Cultures in Schools: For Students, Teachers, and Parents. Thousand Oaks, CA: Corwin.</bibtext> </blist> <blist> <bibtext> Peters, D. H., T. Adam, O. Alonge, I. A. Agyepong, and N. Tran. 2013. "Implementation Research: What It Is and How to Do It." BMJ 347:6753.</bibtext> </blist> <blist> <bibtext> Piper, B., J. Destefano, E. M. Kinyanjui, and S. Ong'ele. 2018. "Scaling Up Successfully: Lessons from Kenya's Tusome National Literacy Program." Journal of Educational Change 19 (3): 293–321.</bibtext> </blist> <blist> <bibtext> Piper, B., and S. S. Zuilkowski. 2015. "Teacher Coaching in Kenya: Examining Instructional Support in Public and Nonformal Schools." Teaching and Teacher Education 47:173–83.</bibtext> </blist> <blist> <bibtext> Piper, B., S. S. Zuilkowski, M. Dubeck, E. Jepkemei, and S. J. King. 2018. "Identifying the Essential Ingredients to Literacy and Numeracy Improvement: Teacher Professional Development and Coaching, Student Textbooks, and Structured Teachers' Guides." World Development 106:324–36.</bibtext> </blist> <blist> <bibtext> Quinn, D. M., and J. S. Kim. 2017. "Scaffolding Fidelity and Adaptation in Educational Program Implementation: Experimental Evidence from a Literacy Intervention." American Educational Research Journal 54 (6): 1187–220.</bibtext> </blist> <blist> <bibtext> Raupp, M., B. Newman, L. Revés, and C. Lauchande. 2015. Impact Evaluation for the USAID/Aprender a Ler Project in Mozambique Year 2 (Midline) IE/RCT: Final Report. Washington, DC: USAID.</bibtext> </blist> <blist> <bibtext> Registe, P. 2017. "Rapport de l'évaluation de base EGRA: Project Haïti Gagne, Lire, Ecrire et Réussir." CARE Haiti, Pétion-Ville.</bibtext> </blist> <blist> <bibtext> Robinson, V., M. J. Claire, A. Lloyd, and K. J. Rowe. 2008. "The Impact of Leadership on Student Outcomes: An Analysis of the Differential Effects of Leadership Types." Educational Administration Quarterly 44 (5): 635–74.</bibtext> </blist> <blist> <bibtext> RTI International. 2015a. Early Grade Reading Assessment (EGRA) Toolkit, Second Edition. Washington, DC: USAID.</bibtext> </blist> <blist> <bibtext> RTI International. 2015b. Tout Timoun Ap Li—ToTAL (All Children Reading): Final Report, REVISED. Washington, DC: USAID.</bibtext> </blist> <blist> <bibtext> Sabarwal, S., and M. Abu-Jawdeh. 2018. "What Teachers Believe: Mental Models about Accountability, Absenteeism, and Student Learning." World Bank Policy Research Working Paper no. 8454, World Bank, Washington, DC.</bibtext> </blist> <blist> <bibtext> Schleicher, A. 2018. World Class. Paris: OECD Publishing.</bibtext> </blist> <blist> <bibtext> Stern, J., M. Jukes, B. Piper, J. DeStefano, J. Mejia, P. Dubeck, B. Carrol, et al. 2021. Learning at Scale: Interim Report. Durham, NC: RTI International.</bibtext> </blist> <blist> <bibtext> Taylor, S., J. Cilliers, C. Prinsloo, B. Fleisch, and V. Reddy. 2017. "The Early Grade Reading Study: Impact Evaluation after Two Years of Interventions." Department of Basic Education, Pretoria.</bibtext> </blist> <blist> <bibtext> UNICEF. 2022. "The State of Global Learning Poverty: A 2022 Update." UNICEF, New York.</bibtext> </blist> <blist> <bibtext> USAID. 2019. "Ghana Early Grade Reading Program Impact Evaluation—2019 Endline Report." USAID, Accra.</bibtext> </blist> <blist> <bibtext> WFP (World Food Program). 2019. "Decentralized Evaluation: Final Evaluation of WFP Haiti's Food for Education and Child Nutrition Programme (2016–2019)." Final Evaluation Report. World Food Program, Port-au-Prince.</bibtext> </blist> <blist> <bibtext> World Bank. 2018. World Development Report 2018: Learning to Realize Education's Promise. Washington, DC: World Bank.</bibtext> </blist> <blist> <bibtext> World Bank. 2020. "Cost Effective Approaches to Improve Global Learning: What Does Recent Evidence Tell Us Are 'Smart Buys' for Improving Learning in Low-And Middle-Income Countries?" World Bank, Washington, DC.</bibtext> </blist> </ref> <aug> <p>By TJ D'Agostino; Danice Brown Guzmán; Paul Perrin; Anasthasie Liberiste-Osirus and Kate Schuenke-Lucien</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib54" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib51" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib22" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib31" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib52" firstref="ref9"></nolink> <nolink nlid="nl6" bibid="bib15" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib26" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib11" firstref="ref13"></nolink> <nolink nlid="nl9" bibid="bib12" firstref="ref16"></nolink> <nolink nlid="nl10" bibid="bib14" firstref="ref17"></nolink> <nolink nlid="nl11" bibid="bib37" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib41" firstref="ref23"></nolink> <nolink nlid="nl13" bibid="bib35" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib49" firstref="ref25"></nolink> <nolink nlid="nl15" bibid="bib39" firstref="ref27"></nolink> <nolink nlid="nl16" bibid="bib24" firstref="ref29"></nolink> <nolink nlid="nl17" bibid="bib28" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib38" firstref="ref34"></nolink> <nolink nlid="nl19" bibid="bib50" firstref="ref35"></nolink> <nolink nlid="nl20" bibid="bib25" firstref="ref36"></nolink> <nolink nlid="nl21" bibid="bib18" firstref="ref37"></nolink> <nolink nlid="nl22" bibid="bib19" firstref="ref39"></nolink> <nolink nlid="nl23" bibid="bib20" firstref="ref40"></nolink> <nolink nlid="nl24" bibid="bib10" firstref="ref41"></nolink> <nolink nlid="nl25" bibid="bib48" firstref="ref42"></nolink> <nolink nlid="nl26" bibid="bib27" firstref="ref43"></nolink> <nolink nlid="nl27" bibid="bib44" firstref="ref45"></nolink> <nolink nlid="nl28" bibid="bib43" firstref="ref50"></nolink> <nolink nlid="nl29" bibid="bib46" firstref="ref51"></nolink> <nolink nlid="nl30" bibid="bib16" firstref="ref53"></nolink> <nolink nlid="nl31" bibid="bib45" firstref="ref55"></nolink> <nolink nlid="nl32" bibid="bib29" firstref="ref59"></nolink> <nolink nlid="nl33" bibid="bib32" firstref="ref61"></nolink> <nolink nlid="nl34" bibid="bib23" firstref="ref62"></nolink> <nolink nlid="nl35" bibid="bib30" firstref="ref65"></nolink> <nolink nlid="nl36" bibid="bib40" firstref="ref67"></nolink> <nolink nlid="nl37" bibid="bib13" firstref="ref76"></nolink> <nolink nlid="nl38" bibid="bib42" firstref="ref77"></nolink> <nolink nlid="nl39" bibid="bib55" firstref="ref80"></nolink> <nolink nlid="nl40" bibid="bib47" firstref="ref81"></nolink> <nolink nlid="nl41" bibid="bib33" firstref="ref87"></nolink> <nolink nlid="nl42" bibid="bib17" firstref="ref89"></nolink> <nolink nlid="nl43" bibid="bib36" firstref="ref90"></nolink> <nolink nlid="nl44" bibid="bib53" firstref="ref91"></nolink> <nolink nlid="nl45" bibid="bib21" firstref="ref98"></nolink> <nolink nlid="nl46" bibid="bib34" firstref="ref99"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1417939
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Explaining Variation in Treatment Effects: An Impact Evaluation and Mixed-Methods Study of Variation in Early Grade Reading Program Effects
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22T%2E+J%2E+D'Agostino%22">T. J. D'Agostino</searchLink><br /><searchLink fieldCode="AR" term="%22Danice+Brown+Guzmán%22">Danice Brown Guzmán</searchLink><br /><searchLink fieldCode="AR" term="%22Paul+Perrin%22">Paul Perrin</searchLink><br /><searchLink fieldCode="AR" term="%22Anasthasie+Liberiste-Osirus%22">Anasthasie Liberiste-Osirus</searchLink><br /><searchLink fieldCode="AR" term="%22Kate+Schuenke-Lucien%22">Kate Schuenke-Lucien</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Comparative+Education+Review%22"><i>Comparative Education Review</i></searchLink>. 2024 68(1):85-112.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Chicago Press. Journals Division, P.O. Box 37005, Chicago, IL 60637. Tel: 877-705-1878; Tel: 773-753-3347; Fax: 877-705-1879; Fax: 773-753-0811; e-mail: subscriptions@press.uchicago.edu; Web site: http://www.press.uchicago.edu
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Early+Reading%22">Early Reading</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Comprehension%22">Reading Comprehension</searchLink><br /><searchLink fieldCode="DE" term="%22Literacy%22">Literacy</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Environment%22">Educational Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Attendance%22">Attendance</searchLink><br /><searchLink fieldCode="DE" term="%22Coaching+%28Performance%29%22">Coaching (Performance)</searchLink><br /><searchLink fieldCode="DE" term="%22Administrator+Attitudes%22">Administrator Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Geographic+Location%22">Geographic Location</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Haiti%22">Haiti</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1086/728393
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0010-4086<br />1545-701X
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This article contributes to understanding the effectiveness of early grade reading interventions in low-income-country contexts and how and why such interventions vary in their effectiveness across schools. It presents the results from an impact evaluation of an early grade reading intervention in Haiti with a mixed-methods analysis of factors that explain variation in program effects. We study program impact using a randomized controlled trial experimental research design and find significant treatment impact on students' literacy skills, including on students' reading comprehension in Creole. We use a mixed-methods design to explore variation in program effects across school sites. Key factors that explain variation in program effects include the frequency of instructional coaching visits; the dosage and uptake of treatment, which was linked to student and teacher absenteeism and the exclusive use of the treatment curriculum; the behaviors of school leaders; the fluency and quality of lesson delivery; and the remoteness of schools.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1417939
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1417939
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1086/728393
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
        StartPage: 85
    Subjects:
      – SubjectFull: Early Reading
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Reading Comprehension
        Type: general
      – SubjectFull: Literacy
        Type: general
      – SubjectFull: Educational Environment
        Type: general
      – SubjectFull: Attendance
        Type: general
      – SubjectFull: Coaching (Performance)
        Type: general
      – SubjectFull: Administrator Attitudes
        Type: general
      – SubjectFull: Geographic Location
        Type: general
      – SubjectFull: Haiti
        Type: general
    Titles:
      – TitleFull: Explaining Variation in Treatment Effects: An Impact Evaluation and Mixed-Methods Study of Variation in Early Grade Reading Program Effects
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: T. J. D'Agostino
      – PersonEntity:
          Name:
            NameFull: Danice Brown Guzmán
      – PersonEntity:
          Name:
            NameFull: Paul Perrin
      – PersonEntity:
          Name:
            NameFull: Anasthasie Liberiste-Osirus
      – PersonEntity:
          Name:
            NameFull: Kate Schuenke-Lucien
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0010-4086
            – Type: issn-electronic
              Value: 1545-701X
          Numbering:
            – Type: volume
              Value: 68
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Comparative Education Review
              Type: main
ResultId 1