View in EDS HTML Full Text PDF Full Text

Comparing the Effectiveness of Two Reciprocal Reading Comprehension Interventions for Primary School Pupils in Disadvantaged Schools

Saved in:
Bibliographic Details
Title: Comparing the Effectiveness of Two Reciprocal Reading Comprehension Interventions for Primary School Pupils in Disadvantaged Schools
Language: English
Authors: O'Hare, Liam (ORCID 0000-0002-4453-2880), Stark, Patrick, Cockerill, Maria, Lloyd, Katrina, McConnellogue, Sheila, Gildea, Aideen, Biggart, Andy, Bower, Christine, Connolly, Paul
Source: British Journal of Educational Psychology. 2023 93(4):1123-1145.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 23
Publication Date: 2023
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Education
Descriptors: Reading Comprehension, Intervention, Reading Instruction, Elementary School Students, Disadvantaged Schools, Program Effectiveness, Reading Difficulties, Cooperative Learning, Comparative Analysis
DOI: 10.1111/bjep.12623
ISSN: 0007-0998
2044-8279
Abstract: Background: Effective reading comprehension teaching is an aspiration of education systems across the world. Teaching incorporating reciprocal reading theory and evidence is an internationally popular approach for improving comprehension. Aims: This paper uses two large cluster randomized controlled trials of similar reciprocal reading interventions implemented in different ways to compare their effectiveness. Sample: The two interventions had the same teacher professional development, reciprocal reading activities and dosage/exposure, but varied in their implementation, with one delivered as a whole-class ('universal') version for pupils aged 8-9 years and the other a small group ('targeted') version for pupils aged 9-11 years with specific comprehension difficulties. Methods: Two large-scale cluster RCTs were conducted in 98 schools with N = 3699 pupils in the universal trial and N = 1523 in the targeted trial. Results: Multi-level models showed significant effects for the targeted version of the intervention on pupil reading comprehension (g = 0.18) and overall reading (g = 0.14). No significant effects were found for the whole class version. A sub-group analyses of disadvantaged pupils showed the targeted intervention's effects were even larger on reading comprehension (g = 0.25). Conclusions: The evidence suggested that this reciprocal reading intervention worked best when implemented in small groups and targeted for pupils with specific comprehension difficulties and particularly for pupils in disadvantaged circumstances. Comments: This evaluation shows that even if a reading comprehension intervention is underpinned by strong theory and evidence-based practice, its effectiveness can still depend on implementation choices.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1399628
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwF2gJgrp01gbJz1HmItx-DzAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDMdizrAW7UPEQquFYwIBEICBm05MHU_RGhxflheAAjV3D96m86lZPvF5Ep_P50aDdsbwgVTkMax4naG9pOrAIiedJ8Fh7RVlQK1J62FzWV1eb3RSQAI3HTnx3b0DCu8z19wjfd_QT8UOqx92EBvUOSoftvOP0L72CwpVpVT1JT50TEiPiY5tCCwyJY63_poWHOAx0Wp-B1pbgC-E9m3LmUvwpTCK7VSSt7ABxYnG
Text:
  Availability: 1
  Value: <anid>AN0173485905;6kx01dec.23;2023Nov10.04:12;v2.2.500</anid> <title id="AN0173485905-1">Comparing the effectiveness of two reciprocal reading comprehension interventions for primary school pupils in disadvantaged schools </title> <p>Background: Effective reading comprehension teaching is an aspiration of education systems across the world. Teaching incorporating reciprocal reading theory and evidence is an internationally popular approach for improving comprehension. Aims: This paper uses two large cluster randomized controlled trials of similar reciprocal reading interventions implemented in different ways to compare their effectiveness. Sample: The two interventions had the same teacher professional development, reciprocal reading activities and dosage/exposure, but varied in their implementation, with one delivered as a whole‐class ('universal') version for pupils aged 8–9 years and the other a small group ('targeted') version for pupils aged 9–11 years with specific comprehension difficulties. Methods: Two large‐scale cluster RCTs were conducted in 98 schools with N = 3699 pupils in the universal trial and N = 1523 in the targeted trial. Results: Multi‐level models showed significant effects for the targeted version of the intervention on pupil reading comprehension (g =.18) and overall reading (g =.14). No significant effects were found for the whole class version. A sub‐group analyses of disadvantaged pupils showed the targeted intervention's effects were even larger on reading comprehension (g =.25). Conclusions: The evidence suggested that this reciprocal reading intervention worked best when implemented in small groups and targeted for pupils with specific comprehension difficulties and particularly for pupils in disadvantaged circumstances. Comments: This evaluation shows that even if a reading comprehension intervention is underpinned by strong theory and evidence‐based practice, its effectiveness can still depend on implementation choices.</p> <p>Keywords: comprehension; implementation; literacy; randomized controlled trial; reciprocal reading</p> <hd id="AN0173485905-2">BACKGROUND</hd> <p>Policymakers in the United States and United Kingdom have placed considerable emphasis on reducing those leaving elementary‐aged schooling with poor levels of literacy. In the United Kingdom, the pedagogical focus has been on systematic synthetic phonics, which has a strong evidence base (Ehri et al., [<reflink idref="bib11" id="ref1">11</reflink>]) but there is increasing recognition that some children successfully learn to decode text but struggle with text comprehension. This was recognized by the National Reading Panel ([<reflink idref="bib22" id="ref2">22</reflink>]) in the United States and as a result they recommended reading comprehension strategies as one of the five key components of reading strategy instruction.</p> <p>As a result of these developments, over the last few decades, the teaching of reading strategies is increasingly a standard part of international reading curricula in elementary and secondary education (Department for Education, [<reflink idref="bib5" id="ref3">5</reflink>]; Okkinga et al., [<reflink idref="bib27" id="ref4">27</reflink>]; Pressley, [<reflink idref="bib30" id="ref5">30</reflink>]). An example of a reading comprehension strategy intervention is reciprocal teaching. Reciprocal teaching is now essentially a worldwide approach (Palincsar, [<reflink idref="bib28" id="ref6">28</reflink>]). However, reciprocal teaching type interventions have not been widely implemented in the United Kingdom as a discrete intervention until recently, but it is now recommended by the UK's Education Endowment Foundation (EEF) teaching toolkit (Education Endowment Foundation, [<reflink idref="bib9" id="ref7">9</reflink>]).</p> <p>Reciprocal teaching is informed by research and development in the United States that was led originally by Palincsar and Brown ([<reflink idref="bib29" id="ref8">29</reflink>]). Although there are a range of interventions based on the principles of reciprocal teaching, the core factors are comprehension strategy instruction through modelling of strategies by an adult and student practice of the modelled strategies (De Corte et al., [<reflink idref="bib4" id="ref9">4</reflink>]; Palincsar & Brown, [<reflink idref="bib29" id="ref10">29</reflink>]). Often, learners are explicitly taught four sequenced evidence‐based strategies, which effective readers are known to use to help them derive meaning from texts (Palincsar & Brown, [<reflink idref="bib29" id="ref11">29</reflink>]). These are: (<reflink idref="bib1" id="ref12">1</reflink>) predicting what will happen next; (<reflink idref="bib2" id="ref13">2</reflink>) clarifying new or unfamiliar words; (<reflink idref="bib3" id="ref14">3</reflink>) asking questions about the text and (<reflink idref="bib4" id="ref15">4</reflink>) summarizing what has been read. The modelling is interactive in that, whilst adult models these strategic actions with the text, there is also a collaborative problem‐solving approach to the text (Palincsar & Brown, [<reflink idref="bib29" id="ref16">29</reflink>]). Predicting is used first in the present study, which differs from Palincsar and Brown where summarizing is used first. The rationale for this is to develop their inference practice using a few clues from the text provided (e.g. text title, the first few sentences and building on text knowledge from a previous session). There is substantive evidence that the ability to draw inferences predetermines reading skills: that is, poor inferencing causes poor comprehension and not vice versa (Kispal, [<reflink idref="bib19" id="ref17">19</reflink>]). FFT Literacy also suggests using prediction first promotes oral discussion from the outset in a group of students who would not be used to this kind of cooperative working together on the reading of a text. For these reasons, the FFT reciprocal reading strategy cycle evaluated in this study begins with predicting and ends with summarizing.</p> <p>Previous meta‐analyses have claimed positive effects for general reading strategy instruction (Chiu, [<reflink idref="bib2" id="ref18">2</reflink>]; Swanson, [<reflink idref="bib34" id="ref19">34</reflink>]) including comprehension strategies (Sencibaugh, [<reflink idref="bib33" id="ref20">33</reflink>]) and reciprocal teaching (Rosenshine & Meister, [<reflink idref="bib32" id="ref21">32</reflink>]). In the latter meta‐analysis of 16 reciprocal teaching studies an average effect size of.32 was found for reading and.88 for reading comprehension (Rosenshine & Meister, [<reflink idref="bib32" id="ref22">32</reflink>]). Despite the large international research base on reading comprehension strategy instruction, the EEF ([<reflink idref="bib10" id="ref23">10</reflink>]) has stated that to date there are few UK‐based evaluations of comprehension programmes and those available generally show lower effects on pupil outcomes than the international (mainly United States) literature.</p> <p>One of the possible explanations of difficulties in replicating large effect sizes of reading comprehension interventions is variation in implementation methods. There are some key reciprocal teaching implementation issues that are worthy of note in the literature. The evidence base for reciprocal teaching generally centres on implementation with specific targeted populations (students with literacy or learning difficulties) and in a recent meta‐analysis of 52 studies, Okkinga et al. ([<reflink idref="bib27" id="ref24">27</reflink>]) showed larger effect sizes for low achievers in reading, rather than typically developing students. Effectiveness of universal (whole‐class) approaches has also been difficult to demonstrate (De Corte et al., [<reflink idref="bib4" id="ref25">4</reflink>]), with research suggesting the intervention is challenging to deliver in a whole‐class setting and still maintains intervention quality (Okkinga et al., [<reflink idref="bib27" id="ref26">27</reflink>]). Also, some studies report that teachers find it hard to instruct students in strategic thinking (Duffy, [<reflink idref="bib7" id="ref27">7</reflink>]). In addition, Okkinga et al. ([<reflink idref="bib27" id="ref28">27</reflink>]) report that teachers are generally less successful than researchers at delivering reading strategy instruction. This all suggests that reciprocal teaching requires a specialist understanding of reading comprehension itself and effective comprehension strategies. Therefore, additional specialist professional development for teachers may be required to deliver reciprocal teaching approaches effectively.</p> <p>There is also a lack of clarity about the optimal age at which reciprocal teaching is beneficial. However, Okkinga et al. ([<reflink idref="bib27" id="ref29">27</reflink>]) did report that reading strategy intervention is particularly beneficial for students aged 8–14 (even in a whole‐class format). Regardless, age, developmental stage and cognitive capacity may be limiting factors for when students can receive a reading strategy‐based intervention.</p> <p>Another implementation issue is the level of disadvantage faced by reading programme participants. Pupil disadvantage has been shown to be a significant predictor of reading level as assessed in terms of accuracy, comprehension and rate (McPhillips & Sheehy, [<reflink idref="bib21" id="ref30">21</reflink>]). However, randomized controlled trials (RCTs) of language interventions in the United Kingdom have shown greater levels of reading improvement for pupils eligible for free school meals (FSM – a proxy for disadvantage) from those not eligible for FSM (Maxwell et al., [<reflink idref="bib20" id="ref31">20</reflink>]; Thurston et al., [<reflink idref="bib35" id="ref32">35</reflink>]).</p> <p>Finally, study methodology has also had an impact on reciprocal teaching evaluation findings in the literature. Okkinga et al. ([<reflink idref="bib27" id="ref33">27</reflink>]) meta‐analyses reported smaller effect sizes (mean ES = .19) from studies using standardized tests compared to researcher‐created proximal reading comprehension tests (mean ES = .43).</p> <hd id="AN0173485905-3">THE PROGRAMME</hd> <p>The Fisher Family Trust Literacy (FFT) <emph>Reciprocal Reading</emph> (RR) programme is based on the principals of reciprocal teaching. The RR programme is implemented with practicing Teachers/Teaching Assistants in mainstream United Kingdom settings and supports them in delivery of RR to pupils aged 8–11 years. In essence, it is a continuing professional development programme (CPD) for primary school educators. All Teachers/Teaching Assistants involved in delivering the programme receive 2 days' off‐site CPD from RR instructors. The CPD covers the knowledge, skills and understanding that practitioners need to deliver RR. The training emphasizes the importance of all readers applying the four common reciprocal teaching strategies of predict, clarify, question, and summarize rather than individual readers taking on separate roles as in other reciprocal teaching programmes, for example, Reading Rockets ([<reflink idref="bib31" id="ref34">31</reflink>]), where one student takes on the role of predictor, another of clarifier etc. The CPD considers the difficulties involved in comprehension and how RR addresses these, how to identify children who will benefit most from RR, as well as giving teachers opportunities to practice using the strategies. Additional sessions include considering how to build challenge into the lesson without losing the emphasis on children's talk and reviewing how best to enhance reciprocal teaching within a comprehension curriculum. FFT also provide in‐situ support throughout delivery of RR to pupils. For example, the CPD is supplemented by school visits which can include demonstration, review of developing practice and advice on embedding the approach into classroom practice.</p> <p>For this study teachers were instructed in the implementation of two different versions of the RR programme, namely: a whole‐class ('universal') version and a 'targeted' programme. The universal version of the RR programme was for elementary school‐aged pupils in England (8‐ to 9‐year‐old children equivalent to UK Year 4 and US Grade 3). This version was to be delivered to a whole class for 20 min at least once per week for a minimum of 12 weeks over a school year (i.e. a minimum of 240 min). The 'targeted' programme was for pupils aged 9–11 years old (UK Years 5 and 6, and US Grade 4 and 5) who were identified as having poor reading comprehension but good decoding skills. This version was to be delivered to groups of approximately six pre‐selected pupils with comprehension difficulties, through at least two sessions of 20 min per week for 12 weeks (minimum recommended 480 min). Both versions of the RR programme were delivered by teachers and teaching assistants who had received the RR CPD training.</p> <p>The programme's logic model (summarized in Figure 1) is a graphical representation of the programme's inputs, outputs, activities and outcomes. Figure 1 also situates the underpinning programme's theory of change, namely: theory of intervention (i.e. programme activities that produce outcome change); theory of change (i.e. developmental pathways in outcome changes); and implementation theory (i.e. implementation factors, like population characteristics and delivery style, that have an influence programme effectiveness). Full details of the programme content and implementation methods are available in the previously published trial protocol (O'Hare et al., [<reflink idref="bib26" id="ref35">26</reflink>]), which includes a TIDieR framework of programme description (Hoffmann et al., [<reflink idref="bib17" id="ref36">17</reflink>]) and a comprehensive programme logic model and programme theories. Further explanation of logic models and programme theory are available here (Connolly et al., [<reflink idref="bib3" id="ref37">3</reflink>] chapter 2).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6KX/01dec23/bjep12623-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjep12623-fig-0001.jpg" title="1 Simplified theory of change and logic model for FFT reciprocal reading programme." /> </p> <p></p> <p>Figure 1 highlights how RR improves a teacher's ability to guide pupils through RR activities, which in turn improves pupils' reading comprehension metacognition (i.e. pupils explicitly applying appropriate reading strategies to support their comprehension) and reading accuracy. The programme's theory of change is that these improvements in reading comprehension metacognition and reading accuracy (secondary outcomes) result in improvements in pupil comprehension abilities that lead to further improvements in overall pupil reading ability and reading attainment (primary outcomes). It is a convention that primary outcomes are the key (primarily important) outcomes that the programme is intended to change. The secondary outcomes are theorized proximal, short‐term changes that lead to distal change in the primary outcome of interest.</p> <hd id="AN0173485905-5">RESEARCH QUESTIONS</hd> <p>This study set out to investigate the effectiveness of FFT Literacy RR intervention and what influence the implementation choices had on the level of effectiveness by answering the following research questions:</p> <p></p> <ulist> <item> Were there different effects of the universal or targeted RR intervention on the primary outcomes of comprehension and overall reading attainment in UK classrooms?</item> <p></p> <item> Were there different effects of the universal or targeted RR intervention on the secondary outcomes of reading accuracy and reading comprehension metacognition alongside the primary outcomes?</item> <p></p> <item> Did implementing the intervention with disadvantaged pupils have an impact on the level of effectiveness?</item> </ulist> <hd id="AN0173485905-6">METHOD</hd> <p>The study consisted of two cluster RCTs that ran between September 2017 and June 2018. Ethical approval for the study was granted by the Research Ethics Committee of XYZ in the United Kingdom. The trial protocol has been published (O'Hare et al., [<reflink idref="bib26" id="ref38">26</reflink>]) and has been registered on the ISRCTN website and summarized here: (O'Hare et al., [<reflink idref="bib24" id="ref39">24</reflink>]).</p> <hd id="AN0173485905-7">Recruitment</hd> <p>Primary schools that met the following inclusion criteria were invited to participate in the trial:</p> <p></p> <ulist> <item> Schools that had not previously participated in the RR programme.</item> <p></p> <item> Schools that were not included in any other ongoing EEF trial.</item> <p></p> <item> Schools that were clustered together within a geographical area with between 8–10 schools from each area, to maximize the probability of having at least three schools from each area randomized to deliver the intervention. Ensuring groups of schools were located within close proximity of one another facilitated CPD sessions such that schools did not have to travel far to receive training.</item> </ulist> <p>The project funder, the EEF, was and continues to be, focused on reducing the attainment gap between disadvantaged schools and more affluent schools. As a result, areas of high disadvantage were targeted to participate in this trial. The aim was that the participating schools had an overall mean of at least the national average of the school population ever in receipt of FSM an indicator of socio‐economic disadvantage (29%). In practice, schools with higher‐than‐average levels of pupils eligible for FSM from the Northeast of England were contacted first and priority was given to these schools when they expressed an interest to participate during recruitment. In addition, other schools could express an interest to participate through the EEF website and were advised that schools with a higher‐than‐average level of FSM were being sought for participation.</p> <hd id="AN0173485905-8">Power</hd> <p>Sample‐size calculations were carried out prior to school recruitment. Effect sizes for literacy interventions evaluated through a good quality RCT design would tend typically be in the range of.2–.3 (Borman et al., [<reflink idref="bib1" id="ref40">1</reflink>]; O'Hare et al., [<reflink idref="bib24" id="ref41">24</reflink>]; Tymms et al., [<reflink idref="bib37" id="ref42">37</reflink>]).</p> <p>At the initial protocol stage, a power calculation for the universal intervention provided by Optimal Design software showed that 94 schools (clusters) would be needed to detect a significant effect size of Hedges g (<emph>g</emph>) = .2 if present with a power of.8.</p> <p>The calculation used estimates as follows: Effect size = .2; <emph>p</emph> = .05; Intra‐cluster correlation coefficient (ICC) = .14; <emph>r</emph><sups>2</sups> = .50 (due to having a pre‐test of New Group Reading Test – NGRT); and average cluster size <emph>n =</emph> 20.</p> <hd id="AN0173485905-9">Participants</hd> <p>We invited 142 schools to participate in the study. Of 119 schools agreeing to participate, five did not meet the inclusion criteria and 14 dropped out prior to pre‐test. We chose to withdraw two further schools, after the pre‐test and prior to randomisation, due to undisclosed non‐compliance with the inclusion criteria (i.e. they were involved in other EEF trials). This left a total of 98 schools for pre‐test and randomisation.</p> <p>We asked participating schools to choose teachers and teaching assistants to be trained in basic FFT Literacy RR CPD. Following their training, they were invited to deliver the universal intervention to pupils in Year 4 (age 8 to 9) and the targeted intervention to pupils in Years 5 and 6 (age 9–11). We did this so we could compare the effectiveness of the two different interventions avoiding contamination across the same year group, that is, pupils having received both interventions. It was not feasible to recruit the additional schools required to conduct a multi‐arm trial at the appropriate power required.</p> <p>Our eligibility criteria for the targeted version required children (aged 9–11) to be selected for the programme on the basis that they were poor at reading comprehension with normal levels of decoding ability. Teachers selected these children before pre‐baseline testing and pre‐randomisation. This ensured that we were comparing equivalent pupils in the intervention and control groups (i.e. selected by teachers in all schools using the same approach).</p> <p>We asked the teachers to select children in June 2017 using guidance and materials which were co‐designed by FFT literacy and the evaluation team. These were based on a simple view of reading (Gough & Tunmer, [<reflink idref="bib14" id="ref43">14</reflink>]; Hoover & Gough, [<reflink idref="bib18" id="ref44">18</reflink>]). It was necessary for us to co‐design a new process because there was no tool for quickly identifying the correct group of pupils available for use by teachers. The main outcome measure used in this trial (NGRT), as described in 'Outcomes and Measures' below, did not provide a measure of decoding for the majority of children and was therefore unsuitable for this selection process. It was also necessary to select children prior to pre‐testing with the outcome measure as they were a sub‐sample of whole classes.</p> <p>Our co‐designed selection guidance allowed teachers to compare each child in their class with two sets of criteria. The first set of criteria described the reading skills possessed by a child who could be classified as having normal‐level decoding skills. The second set of criteria described difficulties that may be experienced by a child who struggles with reading comprehension. By comparing each child with these two sets of criteria, teachers were able to identify children who had a normal level of decoding skill but struggled with reading comprehension. Teachers in each school selected six pupils from each Years 5 and 6 class for the targeted intervention. Although there may have been more than six pupils who met the criteria for receiving the targeted intervention, teachers were asked to select only six.</p> <p>The sessions for targeted pupils were delivered as an additional sessions rather than replacing other taught literacy sessions. However, to facilitate these sessions teachers were required to find the time during the school day which might have involved the children's withdrawal from other non‐literary lessons or activities.</p> <hd id="AN0173485905-10">Randomisation</hd> <p>We performed randomisation after the collection of all baseline data and it was conducted at the school level using 'minimisation' to ensure the control and intervention groups were as evenly matched as possible (Torgerson & Torgerson, [<reflink idref="bib36" id="ref45">36</reflink>]). We performed minimisation stratification for several school‐level covariates: baseline reading comprehension (NGRT passage comprehension score), disadvantage (% of school pupils ever in receipt of FSM) and baseline reading accuracy (NGRT sentence completion score). Minimisation was carried out using the QMinim package. Median values were calculated for reading comprehension score (NGRT passage comprehension), disadvantage and NGRT sentence completion score. These medians were used to code schools as 'High' or 'Low' for each of these characteristics. These variables were then entered into QMinim for each school; and all variables were given a weight of one with the exception of reading comprehension which was double‐weighted as an important predictor of the outcome of interest. The randomisation program then allocated schools to control or intervention groups in a way that minimized differences between groups for these three variables. Control schools acted as control group for both the universal and targeted interventions. Consequently, intervention schools included an intervention group for both the universal and targeted interventions. By assigning different year groups in the same schools to participate in different versions of the programme allowed evaluation of the two different versions of the programme (universal and targeted) within one sample of randomized schools.</p> <hd id="AN0173485905-11">Dosage</hd> <p>Dosage was defined as the total number of minutes of the programme calculated from minutes per week multiplied by the total number of weeks of delivery. Thirty two out of 49 intervention schools returned reported dosage data. Of those children in the universal RR programme, the mean delivery of minutes per week was 66.21 (<emph>SD</emph> = 30.13) with an average duration of the programme of 26.41 weeks (<emph>SD</emph> = 5.06) giving an average total number of minutes of 1748 min. Of those children in the targeted intervention group, the mean delivery of minutes per week of RR was 65.77 (<emph>SD</emph> = 28.16) and the average duration of the programme was 26.03 weeks (<emph>SD</emph> = 6.31) giving an average number of minutes of 1711 min. Both mean dosages are well more than the minimum recommended dosage.</p> <hd id="AN0173485905-12">Measures</hd> <p>The primary outcomes of the trial were two attainment indicators from the digital NGRT (GL Assessment, [<reflink idref="bib12" id="ref46">12</reflink>]): (<reflink idref="bib1" id="ref47">1</reflink>) Overall reading score; and (<reflink idref="bib2" id="ref48">2</reflink>) Comprehension subscale score: (see Table 1). The NGRT is an adaptive test which has high reliability (see Table 1 for the reliability of outcome measures) and measures both reading accuracy, comprehension and overall reading. GL Assessment reports high concurrent validity of the NGRT with correlations between the NGRT and teacher assessments of reading as <emph>r</emph> = .81 and Key Stage 2 SATs of <emph>r</emph> = .75 (GL Assessment, [<reflink idref="bib12" id="ref49">12</reflink>]).</p> <p>1 TABLE Primary and secondary outcome measures.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Measure</th><th align="left">Level of measurement</th><th align="left">Number of items</th><th align="left">Reliability (Cronbach's alpha)</th></tr></thead><tbody valign="top"><tr><td align="left">Primary outcomes</td></tr><tr><td>Reading comprehension</td><td>New Group Reading Test – Passage Comprehension</td><td>Pupil</td><td>27</td><td>.9</td></tr><tr><td>Overall reading ability</td><td>New Group Reading Test – Total Score</td><td>Pupil</td><td>47</td><td>.9</td></tr><tr><td align="left">Secondary outcomes</td></tr><tr><td>Reading accuracy</td><td>New Group Reading Test – Sentence Completion</td><td>Pupil</td><td>20</td><td>.9</td></tr><tr><td>Reading comprehension metacognition</td><td>Assessment of Reading Comprehension Metacognition Scale (ARCMS)</td><td>Pupil</td><td>6</td><td>.7</td></tr></tbody></table> </ephtml> </p> <p>All pupils who took the NGRT test received two subscales out of three possible subscales: sentence completion, passage comprehension and phonics. Sentence completion is the first subscale and was used as the secondary outcome reading accuracy. Pupils then progressed to the passage comprehension subscale (used as the primary outcome measure for reading comprehension attainment), unless they scored at an extremely low level for the sentence completion, in which case they progressed to the more basic phonics subscale instead. As described above, the number of items for each subscale is adaptive, as the difficulty of the next item is chosen based on their performance on items already administered. The maximum number of items depends on pupil performance. The aggregate overall reading score was calculated by GL before the researchers downloaded the scores from the GL software. The calculation was based on the two subscales each pupil received. All Y4, Y5 and Y6 pupils were provided with and asked to complete the NGRT digital test. The adaptive system meant that different test forms for different ages were not necessary. One caveat for the adaptive testing system used by the digital NGRT test was that reading comprehension (passage comprehension) outcome scores were not available for pupils who had scored at an extremely low level on the reading accuracy (sentence completion) subscale; these pupils received the phonics test instead. However, overall reading outcome data was still available for these pupils, as the GL system still generates an overall score, based on sentence completion and phonics elements. There were, therefore, fewer pupils with reading comprehension outcome data than there were pupils with overall reading data.</p> <p>The secondary outcomes of the trial (see Table 1) were reading accuracy scores from the sentence completion subtest of the NGRT and a pupil reading comprehension metacognition using the Assessment of Reading Comprehension Metacognition Scale (ARCMS; O'Hare et al., [<reflink idref="bib25" id="ref50">25</reflink>]) measure developed by the evaluation team. This researcher‐created proximal measure was designed to closely reflect the principles of FFT Literacy RR and was developed to provide specificity matching to the intervention as no previously used standardized measures would be appropriate for this. Therefore, the evaluation team developed six questions that best matched the four key aspects of the programme: predicting, clarifying, questioning and summarizing. The six items related directly to the two overarching concepts of metacognition and comprehension. In this researcher‐created proximal measure, the children were asked to 'think about what kinds of things you can do to help you to understand a story better when you read it' and included items related specifically to the strategies they were taught in the FFT Literacy RR sessions (e.g. 'Before I read a text I ask myself what I already know about the subject of the story (i.e., predicting))' and 'I find ways to help me understand words I am not sure about such as looking them up in a dictionary (i.e., clarifying).' The responses ranged from 1 (Always) to 5 (Never). Internal consistency reliability was measured using Cronbach's alpha and was deemed acceptable (.68) given that the conventional cut‐off is.70 (Nunnally & Bernstein, [<reflink idref="bib23" id="ref51">23</reflink>]). The questions were designed by members of the evaluation team who had expertise in reading programmes to ensure the measure had content validity. Construct validity was assessed using Maximum Likelihood factor analysis as the data were normally distributed. One factor was extracted (eigenvalue 2.30) accounting for 26.46% of the variance. All items loaded.30 or above.</p> <hd id="AN0173485905-13">Data collection</hd> <p>We administered all tests within each school in classroom setting through a trained fieldworker with a class teacher in attendance. The children were each equipped with an individual computer or tablet, which was connected to a pair of headphones. We carried out the testing under exam conditions and the fieldworker delivered clear instructions to the children before the tests commenced. Children were not given additional support or guidance once the tests commenced. In addition, to the post testing, children completed the ARCMS through an on‐line survey.</p> <p>All children (in both the universal and targeted groups) completed a pre‐test (NGRT form A) and a post‐test (NGRT form B) on the outcome measures over the period of the evaluation (May/June 2017 and June/July 2018). The NGRT provides two versions of the digital tests, A and B, and a different test was used for pre‐ and post‐testing to avoid potential practice effects. Of the children (<emph>n</emph> = 4494) who completed both the pre‐ and post‐test, <emph>n</emph> = 3198 were from the universal group and <emph>n</emph> = 1296 were from the targeted group. In addition, schools provided us with data on pupil names, date of Birth, and their Unique Pupil Number (UPN) which was matched with the National Pupil Database (NPD) to obtain accurate information on whether the pupil had ever received FSM which is a proxy for pupil disadvantage (Gorard & Siddiqui, [<reflink idref="bib13" id="ref52">13</reflink>]).</p> <hd id="AN0173485905-14">Statistical methods</hd> <p>Our analysis was conducted on an intention‐to‐treat basis and was carried out using STATA version IC15.1. We estimated the main effects of the intervention using multilevel mixed effects regression modelling to take account of the clustered nature of the data (clustered at the school level) and a series of models were estimated for each outcome (where pupil is level 1 and school is level 2). Mixed effects models account for both fixed (clustered at the school level) and random effects. Our analysis was not conducted at the pupils' individual class level. The analysis methodology, including detail on specific regression models and calculation of effect sizes, has been published in the peer‐reviewed, pre‐publication, analysis plan paper (O'Hare et al., [<reflink idref="bib26" id="ref53">26</reflink>]). As such, we have not deviated from our pre‐published primary and secondary analysis reporting template. The reported results include raw scores for transparency of interpretation alongside effect sizes (Hedges g) and statistical significance. Data were archived and made available to an external and independent analysis team for the purposes of verifying the primary analysis.</p> <p>Firstly, two models were conducted for the universal intervention with the NGRT comprehension score and overall reading score forming the dependent variable of each model and the independent variables including a dummy variable representing whether the child was a member of the intervention or control group (coded '1' and '0' respectively) and pupils' baseline scores at pre‐test. We repeated the same analysis for the targeted intervention. Each primary analysis model was conducted using standardized pre‐test scores as a covariate (<emph>z</emph>‐scores) to control for variation in pre‐test scores between groups (O'Hare et al., [<reflink idref="bib24" id="ref54">24</reflink>]).</p> <p>Those schools that dropped out of delivering the intervention were encouraged to allow post‐testing. Following the intention‐to‐treat methodology, schools and participants that were lost‐to‐follow up were included in as many analyses as possible. The pre‐test data from drop‐out schools was still included in raw means and the maximum number of participants was used for each analysis model, that is, pupils were included in our analysis Model 1 even if they were missing from analysis Model 2.</p> <p>Our trial protocol specified that if missing data was higher than 5%, a 'missing at random' analysis would be carried out to determine if multiple imputation was required. This was carried out by running chi‐squared analyses of group allocation and missingness (missingness was a dummy variable, coding '1' for missing outcome data and '0' for not missing). Multiple imputation was performed using chained iterations to fill in missing values in multiple variables using univariate imputation with fully conditional specification of prediction equations. Twenty imputations were carried out to reduce the risk of the 'Monte Carlo' error (simulation error). Twenty imputations are recommended for between 10% and 30% of missing data (Graham et al., [<reflink idref="bib15" id="ref55">15</reflink>]). A total of 200 iterations (with a burn‐in of 10) were carried out, and estimates were combined using Rubin's pooling rules. This then allowed the primary analysis to be repeated using the imputed data, and the difference in primary effects was compared with the complete case analysis to investigate if the presence of missing data impacted the results. Levels of missing data for each outcome and complete case analysis are reported in the 'Sensitivity analysis: multiple imputation' section of the findings.</p> <hd id="AN0173485905-15">RESULTS</hd> <p></p> <hd id="AN0173485905-16">Participant flow through the study</hd> <p>Participant flow is detailed in Figures 2 and 3. One hundred schools were pretested and randomized (51 intervention and 49 control). Two of these intervention schools were excluded after pretesting and randomisation as they were ineligible for the trial but had not disclosed this on their signed memorandum of understanding. This meant that 98 eligible schools had been pretested and randomized. In the universal intervention sample, <emph>n</emph> = 3699 pupils (<emph>n</emph> = 98 schools) were pretested. The exclusion of two schools = occurred post‐randomisation. At the point of analysis, there were <emph>n</emph> = 3198 pupils. This is an overall attrition rate of 13.5% for the universal intervention sample. Attrition for intervention group was 14.43% and attrition for the control group was 12.56%. In the targeted intervention sample, <emph>n</emph> = 1523 pupils (<emph>n</emph> = 98 schools) were pretested. At the point of analysis, there were <emph>n</emph> = 1296 pupils. This is an overall attrition rate of 14.9% for the targeted intervention sample. In the targeted intervention sample, attrition for the intervention group was 16.4% and attrition for the control group was 13.2%. Any pupils missing from post‐test who were from schools that remained active in the trial were not present for post‐test, either due to absence or leaving the school. The analysed <emph>N</emph> is lower than the baseline <emph>N</emph>, as pupils needed both pre‐test and post‐test scores to be included in the complete case analysis models (both these variables are included in the primary models). Data withdrawal forms were issued before pretesting, and schools were instructed not to pre‐test pupils who had withdrawn from the data collection. The pre‐test data collection was the point at which the evaluation team first received any pupil‐level data from schools. The number of pupil withdrawals is, therefore, not calculated. The analysed N refers to the number of pupils who the evaluation team received both pre‐test and post‐tests for (i.e. who sat both NGRT pre‐tests and post‐tests resulting in an overall reading score for both).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6KX/01dec23/bjep12623-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjep12623-fig-0002.jpg" title="2 Participant flow: Universal intervention." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/6KX/01dec23/bjep12623-fig-0003.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjep12623-fig-0003.jpg" title="3 Participant flow: Targeted intervention." /> </p> <p></p> <hd id="AN0173485905-19">Baseline data</hd> <p>Table 2 displays the baseline data for the universal Year 4 programme evaluation. It shows the school and pupil baseline data for the intervention and control groups. School level characteristics are broken down by Ofsted rating (an independent rating of school quality) and school type. Pupil characteristics include number of pupils per school, % eligibility for FSM (indicator of disadvantage) and pre‐test scores on primary outcome measures. Analysis of pre‐test differences on outcomes show that the effect sizes are very small between control and intervention schools. This means that the groups did not differ strongly for reading scores before the intervention was delivered.</p> <p>2 TABLE Balance at baseline in intervention and control groups: universal intervention.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">School‐level (categorical)</th><th align="left">Intervention group</th><th align="left">Control group</th></tr><tr><th align="left"><italic>n</italic>/<italic>N</italic> (missing)</th><th align="left">Count (%)</th><th align="left"><italic>n</italic>/<italic>N</italic> (missing)</th><th align="left">Count (%)</th></tr></thead><tbody valign="top"><tr><td>Ofsted rating:</td><td>49 (0)</td><td /><td>49 (0)</td><td align="left" /></tr><tr><td>Outstanding</td><td /><td>14.3</td><td /><td align="left">8.2</td></tr><tr><td>Good</td><td /><td>69.4</td><td /><td align="left">83.7</td></tr><tr><td>Satisfactory</td><td /><td>0</td><td /><td align="left">0</td></tr><tr><td>Inadequate</td><td /><td>0</td><td /><td align="left">2</td></tr><tr><td>Requires improvement</td><td /><td>16.3</td><td /><td align="left">6.1</td></tr><tr><td>Academy (converter)</td><td>4</td><td>8.2</td><td>8</td><td align="left">16.33</td></tr><tr><td>Academy (sponsor led)</td><td>4</td><td>8.2</td><td>2</td><td align="left">4.08%</td></tr><tr><td>Community school</td><td>25</td><td>51</td><td>23</td><td align="left">46.94</td></tr><tr><td>Faith school (academy converter)</td><td>1</td><td>2</td><td>1</td><td align="left">2.04</td></tr><tr><td>Faith school (foundation school)</td><td>0</td><td>0.00</td><td>1</td><td align="left">2.04</td></tr><tr><td>Faith school (voluntary aided)</td><td>7</td><td>14.3</td><td>7</td><td align="left">14.29</td></tr><tr><td>Faith school (voluntary controlled)</td><td>3</td><td>6.1</td><td>3</td><td align="left">6.12</td></tr><tr><td>Foundation school</td><td>4</td><td>8.2</td><td>4</td><td align="left">8.16</td></tr><tr><td>Free school</td><td>1</td><td>2</td><td>0</td><td align="left">.00</td></tr><tr><td>School‐level (continuous)</td><td align="left">Mean</td><td align="left">Mean</td></tr><tr><td>Y4 Pupil N per school at baseline</td><td align="left">39</td><td align="left">36</td></tr><tr><td>Pupil‐level (categorical)</td><td>N</td><td>Count (%)</td><td>N</td><td align="left">Count (%)</td></tr><tr><td>Eligible for FSM</td><td>348</td><td>18.34</td><td>418</td><td align="left">20.38</td></tr><tr><td>Pupil‐level (continuous)</td><td>n (missing)</td><td>Mean (SD)</td><td>n (missing)</td><td>Mean (SD)</td><td>Effect size</td></tr><tr><td>Pre‐test NGRT overall reading scale (primary outcome)</td><td>1947 (104)</td><td>248.54 (62.72)</td><td>1752 (146)</td><td>246.10 (64.64)</td><td>.04 (CI: −.03 to.10)</td></tr><tr><td>Pre‐test NGRT reading comprehension (primary outcome)</td><td>1832 (219)</td><td>248.61 (61.39)</td><td>1619 (279)</td><td>247 (62.24)</td><td>0.03 (CI: –.04 to.09)</td></tr></tbody></table> </ephtml> </p> <p>Table 3 shows the baseline data for the targeted Years 5 and 6 programme evaluation. It is noted that there are small non‐significant effect sizes for the difference between control and intervention schools in the targeted sample at baseline on the NGRT pre‐test for primary outcomes. Effect sizes of.2 or over would indicate that the sample groups were substantively different, but it should be noted that these effect sizes were found to be in the within a range of.15 to.18 for the three reading score variables. This means that prior to the intervention being delivered, the intervention schools were already performing at a slightly higher level on the reading outcomes than the control schools. However, the effect sizes calculated for the primary analysis later in the report are calculated in a way that controls for imbalance at pre‐test. Where pre‐test scores were included as a covariate in the analysis models, they were standardized to have a mean of zero, to control for pre‐test imbalances. The imbalance at baseline, therefore, may have meant that the intervention was being delivered in a group of schools who had slightly higher reading attainment, but it does not strongly impact the interpretation of the results at post‐test.</p> <p>3 TABLE Balance at baseline in intervention and control groups: targeted intervention.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">School‐level (categorical)</th><th align="left">Intervention group</th><th align="left">Control group</th></tr><tr><th align="left"><italic>N</italic> (missing)</th><th align="left">Count (%)</th><th align="left"><italic>N</italic> (missing)</th><th align="left">Count (%)</th></tr></thead><tbody valign="top"><tr><td>Ofsted rating:</td><td>49 (0)</td><td /><td>49 (0)</td><td align="left" /></tr><tr><td>Outstanding</td><td /><td>14.3</td><td /><td align="left">8.2</td></tr><tr><td>Good</td><td /><td>69.4</td><td /><td align="left">83.7</td></tr><tr><td>Satisfactory</td><td /><td>0</td><td /><td align="left">0</td></tr><tr><td>Inadequate</td><td /><td>0</td><td /><td align="left">2</td></tr><tr><td>Requires improvement</td><td /><td>16.3</td><td /><td align="left">6.1</td></tr><tr><td>Academy (converter)</td><td>4</td><td>8.2</td><td>8</td><td align="left">16.33</td></tr><tr><td>Academy (sponsor led)</td><td>4</td><td>8.2</td><td>2</td><td align="left">4.08</td></tr><tr><td>Community school</td><td>25</td><td>51</td><td>23</td><td align="left">46.94</td></tr><tr><td>Faith school (academy converter)</td><td>1</td><td>2</td><td>1</td><td align="left">2.04</td></tr><tr><td>Faith school (foundation school)</td><td>0</td><td>0</td><td>1</td><td align="left">2.04</td></tr><tr><td>Faith school (voluntary aided)</td><td>7</td><td>14.3</td><td>7</td><td align="left">14.29</td></tr><tr><td>Faith school (voluntary controlled)</td><td>3</td><td>6.1</td><td>3</td><td align="left">6.12</td></tr><tr><td>Foundation school</td><td>4</td><td>8.2</td><td>4</td><td align="left">8.16</td></tr><tr><td>Free school</td><td>1</td><td>2</td><td>0</td><td align="left">.00</td></tr><tr><td>School‐level (continuous)</td><td align="left">Mean</td><td align="left">Mean</td></tr><tr><td>Y5 & Y6 pupil N per school at baseline</td><td align="left">16</td><td align="left">15</td></tr><tr><td>Pupil‐level (categorical)</td><td>N</td><td>Count (%)</td><td>N</td><td align="left">Count (%)</td></tr><tr><td>Eligible for FSM</td><td>158</td><td>17.67</td><td>182</td><td align="left">22.75</td></tr><tr><td>Pupil‐level (continuous)</td><td>n (missing)</td><td>Mean (SD)</td><td>n (missing)</td><td>Mean (SD)</td><td>Effect Size</td></tr><tr><td>Pre‐test NGRT overall reading Scale (primary outcome)</td><td>811 (83)</td><td>277.05 (45.30)</td><td>712 (88)</td><td>268.57 (47.30)</td><td>.18 (CI:.08–.28)</td></tr><tr><td>Pre‐test NGRT reading comprehension (primary outcome)</td><td>804 (90)</td><td>270.57 (53.59)</td><td>704 (96)</td><td>262.70 (52.33)</td><td>.15 (CI:.05–.25)</td></tr><tr><td>Pre‐test NGRT reading accuracy (secondary outcome)</td><td>811 (83)</td><td>289.76 (43.16)</td><td>712 (88)</td><td>281.59 (46.415)</td><td>.18 (CI:.08–.28)</td></tr></tbody></table> </ephtml> </p> <hd id="AN0173485905-20">Outcomes and estimation</hd> <p>Overall, the targeted version of FFT Literacy RR programme had significant positive effects on both primary and secondary reading outcomes, as shown in Table 4. Specifically, the targeted version of the programme improved primary outcomes of overall reading (effect size <emph>g</emph> = .14) and reading comprehension (effect size <emph>g</emph> = .18) However, the universal version of the programme showed no significant improvements in terms of the primary outcomes with no significant effects on either overall pupil reading or pupil reading comprehension (Table 4). The school‐level intra‐cluster correlation coefficient (ICC) was calculated at the analysis stage for the universal sample as ICC = .04, and as ICC = .18 for the targeted sample.</p> <p>4 TABLE Evaluation of effects on primary outcomes of reading ability and reading comprehension for both universal and targeted programmes: Complete case analysis.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Covariate</th><th align="left">Raw means</th><th align="left" /></tr><tr><th align="left">Outcome</th><th align="left">Treatment group</th><th align="left" char="[">Control group</th><th align="left"><italic>N</italic> in model (intervention; control)</th><th align="left">Effect size</th><th align="left" /></tr><tr><th align="left">Primary outcome</th></tr><tr><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left">Hedges <italic>g</italic> [95% CI]</th><th align="left"><italic>p</italic></th></tr></thead><tbody valign="top"><tr><td align="left">Universal version on primary outcomes</td></tr><tr><td>NGRT overall reading ability</td><td>Pre‐test NGRT overall reading ability</td><td>1770 (281)</td><td>284.98 [57.28]</td><td>1678 (220)</td><td>281.72 [61.81]</td><td>3198 (1666, 1532)</td><td>.00 [−.06,.07]</td><td>.795</td></tr><tr><td>NGRT reading comprehension</td><td>Pre‐test NGRT reading comprehension</td><td>1722 (329)</td><td>288.06 [57.64]</td><td>1605 (293)</td><td>288.52 [57.13]</td><td>2958 (1557, 1401)</td><td>−.02 [−.09,.06]</td><td>.494</td></tr><tr><td align="left">Targeted version on primary outcomes</td></tr><tr><td>NGRT overall reading score</td><td>Pre‐test NGRT overall reading score</td><td>761 (133)</td><td>309.71 [47.16]</td><td>706 (94)</td><td>288.46 [49.98]</td><td>1296 (678, 618)</td><td>.14 [.04,.25]</td><td><.001</td></tr><tr><td>NGRT Reading Comprehension</td><td>Pre‐test NGRT Reading Comprehension</td><td>755 (139)</td><td>308.51 [51.38]</td><td>693 (107)</td><td>285.71 [52.06]</td><td>1270 (668, 602)</td><td>.18 [.07,.29]</td><td><.001</td></tr></tbody></table> </ephtml> </p> <p>Table 5 shows that the targeted programme improved the secondary outcomes of reading accuracy and pupil reading comprehension meta‐cognition. There was one significant positive effect of the universal version on secondary reading outcomes, that is, improved pupil reading comprehension meta‐cognition but there was no significant effect on pupil reading accuracy (Table 5).</p> <p>5 TABLE Evaluation of effects on secondary outcomes of reading accuracy and pupil reading comprehension meta‐cognition for both universal and targeted programmes.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">NGRT reading accuracy</th><th align="left">Coef.</th><th align="left">Std. err.</th><th align="left"><italic>z</italic></th><th align="left">Significance</th><th align="left">95% CI</th></tr></thead><tbody valign="top"><tr><td align="left">Universal version on reading accuracy</td></tr><tr><td>Group</td><td>2.21</td><td>2.26</td><td>.98</td><td>.33</td><td>−2.22</td><td>6.65</td></tr><tr><td>Pre‐test NGRT reading accuracy</td><td>36.43</td><td>.76</td><td>47.68</td><td><.001</td><td>34.93</td><td>37.92</td></tr><tr><td>Constant</td><td>289.32</td><td>1.63</td><td>177.90</td><td><.001</td><td>286.13</td><td>292.50</td></tr><tr><td align="left">Universal version on pupil comprehension meta‐cognition</td></tr><tr><td>Group</td><td>1.19</td><td>.31</td><td>3.84</td><td><.001</td><td>.58</td><td>1.79</td></tr><tr><td>Pre‐test ARCMS reading comprehension meta‐cognition</td><td>.06</td><td>.09</td><td>.61</td><td>.55</td><td>−.13</td><td>.24</td></tr><tr><td>Constant</td><td>18.18</td><td>.22</td><td>82.39</td><td><.001</td><td>17.74</td><td>18.61</td></tr><tr><td align="left">Targeted version on reading accuracy</td></tr><tr><td>Group</td><td>8.57</td><td>4.21</td><td>2.04</td><td>.04</td><td>.32</td><td>16.82</td></tr><tr><td>Pre‐test NGRT reading accuracy</td><td>23.62</td><td>1.61</td><td>14.70</td><td><.001</td><td>20.47</td><td>26.77</td></tr><tr><td>Constant</td><td>293.48</td><td>3.06</td><td>96.02</td><td><.001</td><td>287.49</td><td>299.47</td></tr><tr><td align="left">Targeted version and pupil comprehension meta‐cognition (ARCMS)</td></tr><tr><td>Group</td><td>1.13</td><td>.37</td><td>3.08</td><td><.001</td><td>.41</td><td>1.85</td></tr><tr><td>Pre‐test ARCMS reading comprehension</td><td>−.25</td><td>.17</td><td>−1.50</td><td>.13</td><td>−.58</td><td>.08</td></tr><tr><td>Constant</td><td>18.43</td><td>.26</td><td>71.12</td><td><.001</td><td>17.92</td><td>18.94</td></tr></tbody></table> </ephtml> </p> <hd id="AN0173485905-21">Sensitivity analysis: Multiple imputation</hd> <p>The proportion of missing data at pre‐test and post‐test was greater than 5% for both primary outcomes (see Table 6). The control group showed higher numbers of missing data for pre‐test overall reading (<emph>p</emph> < .01) and for pre‐test reading comprehension (<emph>p</emph> < .01). Intervention showed higher numbers of missing data for post‐test overall reading (<emph>p</emph> < .01). This pattern in missingness means that data was presumed to be missing at random.</p> <p>6 TABLE Missing data for each primary outcome scale for universal and targeted samples.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Sample</th><th align="left">Outcome scale</th><th align="left">% missing</th></tr></thead><tbody valign="top"><tr><td>Universal</td><td>Pre‐test overall reading</td><td>6.33</td></tr><tr><td>Post‐test overall reading</td><td>12.68</td></tr><tr><td>Pre‐test comprehension</td><td>12.61</td></tr><tr><td>Post‐test comprehension</td><td>16.03</td></tr><tr><td>Targeted</td><td>Pre‐test overall reading</td><td>10.09</td></tr><tr><td>Post‐test overall reading</td><td>13.40</td></tr><tr><td>Pre‐test comprehension</td><td>10.98</td></tr><tr><td>Post‐test comprehension</td><td>14.52</td></tr></tbody></table> </ephtml> </p> <p>As per trial protocol, multiple imputation analysis was then carried out comparing primary analysis with imputed data and complete cases data. The pattern of results for primary analysis using the two versions of the data set was the same, that is, all significant and non‐significant results remained the same. This analysis of the data suggests that missing data do not negatively impact the strength of the evidence for efficacy of the targeted intervention or the evidence of a lack of efficacy for the universal intervention.</p> <hd id="AN0173485905-22">Subgroup analysis for disadvantaged pupils</hd> <p>The primary analyses were repeated for disadvantaged pupils, that is, only those in that have ever received FSM.</p> <p>Table 7 shows a significant effect of the universal intervention was found for Overall Reading Ability for FSM pupils (effect size <emph>g</emph> = .08). This means that, although there was no effect of the Universal intervention for the overall sample there was a small positive effect of the universal intervention for FSM pupils. However, no significant effect of the universal intervention was found for Reading Comprehension for FSM pupils.</p> <p>7 TABLE Evaluation of effects on primary outcomes of reading ability and reading comprehension of universal programme for FSM subgroup.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Covariate</th><th align="left">Raw means</th><th align="left" /></tr><tr><th align="left">Outcome</th><th align="left">Treatment group</th><th align="left" char="[">Control group</th><th align="left"><italic>N</italic> in model (intervention; control)</th><th align="left">Effect size</th><th align="left" /></tr><tr><th align="left">Primary outcome</th></tr><tr><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left">Hedges <italic>g</italic> [95% CI]</th><th align="left"><italic>p</italic></th></tr></thead><tbody valign="top"><tr><td>NGRT overall reading ability</td><td>Pre‐test NGRT overall reading ability</td><td>618 (127)</td><td>268.42 [56.39]</td><td>593 (112)</td><td>259.96 [66.23]</td><td>1120 (572; 548)</td><td>.08 [−.04,.2]</td><td align="char" char=".">.05</td></tr><tr><td>NGRT reading comprehension</td><td>Pre‐test NGRT reading comprehension</td><td>594 (151)</td><td>270.76 [57.06]</td><td>547 (158)</td><td>270.76 [52.09]</td><td>989 (512; 477)</td><td>−.05 [−.17,.08]</td><td align="char" char=".">.37</td></tr></tbody></table> </ephtml> </p> <p>Table 8 shows significant positive effects of the Targeted Intervention was found on both overall reading (ES = .2) and reading comprehension (ES = .25) for FSM pupils. This shows that disadvantaged pupils in this study benefitted from the targeted intervention in terms of both overall reading and reading comprehension. These effect sizes are bigger in both cases than in the full population seen in Table 4.</p> <p>8 TABLE Evaluation of effects on primary outcomes of reading ability and reading comprehension of targeted programme for FSM subgroup.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Covariate</th><th align="left">Raw means</th><th align="left" /></tr><tr><th align="left">Outcome</th><th align="left">Treatment group</th><th align="left">Control group</th><th align="left"><italic>N</italic> in model (intervention; control)</th><th align="left">Effect size</th><th align="left" /></tr><tr><th align="left">Primary outcome</th></tr><tr><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left"><italic>n</italic> (missing)</th><th align="left">Mean Post‐test [<italic>SD</italic>]</th><th align="left">Hedges <italic>g</italic> [95% CI]</th><th align="left"><italic>p</italic></th></tr></thead><tbody valign="top"><tr><td>NGRT Overall reading score</td><td>Pre‐test NGRT Overall reading score</td><td align="char" char="(">290 (52)</td><td>302.72 [47.10]</td><td align="char" char="(">305 (54)</td><td>282.54 [49.70]</td><td align="char" char="(">543 (280; 263)</td><td align="char" char="[">.2 [.03,.37]</td><td align="char" char=".">.006</td></tr><tr><td>NGRT Reading Comprehension</td><td>Pre‐test NGRT Reading Comprehension</td><td align="char" char="(">286 (56)</td><td>302.37 [48.43]</td><td align="char" char="(">299 (60)</td><td>279.68 [50.93]</td><td align="char" char="(">529 (273; 256)</td><td align="char" char="[">.25 [.08,.42]</td><td align="char" char="."><.001</td></tr></tbody></table> </ephtml> </p> <hd id="AN0173485905-23">Replication and verification of primary analysis</hd> <p>Primary outcome data was shared with an external and independent analysis team from another institution, who re‐analysed the data and verified the results reported here. The replication analysis was commissioned by the funder (Education Endowment Foundation – EEF) for internal checks and decision‐making before regranting the intervention. The analysis is not publicly available but may be available on request to the EEF.</p> <hd id="AN0173485905-24">DISCUSSION</hd> <p>Looking back at the logic model (Figure 1) the results provide evidence supporting the programme's theory of intervention (i.e. programme activities are producing outcome change) for the targeted version of FFT Literacy RR. That is, RR improves reading comprehension and reading ability of Years 5 and 6 pupils (aged 9–11) in UK schools. However, the findings do not suggest that the universal version of the programme for Year 4 (age 8 to 9) pupils provided the same benefits.</p> <p>So why did the targeted version have better effects than the universal version? From the data and findings, it is not possible to attribute an experimental or causal explanation for why the targeted version worked and the universal version did not, as they differed in more ways than targeted pupil selection on comprehension deficits for example, age of pupils and size of delivery groups were also different across the two interventions. In reality, this is two different trials (with similar methods) rather than two arms being compared within a single trial. However, there is some correlational evidence from the study along with theory and literature that can help us generate some hypotheses (not causality) for the different results. The pupils in the targeted version were specifically chosen as those who best aligned with the programme theory of intervention. That is, it helps develop comprehension ability in those who are typically developing in reading accuracy/decoding but below average in reading comprehension. This notion is also supported by the fact that the significant effect on comprehension was slightly larger (effect size <emph>g</emph> = .18) than on overall reading (effect size <emph>g</emph> = .14) within the targeted group.</p> <p>Another hypothesis could be that the older age of the readers 9–11 (Years 5 and 6) in the targeted intervention were better able to understand and apply the four core concepts of the programme (predicting, clarifying, questioning and summarizing) than the younger readers in the universal version aged 8–9 (Year 4). This would concur with the previous research on younger students being less able to engage with meta‐cognitive or strategic thinking about reading (Duffy, [<reflink idref="bib7" id="ref56">7</reflink>]; Hacker & Tenent, [<reflink idref="bib16" id="ref57">16</reflink>]).</p> <p>A common explanation for variation in reading intervention success is variation in dosage, that is, the higher the dosage, the more effective the intervention. In this study, however, the universal intervention was delivered at a higher dosage but was less effective than the targeted intervention. Although the recommended dosage of the targeted intervention is higher than the recommended dosage of the universal intervention, this does not mean that any deviation from this makes a school non‐compliant with the intervention. The much higher‐than‐minimum dosage found in all intervention schools underlines how feasible and successfully implemented the programme was in this study. Again, we cannot be sure that the minimum effective dosage declared by the developers is in fact 'effective', and the minimum effective dosage remains untested due to the high dosage in this study.</p> <p>The findings also provide some evidence to support the theory of change within the logic model (i.e. developmental pathways between outcome changes) in that there were improvements in both the proximal secondary outcomes of reading comprehension metacognition and improved reading accuracy and distal primary outcomes of reading comprehension and overall reading attainment. This chain of outcome change is still a working hypothesis and, again, further analysis could be helpful to strengthen this theory of change.</p> <p>The sub‐group analysis suggested that the benefits of RR are even greater for disadvantaged pupils. Even the universal version provided some significant benefits to disadvantaged pupils. These findings suggest that socio‐economic disadvantage could be considered in the selection process for targeted intervention. So, in addition to selecting pupils based on teacher's perceptions of pupils' poor reading comprehension ability and normal decoding skills, the teachers could also use disadvantage as an additional variable for selecting pupils to receive the targeted programme. Inclusion of disadvantage could improve the reliability and validity of the targeting process as pupils selected solely on teacher perceptions of pupil reading accuracy and comprehension is open to teacher perception bias or error. One caveat to this is, it may be that the disadvantage benefits of the programme are acting at the school level and therefore, selecting schools for the intervention based on higher levels of disadvantage (rather than individual pupils) is enough to confer these additional benefits. Generally, more study is required to clarify the benefits of RR for addressing disadvantages and inequality in outcomes.</p> <p>So overall, this evaluation provides strong support for further implementation of the targeted version of FFT Literacy RR intervention. The main reasons being the consistent evidence of its effect on reading outcomes; its focus on pupils who benefit most through a targeting process; its potential for reducing the attainment gap between disadvantaged and non‐disadvantaged pupils; the lack of strong evidence for other reading comprehension interventions; and it is low cost because it uses a teacher professional development model.</p> <hd id="AN0173485905-25">Limitations</hd> <p>The generalisability of these findings is quite good for pupils of similar characteristics to the sample tested, that is, Years 4, 5 and 6 pupils in English primary schools with above average levels of pupils in receipt of FSM, due to the scale and size of this study. However, generalisability to other age groups may be less certain.</p> <p>A limitation of this study is that there was some potential for teacher selection bias in their identification of pupils in receipt of the targeted version of the programme. However, both control and intervention teachers did make their selections before randomisation and their choices were based on standardized guidance based on the simple view of reading (Gough & Tunmer, [<reflink idref="bib14" id="ref58">14</reflink>]; Hoover & Gough, [<reflink idref="bib18" id="ref59">18</reflink>]). It could be argued that an objective method with evidence of reliability and validity is required. There is merit in this argument and there is a need for more research on the selection method. However, there is an overriding practical concern that if the programme is to be scaled up and available to more teachers, they need a simple method and does not involve complicated selection instruments.</p> <p>A threat to internal validity is the risk associated with children joining the trial after pre‐testing and randomisation, thus introducing group imbalance beyond that at the point of pre‐testing. The number of children who were missing from the pre‐test and randomisation data was <emph>n</emph> = 104 and <emph>n</emph> = 146 for intervention and control in the universal intervention sample and <emph>n</emph> = 83 and <emph>n</emph> = 88 for the targeted intervention sample.</p> <p>A limitation of the analysis is that class‐level clustering was not included in the multi‐level models. We collected data at the school level via digital NGRT testing but data was not gathered on class (for the Universal intervention) or for targeted delivery group (for the Targeted intervention). Considering the intervention was delivered in a clustered way, below school level, that is, class or targeted group it is possible that variance would have been observed between teachers/teaching assistants who delivered the intervention. Secondly, this means that class or group‐level peer effects are not considered. This must be addressed in future evaluations of Reciprocal Reading and reduces the explanatory power of the present analysis.</p> <p>Another potential issue with this evaluation was that the NGRT test uses adaptive testing. Therefore, pupils who perform very poorly on accuracy questions may not get access to the comprehension questions, although this may not have impacted upon the pupils in the targeted group as they were selected on normal accuracy ability. Also, distributions of the pre‐test and post‐test of primary outcomes showed no floor or ceiling effects.</p> <p>One further issue about measures is that the secondary outcome measures were researcher‐created proximal measures as no existing tests had the specificity to measure these outcomes (e.g. pupil comprehension meta‐cognition). However, these tests were designed with scrutiny of content validity and post hoc tests of reliability and validity were conducted. In addition, these secondary measures are mainly to explore the theory of change rather than effectiveness of the intervention. Effectiveness of the programme should mainly rest on the performance of the intervention on the primary standardized outcome measures of comprehension and overall reading attainment.</p> <p>A limitation to the dosage analysis is that it is possible that high‐dosage schools were more likely to submit dosage data, thus under‐representing schools with low‐dosage in the analysis and interpretation.</p> <p>Finally, the sample sizes for the subgroup analyses are comparatively smaller than the primary analyses in the main report, but still of a substantial size. However, the interpretation of the results should take sample size into consideration.</p> <hd id="AN0173485905-26">CONCLUSION</hd> <p>In conclusion, it is an important finding that this study demonstrated an effective application of RR instruction. This importance is enhanced due to the proportion of language and literacy‐focused interventions rigorously evaluated in the United Kingdom not showing significant effects on reading outcomes (Education Endowment Foundation, [<reflink idref="bib8" id="ref60">8</reflink>]). Furthermore, with the Department for Education ([<reflink idref="bib6" id="ref61">6</reflink>]) in the UK advocating a universal focus on comprehension instruction, a targeted version of FFT Literacy RR can offer additional support to those who may not benefit from the curriculum's universal approach.</p> <p>Finally, this evaluation shows that even if an intervention is underpinned by evidence‐based practice and has a strong theory of change, implementation factors such as appropriate targeting and the developmental stage of participants can still determine the effectiveness of reading comprehension interventions.</p> <hd id="AN0173485905-27">AUTHOR CONTRIBUTIONS</hd> <p> <bold>Liam O'Hare:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Patrick Stark:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Maria Cockerill:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Katrina Lloyd:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Shelia McConnellogue:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Aideen Gildea:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Andy Biggart:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing. <bold>Christine Bower:</bold> Data curation; formal analysis; investigation; methodology; project administration; writing – original draft; writing – review and editing. <bold>Paul Connolly:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; supervision; validation; visualization; writing – original draft; writing – review and editing.</p> <hd id="AN0173485905-28">ACKNOWLEDGEMENTS</hd> <p>The authors would like to acknowledge the contributions and participation of all the schools, teachers and pupils involved in this study. They also gratefully acknowledge the funding provided by the Education Endowment Foundation to conduct the study.</p> <hd id="AN0173485905-29">CONFLICT OF INTEREST STATEMENT</hd> <p>None declared.</p> <hd id="AN0173485905-30">DATA AVAILABILITY STATEMENT</hd> <p>The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.</p> <ref id="AN0173485905-31"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref12" type="bt">1</bibl> <bibtext> Borman, G. D., Slavin, R. E., Cheung, A. C., Chamberlain, A. M., Madden, N. A., & Chambers, B. (2007). Final reading outcomes of the national randomized field trial of success for all. American Educational Research Journal, 44 (3), 701 – 731. https://doi.org/10.3102/0002831207306743</bibtext> </blist> <blist> <bibl id="bib2" idref="ref13" type="bt">2</bibl> <bibtext> Chiu, C. W. T. (1998). Synthesizing metacognitive interventions: What training characteristics can improve reading performance? [annual meeting presentation]. American Educational Research Association annual meeting, San Diego, CA. r.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref14" type="bt">3</bibl> <bibtext> Connolly, P., Biggart, A., Miller, S., O'Hare, L., & Thurston, A. (2017). Using Randomised Controlled Trials in Education. SAGE.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref9" type="bt">4</bibl> <bibtext> De Corte, E., Verschaffel, L., & Van De Ven, A. (2001). Improving text comprehension strategies in upper primary school children: A design experiment. British Journal of Educational Psychology, 71 (4), 531 – 559. https://doi.org/10.1348/000709901158668</bibtext> </blist> <blist> <bibl id="bib5" idref="ref3" type="bt">5</bibl> <bibtext> Department for Education. (2013a). English programmes of study: key stages 1 and 2 National curriculum in England. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/335186/PRIMARY_national_curriculum_‐_English_220714.pdf</bibtext> </blist> <blist> <bibl id="bib6" idref="ref61" type="bt">6</bibl> <bibtext> Department for Education. (2013b). The national curriculum in England: Framework document. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/381344/Master_final_national_curriculum_28_Nov.pdf</bibtext> </blist> <blist> <bibl id="bib7" idref="ref27" type="bt">7</bibl> <bibtext> Duffy, G. G. (1993). Teachers' progress toward becoming expert strategy teachers. The Elementary School Journal, 94 (2), 109 – 120. https://doi.org/10.1086/461754</bibtext> </blist> <blist> <bibl id="bib8" idref="ref60" type="bt">8</bibl> <bibtext> Education Endowment Foundation. (2018). Completed projects. London: Education Endowment Foundation. https://educationendowmentfoundation.org.uk/projects‐and‐evaluation/reports/</bibtext> </blist> <blist> <bibl id="bib9" idref="ref7" type="bt">9</bibl> <bibtext> Education Endowment Foundation. (2022a). Sutton trust‐education endowment foundation teaching and learning toolkit. London: Education Endowment Foundation. https://educationendowmentfoundation.org.uk/evidence‐summaries/teaching‐learning‐toolkit</bibtext> </blist> <blist> <bibtext> Education Endowment Foundation. (2022b). Reading comprehension strategies. London: Education Endowment Foundation. https://educationendowmentfoundation.org.uk/education‐evidence/teaching‐learning‐toolkit/reading‐comprehension‐strategies</bibtext> </blist> <blist> <bibtext> Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel's meta‐analysis. Review of Educational Research, 71 (3), 393 – 447. https://doi.org/10.3102/003465430710033</bibtext> </blist> <blist> <bibtext> GL Assessment. (2018). Technical information. New Group Reading Test (NGRT) Digital Edition. https://support.gl‐assessment.co.uk/knowledge‐base/assessments/ngrt‐support/general‐information/technical‐guidance/</bibtext> </blist> <blist> <bibtext> Gorard, S., & Siddiqui, N. (2019). How trajectories of disadvantage help explain school attainment. SAGE Open, 9 (1), 215824401882517. https://doi.org/10.1177/2158244018825171</bibtext> </blist> <blist> <bibtext> Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and Special Education, 7 (1), 6 – 10. https://doi.org/10.1177/074193258600700104</bibtext> </blist> <blist> <bibtext> Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8 (3), 206 – 213. https://doi.org/10.1007/s11121‐007‐0070‐9</bibtext> </blist> <blist> <bibtext> Hacker, D. J., & Tenent, A. (2002). Implementing reciprocal teaching in the classroom: Overcoming obstacles and making modifications. Journal of Educational Psychology, 94 (4), 699 – 718. https://doi.org/10.1037/0022‐0663.94.4.699</bibtext> </blist> <blist> <bibtext> Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., Lamb, S. E., Dixon‐Woods, M., McCulloch, P., Wyatt, J. C., Chan, A., & Michie, S. (2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ, 348, g1687. https://doi.org/10.1136/bmj.g1687</bibtext> </blist> <blist> <bibtext> Hoover, W. A., & Gough, P. B. (1990). The simple view of reading. Reading and Writing, 2 (2), 127 – 160. https://doi.org/10.1007/BF00401799</bibtext> </blist> <blist> <bibtext> Kispal, A. (2008). Effective teaching of inference skills for children: Literature review. Research Report DCSF‐RR031. National Foundation for Educational Research. https://<ulink href="http://www.nfer.ac.uk/publications/edr01/edr01.pdf">www.nfer.ac.uk/publications/edr01/edr01.pdf</ulink></bibtext> </blist> <blist> <bibtext> Maxwell, B., Connolly, P., Demack, S., O'Hare, L., Stevens, A., & Clague, L. (2014). Summer active reading programme: evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> McPhillips, M., & Sheehy, N. (2004). Prevalence of persistent primary reflexes and motor problems in children with reading difficulties. Dyslexia, 10 (4), 316 – 338. https://doi.org/10.1002/dys.282</bibtext> </blist> <blist> <bibtext> National Reading Panel. (2000). Teaching children to read: An evidence‐based assessment of the scientific research literature on reading and its implications for reading instruction. National Institute of Child Health and Human Development.</bibtext> </blist> <blist> <bibtext> Nunnally, J., & Bernstein, L. (1994). Psychometric theory (3rd ed.). McGraw‐Hill Higher.</bibtext> </blist> <blist> <bibtext> O'Hare, L., Lloyd, K., Stark, P., McConnellogue, S., Biggart, A., & Cockerill, M. (2017). Reciprocal Reading: A training programme for teachers aimed at improving reading skills of pupils. ISRCTN Trial Registry. https://<ulink href="http://www.isrctn.com/ISRCTN81582662">www.isrctn.com/ISRCTN81582662</ulink></bibtext> </blist> <blist> <bibtext> O'Hare, L., Stark, P., Cockerill, M., Lloyd, K., McConnellogue, S., Gildea, A., Biggart, A., Connolly, P., & Bower, C. (2019). Reciprocal reading: Evaluation report. Education Endowment Foundation. https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Reports/Reciprocal_Reading.pdf</bibtext> </blist> <blist> <bibtext> O'Hare, L., Stark, P., McConnellogue, S., Lloyd, K., Cockerill, M., & Biggart, A. (2018). Protocol: A cluster randomised controlled trial of Reciprocal Reading: A teacher training comprehension programme. International Journal of Educational Research, 92, 30 – 42.</bibtext> </blist> <blist> <bibtext> Okkinga, M., van Steensel, A., van Gelderen, A., & van Schooten, E. (2018). Effectiveness of reading strategy intervention in whole classrooms: A meta‐analysis. Educational Psychology Review, 30 (4), 1215 – 1239. https://doi.org/10.1007/s10648‐018‐9445‐7</bibtext> </blist> <blist> <bibtext> Palincsar, A. S. (2013). Reciprocal teaching. In J. Hattie & E. M. Anderman (Eds.), International guide to student achievement (pp. 369 – 371). Routledge.</bibtext> </blist> <blist> <bibtext> Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension‐fostering and comprehension‐monitoring activities. Cognition and Instruction, 1 (2), 117 – 175. https://doi.org/10.1207/s1532690xci0102_1</bibtext> </blist> <blist> <bibtext> Pressley, M. (2002). Reading instruction that works: The case for balanced teaching (4th ed.). Guilford Press.</bibtext> </blist> <blist> <bibtext> Rockets, R. (2023). How to use reciprocal reading. https://<ulink href="http://www.readingrockets.org/strategies/reciprocal%5fteaching">www.readingrockets.org/strategies/reciprocal%5fteaching</ulink></bibtext> </blist> <blist> <bibtext> Rosenshine, B., & Meister, C. (1994). Reciprocal teaching: A review of the research. Review of Educational Research, 64 (4), 479 – 530. https://doi.org/10.3102/00346543064004479</bibtext> </blist> <blist> <bibtext> Sencibaugh, J. M. (2007). Meta‐analysis of reading comprehension interventions for students with learning disabilities: Strategies and implications. Reading Improvement, 44 (10), 6 – 22.</bibtext> </blist> <blist> <bibtext> Swanson, H. (1999). Reading research for students with LD: A meta‐analysis of intervention outcomes. Journal of Learning Disabilities, 32 (6), 504 – 532. https://doi.org/10.1177/002221949903200605</bibtext> </blist> <blist> <bibtext> Thurston, A., Roseth, C., O'Hare, L., Davison, J., & Stark, P. (2016). Talk of the town. Evaluation report and executive summary. Education Endowment Foundation.</bibtext> </blist> <blist> <bibtext> Torgerson, C. J., & Torgerson, D. J. (2007). The use of minimization to form comparison groups in educational research. Educational Studies, 33 (3), 333 – 337. https://doi.org/10.1080/03055690701423267</bibtext> </blist> <blist> <bibtext> Tymms, P., Merrell, C., Thurston, A., Andor, J., Topping, K., & Miller, D. (2011). Improving attainment across a whole district: School reform through peer tutoring in a randomized controlled trial. School Effectiveness and School Improvement, 22 (3), 265 – 289. https://doi.org/10.1080/09243453.2011.589859</bibtext> </blist> </ref> <aug> <p>By Liam O'Hare; Patrick Stark; Maria Cockerill; Katrina Lloyd; Sheila McConnellogue; Aideen Gildea; Andy Biggart; Christine Bower and Paul Connolly</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib11" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib22" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib27" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib30" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib28" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib29" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib19" firstref="ref17"></nolink> <nolink nlid="nl8" bibid="bib34" firstref="ref19"></nolink> <nolink nlid="nl9" bibid="bib33" firstref="ref20"></nolink> <nolink nlid="nl10" bibid="bib32" firstref="ref21"></nolink> <nolink nlid="nl11" bibid="bib10" firstref="ref23"></nolink> <nolink nlid="nl12" bibid="bib21" firstref="ref30"></nolink> <nolink nlid="nl13" bibid="bib20" firstref="ref31"></nolink> <nolink nlid="nl14" bibid="bib35" firstref="ref32"></nolink> <nolink nlid="nl15" bibid="bib31" firstref="ref34"></nolink> <nolink nlid="nl16" bibid="bib26" firstref="ref35"></nolink> <nolink nlid="nl17" bibid="bib17" firstref="ref36"></nolink> <nolink nlid="nl18" bibid="bib24" firstref="ref39"></nolink> <nolink nlid="nl19" bibid="bib37" firstref="ref42"></nolink> <nolink nlid="nl20" bibid="bib14" firstref="ref43"></nolink> <nolink nlid="nl21" bibid="bib18" firstref="ref44"></nolink> <nolink nlid="nl22" bibid="bib36" firstref="ref45"></nolink> <nolink nlid="nl23" bibid="bib12" firstref="ref46"></nolink> <nolink nlid="nl24" bibid="bib25" firstref="ref50"></nolink> <nolink nlid="nl25" bibid="bib23" firstref="ref51"></nolink> <nolink nlid="nl26" bibid="bib13" firstref="ref52"></nolink> <nolink nlid="nl27" bibid="bib15" firstref="ref55"></nolink> <nolink nlid="nl28" bibid="bib16" firstref="ref57"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1399628
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Comparing the Effectiveness of Two Reciprocal Reading Comprehension Interventions for Primary School Pupils in Disadvantaged Schools
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22O'Hare%2C+Liam%22">O'Hare, Liam</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4453-2880">0000-0002-4453-2880</externalLink>)<br /><searchLink fieldCode="AR" term="%22Stark%2C+Patrick%22">Stark, Patrick</searchLink><br /><searchLink fieldCode="AR" term="%22Cockerill%2C+Maria%22">Cockerill, Maria</searchLink><br /><searchLink fieldCode="AR" term="%22Lloyd%2C+Katrina%22">Lloyd, Katrina</searchLink><br /><searchLink fieldCode="AR" term="%22McConnellogue%2C+Sheila%22">McConnellogue, Sheila</searchLink><br /><searchLink fieldCode="AR" term="%22Gildea%2C+Aideen%22">Gildea, Aideen</searchLink><br /><searchLink fieldCode="AR" term="%22Biggart%2C+Andy%22">Biggart, Andy</searchLink><br /><searchLink fieldCode="AR" term="%22Bower%2C+Christine%22">Bower, Christine</searchLink><br /><searchLink fieldCode="AR" term="%22Connolly%2C+Paul%22">Connolly, Paul</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22British+Journal+of+Educational+Psychology%22"><i>British Journal of Educational Psychology</i></searchLink>. 2023 93(4):1123-1145.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 23
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Reading+Comprehension%22">Reading Comprehension</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Instruction%22">Reading Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Disadvantaged+Schools%22">Disadvantaged Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Difficulties%22">Reading Difficulties</searchLink><br /><searchLink fieldCode="DE" term="%22Cooperative+Learning%22">Cooperative Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/bjep.12623
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0007-0998<br />2044-8279
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Effective reading comprehension teaching is an aspiration of education systems across the world. Teaching incorporating reciprocal reading theory and evidence is an internationally popular approach for improving comprehension. Aims: This paper uses two large cluster randomized controlled trials of similar reciprocal reading interventions implemented in different ways to compare their effectiveness. Sample: The two interventions had the same teacher professional development, reciprocal reading activities and dosage/exposure, but varied in their implementation, with one delivered as a whole-class ('universal') version for pupils aged 8-9 years and the other a small group ('targeted') version for pupils aged 9-11 years with specific comprehension difficulties. Methods: Two large-scale cluster RCTs were conducted in 98 schools with N = 3699 pupils in the universal trial and N = 1523 in the targeted trial. Results: Multi-level models showed significant effects for the targeted version of the intervention on pupil reading comprehension (g = 0.18) and overall reading (g = 0.14). No significant effects were found for the whole class version. A sub-group analyses of disadvantaged pupils showed the targeted intervention's effects were even larger on reading comprehension (g = 0.25). Conclusions: The evidence suggested that this reciprocal reading intervention worked best when implemented in small groups and targeted for pupils with specific comprehension difficulties and particularly for pupils in disadvantaged circumstances. Comments: This evaluation shows that even if a reading comprehension intervention is underpinned by strong theory and evidence-based practice, its effectiveness can still depend on implementation choices.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1399628
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1399628
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/bjep.12623
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 23
        StartPage: 1123
    Subjects:
      – SubjectFull: Reading Comprehension
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Reading Instruction
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Disadvantaged Schools
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Reading Difficulties
        Type: general
      – SubjectFull: Cooperative Learning
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
    Titles:
      – TitleFull: Comparing the Effectiveness of Two Reciprocal Reading Comprehension Interventions for Primary School Pupils in Disadvantaged Schools
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: O'Hare, Liam
      – PersonEntity:
          Name:
            NameFull: Stark, Patrick
      – PersonEntity:
          Name:
            NameFull: Cockerill, Maria
      – PersonEntity:
          Name:
            NameFull: Lloyd, Katrina
      – PersonEntity:
          Name:
            NameFull: McConnellogue, Sheila
      – PersonEntity:
          Name:
            NameFull: Gildea, Aideen
      – PersonEntity:
          Name:
            NameFull: Biggart, Andy
      – PersonEntity:
          Name:
            NameFull: Bower, Christine
      – PersonEntity:
          Name:
            NameFull: Connolly, Paul
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 0007-0998
            – Type: issn-electronic
              Value: 2044-8279
          Numbering:
            – Type: volume
              Value: 93
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: British Journal of Educational Psychology
              Type: main
ResultId 1