View in EDS HTML Full Text PDF Full Text

Evaluating the Impact of National Educational Policy to Reduce Retention and Increase Achievement in Compulsory Education

Saved in:
Bibliographic Details
Title: Evaluating the Impact of National Educational Policy to Reduce Retention and Increase Achievement in Compulsory Education
Language: English
Authors: Barata, M. Clara, Calheiros, M. Manuela, Patrício, Joana Nunes, Graça, João, Lima, M. Luisa
Source: Elementary School Journal. Sep 2015 116(1):149-171.
Availability: University of Chicago Press. Journals Division, P.O. Box 37005, Chicago, IL 60637. Tel: 877-705-1878; Tel: 773-753-3347; Fax: 877-705-1879; Fax: 773-753-0811; e-mail: subscriptions@press.uchicago.edu; Web site: http://www.press.uchicago.edu
Peer Reviewed: Y
Page Count: 23
Publication Date: 2015
Document Type: Journal Articles
Reports - Research
Descriptors: Foreign Countries, Educational Policy, Public Policy, School Holding Power, Academic Persistence, Academic Achievement, Compulsory Education, Program Effectiveness, Class Size, Student Characteristics, Individualized Instruction, Grade Inflation
Geographic Terms: Portugal
DOI: 10.1086/682748
ISSN: 0013-5984
Abstract: Despite the accumulated evidence that retention is an ineffective and potentially harmful remedial strategy, several countries struggle with high levels of retention in compulsory schooling. This article provides evidence of the impact of the Portuguese national educational policy "Programa Mais Sucesso Escolar" (PMSE) using class size, class composition, and differentiated instruction to reduce student retention and increase achievement. We found that PMSE significantly reduced retention, but had mixed effects on educational achievement, pointing to evidence of grade inflation on teacher-determined measures of achievement. In the discussion, we analyze the implications of these results for policy, practice, and program evaluation research.
Abstractor: As Provided
Number of References: 56
Entry Date: 2015
Accession Number: EJ1077335
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFyTc1PuXltHsitMU7nJBVMAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDMRGnz3Lh7fGmo_quAIBEICBmxnVBmNUPqTalOPSggZLOGlOVU2G9khvueb8ilUlYcljpoAIS5nQ6Qel-56u9uQ9lgoarH9XQfUp0FUxbrwBtLBvb5P9erJbolvCYPEOslqlHCyrqi2u55qQzpwPnuHkWGUfa3Fb52fYR8Hte3KNGHpnEJmEzSbTbwM4ldxfNG_AuvCGgMGQfzwbGAVBtCnX8dFCgpVO_PRza7q2
Text:
  Availability: 1
  Value: <anid>AN0109514424;esj01sep.15;2018Oct10.09:30;v2.2.500</anid> <title id="AN0109514424-1">EVALUATING THE IMPACT OF NATIONAL EDUCATIONAL POLICY TO REDUCE RETENTION AND INCREASE ACHIEVEMENT IN COMPULSORY EDUCATION. </title> <p>Despite the accumulated evidence that retention is an ineffective and potentially harmful remedial strategy, several countries struggle with high levels of retention in compulsory schooling. This article provides evidence of the impact of the Portuguese national educational policy "Programa Mais Sucesso Escolar" (PMSE) using class size, class composition, and differentiated instruction to reduce student retention and increase achievement. We found that PMSE significantly reduced retention, but had mixed effects on educational achievement, pointing to evidence of grade inflation on teacher-determined measures of achievement. In the discussion, we analyze the implications of these results for policy, practice, and program evaluation research.</p> <p>Despite the accumulated evidence that retention is an ineffective and potentially harmful pedagogical strategy (e.g., Jimerson, [<reflink idref="bib26" id="ref1">26</reflink>]; Shepard & Smith, [<reflink idref="bib49" id="ref2">49</reflink>]), and in spite of changes to state and national policy and regulation to minimize it, grade retention (or grade repetition) in compulsory education is a common practice in many countries. In Europe, retention rates vary significantly between countries, from 2% to 22.4% in primary education, and up to 31.9% in lower secondary education (OECD, [<reflink idref="bib42" id="ref3">42</reflink>]). In the United States, about 10% of students are retained between kindergarten and eighth grade (National Center for Education Statistic, [<reflink idref="bib39" id="ref4">39</reflink>]).</p> <p>To reduce retention and increase school achievement in compulsory education, the Portuguese Ministry of Education (ME) designed a policy designated "Programa Mais Sucesso Escolar" (PMSE) in 2009. At the core of the program were a set of organizational strategies affecting class size, class composition, and differentiated instruction, as well as a small financial incentive to lower retention rates.</p> <p>In this article we present the results of the first and second years of impact of the PMSE on teacher-determined measures of achievement, including success and excellence in the core subject areas (mathematics, Portuguese, and English) and on external standardized measures (i.e., national low-stakes and high-stakes exams). We used a comparison group of schools that applied for the program but did not get it. We then used propensity score weighting of both the PMSE and comparison schools to further reduce any potential differences in the educational achievement of the students from the two groups that were not attributable to the program. The results indicated that PMSE significantly reduced retention, but otherwise had mixed effects on educational achievement, pointing to evidence of grade inflation on teacher-determined measures of achievement. This study contributes to the discussion of what constitutes and how to measure success in policy for reducing grade retention and increasing school achievement in compulsory education.</p> <hd id="AN0109514424-2">The Policy and Practice of Grade Retention</hd> <p>In a large number of countries, at the end of a school year, grade retention can be an option for students who, in spite of the support measures implemented during the course of the year, have not been able to make sufficient progress. In Europe, there is considerable variation in retention rates at the primary level. Some countries such as Greece (2.0%) and Austria (4.9%) have low grade retention rates, while other countries such as France (17.8%), Portugal, and the Netherlands (both at 22.4%) reveal much higher rates. At the lower secondary level, these trends persist with variations between countries' rates ranging from 0.5% in Finland to 31.9% in Spain (OECD, [<reflink idref="bib42" id="ref5">42</reflink>]).</p> <p>In the United States, 5%–10% of students are retained between kindergarten and eighth grade (National Center for Education Statistic, [<reflink idref="bib39" id="ref6">39</reflink>]). Data from the 2004 Current Population Survey showed that grade retention was most common at kindergarten (12%), first grade (20%), and ninth grade (13%) (Davis & Bauman, [<reflink idref="bib7" id="ref7">7</reflink>]). With states facing increasing pressure to ensure that students meet minimum proficiency levels under the No Child Left Behind Act (NCLB) of 2001, and with grade retention viewed as an important alternative to social promotion, some U.S. states have seen sharp increases in the practice of grade retention, particularly at early ages (Hauser, Frederick, & Andrew, [<reflink idref="bib21" id="ref8">21</reflink>]; Zinth, [<reflink idref="bib57" id="ref9">57</reflink>]).</p> <p>There is considerable evidence that retention may have a negative impact on student achievement (Jimerson, [<reflink idref="bib26" id="ref10">26</reflink>]; Jimerson, Anderson, & Whipple, [<reflink idref="bib27" id="ref11">27</reflink>]; OECD, [<reflink idref="bib44" id="ref12">44</reflink>]; Shepard & Smith, [<reflink idref="bib49" id="ref13">49</reflink>]; Silberglitt, Jimerson, Burns, & Appleton, [<reflink idref="bib50" id="ref14">50</reflink>]). In a survey of 47 empirical studies with a variety of academic achievement measures, Holmes ([<reflink idref="bib23" id="ref15">23</reflink>]) found that retained students scored 0.19 to 0.31 standard deviations below comparable students who had not been retained. A variety of studies have also found that retention is associated with an increased likelihood of dropping out (Jimerson et al., [<reflink idref="bib27" id="ref16">27</reflink>]; Roderick, [<reflink idref="bib45" id="ref17">45</reflink>]; Rumberger [<reflink idref="bib47" id="ref18">47</reflink>]). Furthermore, one of the primary criticisms of grade retention is that it disrupts a student's peer and cohort group, and potentially affects their socioemotional development (Fruehwirth, Navarro, & Takahashi, [<reflink idref="bib18" id="ref19">18</reflink>]; Shepard & Smith, [<reflink idref="bib49" id="ref20">49</reflink>]).</p> <p>Despite this evidence, retention is generally seen as a consensual remedial policy intended to provide students additional time to master the skills at their current grade level, with the assumption that repeating a year is beneficial to pupils' learning. This "culture of grade retention" (OECD, [<reflink idref="bib42" id="ref21">42</reflink>], p. 60) seems to determine high rates of retention in some European countries. In Portugal, belief in the positive benefits of repeating a year for underperforming students is still quite common (OECD, [<reflink idref="bib44" id="ref22">44</reflink>]), explaining why the practice is still used, often in spite of limitations imposed by official regulations. For example, the common educational law of 1986 that decreed that teachers should focus practices of student retention at the end of the cycles of schooling, rather than at the end of each school year (Lei n.º 46/86, D.R. nº 237, Série I, de 1986-10-14), seems to have had limited impact on retention rates (Favinha, Silvestre, & Magro-C, [<reflink idref="bib15" id="ref23">15</reflink>]). Data show that in upper-secondary education, retention rates have been reduced by 13.2% in the last 15 years, but still reached rates above 20% in 2010. In compulsory schooling, the reduction in retention rates during the same period was more modest (5.2%), and the retention rate has been stable at around 8%. Despite decreases, in 2012, 34.3% of students in Portugal still reported having repeated a grade at least once in primary, lower secondary, or upper secondary education, making this proportion the fourth highest among OECD countries (OECD, [<reflink idref="bib43" id="ref24">43</reflink>]).</p> <hd id="AN0109514424-3">The "Programa Mais Sucesso Escolar" for Portuguese Compulsory Education</hd> <p>To facilitate changes in retention culture and practice, the Portuguese Ministry of Education (ME) designed in 2009 a policy designated "Programa Mais Sucesso Escolar," a national school-based four-year initiative to reduce student retention and increase school achievement in compulsory education. In 2009, the Portuguese compulsory education system started at age 6 and included three cycles of schooling that lasted a total of 9 years: the first cycle, or primary school, included grades 1–4; the second cycle, or middle school, included grades 5–6; and the third cycle, or lower secondary school, included grades 7–9. PMSE encompassed all three cycles. Transitions between grades and cycles were conditional on the successful completion of the previous grade and cycle, as determined by teachers in the <emph>conselho de turma</emph> (class council), except in ninth grade. Students faced external standardized testing in ninth grade that limited students' access to upper-secondary education (i.e., high stakes). In contrast, students also faced external standardized testing in sixth grade, but this testing was low stakes because it did not limit students' progress to the next cycle.</p> <p>A particularly unusual aspect of PMSE in the context of the Portuguese education system is that it required an application from schools wanting to adhere to the policy. This system of policy adherence was in striking contrast with the otherwise generally centralized national system of education administration, characterized by a top-down approach to the implementation of policy, distribution of human resources, curriculum decision making, and so on (OECD, [<reflink idref="bib41" id="ref25">41</reflink>]). To apply to PMSE, schools had to describe (<reflink idref="bib1" id="ref26">1</reflink>) the human resources that would be involved (e.g., coordinator, team members, other experts); (<reflink idref="bib2" id="ref27">2</reflink>) subject areas, grade levels, and students affected by retention; (<reflink idref="bib3" id="ref28">3</reflink>) goals the school planned to achieve in teacher-determined and external standardized measures for these groups; (<reflink idref="bib4" id="ref29">4</reflink>) a detailed plan of strategies (including professional development) necessary to achieve these goals; (<reflink idref="bib5" id="ref30">5</reflink>) the costs of implementation; and, finally, (<reflink idref="bib6" id="ref31">6</reflink>) the process-evaluation strategies the school would employ to assess whether the intervention was working (Office of the Minister of Education, [<reflink idref="bib40" id="ref32">40</reflink>]).</p> <p>Most of the applications presented by schools were inspired by two intervention approaches that had been recently developed in Portuguese public schools, the "TurmaMais" or "MoreClassroom" (Fialho & Salgueiro, [<reflink idref="bib15" id="ref33">15</reflink>]; Fialho & Verdasca, [<reflink idref="bib16" id="ref34">16</reflink>]; Verdasca, [<reflink idref="bib51" id="ref35">51</reflink>]; Verdasca & Cruz, [<reflink idref="bib53" id="ref36">53</reflink>]) and the "Fénix" or "Phoenix" (Alves & Moreira, [<reflink idref="bib2" id="ref37">2</reflink>]; Azevedo & Alves, [<reflink idref="bib4" id="ref38">4</reflink>]; Moreira, [<reflink idref="bib34" id="ref39">34</reflink>], [<reflink idref="bib35" id="ref40">35</reflink>], [<reflink idref="bib36" id="ref41">36</reflink>]). These two intervention approaches had proven successful in small-scale observational studies. A preliminary evaluation of the "TurmaMais" pilot intervention indicated a 40% reduction in retention rates and a 40% increase in achievement at the excellence level of two cohorts of students (2003/04 and 2004/05) enrolled in mathematics and Portuguese in grades 7 through 9, when compared to the national averages for the same years, and the averages of the school prior to the intervention (Verdasca, [<reflink idref="bib51" id="ref42">51</reflink>]). The evaluation of the "Fénix" pilot intervention indicated a 97% reduction in retention rates (i.e., only one student was retained in the last cohort) and a 19% increase in achievement at the excellence level of three cohorts of students (2006/07, 2007/08, and 2008/09) enrolled in mathematics and Portuguese in grades 5 through 8, when compared to the averages of the school 2 years prior to the intervention (Martins, [<reflink idref="bib29" id="ref43">29</reflink>]; Martins & Alves, [<reflink idref="bib30" id="ref44">30</reflink>]). Impacts of these pilot interventions on standardized assessments were less clear and based on limited samples, with both increases and decreases in the sixth-grade low-stakes assessments for mathematics (Fénix) and ninth-grade high-stakes assessment (TurmaMais), when compared to the national averages for the same years (Martins, [<reflink idref="bib29" id="ref45">29</reflink>]; Martins & Alves, [<reflink idref="bib30" id="ref46">30</reflink>]; Verdasca, [<reflink idref="bib51" id="ref47">51</reflink>]).</p> <p>Three review panels (in the north, center, and south of Portugal), including ME representatives as well as representatives of the two intervention approaches, selected 123 out of the 375 applications to join PMSE according to the description of all indicators expressed in the application and the coherence and consistency of the plans of recuperation and development for failing students presented by each school. After being accepted into PMSE, all schools had to commit to lowering retention rates by one-third each year, for 4 years. Because in Portugal teachers make decisions on matters concerning progression and repetition of a year based on teacher-determined measures of achievement, this mandate compounded the PMSE policy strategies with grade-level retention.</p> <p>To help PMSE schools achieve the retention goals, the ME paid for a small number of additional teaching hours, equivalent to raising one member of the teaching staff from part-time to full-time per school (16 to 35 hours per school, depending on the number of classes/students involved). Participation in the PMSE also required that schools choose up to four subject areas and two grade levels in which to invest their extra teaching time. Finally, the ME also committed to providing a team of technical and academic experts to advise on aspects of implementation and professional development. Overall, the PMSE policy had a moderate cost, particularly in comparison to the potential savings that could result from a reduction in retention costs (Ministry of Education, [<reflink idref="bib33" id="ref48">33</reflink>]; OECD, [<reflink idref="bib44" id="ref49">44</reflink>]).</p> <p>The funding and implementation guidance offered by the PMSE policy provided schools the opportunity to create small subgroups of students that worked together for a reduced period of time (usually 6 weeks). The students were selected from all other classes in the same grade level according to their achievement level to create more homogeneous subgroups of students. Throughout the year the students rotated through this additional subgroup of students, and then returned to their original classrooms. In sum, the theory of action behind PMSE implied that a temporary decrease in class size and increase in class homogeneity in both the original classroom, as well as in the smallest subgroups of students, would improve achievement. PMSE teachers could administer these changes to class size and composition to improve academic achievement, and therefore hope to increase the chances of progression for every student.</p> <p>Another feature of PMSE was the recourse to teacher professional development directed at increasing differentiated instruction. Differentiated instruction generally consists of changes to teaching and learning routines in order to address the academic diversity usually found in classrooms (Heacox, [<reflink idref="bib22" id="ref50">22</reflink>]; Morgado, [<reflink idref="bib37" id="ref51">37</reflink>]). It involves a set of strategies to modify curricula, resources, teaching methods, and learning tasks in a planned way so that the classroom environment is closer to the zone of proximal development of each student or group of students (Vygotsky, [<reflink idref="bib54" id="ref52">54</reflink>]) and learning is optimal. In PMSE schools, professional development in differentiated instruction focused on many strategies, such as approaches to student group work (such as peer tutoring), formative assessments and student evaluation focused on individual progress, curriculum management focused on interests of small groups, use of teaching materials of more concrete or abstract level based on the level of small-group achievement, and flexibility in timing and opportunities for learning based on specific group needs, among others (Barata, Calheiros, Patrício, Graça, & Lima, 2012). PMSE offered teachers differentiated instruction strategies in order to take advantage of the class size and class composition changes and maximize the effects on achievement. Results from a complete evaluation of the implementation of PMSE in the first 2 years confirmed that class size and composition and differentiated instruction were in fact three of the few significantly distinctive features of PMSE schools, when compared to non-PMSE schools (effect sizes equal to 2.28 for class size and composition assessed together, and 0.37 for differentiated instruction, respectively; Barata et al., [<reflink idref="bib5" id="ref53">5</reflink>]).</p> <hd id="AN0109514424-4">Impact of Class Size and Composition Policies</hd> <p>Available research on the impact of class size policies on achievement offers mostly contradictory results. In a summary of all empirical evidence available until 1994, including 277 estimates from 59 rigorous studies of the impact of class size on academic achievement, Hanushek ([<reflink idref="bib20" id="ref54">20</reflink>]) reported that only 15% of all studies presented significant positive estimates of impact, while 13% of studies presented significant negative estimates—a pattern most likely to represent a null effect of the policy. Experimental evidence from the Student/Teacher Achievement Ratio, or STAR, experiment (Finn & Achilles, [<reflink idref="bib17" id="ref55">17</reflink>]; Hanushek, [<reflink idref="bib20" id="ref56">20</reflink>]; Krueger, [<reflink idref="bib28" id="ref57">28</reflink>]) indicated that performance on standardized tests increased significantly for students in small classes compared to students in regular classes, particularly after just one year of class size intervention (Krueger, [<reflink idref="bib28" id="ref58">28</reflink>]). Other studies exploring natural class size variation have demonstrated that reductions of 30 to 15 students did not have a significant impact on student performance on state tests (Hoxby, [<reflink idref="bib24" id="ref59">24</reflink>], [<reflink idref="bib25" id="ref60">25</reflink>]). Overall the results are mixed.</p> <p>Studies examining the effects of class composition are less frequent, and their results are less clear. Because class size reduction is often associated with class composition policies (i.e., after determining that a teacher will be given fewer students to teach, oftentimes a new criterion is used to reorganize students in classes), the effect of both policies is often combined. For example, a study of fourth-, sixth-, and eighth-grade Dutch children by Dobbelsteen, Levin, and Oosterbeck ([<reflink idref="bib14" id="ref61">14</reflink>]) found that students in smaller classes did not present higher academic achievement than students in larger classes. However, when the authors took into account changes in class composition, they found that more homogeneous IQ classes presented higher academic achievement; furthermore, when accounting for class composition, students in smaller classes did perform better than students in larger classes (Dobbelsteen et al., [<reflink idref="bib14" id="ref62">14</reflink>]).</p> <p>In addition to addressing the gap in the literature regarding the effects of class size and composition together, the study by Dobbelsteen and colleagues ([<reflink idref="bib14" id="ref63">14</reflink>]) also provided positive evidence of class size effects beyond elementary school. Studies on class size and composition tend to focus on elementary settings because class size is generally more stable in elementary rather than later grades (Hoxby, [<reflink idref="bib25" id="ref64">25</reflink>]). However, class size is also more likely to increase in later grades, hindering teachers' effectiveness in addressing every student's needs.</p> <p>To our knowledge, there is no evidence of the combined effect of class size, class composition, and differentiated instruction practices on achievement. However, the lack of significant effects of class size in natural class reduction, when compared to the positive results of the STAR experiment, has been attributed to the fact that teachers were not equipped to deal with natural variations in class size (Hoxby, [<reflink idref="bib25" id="ref65">25</reflink>]). When arbitrarily given a smaller class, teachers were likely to continue using the same instructional strategies they employed for larger class sizes, and it was only under the atypical conditions of an experiment that such strategies made no difference (Hoxby, [<reflink idref="bib25" id="ref66">25</reflink>]). Training in differentiated instruction practices may help teachers make the most of natural and programmed variations in class size and composition.</p> <hd id="AN0109514424-5">Evaluating Policy and Program Impact</hd> <p>In the past few decades there has been substantial development in program evaluation research, as well as in methods for statistical data analysis of program impact. In the United States, the Institute for Educational Sciences funds and supervises large-scale experimental and quasi-experimental trials of educational policy. However, in the majority of European and other countries, causal methods for program evaluation are still seldom applied to educational programs at scale, and most of the empirical evidence developed by the MEs does not answer the question, What is the true impact (or cause-effect relationship) of an educational program in students' achievement? This empirical problem is due to the fact that a considerable number of evaluation studies of policy in education use only qualitative data and small samples, and lack a comparison group that makes these inferences about the effect of the policy unlikely.</p> <p>In current educational research, even when these problems are overcome by collecting adequate quantitative data in groups affected and not affected by the policy, and comparing these groups, participation in most educational programs depends on choices made by parents, teachers, legislators, or other stakeholders, making it hard to attribute program differences to actual programs. These choices make the participation in educational programs the product of a process of self-selection, instead of being randomly determined (Murnane & Willett, [<reflink idref="bib38" id="ref67">38</reflink>]; Shadish, Campbell, & Cook, [<reflink idref="bib48" id="ref68">48</reflink>]). As such, the variation in educational programs is potentially correlated with other determinants of educational achievement, such as teachers' expectations or parents' investment, producing biased results of the effects of the program.</p> <p>Because class size and class composition policies are such a costly investment (Hoxby, [<reflink idref="bib25" id="ref69">25</reflink>]), it is essential that we know the true impact of this policy. The best way to solve this empirical problem would be to randomly assign schools that applied to the program to PMSE or a comparison group, and then evaluate their outcomes, as was the case with the Student/Teacher Achievement Ratio, or STAR, experiment (Finn & Achilles, [<reflink idref="bib17" id="ref70">17</reflink>]; Hanushek, [<reflink idref="bib20" id="ref71">20</reflink>]; Krueger, [<reflink idref="bib28" id="ref72">28</reflink>]). However, because randomization is not applicable to programs already at scale, such as PMSE, researchers have applied econometric techniques that take advantage of the natural variation in educational processes to achieve estimates of the true impact of class size policies (Angrist & Lavy, [<reflink idref="bib3" id="ref73">3</reflink>]; Dobbelsteen et al., [<reflink idref="bib14" id="ref74">14</reflink>]; Hoxby, [<reflink idref="bib24" id="ref75">24</reflink>], [<reflink idref="bib25" id="ref76">25</reflink>]; McEwan & Urquiola, [<reflink idref="bib31" id="ref77">31</reflink>]).</p> <p>One econometric technique employed in nonexperimental impact studies of educational programs to reduce the differences in educational achievement between intervention and comparison groups that are not attributable to the program is propensity scores estimation (PSE) (Rosenbaum & Rubin, [<reflink idref="bib46" id="ref78">46</reflink>]). Several studies have now examined the conditions under which PSE estimates can be reliable and can approximate experimental estimates, with mixed results (Agodini & Dynarsky, [<reflink idref="bib1" id="ref79">1</reflink>]; Cook, Shadish, & Wong, [<reflink idref="bib6" id="ref80">6</reflink>]; Dehejia & Wahba, [<reflink idref="bib8" id="ref81">8</reflink>], [<reflink idref="bib9" id="ref82">9</reflink>]; Diaz & Handa, [<reflink idref="bib12" id="ref83">12</reflink>]; Michalopoulos, Bloom, & Hill, [<reflink idref="bib32" id="ref84">32</reflink>]; Wilde & Hollister, [<reflink idref="bib55" id="ref85">55</reflink>]). One limitation of PSE is that measured effects can be confounded with unobservable or unobserved characteristics of participants. However, studies have demonstrated that, when used properly, PSE estimates can be reliable and can approach experimental estimates (Dehejia & Wahba, [<reflink idref="bib8" id="ref86">8</reflink>], [<reflink idref="bib9" id="ref87">9</reflink>]; Diaz & Handa, [<reflink idref="bib12" id="ref88">12</reflink>]). For proper use of PSE it is necessary to adequately control for all key variables that would likely contribute to the selection of schools into the intervention. To our knowledge, PSE has been used to simulate selection bias in impact studies of class size and composition only once (Wilde & Hollister, [<reflink idref="bib55" id="ref89">55</reflink>]).</p> <hd id="AN0109514424-6">The Present Study</hd> <p>The goal of this study was to evaluate the impact of the Portuguese national policy "Programa Mais Sucesso Escolar." Capitalizing on the application process, we used a comparison group of schools that applied for the program but did not get accepted. We then used propensity score weighting of both the PMSE and comparison schools to further reduce any potential differences in the educational achievement of the students from the two groups that were not attributable to the program. This methodology allowed us to design an evaluation that worked toward providing the necessary empirical evidence for policy-makers to make informed choices regarding the program. In PMSE, schools could apply the class size and composition model to all grades 1 through 9. Therefore, the results of the study contribute to the discussion on the combined effects of class size, composition, and differentiated instruction on retention and achievement. Finally, this study contributes to the discussion of what constitutes and how to measure success in policy for reducing grade retention and increasing achievement in all grades of compulsory education. The following research question was addressed: What was the impact of the first 2 years of PMSE on teacher-determined measures and external standardized measures of educational achievement, at the school and student level?</p> <hd id="AN0109514424-7">Method</hd> <p></p> <hd id="AN0109514424-8">Sample</hd> <p>The study sample comprised 123 intervention schools and 252 comparison schools in the first year of PMSE. At the end of the first year, 12 schools in the intervention group did not achieve their retention goals and were eliminated from PMSE; another four schools from the south region of Portugal were selected from the comparison group to join the intervention group.[<reflink idref="bib2" id="ref90">2</reflink>] In accordance with an intent-to-treat analysis, these schools were kept in their initial assigned group.</p> <hd id="AN0109514424-9">Procedures</hd> <p>In collaboration with the ME, and after extensive interviews with PMSE stakeholders at the ministry, regional, school district, and school level, the research team defined a list of indicators to measure the impact of PMSE and to further reduce any potential differences in the educational achievement of the students from the intervention and comparison schools. All indicators, except those pertaining to the high-stakes ninth-grade exam, and covariates were provided by the ME at the school level. Indicators pertaining to the high-stakes ninth-grade external standardized assessment were downloaded from the Júri Nacional de Exames website at the student level, therefore supplementing school-level performance data.</p> <hd id="AN0109514424-10">Measures</hd> <p></p> <hd id="AN0109514424-11">Teacher-determined educational achievement</hd> <p></p> <hd id="AN0109514424-12">Grade-level transitions.</hd> <p>Indicators for educational achievement included percentage of grade-level transitions in the second to ninth grade in compulsory education, by school. Grade-level transitions were computed by dividing the total number of transitions by the total number of registered students per grade level, per school. This indicator was computed for every school in the sample.</p> <hd id="AN0109514424-13">Percentage of success in mathematics, Portuguese, and English.</hd> <p>Indicators for educational achievement also included percentages of success in mathematics, Portuguese, and English, by school. Students were evaluated on a scale of 1 (worst performance) to 5 (best performance) in the fifth to ninth grade in compulsory education. A performance lower than 3 is equivalent to failing a subject. Most PMSE schools invested their extra teaching time in mathematics, Portuguese, and English (of all subject areas, approximately 32% were Portuguese, 29% were mathematics, and 23% were English in the first and second years). These indicators were computed for schools intervening in grades 5 through 9 (84% and 77% of all grade levels included in the analysis, for the first and second years, respectively).</p> <hd id="AN0109514424-14">Teacher-determined high-performance educational achievement</hd> <p></p> <hd id="AN0109514424-15">Percentage of fours and fives in mathematics, Portuguese, and English.</hd> <p>To quantify the impact of PMSE at excellence levels of educational achievement (i.e., not just passing, but excelling at the specific discipline), we also included percentage of fours and fives in mathematics, Portuguese, and English, by year and by school. These indicators were computed for schools intervening in grades 5 through 9 (84% and 77% of all grade levels included in the analysis, for the first and second years, respectively).</p> <hd id="AN0109514424-16">Sixth-grade low-stakes standardized assessment, school level</hd> <p></p> <hd id="AN0109514424-17">Percentage of success in the mathematics and Portuguese exam.</hd> <p>The indicators for success in the sixth-grade low-stakes standardized assessment at the school level were the percentage of success in mathematics and Portuguese standardized assessment, by school. Students were evaluated on a scale of 1 (worst performance) to 5 (best performance) in the sixth-grade external standardized assessments. A performance lower than 3 is equivalent to failing the standardized assessments. These indicators were computed for schools intervening in grade 6 (5% and 13% of all grade levels included in the analysis, for the first and second years, respectively).</p> <hd id="AN0109514424-18">Ninth-grade high-stakes standardized assessment, student level</hd> <p></p> <hd id="AN0109514424-19">Performance in the mathematics and Portuguese exam.</hd> <p>Indicators for success in high-stakes standardized assessments at the student level included performance in the mathematics and Portuguese ninth-grade exam, per student. Students were evaluated on a scale of 1 (worst performance) to 5 (best performance) in the ninth-grade standardized assessments. These indicators were computed for schools intervening in grade 9 (6% and 7% of all grade levels included in the analysis, for the first and second years, respectively).</p> <hd id="AN0109514424-20">Covariates.</hd> <p>Covariates included baseline school-level indicators pertaining to the year of application on (<emph>a</emph>) school and human resources, such as whether the school was recently integrated in a large school district administrative unit, included in the same building as preschool and/or secondary school, total number of faculty, average age of faculty, average years of teaching experience of faculty, number of faculty with a master's degree or higher, percentage of female faculty, and total number of staff; (<emph>b</emph>) student characteristics, such as total number of students, percentage of students receiving government support (SASE A, SASE B, and SASE C), percentage of students with a computer at home, percentage of students with internet at home, and number of school violence events; (<emph>c</emph>) student achievement, such as the average retention rate, percentage of success in the high-stakes ninth-grade exam in mathematics, percentage of success in the high-stakes ninth-grade exam in Portuguese; and (<emph>d</emph>) school geographic location (i.e., geographic location of the school in five major regions in Portugal) and whether the school was located in an urban, rural, or mixed urban-rural district.</p> <hd id="AN0109514424-21">Data-Analytic Strategy</hd> <p>To evaluate PMSE, we chose to capitalize on the variation created by the selection process into PMSE, comparing schools chosen to enter the program to schools that applied but were not chosen to enter the program (i.e., comparison schools). Given the complex nature of the data available for these two groups of schools, we used several data-analytic strategies designed to utilize all data available, and also reduce bias. Because PMSE schools were directed by the ME to choose up to four subject areas and two grade levels in which to invest their extra teaching time, each PMSE school contributed with a predetermined set of data units for the analysis of impact. In contrast, comparison schools could potentially contribute with information from all grade levels available. Data on intervention schools were restricted to grade levels in which the PMSE was implemented. Forty-nine schools (39.84% of the total) in the first year of PMSE, and 46 schools (37.40% of the total) in the second year of PMSE, chose to implement PMSE on two grade levels. In each of the years, four schools chose to also implement the program in first grade, a grade level for which there is no retention or reported grades by law; these four grade levels were dropped. In the comparison schools, the grade levels were randomly selected based on the percentage of total grade levels (first and second choice) available in the intervention schools in the first year (approximately 5% at first grade, 9% at second grade, 2% at third grade, 1% at fourth grade, 17% at fifth grade, 5% at sixth grade, 48% at seventh grade, 8% at eighth grade, and 6% at ninth grade).</p> <p>The 12 schools that dropped out of the program were assumed to have followed the cohort and implemented the program in the subsequent grade levels in year 2. The four schools that joined the program in year 2 remained in the comparison sample for the 2 years, and, in similarity to all other comparison schools, the grade levels of these four schools were also randomly selected from all grade levels available in the comparison schools. The actual grade levels chosen for implementation in year 2 for these four schools were ignored because of potential bias.</p> <p>We used multilevel models with random effects for schools to adjust for the clustering of multiple grade levels within schools. We also adjusted all estimates for school and school district characteristics, as well as baseline grade-level/school values (or averages, when only these were available). Our primary equation for fitting these multilevel models was as follows:</p> <p></p> <p>where OUTCOME<subs><emph>ij</emph></subs> is the achievement performance for grade level <emph>i</emph> in school <emph>j</emph>, INTERV<subs><emph>j</emph></subs> is a dummy variable that takes on the value of 1 if the school was in the intervention group and 0 if not, OUTMEAN<subs><emph>ij</emph></subs> is the baseline outcome value for grade level <emph>i</emph> in school <emph>j</emph>, SCHOOLV<subs><emph>j</emph></subs> represents a vector of school and school district characteristics, ε<subs><emph>ij</emph></subs> is the error term for grade level <emph>i</emph> in school <emph>j</emph>, and μ<subs><emph>j</emph></subs> is the school-level error term. β<subs>1</subs> is the estimate of interest.</p> <p>A similar process was used for the student-level data available for success on the ninth-grade high-stakes exam. In these models we adjusted for student gender and age, in addition to school and school district characteristics, as well as baseline school values. Our primary equation for fitting these multilevel models was as follows:</p> <p></p> <p>where OUTCOME<subs><emph>ij</emph></subs> is the achievement performance for student <emph>i</emph> in school <emph>j</emph>, INTERV<subs><emph>j</emph></subs> is a dummy variable that takes on the value of 1 if the school was in the intervention group and 0 if not, GENDER<subs><emph>ij</emph></subs> is the gender for student <emph>i</emph> in school <emph>j</emph>, AGE<subs><emph>ij</emph></subs> is the age for student <emph>i</emph> in school <emph>j</emph>, OUTMEAN<subs><emph>j</emph></subs> is the baseline outcome value for school <emph>j</emph>, SCHOOLV<subs><emph>j</emph></subs> represents a vector of school and school district characteristics, ε<subs><emph>ij</emph></subs> is the error term for student <emph>i</emph> in school <emph>j</emph>, and μ<subs><emph>j</emph></subs> is the school-level error term. β<subs>1</subs> is the estimate of interest. We also used multivariate analysis to compare PMSE and comparison schools on educational achievement when only one point of data was available per school for a particular indicator (e.g., sixth-grade exam performance indicators).</p> <p>Finally, because schools chosen to enter PMSE are likely to be different from schools not chosen to enter the program (i.e., comparison schools), we used propensity scores estimation (PSE) to reduce the differences in educational achievement between intervention and comparison groups that are not attributable to PMSE (Dehejia & Wahba, [<reflink idref="bib8" id="ref91">8</reflink>], [<reflink idref="bib9" id="ref92">9</reflink>]; Diaz & Handa, [<reflink idref="bib12" id="ref93">12</reflink>]; Rosenbaum & Rubin, [<reflink idref="bib46" id="ref94">46</reflink>]). We implemented a version of PSE, inverse propensity score weighting (Murnane & Willett, [<reflink idref="bib38" id="ref95">38</reflink>]), by using the "pscore" procedure in Stata 12. Pscore estimated the propensity score of the treatment (i.e., PMSE) on a list of variables using a logit model and stratified schools in blocks according to the pscore. The list comprised a rich set of characteristics (Diaz & Handa, [<reflink idref="bib12" id="ref96">12</reflink>]), including a subset of the covariates for the year prior to the first year of implementation of PMSE (2008/09) described above, that is, (<emph>a</emph>) school and human resources, (<emph>b</emph>) student characteristics, (<emph>c</emph>) educational achievement, and (<emph>d</emph>) school geographic location. These variables are likely to be correlated with a school's administrative capacity to put together a successful implementation plan, as well as student outcomes, and therefore should approximate the selection process. Moreover, these variables are uniformly collected by the ME, and so are defined and measured the same way across all schools (Wilde & Hollister, [<reflink idref="bib55" id="ref97">55</reflink>]).</p> <p>The procedure then checked whether PMSE and comparison schools were balanced within pscore strata (i.e., balancing property). When the balancing property was satisfied, the pscores were inverted for the intervention group and subtracted from 1 and inverted for the comparison group. These new pscores were used as probability weights ("pweights" in Stata) to adjust the estimates from the impact models.</p> <p>We compared intervention and comparison schools on all our covariates pertaining to the year of application, adjusting for the propensity weight. Intervention and comparison schools were significantly different on one out of 27 indicators (Table 1).</p> <p>Graph</p> <p>Table 1. Weighted Baseline Data on Students, Schools, and School Districts Comparing PMSE and Control Schools (N = 375)</p> <p> <ephtml> <table><tr><td valign="bottom" /><td valign="bottom">Weighted Baseline Mean of Control Schools</td><td valign="bottom">Weighted Baseline Mean of Intervention Schools</td><td valign="bottom">Weighted Baseline Diff.</td><td valign="bottom">p Value</td><td valign="bottom">Weighted Diff./Control SD</td></tr><tr><td>Human and school resources:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td> Schools in large administrative unit</td><td char=".">.801</td><td char=".">.801</td><td char=".">.000</td><td char=".">.997</td><td char=".">.000</td></tr><tr><td> Includes preschool</td><td char=".">.052</td><td char=".">.060</td><td char=".">.008</td><td char=".">.784</td><td char=".">.039</td></tr><tr><td> Includes secondary school</td><td char=".">.391</td><td char=".">.518</td><td char=".">.127</td><td char=".">.041*</td><td char=".">.262</td></tr><tr><td> Total no. of faculty</td><td char=".">88.671</td><td char=".">88.371</td><td char=".">−.300</td><td char=".">.942</td><td char=".">−.008</td></tr><tr><td> Average age of faculty</td><td char=".">41.805</td><td char=".">41.749</td><td char=".">−.056</td><td char=".">.855</td><td char=".">−.021</td></tr><tr><td> Average teaching experience of faculty</td><td char=".">17.110</td><td char=".">17.069</td><td char=".">−.041</td><td char=".">.903</td><td char=".">−.013</td></tr><tr><td> No. of faculty with master's degree or higher</td><td char=".">4.499</td><td char=".">4.464</td><td char=".">−.035</td><td char=".">.943</td><td char=".">−.008</td></tr><tr><td> Percentage female faculty</td><td char=".">73.305</td><td char=".">73.626</td><td char=".">.321</td><td char=".">.695</td><td char=".">.050</td></tr><tr><td> Total no. of staff</td><td char=".">34.062</td><td char=".">34.914</td><td char=".">.852</td><td char=".">.383</td><td char=".">.094</td></tr><tr><td /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>Student indicators:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td> Total no. of students</td><td char=".">685.576</td><td char=".">682.771</td><td char=".">−2.805</td><td char=".">.947</td><td char=".">−.008</td></tr><tr><td> Percentage students receiving government support:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>  SASE A</td><td char=".">25.682</td><td char=".">25.276</td><td char=".">−.407</td><td char=".">.762</td><td char=".">−.033</td></tr><tr><td>  SASE B</td><td char=".">18.130</td><td char=".">17.039</td><td char=".">−1.092</td><td char=".">.169</td><td char=".">−.152</td></tr><tr><td>  SASE C</td><td char=".">2.130</td><td char=".">1.728</td><td char=".">−.402</td><td char=".">.214</td><td char=".">−.131</td></tr><tr><td> Percentage students with computer at home</td><td char=".">58.887</td><td char=".">59.210</td><td char=".">.323</td><td char=".">.900</td><td char=".">.016</td></tr><tr><td> Percentage students with internet at home</td><td char=".">38.830</td><td char=".">39.525</td><td char=".">.694</td><td char=".">.778</td><td char=".">.038</td></tr><tr><td> No. of school violence events</td><td char=".">3.364</td><td char=".">3.101</td><td char=".">−.263</td><td char=".">.835</td><td char=".">−.024</td></tr><tr><td /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>Student achievement:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td> Average retention rate</td><td char=".">87.312</td><td char=".">87.666</td><td char=".">.354</td><td char=".">.653</td><td char=".">.050</td></tr><tr><td> Percentage success at high-stakes ninth-grade exam in mathematics</td><td char=".">68.384</td><td char=".">68.426</td><td char=".">.042</td><td char=".">.982</td><td char=".">.003</td></tr><tr><td> Percentage success at high-stakes ninth-grade exam in Portuguese</td><td char=".">62.001</td><td char=".">61.992</td><td char=".">−.009</td><td char=".">.997</td><td char=".">−.001</td></tr><tr><td /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>Geographic location:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td> Center region</td><td char=".">.200</td><td char=".">.205</td><td char=".">.004</td><td char=".">.928</td><td char=".">.009</td></tr><tr><td> Lisbon region</td><td char=".">.306</td><td char=".">.299</td><td char=".">−.006</td><td char=".">.911</td><td char=".">−.013</td></tr><tr><td> Alentejo region</td><td char=".">.135</td><td char=".">.133</td><td char=".">−.002</td><td char=".">.971</td><td char=".">−.009</td></tr><tr><td> Algarve region</td><td char=".">.070</td><td char=".">.079</td><td char=".">.009</td><td char=".">.780</td><td char=".">.035</td></tr><tr><td> Urban-rural qualification:</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>  Moderately urban district</td><td char=".">.219</td><td char=".">.230</td><td char=".">.011</td><td char=".">.832</td><td char=".">.027</td></tr><tr><td>  Predominantly urban district</td><td char=".">.706</td><td char=".">.696</td><td char=".">−.010</td><td char=".">.857</td><td char=".">−.022</td></tr></table> </ephtml> </p> <p>1 <emph>p</emph> <.05.</p> <p>In the Results section below we discuss the results of the final model for each indicator, in most cases a multilevel model, adjusted for pweights, all covariates (with the exception of the indicators for school achievement), as well as grade-level/baseline school averages for that indicator. In Tables 2 and 3, we report the sample size, the adjusted mean of the intervention schools,[<reflink idref="bib3" id="ref98">3</reflink>] the adjusted mean of the comparison schools, the adjusted difference between these groups, the standard error of the adjusted difference, the level of significance of the difference, and the effect size based on these final models that are of prime interest for policy decisions about PMSE.[<reflink idref="bib4" id="ref99">4</reflink>] The effect size was computed by dividing the estimated adjusted difference between groups by the standard deviation of the indicator for the comparison group (Gormley, Gayer, Phillips, & Dawson, [<reflink idref="bib19" id="ref100">19</reflink>]; Wong, Cook, Barnett, & Jung, [<reflink idref="bib56" id="ref101">56</reflink>]) and indicates the magnitude of the PMSE effect regardless of the instrument or method used. Data were analyzed in Stata (Version 13).</p> <p>Graph</p> <p>Table 2. Impact on Teacher-Determined Educational Achievement (Panel A) and High-Performance Educational Achievement (Panel B) in Years 1 and 2 of PMSE</p> <p> <ephtml> <table><tr><td>Variables</td><td valign="bottom">N Grade Levels</td><td valign="bottom">N Schools</td><td valign="bottom">Adjusted Mean of Intervention Schools</td><td valign="bottom">Adjusted Mean of Control Schools</td><td valign="bottom">Adjusted Diff.</td><td valign="bottom">Standard Error</td><td valign="bottom">p-Value</td><td valign="bottom">Effect Size</td></tr><tr><td>Panel A: Educational Achievement</td></tr><tr><td>Percentage of grade-level transitions:</td></tr><tr><td> Year 1</td><td char=".">450</td><td char=".">347</td><td char=".">90.713</td><td char=".">86.415</td><td char=".">4.298</td><td char=".">.708</td><td char=".">.000***</td><td char=".">.473</td></tr><tr><td> Year 2</td><td char=".">449</td><td char=".">343</td><td char=".">89.167</td><td char=".">89.739</td><td char=".">−.572</td><td char=".">.444</td><td char=".">.209</td><td char=".">−.109</td></tr><tr><td>Percentage of success in mathematics:</td></tr><tr><td> Year 1</td><td char=".">386</td><td char=".">299</td><td char=".">77.142</td><td char=".">72.089</td><td char=".">5.053</td><td char=".">1.300</td><td char=".">.000***</td><td char=".">.423</td></tr><tr><td> Year 2</td><td char=".">387</td><td char=".">297</td><td char=".">75.215</td><td char=".">70.372</td><td char=".">4.844</td><td char=".">1.480</td><td char=".">.001**</td><td char=".">.405</td></tr><tr><td>Percentage of success in Portuguese:</td></tr><tr><td> Year 1</td><td char=".">391</td><td char=".">305</td><td char=".">87.569</td><td char=".">81.581</td><td char=".">5.988</td><td char=".">.956</td><td char=".">.000***</td><td char=".">.557</td></tr><tr><td> Year 2</td><td char=".">396</td><td char=".">306</td><td char=".">89.230</td><td char=".">84.672</td><td char=".">4.558</td><td char=".">.848</td><td char=".">.000***</td><td char=".">.467</td></tr><tr><td>Percentage of success in English:</td></tr><tr><td> Year 1</td><td char=".">361</td><td char=".">282</td><td char=".">84.406</td><td char=".">79.317</td><td char=".">5.088</td><td char=".">1.218</td><td char=".">.000***</td><td char=".">.477</td></tr><tr><td> Year 2</td><td char=".">361</td><td char=".">277</td><td char=".">85.378</td><td char=".">80.369</td><td char=".">5.009</td><td char=".">1.373</td><td char=".">.000***</td><td char=".">.455</td></tr><tr><td /><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Panel B: High-Performance Educational Achievement</td></tr><tr><td>Percentage of fours and fives in mathematics:</td></tr><tr><td> Year 1</td><td char=".">386</td><td char=".">299</td><td char=".">29.701</td><td char=".">28.321</td><td char=".">1.379</td><td char=".">.995</td><td char=".">.166</td><td char=".">.126</td></tr><tr><td> Year 2</td><td char=".">387</td><td char=".">297</td><td char=".">27.983</td><td char=".">26.092</td><td char=".">1.891</td><td char=".">1.069</td><td char=".">.077+</td><td char=".">.180</td></tr><tr><td>Percentage of fours and fives in Portuguese:</td></tr><tr><td> Year 1</td><td char=".">391</td><td char=".">305</td><td char=".">31.475</td><td char=".">27.318</td><td char=".">4.157</td><td char=".">1.063</td><td char=".">.000***</td><td char=".">.359</td></tr><tr><td> Year 2</td><td char=".">396</td><td char=".">306</td><td char=".">31.019</td><td char=".">28.180</td><td char=".">2.839</td><td char=".">1.232</td><td char=".">.021*</td><td char=".">.249</td></tr><tr><td>Percentage of fours and fives in English:</td></tr><tr><td> Year 1</td><td char=".">361</td><td char=".">282</td><td char=".">37.061</td><td char=".">33.726</td><td char=".">3.335</td><td char=".">1.224</td><td char=".">.006**</td><td char=".">.295</td></tr><tr><td> Year 2</td><td char=".">361</td><td char=".">277</td><td char=".">34.109</td><td char=".">33.247</td><td char=".">.862</td><td char=".">1.368</td><td char=".">.529</td><td char=".">.073</td></tr></table> </ephtml> </p> <ulist> <item>21 The effect size was computed by dividing the estimated adjusted difference between groups by the standard deviation of the indicator for the comparison group.</item> <item>2 <emph>p</emph> <.10.</item> <item>2 <emph>p</emph> <.05.</item> <item>2 <emph>p</emph> <.01.</item> <item>2 <emph>p</emph> <.001.</item> </ulist> <p>Graph</p> <p>Table 3. Impact on External Standardized Measures of Educational Achievement for the Sixth-Grade Low-Stakes Assessment at the School Level (Panel A) and Ninth-Grade High-Stakes Assessment at the Student Level (Panel B) in Years 1 and 2 of PMSE</p> <p> <ephtml> <table><tr><td valign="bottom">Variables</td><td valign="bottom">N Students</td><td valign="bottom">N Schools</td><td valign="bottom">Adjusted Mean of Intervention Schools</td><td valign="bottom">Adjusted Mean of Control Schools</td><td valign="bottom">Adjusted Diff.</td><td valign="bottom">Standard Error</td><td valign="bottom">p-Value</td><td valign="bottom">Effect Size</td></tr><tr><td>Panel A: Sixth-Grade Low-Stakes Standardized Assessment, School Level</td></tr><tr><td>Percentage of success in the mathematics exam:</td></tr><tr><td> Year 1</td><td>N/A</td><td char=".">24</td><td char=".">71.666</td><td char=".">78.554</td><td char=".">−6.888</td><td char=".">10.385</td><td char=".">.575</td><td char=".">−.548</td></tr><tr><td> Year 2</td><td>N/A</td><td char=".">70</td><td char=".">63.855</td><td char=".">62.085</td><td char=".">1.770</td><td char=".">2.519</td><td char=".">.486</td><td char=".">.116</td></tr><tr><td>Percentage of success in the Portuguese exam:</td></tr><tr><td> Year 1</td><td>N/A</td><td char=".">24</td><td char=".">91.049</td><td char=".">84.828</td><td char=".">6.221</td><td char=".">4.690</td><td char=".">.316</td><td char=".">.608</td></tr><tr><td> Year 2</td><td>N/A</td><td char=".">71</td><td char=".">82.131</td><td char=".">81.854</td><td char=".">.277</td><td char=".">2.120</td><td char=".">.896</td><td char=".">.030</td></tr><tr><td /><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Panel B: Ninth-Grade High-Stakes Standardized Assessment, Student Level</td></tr><tr><td>Performance in the mathematics exam:</td></tr><tr><td> Year 1</td><td char=".">1,955</td><td char=".">33</td><td char=".">2.545</td><td char=".">2.745</td><td char=".">−.200</td><td char=".">.115</td><td char=".">.081+</td><td char=".">−.194</td></tr><tr><td> Year 2</td><td char=".">2,649</td><td char=".">36</td><td char=".">2.362</td><td char=".">2.310</td><td char=".">.052</td><td char=".">.091</td><td char=".">.568</td><td char=".">.052</td></tr><tr><td>Performance in the Portuguese exam:</td></tr><tr><td> Year 1</td><td char=".">1,942</td><td char=".">33</td><td char=".">2.838</td><td char=".">2.959</td><td char=".">−.121</td><td char=".">.082</td><td char=".">.140</td><td char=".">−.159</td></tr><tr><td> Year 2</td><td char=".">2,684</td><td char=".">36</td><td char=".">2.586</td><td char=".">2.755</td><td char=".">−.169</td><td char=".">.082</td><td>.041*</td><td char=".">−.222</td></tr></table> </ephtml> </p> <ulist> <item>31 The effect size was computed by dividing the estimated adjusted difference between groups by the standard deviation of the indicator for the comparison group.</item> <item>3 <emph>p</emph> <.10.</item> <item>3 <emph>p</emph> <.05.</item> </ulist> <hd id="AN0109514424-22">Results</hd> <p></p> <hd id="AN0109514424-23">Impact on Teacher-Determined Measures of Educational Achievement</hd> <p>In Table 2, Panel A, we present the impact results on teacher-determined measures of educational achievement in the first and second years of PMSE for the indicators percentage of grade-level transitions, and percentage of success in mathematics, Portuguese, and English. Accounting for the multilevel structure of the data, and adjusting for the pscore weights, school, and school district characteristics, in the first year of implementation, grade-level transitions from the second to the ninth-grade level were 4.30 percentage points higher in PMSE schools than comparison schools (<emph>SE</emph> = 0.708, <emph>p</emph> <.001). Moreover, success was 5.06 (<emph>SE</emph> = 1.300, <emph>p</emph> <.001), 5.99 (<emph>SE</emph> = 0.956, <emph>p</emph> <.001), and 5.09 (<emph>SE</emph> = 1.218, <emph>p</emph> <.001) percentage points higher in mathematics, Portuguese, and English, respectively, in PMSE schools than in comparison schools.</p> <p>In the second year of implementation, success was 4.84 (<emph>SE</emph> = 1.480, <emph>p</emph> <.01), 4.56 (<emph>SE</emph> = 0.848, <emph>p</emph> <.001), and 5.01 (<emph>SE</emph> = 1.373, <emph>p</emph> <.001) percentage points higher in mathematics, Portuguese, and English, respectively, in PMSE schools than in comparison schools.</p> <p>In Table 2, Panel B, we present the impact results on teacher-determined measures of high-performance educational achievement in the first and second years of PMSE. In the first year of the program, the percentage of fours and fives was 4.16 (<emph>SE</emph> = 1.063, <emph>p</emph> <.001) and 3.34 (<emph>SE</emph> = 1.224, <emph>p</emph> <.01) percentage points higher in Portuguese and in English, respectively, in PMSE schools than in comparison schools. There was no significant effect on the percentage of fours and fives in mathematics in the first year.</p> <p>In the second year of the program, the percentage of fours and fives was 1.89 (<emph>SE</emph> = 1.069, <emph>p</emph> =.077) and 2.84 (<emph>SE</emph> = 1.232, <emph>p</emph> <.05) percentage points higher in mathematics and Portuguese, respectively, in PMSE schools than in comparison schools.</p> <hd id="AN0109514424-24">Impact on External Standardized Measures of Educational Achievement</hd> <p>In Table 3, we present the impact results on the sixth- and ninth-grade high-stakes standardized assessments for the percentage of success in the mathematics and Portuguese exam. All estimates of impact on external standardized assessments were based on a much smaller sample of schools than the analysis of impact on teacher-determined measures of educational achievement because there is standardized testing only at grades 6 and 9, and most schools chose not to intervene in these grade levels in the first year of the program, hoping to improve standardized outcomes over time.</p> <p>As can be seen in Table 3, Panel A, there is no statistical evidence of a significant impact of PMSE in the sixth-grade high-stakes standardized assessments at the school level, in the first or in the second year of PMSE. In Table 3, Panel B, we present the impact results on the ninth-grade standardized assessments for the indicators performance in the mathematics and Portuguese ninth-grade exam, per student. There is a negative impact of PMSE in the ninth-grade high-stakes standardized assessment at the individual level. In the first year of PMSE, the performance in the mathematics standardized assessment was 0.20 percentage points (<emph>SE</emph> = 0.115, <emph>p</emph> = 0.081) lower for the students in the intervention schools than the students in the comparison schools (marginal effect). In the second year of PMSE, the performance in the Portuguese standardized was 0.17 percentage points (<emph>SE</emph> = 0.082, <emph>p</emph> <.05) lower for the students in the intervention schools than the students in the comparison schools.</p> <hd id="AN0109514424-25">Discussion</hd> <p>The goal of this study was to evaluate the impact of the Portuguese national policy "Programa Mais Sucesso Escolar" on retention and educational achievement during the first 2 years of the program. The study showed that PMSE had mixed effects on educational achievement, with some significant positive impacts on achievement measures that were determined by teachers, and nonsignificant to significant negative impact on external standardized assessments. Specifically, we found that PMSE schools generally presented higher percentages of grade-level transitions when compared to comparison schools. In the first year of the policy, on average, approximately 91% of students transitioned to the following grade in PMSE schools, compared to only 86% in comparison schools; there were no significant differences on average in the second year.</p> <p>The study also indicated that PMSE schools had higher percentages of success in mathematics, Portuguese and English than comparison schools (observed effect sizes varied between −.11 and 0.56). On average, an additional 5% of students met the minimal level of performance for passing a subject in PMSE compared to comparison schools; in the second year, this difference was only slightly smaller. PMSE also had a significant positive impact on educational achievement at the excellence level in these three subject areas (ES between 0.08 and 0.36). In Portuguese compulsory education levels, students are evaluated on a scale of 1 (worst performance) to 5 (best performance). On average, an additional 3% in the first year, and 2% in the second year, of students achieved high levels of performance (fours or fives) in mathematics, Portuguese, and English in PMSE compared to comparison schools.</p> <p>Finally, we found that the performance of students in the intervention schools was not significantly different in the low-stakes sixth-grade exams (ES between −.55 and.61), but tended to be significantly lower in the ninth-grade high-stakes mathematics and Portuguese standardized assessments when compared to students in the comparison schools (ES between −.22 and.05). In the ninth-grade standardized assessments, students are evaluated on a scale of 1 (worst performance) to 5 (best performance). On average, and across the 2 years, students in PMSE schools scored 0.1 points lower than students in comparison schools in the ninth-grade high-stakes mathematics and Portuguese standardized assessments.</p> <p>The results of the policy are complex. At first glance, the findings seem to indicate that PMSE achieved its main goal, namely, the reduction of student retention, but at the cost of students' performance on national standardized assessments. There are a few potential explanations for this pattern of mixed results. One hypothesis is that in an attempt to lower retention, PMSE schools may have moved a larger number of students with lower achievement through compulsory school, allowing them to take the final standardized assessments. This strategy could effectively lower PMSE schools' average performance on high-stakes standardized assessments when compared to other schools, as observed.</p> <p>A second hypothesis is that the positive effects of PMSE in teacher-determined measures of educational achievement may have been (at least partially) a function of grade inflation. In effect, it is no surprise that PMSE schools presented lower retention rates given that they were mandated to do so to continue in the PMSE policy and continue to receive resources from the ME. The positive (albeit moderate) significant effects on measures for which there was no grade-inflation incentive, such as success in the different subject areas (mathematics, Portuguese, and English) or at the high levels of achievement, indicate that the effects may go beyond grade inflation, but this does not constitute enough evidence against the grade-inflation hypothesis.</p> <p>A third hypothesis is based in the autonomy given to schools by the policy to determine how to invest the additional resources (i.e., teaching time, target groups) to reduce retention and increase student achievement. The results from the external standardized measures suggest that, at least for the majority of schools, their choices may not have led to the expected results. The null results on the sixth-grade exams can also indicate that PMSE may be a more fruitful strategy in the earlier grade levels, where grade retention tends to be lower. In summary, new data are necessary to examine grade inflation, as well as the aforementioned hypotheses about the behavior of teachers and schools.</p> <p>It is important to note that all estimates of impact on external standardized assessments were based on a much smaller sample of schools than the analysis of impact on teacher-determined measures of educational achievement because of three reasons. First, estimates for the impact on the standardized measures refer to students' performance at a single grade level (i.e., sixth or ninth), whereas the remaining outcomes refer to the performance at the two grade levels in which PMSE schools chose to invest their PMSE resources each year. Second, because PMSE schools were directed by the ME to choose only two grade levels, most schools—but not all—chose to intervene in the beginning of a cycle of schooling in the first year of the program (first, fifth, and seventh grade), and then follow that cohort into the second year (most likely second, sixth, and eighth), and so on. Moreover, because there is standardized testing at grades 6 and 9, most schools chose not to intervene in these grade levels in the first year of the program, hoping to train the cohort with the new PMSE strategies and improve standardized outcomes over time. Therefore, these estimates of impact on external standardized assessments need to be interpreted with caution and confirmed with additional data.</p> <p>Overall, the mixed evidence from this study indicates that policies such as PMSE can in fact reduce retention and increase educational achievement in certain domains, but can also have a negative impact on students' performance on high-stakes standardized assessments. Policy-makers in countries with high retention rates, such as Portugal, France, and the Netherlands, should be well informed of the mixed impact of such policies and consider the impact of incentives to lowering retention, while monitoring evidence of grade inflation during implementation and scale-up.</p> <p>To better inform policy design, we also need estimates of the cost effectiveness of PMSE-type policies. It is possible that the positive effects of the PMSE policy, together with the moderate cost of its implementation, can potentially outweigh the net effect of the negative impact on standardized assessments, particularly taking into consideration the potential savings that result from a reduction in retention costs (OECD, [<reflink idref="bib44" id="ref102">44</reflink>]).</p> <p>On the policy side, following the dissemination of study results, the ME decided to allow PMSE to continue in schools through the 4 years of implementation as predicted in 2009 (Despacho n.º 13825/[<reflink idref="bib10" id="ref103">10</reflink>]), despite a change in government. The new Minister of Education and Sciences also proposed a new set of policies endorsing the use of homogeneous class composition based on prior achievement as a strategy to promote school achievement (Despacho normativo n.º 13-A/[<reflink idref="bib11" id="ref104">11</reflink>]). It is important to note that this new set of policies includes a system of performance incentives for schools that relies heavily on standardized assessment as a measure of school performance. Given the results described in this study, the proposed system of performance may expose new PMSE schools to be perceived as generally "failing." However, it is important to note that this study offers little evidence concerning what to expect of a class composition strategy in the absence of a class size recommendation, as described in this new policy. The current study of PMSE indicates only what the expected impacts are when all PMSE strategies are applied in combination, not separately.</p> <p>On the practice side, the PMSE implementation team is now considering adaptations of PMSE strategies that require no additional financial resources from the ME, including the additional teaching time, and no mandate for reducing retention. Some of these adaptations are already being experimented upon in schools (Magro-C., personal communication, 2013; Moreira, [<reflink idref="bib36" id="ref105">36</reflink>]). These adaptations hold promise for the continuous innovation of teaching practices to further reduce the retention problem, despite a climate of increasing economic austerity and decreasing resources. Moreover, a careful examination of the impact of these strategies may help shed some light on the effects of grade inflation on PMSE-like policies in the absence of incentives.</p> <p>Finally, in the field of program evaluation, this study has contributed to the national and international discussion of what constitutes appropriate evidence of the true impact of an educational program, specifically, a policy targeting class size, class composition, and differentiated instruction policy concurrently, which mandated the reduction of student retention and aimed to increase achievement. The discrepancy between the impact results on measures of teacher-determined and external standardized measures of achievement was made visible because these indicators were used to estimate the impact of the policy. It is therefore important to analyze prior and future evidence of such policies in light of the combination of a varied set of indicators. New indicators of school performance that combine results on external evaluations with other indicators, such as retention rates, would also help quantify the aggregate impact of educational policies implemented at scale.</p> <p>The limitations of using these data and method as a means of assessing the impact of the PMSE should be explicitly stated. First and foremost, there is no way to assess how the PSE estimates for PMSE approximate experimental estimates or other estimates based on econometric techniques. In fact, there was no explicit plan to evaluate this policy at its onset; therefore, all evaluation and data decisions were made ad hoc after 2 years of PMSE intervention.</p> <p>Second, there was some evidence of cross-over as well as potential differential attrition in the intervention and comparison groups. For example, in the second year, 12 schools were dropped from the intervention sample for failure to reduce grade-level retention rates according to mandate, and four schools were selected from the comparison group to join the intervention group. It is possible that some of this selection process in the second year may have made the PMSE group stronger by removing low-performing treatment cases from the intervention and moving top-performing comparison cases to the PMSE group. We need further testing to estimate the impact that this selection has had in the second-year impact estimates of PMSE.</p> <p>Third, it is debatable whether we can use a wide set of school characteristics to simulate the selection process of PMSE schools. PSE is always imperfect because one is unsure whether schools were weighted on important unobservables, and it is possible that unobserved variables could explain some of the findings. Moreover, the liberty given to PMSE schools to design their own treatment created another source of variance, again posing additional challenges to the adequate weighting on unobservables. Because we elected to compare schools chosen to enter the program to schools that applied but were not chosen to enter the program, it would have been preferable to use measures of administrative leadership and resources prior to the policy, which are likely to predict a school's administrative capacity to put together a successful implementation plan, as well as student outcomes. However, no such measures are collected uniformly by the ME across the universe of all schools. It would also have been advisable to carefully monitor all implementation practices in detail. As the Portuguese ME makes further investments to produce yearly data that are valid for all schools, it is important to continue the discussion of what these data should include to really support policy decisions. Moreover, in resemblance to the incentive measures used in other countries to increase the quality and rigor of policy impact evaluation studies (e.g., What Works Clearinghouse for U.S. education policy), we hope that new opportunities will come forth to design randomized trials that will improve causal inferences about current and future policies and facilitate the decisions of policy-makers regarding the Portuguese and other education systems.</p> <p>Recognizing these limitations, however, there are many situations in which decisions are to be made about the future of a policy, and researchers and decision makers would face similar limitations. For example, if one were testing other policies already at scale, it is often difficult to characterize and model the selection process when there is a system of application and selection, or take advantage of an exogenous natural source of variation when there is not. In this study, the use of PSE allowed us to design an evaluation that worked toward providing the necessary empirical evidence for policy-makers to make informed choices regarding the program, while discussing policy alternatives in Portugal and abroad aiming to reduce grade retention and increase achievement across all levels of compulsory education.</p> <hd id="AN0109514424-26">Appendix</hd> <p>Graph</p> <p>Table A1.  Impact on Teacher-Determined Educational Achievement (Panel A) and High-Performance Educational Achievement (Panel B) in Years 1 and 2 of PMSE</p> <p> <ephtml> <table><tr><td valign="bottom">Variables</td><td valign="bottom">N Grade Levels</td><td valign="bottom">N Schools</td><td valign="bottom">Unadjusted Mean of Intervention Schools</td><td valign="bottom">Adjusted Mean of Control Schools</td><td valign="bottom">Adjusted Diff.</td><td valign="bottom">Standard Error</td><td valign="bottom">p-Value</td><td valign="bottom">Effect Size</td></tr><tr><td>Panel A: Educational Achievement</td></tr><tr><td>Percentage of grade-level transitions:</td></tr><tr><td> Year 1</td><td char=".">450</td><td char=".">347</td><td char=".">90.747</td><td char=".">86.449</td><td char=".">4.298</td><td char=".">.708</td><td>.000***</td><td char=".">.473</td></tr><tr><td> Year 2</td><td char=".">449</td><td char=".">343</td><td char=".">89.106</td><td char=".">89.678</td><td char=".">−.572</td><td char=".">.444</td><td>.209</td><td char=".">−.109</td></tr><tr><td>Percentage of success in mathematics:</td></tr><tr><td> Year 1</td><td char=".">386</td><td char=".">299</td><td char=".">78.393</td><td char=".">73.340</td><td char=".">5.053</td><td char=".">1.300</td><td>.000***</td><td char=".">.423</td></tr><tr><td> Year 2</td><td char=".">387</td><td char=".">297</td><td char=".">77.439</td><td char=".">72.595</td><td char=".">4.844</td><td char=".">1.480</td><td>.001**</td><td char=".">.405</td></tr><tr><td>Percentage of success in Portuguese:</td></tr><tr><td> Year 1</td><td char=".">391</td><td char=".">305</td><td char=".">87.973</td><td char=".">81.985</td><td char=".">5.988</td><td char=".">.956</td><td>.000***</td><td char=".">.557</td></tr><tr><td> Year 2</td><td char=".">396</td><td char=".">306</td><td char=".">89.454</td><td char=".">84.896</td><td char=".">4.558</td><td char=".">.848</td><td>.000***</td><td char=".">.467</td></tr><tr><td>Percentage of success in English:</td></tr><tr><td> Year 1</td><td char=".">361</td><td char=".">282</td><td char=".">84.573</td><td char=".">79.485</td><td char=".">5.088</td><td char=".">1.218</td><td>.000***</td><td char=".">.477</td></tr><tr><td> Year 2</td><td char=".">361</td><td char=".">277</td><td char=".">85.963</td><td char=".">80.954</td><td char=".">5.009</td><td char=".">1.373</td><td>.000***</td><td char=".">.455</td></tr><tr><td /><td /><td /><td /><td /><td /><td /><td /><td /></tr><tr><td>Panel B: High-Performance Educational Achievement</td></tr><tr><td>Percentage of fours and fives in mathematics:</td></tr><tr><td> Year 1</td><td char=".">386</td><td char=".">299</td><td char=".">29.476</td><td char=".">28.097</td><td char=".">1.379</td><td char=".">.995</td><td char=".">.166</td><td char=".">.126</td></tr><tr><td> Year 2</td><td char="."> 387</td><td char=".">297</td><td char=".">28.691</td><td char=".">26.800</td><td char=".">1.891</td><td char=".">1.069</td><td char=".">.077+</td><td char=".">.180</td></tr><tr><td>Percentage of fours and fives in Portuguese:</td></tr><tr><td> Year 1</td><td char=".">391</td><td char=".">305</td><td char=".">30.990</td><td char=".">26.833</td><td char=".">4.157</td><td char=".">1.063</td><td>.000***</td><td char=".">.359</td></tr><tr><td> Year 2</td><td char=".">396</td><td char=".">306</td><td char=".">31.180</td><td char=".">28.341</td><td char=".">2.839</td><td char=".">1.232</td><td>.021*</td><td char=".">.249</td></tr><tr><td>Percentage of fours and fives in English:</td></tr><tr><td> Year 1</td><td char=".">361</td><td char=".">282</td><td char=".">35.112</td><td char=".">31.777</td><td char=".">3.335</td><td char=".">1.224</td><td>.006**</td><td char=".">.295</td></tr><tr><td> Year 2</td><td char=".">361</td><td char=".">277</td><td char=".">33.580</td><td char=".">32.718</td><td char=".">.862</td><td char=".">1.368</td><td>.529</td><td char=".">.073</td></tr></table> </ephtml> </p> <p>1 The effect size was computed by dividing the estimated adjusted difference between groups by the standard deviation of the indicator for the comparison group.</p> <ulist> <item>2 <emph>p</emph> <.10.</item> <item>3 <emph>p</emph> <.05.</item> <item>4 <emph>p</emph> <.01.</item> <item>5 <emph>p</emph> <.001.</item> </ulist> <p>Graph</p> <p>Table A2.  Impact on External Standardized Measures of Educational Achievement for the Sixth-Grade Low-Stakes Assessment at the School Level (Panel A) and Ninth-Grade High-Stakes Assessment at the Student Level (Panel B) in Years 1 and 2 of PMSE</p> <p> <ephtml> <table><tr><td valign="bottom">Variables</td><td valign="bottom">N Students</td><td valign="bottom">N Schools</td><td valign="bottom">Unadjusted Mean of Intervention Schools</td><td valign="bottom">Adjusted Mean of Control Schools</td><td valign="bottom">Adjusted Diff.</td><td valign="bottom">Standard Error</td><td valign="bottom">p-Value</td><td valign="bottom">Effect Size</td></tr><tr><td>Panel A: Sixth-Grade Low-Stakes Standardized Assessment, School Level</td></tr><tr><td>Percentage of success in the mathematics exam:</td></tr><tr><td> Year 1</td><td>N/A</td><td char=".">24</td><td char=".">78.363</td><td char=".">85.251</td><td char=".">−6.888</td><td char=".">10.385</td><td char=".">.575</td><td char=".">−.548</td></tr><tr><td> Year 2</td><td>N/A</td><td char=".">70</td><td char=".">60.608</td><td char=".">58.838</td><td char=".">1.770</td><td char=".">2.519</td><td char=".">.486</td><td char=".">.116</td></tr><tr><td>Percentage of success in the Portuguese exam:</td></tr><tr><td> Year 1</td><td>N/A</td><td char=".">24</td><td char=".">89.421</td><td char=".">83.200</td><td char=".">6.221</td><td char=".">4.690</td><td char=".">.316</td><td char=".">.608</td></tr><tr><td> Year 2</td><td>N/A</td><td char=".">71</td><td char=".">79.635</td><td char=".">79.358</td><td char=".">.277</td><td char=".">2.120</td><td char=".">.896</td><td char=".">.030</td></tr><tr><td /><td valign="top" /><td valign="top" /><td valign="top" /><td valign="top" /><td valign="top" /><td valign="top" /><td valign="top" /><td valign="top" /></tr><tr><td>Panel B: Ninth-Grade High-Stakes Standardized Assessment, Student Level</td></tr><tr><td>Performance in the mathematics exam:</td></tr><tr><td> Year 1</td><td char=".">1,955</td><td char=".">33</td><td char=".">2.509</td><td char=".">2.709</td><td char=".">−.200</td><td char=".">.115</td><td char=".">.081+</td><td char=".">−.194</td></tr><tr><td> Year 2</td><td char=".">2,649</td><td char=".">36</td><td char=".">2.195</td><td char=".">2.143</td><td char=".">.052</td><td char=".">.091</td><td char=".">.568</td><td char=".">.052</td></tr><tr><td>Performance in the Portuguese exam:</td></tr><tr><td> Year 1</td><td char=".">1,942</td><td char=".">33</td><td char=".">2.839</td><td char=".">2.960</td><td char=".">−.121</td><td char=".">.082</td><td char=".">.140</td><td char=".">−.159</td></tr><tr><td> Year 2</td><td char=".">2,684</td><td char=".">36</td><td char=".">2.641</td><td char=".">2.810</td><td char=".">−.169</td><td char=".">.082</td><td char=".">.041*</td><td char=".">−.222</td></tr></table> </ephtml> </p> <ulist> <item>6 The effect size was computed by dividing the estimated adjusted difference between groups by the standard deviation of the indicator for the comparison group.</item> <item>7 <emph>p</emph> <.10.</item> <item>8 <emph>p</emph> <.05.</item> </ulist> <ref id="AN0109514424-27"> <title> Notes </title> <blist> <bibl id="bib1" idref="ref26" type="bt">1</bibl> <bibtext> This study was funded by the Portuguese Ministry of Education and Science Procedimento n.º 2011/109/GEPE/ABS, through a grant from the POAT-European Social Fund, European Commission. The authors would like to thank the editor and four anonymous reviewers for their constructive feedback, which helped us improve the manuscript. We would also like to thank participating students, teachers, and principals. M. Clara Barata acknowledges funding from the European Union under the Marie Curie International Incoming Fellowship (FP7-PEOPLE-2011-IIF 303242). M. Clara Barata is a researcher at the Universidade de Coimbra, Coimbra, Portugal. Joana Nunes Patrício and João Graça are researchers, M. Manuela Calheiros is assistant professor, and M Luisa Lima is full professor at the Instituto Universitário de Lisboa (ISCTE-IUL), Cis-IUL, Lisbon, Portugal. Address all correspondence to Dr. M. Clara Barata, Universidade de Coimbra, Faculdade de Psicologia e Ciências de Educação—Pólo 1, Gab.2.11. 3001-802 Coimbra, Portugal; e-mail: mclarabarata@fpce.uc.pt.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref27" type="bt">2</bibl> <bibtext> According to the ME's second-year PMSE implementation report, the schools in the south region of Portugal were underrepresented in the initial application pool because of a delay in access to the application process in this region. Therefore, the ME team decided to add four additional schools from this region to the PMSE sample in the second year (DGIDC, [13]).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref28" type="bt">3</bibl> <bibtext> In the appendix, we present alternative Tables A1 and A2 with unadjusted means for the intervention schools and adjusted means for the comparison schools based on the covariate values of the intervention group. This alternative way of reporting the effects of the PMSE intervention has no impact on the estimated adjusted difference between the groups, and only a very small impact on the interpretation of this difference. The distinction between the two ways of reporting the means is larger for estimates based on smaller samples (such as the low-stakes standardized assessment), and almost negligible for the remainder estimated results.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref29" type="bt">4</bibl> <bibtext> Full estimation results are available upon request.</bibtext> </blist> </ref> <ref id="AN0109514424-28"> <title> References </title> <blist> <bibtext> Agodini, R., & Dynarski, M. (2004). Are experiments the only option? A look at dropout prevention programs. Review of Economics and Statistics, 86, 180–194.</bibtext> </blist> <blist> <bibtext> Alves, J. M., & Moreira, L. T. (2011). Projecto Fénix—Relatos que contam o sucesso. Porto, Portugal: Faculdade de Educação e Psicologia da Universidade Católica Portuguesa.</bibtext> </blist> <blist> <bibtext> Angrist, J. D., & Lavy, V. (1999). Using Maimonides' rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 114(2), 533–575. doi:10.1162/003355399556061</bibtext> </blist> <blist> <bibtext> Azevedo, J., & Alves, J. M. (2010). Projecto Fénix—Mais Sucesso Para Todos: Memórias e dinâmicas da construção do sucesso escolar. Porto, Portugal: Faculdade de Educação e Psicologia da Universidade Católica Portuguesa.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref30" type="bt">5</bibl> <bibtext> Barata, M. C., Calheiros, M. C., Patrício, J., Graça, J., & Lima, M.L. (2012). Avaliação do programa mais sucesso escolar. Lisboa: Direção-Geral de Estatísticas de Educação e Ciência—Ministério da Educação e Ciência. Retrieved from <ulink href="http://www.gepe.min-edu.pt/np4/?newsId=670&fileName=PMSE%5fAlt%5fPDF.pdf">http://www.gepe.min-edu.pt/np4/?newsId=670&fileName=PMSE%5fAlt%5fPDF.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib6" idref="ref31" type="bt">6</bibl> <bibtext> Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiment and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref7" type="bt">7</bibl> <bibtext> Davis, J. W., & Bauman, K. (2011). School enrollment in the United States: 2008. Current Population Reports P20–564. Washington, DC: U.S. Census Bureau.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref81" type="bt">8</bibl> <bibtext> Dehejia, R., & Wahba, S. (1999). Causal effects in non-experimental studies: Re-evaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref82" type="bt">9</bibl> <bibtext> Dehejia, R., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84(1), 151– 161.</bibtext> </blist> <blist> <bibtext> Despacho n.º 13825/2011, D.R. n.º 198, Série II, de 2011-10-14 [Decree No. 13825/2011 of October 14, 2011].</bibtext> </blist> <blist> <bibtext> Despacho normativo n.º 13-A/2012, D.R. 109, Série II, de 2012-06-05 [Legislative Decree No. 13-A/2012 of June 5, 2012].</bibtext> </blist> <blist> <bibtext> Diaz, J., & Handa, S. (2006). An assessment of propensity score matching as a nonexperimental impact estimator: Evidence from Mexico's PROGRESA program. Journal of Human Resources, 41(2), 319–345. doi:10.3368/jhr.XLI.2.319</bibtext> </blist> <blist> <bibtext> Direcção-Geral de Inovação e de Desenvolvimento Curricular (DGIDC) (2010). Programa Mais Sucesso Escolar: Relatório final. Retrieved from <ulink href="http://www.dgidc.min-edu.pt/outrosprojetos/index.php?s=directorio&pid=122#i">http://www.dgidc.min-edu.pt/outrosprojetos/index.php?s=directorio&pid=122#i</ulink></bibtext> </blist> <blist> <bibtext> Dobbelsteen, S., Levin, J., & Oosterbeek, H. (2002). The causal effect of class size on scholastic achievement: Distinguishing the pure class size effect from the effect of changes in class composition. Oxford Bulletin of Economics and Statistics, 64, 17–38.</bibtext> </blist> <blist> <bibtext> Favinha, M., Silvestre, M., & Magro-C, T. (2011). Monitorizar resultados para avaliar segundo a lógica de ciclo: Tecnologia de suporte avaliativo do projecto TurmaMais. In I. Fialho & H. Salgueiro (Org.), TurmaMais e sucesso escolar: Contributos teóricos e práticos (pp. 125–134). Évora: CIEP/UE.</bibtext> </blist> <blist> <bibtext> Fialho, I., & Verdasca, J. (2013). TurmaMais e sucesso escolar: Trajetórias para uma nova cultura de escola. Évora: Centro de Investigação em Educação e Psicologia–Universidade de Évora.</bibtext> </blist> <blist> <bibtext> Finn, J., & Achilles, C. (1999). Tennessee's class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21(2), 97–109. doi:10.3102/01623737021002097</bibtext> </blist> <blist> <bibtext> Fruehwirth, J. C., Navarro, S., & Takahashi, Y. (2011). How the timing of grade retention affects outcomes: Identification and estimation of time-varying treatment effects (Working Paper 2011-015, Human Capital and Economic Opportunity Working Group).</bibtext> </blist> <blist> <bibtext> Gormley, W., Gayer, T., Phillips, D., & Dawson, B. (2005). The effects of universal pre-K on cognitive development. Developmental Psychology, 41(6), 872–884.</bibtext> </blist> <blist> <bibtext> Hanushek, E. (1999). Some findings from an independent investigation of the Tennessee STAR experiment and from other investigations of class size effects. Educational Evaluation and Policy Analysis, 21(2), 143–163. doi:10.3102/01623737021002143</bibtext> </blist> <blist> <bibtext> Hauser, R. M., Frederick, C. B., & Andrew, M. (2007). Grade retention in the age of standards-based reform. In A. Gamoran (Ed.), Will standards-based reform in education help close the poverty gap? Washington, D.C.: Brookings Institution Press.</bibtext> </blist> <blist> <bibtext> Heacox, D. (2006). Diferenciação curricular na sala de aula. Porto: Porto Editora.</bibtext> </blist> <blist> <bibtext> Holmes, C. T. (1989). Grade-level retention effects: A meta-analysis of research studies. In L. A. Shepard & M. L. Smith (Eds.), Flunking grades: Research and policies on retention (pp. 16–33). London: Falmer.</bibtext> </blist> <blist> <bibtext> Hoxby, C. (1998). The effects of class size and composition on student achievement: New evidence from natural population variation (NBER Working Paper No. 6869, National Bureau of Economic Research).</bibtext> </blist> <blist> <bibtext> Hoxby, C. (2000). The effects of class size on student achievement: New evidence from population variation. Quarterly Journal of Economics, 115(4), 1239–1285.</bibtext> </blist> <blist> <bibtext> Jimerson, S. (2001). Meta-analysis of grade retention research: Implications for practice in the 21st century. School Psychology Review, 30(3), 420.</bibtext> </blist> <blist> <bibtext> Jimerson, S. R., Anderson, G., & Whipple, A. (2002). Winning the battle and losing the war: Examining the relation between grade retention and dropping out of high school. Psychology in the Schools, 39(4), 441–457. doi:10.1002/pits.10046</bibtext> </blist> <blist> <bibtext> Krueger, A. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532.</bibtext> </blist> <blist> <bibtext> Martins, M. (2009). A escola faz a diferença: O projeto fénix e as dinâmicas de sucesso—estudo de caso. Master's dissertation, Universidade Católica Portuguesa, Porto.</bibtext> </blist> <blist> <bibtext> Martins, M., & Alves, J. (2009). O projeto fénix, a alteração da gramática escolar e a promoção do sucesso: Evidências de um percurso de avaliação. Revista Portuguesa de Investigação Educacional, 8, 5–46.</bibtext> </blist> <blist> <bibtext> McEwan, P. J., & Urquiola, M. (2005). Economic behavior and the regression-discontinuity design: Evidence from class size reduction. Unpublished manuscript.</bibtext> </blist> <blist> <bibtext> Michalopoulos, C., Bloom, H. S., & Hill, C. J. (2004). Can propensity-score methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? Review of Economics and Statistics, 86(1), 156–179.</bibtext> </blist> <blist> <bibtext> Ministry of Education (2009). Mais sucesso escolar: Projectos de combate ao insucesso escolar no ensino básico [Mais sucesso escolar: Projects to address school failure in compulsory schooling]. Boletim dos Professores. 15.</bibtext> </blist> <blist> <bibtext> Moreira, M. L. (2009). Projecto fénix—um projecto de inovação pedagógica: Operacionalização e impacto no agrupamento Campo Aberto—Beiriz (Tese de Mestrado não publicada). Porto, Portugal: Universidade Católica Portuguesa.</bibtext> </blist> <blist> <bibtext> Moreira, L. T. (2010). Projeto fénix—um projeto de inovação pedagógica: Estudo avaliativo da operacionalização e do impacto no agrupamento Campo Aberto—Beiriz. In J. Azevedo & J. M. Alves (Org.), Projecto fénix—mais sucesso para todos: Memórias e dinâmicas da construção do sucesso escolar (pp. 75–110). Porto: Universidade Católica Portuguesa, Faculdade de Educação e Psicologia.</bibtext> </blist> <blist> <bibtext> Moreira, L. T. (2014). Projeto fénix: Sentidos de um projeto emergente na construção do sucesso escolar. Porto: Universidade Católica Portuguesa, Faculdade de Educação e Psicologia.</bibtext> </blist> <blist> <bibtext> Morgado, J. (2005). A model of differentiated classroom management. In Proceedings of International Special Education Congress "Inclusion: Celebrating Diversity" (p. 142). Glasgow.</bibtext> </blist> <blist> <bibtext> Murnane, R., & Willett. J. (2010). Methods matter. Oxford: Oxford University Press.</bibtext> </blist> <blist> <bibtext> National Center for Education Statistic. (2009). Condition of Education 2009, Indicator 18. Retrieved from <ulink href="http://nces.ed.gov/programs/coe/2009/section3/indicator18.asp">http://nces.ed.gov/programs/coe/2009/section3/indicator18.asp</ulink>.</bibtext> </blist> <blist> <bibtext> Office of the Minister of Education. (2009). Edital mais sucesso escolar [Public notice mais sucesso escolar]. Retrieved from <ulink href="http://www.dgidc.min-edu.pt/outrosprojetos/data/outrosprojectos/Programa%5fMais%5fSucesso%5fEscolar/Documentos/edital%5fregulamento%5fmais%5fsucesso%5fescolar.pdf">http://www.dgidc.min-edu.pt/outrosprojetos/data/outrosprojectos/Programa%5fMais%5fSucesso%5fEscolar/Documentos/edital%5fregulamento%5fmais%5fsucesso%5fescolar.pdf</ulink></bibtext> </blist> <blist> <bibtext> Organisation for Economic Cooperation and Development (OECD). (2007). Improving school leadership. Retrieved from <ulink href="http://www.oecd.org/dataoecd/44/56/40710632.pdf">http://www.oecd.org/dataoecd/44/56/40710632.pdf</ulink></bibtext> </blist> <blist> <bibtext> Organisation for Economic Cooperation and Development (OECD). (2011). Grade retention during compulsory education in Europe: Regulations and statistics. Paris: OECD Publishing. <ulink href="http://eacea.ec.europa.eu/education/eurydice/documents/thematic%5freports">http://eacea.ec.europa.eu/education/eurydice/documents/thematic%5freports</ulink> /126EN.pdf</bibtext> </blist> <blist> <bibtext> Organisation for Economic Cooperation and Development (OECD). (2013). PISA 2012 Results: What makes schools successful? Resources, policies and practices (Vol. 4). Paris: OECD Publishing. Retrieved from <ulink href="http://dx.doi.org/10.1787/9789264201156-en">http://dx.doi.org/10.1787/9789264201156-en</ulink></bibtext> </blist> <blist> <bibtext> Organisation for Economic Cooperation and Development (OECD). (2015). OECD skills strategy diagnostic report Portugal 2015. Paris: OECD Publishing. Retrieved from <ulink href="http://skills.oecd.org/developskills/documents/Portugal-Diagnostic-Report-web.pdf">http://skills.oecd.org/developskills/documents/Portugal-Diagnostic-Report-web.pdf</ulink></bibtext> </blist> <blist> <bibtext> Roderick, M. (1994). Grade retention and school dropout: Investigating the association. American Educational Research Journal, 31(4), 729–759. doi:10.3102/00028312031004729</bibtext> </blist> <blist> <bibtext> Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.</bibtext> </blist> <blist> <bibtext> Rumberger, R. W. (1987). High school dropouts: A review of issues and evidence. Review of Educational Research, 57, 101–121. doi:10.3102/00346543057002101</bibtext> </blist> <blist> <bibtext> Shadish, W., Campbell, D., & Cook, T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin.</bibtext> </blist> <blist> <bibtext> Shepard, L. A., & Smith, M. L. (1990). Synthesis of research on grade retention. Educational Leadership, 47(8), 84.</bibtext> </blist> <blist> <bibtext> Silberglitt, B., Jimerson, S. R., Burns, M. K., & Appleton, J. J. (2006). Does the timing of grade retention make a difference? Examining the effects of early versus later retention. School Psychology Review, 35(1), 134–141.</bibtext> </blist> <blist> <bibtext> Verdasca, J. (2008). TurmaMais: Uma tecnologia organizacional para a promoção do sucesso escolar. In Manuel Villaverde Cabral (Org.), Conferência internacional sucesso e insucesso: Escola, economia e sociedade (pp. 139–176). Lisbon: Fundação Calouste Gulbenkian.</bibtext> </blist> <blist> <bibtext> Verdasca, J. (2010). Programa Mais Sucesso Escolar: Um desafio na afirmação da autonomia da escola. In J. Azevedo & J. M. Alves (Eds.), Projecto Fénix—Mais sucesso para todos (pp. 31–35). Porto, Portugal: Universidade Católica Portuguesa.</bibtext> </blist> <blist> <bibtext> Verdasca, J., & Cruz, T. (2006). O projecto TurmaMais: Dialogando em torno de uma experiência no combate ao insucesso e abandono escolares. Revista Portuguesa de Investigação Educacional, 4, 113–128.</bibtext> </blist> <blist> <bibtext> Vygotsky, L. (1978). Mind and society: The development of higher mental processes. Cambridge, MA: Harvard University Press.</bibtext> </blist> <blist> <bibtext> Wilde, E. T., & Hollister, R. (2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management, 26, 455–477. doi:10.1002/pam.20262</bibtext> </blist> <blist> <bibtext> Wong, V., Cook, T., Barnett, S., & Jung, K. (2008). An effectiveness-based evaluation of five state pre-kindergarten programs. Journal of Policy Analysis and Management, 27(1), 122–154.</bibtext> </blist> <blist> <bibtext> Zinth, K. (2005). Student promotion/retention policies. Denver: Education Commission of the States.</bibtext> </blist> </ref> <aug> <p>By M. Clara Barata; M. Manuela Calheiros; Joana Nunes Patrício; João Graça and M. Luisa Lima</p> </aug> <nolink nlid="nl1" bibid="bib26" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib49" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib42" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib39" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib21" firstref="ref8"></nolink> <nolink nlid="nl6" bibid="bib57" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib27" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib44" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib50" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib23" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib45" firstref="ref17"></nolink> <nolink nlid="nl12" bibid="bib47" firstref="ref18"></nolink> <nolink nlid="nl13" bibid="bib18" firstref="ref19"></nolink> <nolink nlid="nl14" bibid="bib15" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib43" firstref="ref24"></nolink> <nolink nlid="nl16" bibid="bib41" firstref="ref25"></nolink> <nolink nlid="nl17" bibid="bib40" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib16" firstref="ref34"></nolink> <nolink nlid="nl19" bibid="bib51" firstref="ref35"></nolink> <nolink nlid="nl20" bibid="bib53" firstref="ref36"></nolink> <nolink nlid="nl21" bibid="bib34" firstref="ref39"></nolink> <nolink nlid="nl22" bibid="bib35" firstref="ref40"></nolink> <nolink nlid="nl23" bibid="bib36" firstref="ref41"></nolink> <nolink nlid="nl24" bibid="bib29" firstref="ref43"></nolink> <nolink nlid="nl25" bibid="bib30" firstref="ref44"></nolink> <nolink nlid="nl26" bibid="bib33" firstref="ref48"></nolink> <nolink nlid="nl27" bibid="bib22" firstref="ref50"></nolink> <nolink nlid="nl28" bibid="bib37" firstref="ref51"></nolink> <nolink nlid="nl29" bibid="bib54" firstref="ref52"></nolink> <nolink nlid="nl30" bibid="bib20" firstref="ref54"></nolink> <nolink nlid="nl31" bibid="bib17" firstref="ref55"></nolink> <nolink nlid="nl32" bibid="bib28" firstref="ref57"></nolink> <nolink nlid="nl33" bibid="bib24" firstref="ref59"></nolink> <nolink nlid="nl34" bibid="bib25" firstref="ref60"></nolink> <nolink nlid="nl35" bibid="bib14" firstref="ref61"></nolink> <nolink nlid="nl36" bibid="bib38" firstref="ref67"></nolink> <nolink nlid="nl37" bibid="bib48" firstref="ref68"></nolink> <nolink nlid="nl38" bibid="bib31" firstref="ref77"></nolink> <nolink nlid="nl39" bibid="bib46" firstref="ref78"></nolink> <nolink nlid="nl40" bibid="bib12" firstref="ref83"></nolink> <nolink nlid="nl41" bibid="bib32" firstref="ref84"></nolink> <nolink nlid="nl42" bibid="bib55" firstref="ref85"></nolink> <nolink nlid="nl43" bibid="bib19" firstref="ref100"></nolink> <nolink nlid="nl44" bibid="bib56" firstref="ref101"></nolink> <nolink nlid="nl45" bibid="bib10" firstref="ref103"></nolink> <nolink nlid="nl46" bibid="bib11" firstref="ref104"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1077335
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluating the Impact of National Educational Policy to Reduce Retention and Increase Achievement in Compulsory Education
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Barata%2C+M%2E+Clara%22">Barata, M. Clara</searchLink><br /><searchLink fieldCode="AR" term="%22Calheiros%2C+M%2E+Manuela%22">Calheiros, M. Manuela</searchLink><br /><searchLink fieldCode="AR" term="%22Patrício%2C+Joana+Nunes%22">Patrício, Joana Nunes</searchLink><br /><searchLink fieldCode="AR" term="%22Graça%2C+João%22">Graça, João</searchLink><br /><searchLink fieldCode="AR" term="%22Lima%2C+M%2E+Luisa%22">Lima, M. Luisa</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Elementary+School+Journal%22"><i>Elementary School Journal</i></searchLink>. Sep 2015 116(1):149-171.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Chicago Press. Journals Division, P.O. Box 37005, Chicago, IL 60637. Tel: 877-705-1878; Tel: 773-753-3347; Fax: 877-705-1879; Fax: 773-753-0811; e-mail: subscriptions@press.uchicago.edu; Web site: http://www.press.uchicago.edu
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 23
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2015
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Policy%22">Educational Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Public+Policy%22">Public Policy</searchLink><br /><searchLink fieldCode="DE" term="%22School+Holding+Power%22">School Holding Power</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Persistence%22">Academic Persistence</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Compulsory+Education%22">Compulsory Education</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Class+Size%22">Class Size</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Individualized+Instruction%22">Individualized Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+Inflation%22">Grade Inflation</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Portugal%22">Portugal</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1086/682748
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0013-5984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Despite the accumulated evidence that retention is an ineffective and potentially harmful remedial strategy, several countries struggle with high levels of retention in compulsory schooling. This article provides evidence of the impact of the Portuguese national educational policy "Programa Mais Sucesso Escolar" (PMSE) using class size, class composition, and differentiated instruction to reduce student retention and increase achievement. We found that PMSE significantly reduced retention, but had mixed effects on educational achievement, pointing to evidence of grade inflation on teacher-determined measures of achievement. In the discussion, we analyze the implications of these results for policy, practice, and program evaluation research.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 56
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2015
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1077335
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1077335
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1086/682748
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 23
        StartPage: 149
    Subjects:
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Educational Policy
        Type: general
      – SubjectFull: Public Policy
        Type: general
      – SubjectFull: School Holding Power
        Type: general
      – SubjectFull: Academic Persistence
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Compulsory Education
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Class Size
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Individualized Instruction
        Type: general
      – SubjectFull: Grade Inflation
        Type: general
      – SubjectFull: Portugal
        Type: general
    Titles:
      – TitleFull: Evaluating the Impact of National Educational Policy to Reduce Retention and Increase Achievement in Compulsory Education
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Barata, M. Clara
      – PersonEntity:
          Name:
            NameFull: Calheiros, M. Manuela
      – PersonEntity:
          Name:
            NameFull: Patrício, Joana Nunes
      – PersonEntity:
          Name:
            NameFull: Graça, João
      – PersonEntity:
          Name:
            NameFull: Lima, M. Luisa
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2015
          Identifiers:
            – Type: issn-print
              Value: 0013-5984
          Numbering:
            – Type: volume
              Value: 116
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Elementary School Journal
              Type: main
ResultId 1