View in EDS HTML Full Text PDF Full Text

The Impact of Transition Intervention in High School on Pathways through College

Saved in:
Bibliographic Details
Title: The Impact of Transition Intervention in High School on Pathways through College
Language: English
Authors: Xu, Zeyu, Backes, Ben (ORCID 0000-0003-3485-1586), Goldhaber, Dan (ORCID 0000-0003-4260-4040)
Source: Community College Review. Apr 2023 51(2):216-245.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 30
Publication Date: 2023
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305A160188
Document Type: Journal Articles
Reports - Research
Education Level: High Schools
Secondary Education
Higher Education
Postsecondary Education
Descriptors: High School Students, College Bound Students, College Readiness, College Students, Transitional Programs, Intervention, College Credits, Program Effectiveness, Remedial Instruction, Academic Achievement, College Preparation
Geographic Terms: Kentucky
DOI: 10.1177/00915521221145325
ISSN: 0091-5521
1940-2325
Abstract: Objective: In 2009, the Kentucky General Assembly found unacceptable and costly the ongoing high numbers of high school students requiring remediation once they enter higher education. The state passed legislation to better align secondary and college education, establishing a diagnostic cycle that would become its Targeted Interventions (TI) program. Using 11 years of panel data, this study tracked college progression of seven cohorts of students in order to estimate the impacts of this program. Method: Using student-level administrative data from the state of Kentucky that tracks students from high school through college, a difference-in-regression-discontinuity design was used to compare how students just below college readiness benchmarks fared relative to those just above once TI was implemented. Results: The TI program significantly increased the likelihood that students took at least 15 credits during their first term, a key predictive measure for college completion. However, these early effects did not translate into detectable impacts on the likelihood of earning enough credits to graduate from college or likelihood of transfers from a 2-year to a 4-year college. One possible explanation for this pattern is that TI appears to have crowded out other core courses in high school, especially in math, without increasing total instructional time. Findings suggest that the standards used by high schools to judge student progress toward college readiness may be consistent with the skills needed to place out of developmental courses, but not sufficient to better prepare students for college-level instruction. Contributions: To our knowledge, this is the first study to explore how TI shapes longer term college outcomes. The transition curriculum, while helping students avoid the need for college developmental courses, did not help a measurable share of students develop necessary skills to progress through college relative to what they would have otherwise taken. A possible explanation for these findings is that high school-to-college transition interventions that do not increase total instruction time do not sufficiently move the needle on the college preparedness among high school graduates. For states concerned with the number of students entering college deemed not college ready, it appears that high school-to-college transition interventions that supplant instead of supplement regular high school curriculum have a limited scope for impact on long-run college success.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2023
Accession Number: EJ1369483
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFZQQ0i6ukgPBRIy2-UdnkMAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDKdviwwrLv5bWrG3JgIBEICBm12zazjQsv78WbO5ZatI7U2dZ6J14DwwiAtmjEXLTU1GCafHpXk5ksxgOOMcLqeA4XOTQIN5P43W5PpyOvbh3URFfzFus2CtqN1pRPQCzWoEH9r7gMxUEApPd8L56oFPVYgKOSSFBh5shQFXga9wRzMJolAXtlOogSCda3aTD9LzWa2r8I0lVzBGnn8_OWZiLi7zXCnKzCKmxlAe
Text:
  Availability: 1
  Value: <anid>AN0162243559;ccr01apr.23;2023Mar08.05:48;v2.2.500</anid> <title id="AN0162243559-1">The Impact of Transition Intervention in High School on Pathways Through College </title> <p>Objective: In 2009, the Kentucky General Assembly found unacceptable and costly the ongoing high numbers of high school students requiring remediation once they enter higher education. The state passed legislation to better align secondary and college education, establishing a diagnostic cycle that would become its Targeted Interventions (TI) program. Using 11 years of panel data, this study tracked college progression of seven cohorts of students in order to estimate the impacts of this program. Method: Using student-level administrative data from the state of Kentucky that tracks students from high school through college, a difference-in-regression-discontinuity design was used to compare how students just below college readiness benchmarks fared relative to those just above once TI was implemented. Results: The TI program significantly increased the likelihood that students took at least 15 credits during their first term, a key predictive measure for college completion. However, these early effects did not translate into detectable impacts on the likelihood of earning enough credits to graduate from college or likelihood of transfers from a 2-year to a 4-year college. One possible explanation for this pattern is that TI appears to have crowded out other core courses in high school, especially in math, without increasing total instructional time. Findings suggest that the standards used by high schools to judge student progress toward college readiness may be consistent with the skills needed to place out of developmental courses, but not sufficient to better prepare students for college-level instruction. Contributions: To our knowledge, this is the first study to explore how TI shapes longer term college outcomes. The transition curriculum, while helping students avoid the need for college developmental courses, did not help a measurable share of students develop necessary skills to progress through college relative to what they would have otherwise taken. A possible explanation for these findings is that high school-to-college transition interventions that do not increase total instruction time do not sufficiently move the needle on the college preparedness among high school graduates. For states concerned with the number of students entering college deemed not college ready, it appears that high school-to-college transition interventions that supplant instead of supplement regular high school curriculum have a limited scope for impact on long-run college success.</p> <p>Keywords: remedial/developmental education; student outcomes; work with K-12 schools; quantitative methods; student retention/achievement</p> <hd id="AN0162243559-2">Introduction</hd> <p>Many students who enroll in college do not earn a degree. Among first-time, full-time college students, only 30% graduate from 2-year colleges within 3 years of initial enrollment and only 60% graduate from 4-year colleges within 6 years.[<reflink idref="bib4" id="ref1">4</reflink>] Weak pre-college preparation is thought to be one of the main reasons for poor degree attainment ([<reflink idref="bib9" id="ref2">9</reflink>]). Traditionally, entering underprepared college students were placed in developmental education (DE) before they were able to enroll in college-level courses. However, DE has been found to be both costly and ineffective ([<reflink idref="bib11" id="ref3">11</reflink>]; [<reflink idref="bib35" id="ref4">35</reflink>]), and some scholars and administrators consider getting students ready for college a shared responsibility between high schools and colleges ([<reflink idref="bib7" id="ref5">7</reflink>]).[<reflink idref="bib5" id="ref6">5</reflink>]</p> <p>In response to this concern, an increasing number of states have implemented transition intervention programs for high school students designed to help them graduate high school ready for college. Between 2013 and 2017, the number of statewide programs more than doubled, up to 17, with another 22 states offering local transition intervention programs ([<reflink idref="bib6" id="ref7">6</reflink>]). Transition intervention programs vary in scope, placement mechanism, delivery format, and whether successful completion automatically advances students to college-level coursework, but most include a screening assessment for college readiness, a transition curriculum for those deemed to need the intervention, and an exit evaluation ([<reflink idref="bib15" id="ref8">15</reflink>]).</p> <p>But while high school-to-college transition intervention programs are viewed as holding the promise of improving student college preparedness (see Barnett et al., 2016), there is limited evidence (we describe this below) on the effectiveness of this type of intervention, with existing findings mixed at best. In this paper, we explored a statewide transition intervention program in Kentucky—Targeted Interventions (TI)—and its impact on longer term college outcomes. This is an especially interesting case to investigate because there is evidence ([<reflink idref="bib36" id="ref9">36</reflink>]) that the Kentucky TI program has reduced the need for DE of initially low-achieving students who attend college in the state. To our knowledge, this is the first study of a statewide transition intervention program in the U.S. to explore longer term college outcomes, including the likelihood of transferring from a 2-year to a 4-year college and obtaining enough credits to earn a college degree.</p> <p>In addition to the main research question about the effect of TI on the likelihood of college completion and transfers, this study also examined the extent to which TI impacted early college outcomes including initial major choices, frequency of major switching, and early momentum as measured by taking 15 or more credits during the first term in college. Our study found that TI significantly increased the likelihood that students would take at least 15 credits during the first term in college—a key measure of early college progress that has been shown to be predictive of college completion (e.g., [<reflink idref="bib3" id="ref10">3</reflink>]). But improved early momentum did not lead to detectible improvements in the likelihood of transfers from a 2-year to a 4-year college or the likelihood of completing the minimum number of credits required for graduation (60 credits for 2-year college and 120 credits for 4-year college) within 150% of the normal time for completion (i.e., 3 years for those who start in a 2-year college and 6 years for those who start in a 4-year college).</p> <p>In what follows, we first introduce transition intervention in general, describe its design and implementation in Kentucky, and summarize existing evidence on the effectiveness of transition intervention programs in section 2. This is followed by a brief description of the data, samples, and research method in section 3. This section describes the difference-in-regression discontinuity (DiRD) framework and examines the validity of its assumption in the current context. In Section 4, we present the estimated impact of TI on college outcomes. Section 5 explores potential explanations for the tapered effects of TI (i.e., the lack of statistically significance effect on long-term college outcomes).</p> <hd id="AN0162243559-3">Transition Intervention, Implementation in Kentucky, and Prior Evidence</hd> <p>Although college access has improved for high school graduates ([<reflink idref="bib29" id="ref11">29</reflink>]), roughly 60% of high school graduates are considered not fully prepared for college (ACT, 2019). Misalignment between K-12 and postsecondary education systems has been identified as main reason why completing a high school diploma does not necessarily adequately prepare a student for college ([<reflink idref="bib9" id="ref12">9</reflink>]). To bridge the disconnect between high school graduation requirements and what students need to succeed in college, most states adopted the Common Core State Standards (CCSS) in the early 2010s. College- and career-ready for all students became a top education policy priority, and transition intervention programs were designed to provide extra help for students who were deemed not on track to be college- or career-ready when graduating high school.</p> <p>Transition intervention programs are thought to improve college readiness through several channels. First, these programs typically use college placement tests to identify high school juniors for supplemental support ([<reflink idref="bib15" id="ref13">15</reflink>]). By using the same benchmarks that colleges use for DE placement, transition intervention programs can bridge the disconnect between high school and college expectations. Second, once students whose test scores qualified them for intervention, transition intervention programs often administer additional diagnostic assessments to identify specific areas of deficiency. Such information can be used to provide more personalized instruction and to monitor progress. By helping students meet college readiness criteria before college matriculation, transition intervention programs potentially help reduce the need for DE in college, which is considered a diversion that disrupts student progress through college ([<reflink idref="bib32" id="ref14">32</reflink>]).</p> <p>Transition interventions, however, have an opportunity cost. Because most programs do not provide additional school time for these interventions ([<reflink idref="bib15" id="ref15">15</reflink>]), they displace other opportunities for students to learn. As documented in the literature, transition interventions tend to crowd out advanced courses like pre-calculus and English IV that high school seniors typically take (Kane et al., 2021). Additionally, because most career technical education (CTE)-oriented high school students tend to score in a narrow range around college readiness benchmarks ([<reflink idref="bib24" id="ref16">24</reflink>]), transition interventions may also compete with career technical education. The tradeoff among these learning opportunities is unclear, and we empirically investigate this question at length in Section 5 of this study.</p> <hd id="AN0162243559-4">Targeted Transitional Interventions (TI) in Kentucky</hd> <p>In 2009, the Kentucky General Assembly ([<reflink idref="bib20" id="ref17">20</reflink>], p. 33) found "the continuing high rates of high school students who require remediation at the postsecondary education level totally unacceptable and an unwarranted additional expense to the state." As a result, the state passed legislation intended to better align secondary and college education, including establishing the diagnostic cycle that became TI. The postsecondary and secondary levels were charged to "develop a unified strategy to reduce college remediation rates by at least 50 percent by 2014... and increase the college completion rates of students enrolled in one or more remedial classes by three percent (3%) annually from 2009 to 2014." ([<reflink idref="bib20" id="ref18">20</reflink>], p. 33).</p> <p>The TI program was first implemented for high school math in 2010 to 2011 and high school English in 2011 to 2012. Using test scores from the ACT taken by all 11th grade students in Kentucky public schools, TI uses predetermined cutoffs (19 for math and 18 for English) to identify students who may not be college-ready at the end of high school and provides supplemental instruction to help those students meet college readiness expectations. By these standards, 59% of students were deemed not on track to be ready for college-level math and 43% were not on track to be ready for college English ([<reflink idref="bib36" id="ref19">36</reflink>]).</p> <p>Detailed student-level intervention data were not collected until the 2013 to 2014 cohort of 11th grade students. These data, supplemented with interviews with program administrators and teachers conducted by [<reflink idref="bib36" id="ref20">36</reflink>], provide information on the format of TI delivery. Each intervention cycle starts with a diagnostic pretest. Diagnostic results show that the areas in which students most frequently needed help include algebraic thinking, math reasoning, math computation, writing mechanics, and writing content. These results were used to inform the development of instructional targets and their associated formative assessments. Interventions were primarily delivered in the form of transition courses, which were developed by the Kentucky Department of Education, postsecondary institutions, or the Southern Regional Educational Board. Transition courses were designed to be flexible. As discussed in [<reflink idref="bib36" id="ref21">36</reflink>], these courses could be offered as full-semester courses or courses in which students could enter and exit flexibly based on diagnostic test scores (65%), or as interventions before or after school (22%). Teachers were responsible for designing instructional plans based on these course curricula, and they were provided with best practices guidance, sample problems, and additional resources (e.g., websites, online activities, or videos). In practice, intervention services were offered four times a week on average, with each session lasting 55 minutes. Most intervention teachers used a combination of self-developed materials and online curricula such as ALEX, Dreambox, Edgenuity, and IXL. Finally, at the end of each intervention cycle, a posttest was administered to determine the completion status of targeted students. Administrative records show that 57% of targeted students successfully completed interventions in math and 50% successfully completed interventions in English.</p> <hd id="AN0162243559-5">Existing Evidence on the Effectiveness of Transition Interventions</hd> <p>Prior research has found mixed results about the impact of transition programs, such as the one in Kentucky. In looking at Kentucky's model, [<reflink idref="bib36" id="ref22">36</reflink>] found that, consistent with the goals of the program, TI reduced the likelihood that Kentucky students would enroll in college DE by 8 to 10 percentage points in math in both 2- and 4-year institutions among students who just missed the college-ready benchmark in 11th grade, and that the cost of the program was estimated to be about $600 per student ([<reflink idref="bib21" id="ref23">21</reflink>]). In further exploration, the authors also found that TI increased the rate at which these students pass introductory college math within the first year of college by 4 percentage points in 4-year institutions. The positive effects of TI were even stronger among free/reduced-price lunch–eligible students, reducing enrollment in developmental math courses by 11 percentage points and increasing the rate of passing college math by 9 percentage points by the end of the first year in college.</p> <p>In addition to Kentucky, state-level transition intervention programs have been evaluated in West Virginia ([<reflink idref="bib27" id="ref24">27</reflink>]), Florida ([<reflink idref="bib25" id="ref25">25</reflink>]), and Tennessee ([<reflink idref="bib18" id="ref26">18</reflink>]). Findings in those states, all based on regression discontinuity (RD) designs, are less promising than in Kentucky. In all three states, transition interventions produced either no significant effect (Florida and Tennessee) or a negative effect (West Virginia) on students' likelihood of passing introductory college courses (a key college graduation requirement). In Tennessee, the transition intervention produced an initial effect of a 30% reduction in developmental math enrollment. But the effect was not the result of improved math skills, as [<reflink idref="bib18" id="ref27">18</reflink>] demonstrated using a post-intervention assessment, but due to automatic exemption from DE in college upon successful completion of a transition curriculum in high school.[<reflink idref="bib6" id="ref28">6</reflink>] In West Virginia and Florida, transition interventions had no detectable impact on DE enrollment.</p> <hd id="AN0162243559-6">Data, Measures, and Methods</hd> <p>In this section, we begin by describing the administrative data used in the paper, followed by the college outcomes used in the subsequent analyses, and finally detail the difference-in-regression discontinuity strategy used for the main results.</p> <hd id="AN0162243559-7">Data</hd> <p>Because Kentucky's TI program delivered positive early college outcomes ([<reflink idref="bib36" id="ref29">36</reflink>]), it is important to investigate whether these early impacts ultimately improved the chances of college completion and—for 2-year college students—successful transfer to a 4-year college. This study utilizes individual-level administrative records provided by the Kentucky Department of Education and Kentucky Council on Postsecondary Education to address these questions. We focus on seven cohorts of students who were high school juniors between the 2008 and 2009 and 2014 and 2015 school years. These cohorts span both pre- and post-treatment periods, with pre-treatment cohorts represented by the earliest two cohorts for math TI and the earliest three for English TI (because English TI was implemented 1 year before math). Each cohort consists of about 43,000 students who are observed annually through high school and college. With 11 years of panel data, this study tracks student progress through college until 3 years after initial enrollment in a 2-year college for all seven cohorts of high school juniors and 6 years after initial enrollment among 4-year college students for four cohorts of high school juniors (2008–2009 to 2011–2012).</p> <p>High school data include ACT scores from the mandatory statewide administration in the spring of the 11th grade, which were used by TI to refer students for supplemental services, as well as student gender, race/ethnicity, free/reduced-price lunch (FRL) eligibility, and high school completion status. Postsecondary data cover all enrollments in Kentucky institutions (both public and private).[<reflink idref="bib7" id="ref30">7</reflink>] These data include records on course enrollment, grades, credits attempted and earned, programs of study, transfers, and completion.</p> <p>As summarized in [<reflink idref="bib36" id="ref31">36</reflink>] and reproduced as Table 1, the study population is predominantly white (85%), with an FRL eligibility rate of close to 50%. The average ACT score is 18.8 in math and 18.5 in English. Over 90% of 11th grade students with a valid ACT score graduated high school on time. The rate of college enrollment immediately after high school graduation varied by ACT score, ranging from about 50% among students scoring three points below the TI cutoffs to about 65% among students scoring three points above the cutoffs.</p> <p>Graph</p> <p>Table 1. Descriptive Statistics of Study Samples: 2009 to 2016 (Xu et al., 2022).</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center">Full sample</th><th align="center">Math RD sample (ACT-M 16-21)</th><th align="center">English RD sample (ACT-E 15-20)</th></tr></thead><tbody><tr><td colspan="4"><italic>Demographics (%)</italic></td></tr><tr><td>Female</td><td>50</td><td>52</td><td>51</td></tr><tr><td>White</td><td>84</td><td>85</td><td>85</td></tr><tr><td>Black</td><td>10</td><td>10</td><td>10</td></tr><tr><td>Hispanic</td><td>5</td><td>5</td><td>5</td></tr><tr><td>FRL</td><td>48</td><td>50</td><td>51</td></tr><tr><td colspan="4"><italic>Test scores (mean and standard deviations)</italic></td></tr><tr><td>ACT Math</td><td>18.8 (4.5)</td><td>17.6 (1.6)</td><td>17.7 (2.8)</td></tr><tr><td>ACT Reading</td><td>19.4 (5.9)</td><td>18.7 (4.6)</td><td>18.1 (3.6)</td></tr><tr><td>ACT English</td><td>18.5 (6.3)</td><td>17.9 (4.6)</td><td>17.4 (1.8)</td></tr></tbody></table> </ephtml> </p> <hd id="AN0162243559-8">Measures</hd> <p>The key measure of success for this study is earning the minimum number of credits required to graduate from college. While we are able to track student transfers throughout the study period, reliable degree completion records are not available before 2014. As a result, we rely on credit accumulation as a proxy measure of college completion. Specifically, we calculate whether a student has accumulated at least 120 credits in a 4-year college or at least 60 credits in a 2-year college as an indicator of students' likely college completion status. These thresholds are based on a review of graduation requirements in academic catalogs published during the study period and represent the minimum thresholds below which degree completion is unlikely. Students typically need to satisfy additional requirements such as minimum GPA and the completion of core credits to graduate, and some Associate of Applied Science degree programs require up to 68 credits. Because about 80% of students who enroll in 2-year institutions intend to eventually earn a bachelor's degree, we also measure transfer from a 2-year to a 4-year institution as an outcome.[<reflink idref="bib8" id="ref32">8</reflink>]</p> <p>In order to examine how well credit accumulation thresholds approximate actual degree completion, we verify the relationship between credit accumulation and degree completion using available completion data from 2014 and later. As expected based on the known degree requirements discussed above, falling short of the threshold almost perfectly predicts incompletion. However, earning more credits than the thresholds does not always guarantee graduation. For example, 99% of 2-year college students who did not accumulate 60 credits 3 years after initial enrollment failed to attain any degree. On the other hand, 34% of students who earned at least 60 credits did not graduate or transfer to a 4-year institution.</p> <p>Measures of progress at critical junctures along the pathway through college can also help pinpoint where students may need more help. For example, research shows that many students start college not knowing what to study ([<reflink idref="bib9" id="ref33">9</reflink>]) and switch programs excessively in subsequent terms ([<reflink idref="bib17" id="ref34">17</reflink>]). In Kentucky 2-year colleges, for example, 27% of first-time freshman under the age of 24 enroll without a declared major; among those who have declared a major at the start, about 20% to 30% switch among six broadly defined programs of study (STEM, health, business, liberal arts/social sciences, other occupational fields, and certificate/diploma programs) ([<reflink idref="bib17" id="ref35">17</reflink>]). Some switches may be made as students gain more information about the field and about themselves. But there is an opportunity cost, and uncertainty about what field to pursue may explain why one third of 2-year college students in our data completed more credits than required without graduating or transferring to a 4-year college. Transition interventions like TI have the potential to improve the efficiency of program choice by exposing students to college-level content and expectations. We thus measure the likelihood of beginning college without a declared major and the number of times a student switches programs in their first year of enrollment.</p> <p>Empirical evidence also suggests that carrying an adequate course load early on is a strong predictor of college completion and successful transfer from a 2-year institution to a 4-year college (e.g., [<reflink idref="bib2" id="ref36">2</reflink>]). National and college administrative data both suggest that a 15-credit course load during the first semester is a critical threshold with the strongest predictive power of college attainment ([<reflink idref="bib3" id="ref37">3</reflink>]; [<reflink idref="bib8" id="ref38">8</reflink>]). Students attempting at least 15 credits during the first semester are 9 percentage points more likely to graduate ([<reflink idref="bib3" id="ref39">3</reflink>]). TI significantly reduced the need for DE in college, and we investigate whether this led to an increase in overall course load by measuring the impact of TI on the likelihood that students take at least 15 credits during the first term in college.</p> <hd id="AN0162243559-9">Research Methods</hd> <p>The use of predefined ACT cut scores for TI assignment lends itself to a regression discontinuity (RD) design. However, the use of these cut scores in an RD framework is complicated by these same cut scores being used at the postsecondary level for assignment to developmental courses upon arrival to college. Therefore, following [<reflink idref="bib36" id="ref40">36</reflink>], we use a difference-in-regression discontinuity (DiRD) method to estimate the effect of TI. The crucial assumption of the DiRD strategy is that college placement policies or any other factors which affect the outcomes of students just above and below the cutoffs remained unchanged over time for the 11th grade cohorts under consideration. Under this assumption, DiRD uses pre-TI cohorts to net out the college placement policy effect in order to isolate the TI effect. Specifically, we estimate the RD effect of falling just below the ACT cutoff for the pre- and post-TI periods using</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mi>i</mi></msub><mo>=</mo><msub><mi>α</mi><mn>0</mn></msub><mo>+</mo><msub><mi>α</mi><mn>1</mn></msub><msub><mi>B</mi><mi>i</mi></msub><mo>+</mo><mi>k</mi><mrow><mo>(</mo><mrow><msub><mi>S</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow><mo>+</mo><msub><mi>α</mi><mn>2</mn></msub><msub><mi>B</mi><mi>i</mi></msub><mo>*</mo><mi>k</mi><mrow><mo>(</mo><mrow><msub><mi>S</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow><mo>+</mo><msub><mi>ε</mi><mi>i</mi></msub><mo>,</mo></mrow></math> </ephtml> </p> <p>Graph</p> <p>where <emph>Y<subs>i</subs></emph> is the outcome for student <emph>i</emph> and <emph>k(.)</emph> is a function of the ACT score of student <emph>i</emph>, <emph>S<subs>i</subs></emph>, that is centered around the cutoff (19 for math and 18 for English) such that negative values indicate scores below cutoff. <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>B</mi><mi>i</mi></msub></mrow></math> </ephtml> is an indicator for whether student <emph>i</emph>'s score falls below the cutoff. The DiRD estimate is then the difference in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>α</mi><mn>1</mn></msub></mrow></math> </ephtml> between the post period and the pre period:</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>β</mi><mo stretchy="true">^</mo></mover><mo>=</mo><msup><mrow><msub><mrow><mover accent="true"><mi>α</mi><mo stretchy="true">^</mo></mover></mrow><mn>1</mn></msub></mrow><mrow><mi>p</mi><mi>o</mi><mi>s</mi><mi>t</mi></mrow></msup><mo>−</mo><msup><mrow><mrow><msub><mrow><mover accent="true"><mi>α</mi><mo stretchy="true">^</mo></mover></mrow><mn>1</mn></msub></mrow></mrow><mrow><mi>p</mi><mi>r</mi><mi>e</mi></mrow></msup></mrow></math> </ephtml> </p> <p>Graph</p> <p>As shown in Figure 1, the discontinuity of TI participation is not sharp near the cutoff. A large percentage of students who scored below the ACT cutoffs did not participate in TI in a given subject ("no shows"), and many students who scored above the cutoffs did participate ("crossovers").[<reflink idref="bib9" id="ref41">9</reflink>] The estimated discontinuity in program participation around the cutoff is about 40 percentage points in both subjects. Therefore, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>β</mi><mo>̂</mo></mover></mrow></math> </ephtml> estimates the intent-to-treat (ITT) effect of TI on student college outcomes. Under the assumption that the introduction of TI was the only policy change that relied on these specific ACT cut points during the study period, DiRD produces an unbiased estimate of the TI effect on student outcomes.</p> <p>Graph: Figure 1. Targeted intervention (TI) participation rate by ACT score, by subject. Note. Based on ACT math and English tests taken by high school juniors in 2014, 2015, and 2016. Test scores are centered around 19 for math and 18 for English.</p> <p>Our data include integer values of ACT scores and should thus be viewed as discrete rather than continuous. Due to this discreteness in the running variable—each point of difference on the ACT is equivalent to 0.17 to 0.20 standard deviations—we are unable to get arbitrarily close to the cutoff and have to rely on parametric assumptions about the relationship between the outcome and the ACT score near the cutoff, incurring additional uncertainty about impact estimates. As a result, we report Eicker–Huber–White (EHW) standard errors for impact estimates following [<reflink idref="bib23" id="ref42">23</reflink>] and [<reflink idref="bib19" id="ref43">19</reflink>]. In addition, DiRD mitigates the uncertainty associated with parametric assumptions to the extent that the parametric relationship between ACT test scores and student outcomes remains unchanged before and after TI took effect.</p> <p>Using the bandwidth selection procedure implemented by [<reflink idref="bib12" id="ref44">12</reflink>], we obtain optimal bandwidths for each college outcome, subject, and institution type. In line with prior RD studies that use ACT scores as a running variable (e.g., [<reflink idref="bib10" id="ref45">10</reflink>]), optimal bandwidths range mostly between 2 and 5 points. We also investigate the robustness of key findings to different bandwidths and find that the choice of bandwidth is generally not substantively important. Finally, given the small bandwidths, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>k</mi><mrow><mo>(</mo><mrow><msub><mi>S</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow></mrow></math> </ephtml> is entered into Equation 1 as a linear function of test scores.</p> <hd id="AN0162243559-10">Validity</hd> <p>In addition to the usual assumptions of an RD—the existence of significant discontinuity in treatment receipt at the cutoff, the integrity of the running variable, no differential sample attrition at the cutoff—DiRD also assumes that the discontinuity in student outcomes at the cutoff due to college placement policies would have remained constant throughout the study period in the absence of TI. As demonstrated in [<reflink idref="bib36" id="ref46">36</reflink>], which uses the same identification approach we rely on here, there is no evidence that any of these assumptions are violated. The discontinuity in TI participation around ACT cutoffs is about 40 percentage points for both subjects. Manipulation of ACT scores is unlikely since teachers and schools were not involved in scoring and cutoffs were predetermined. We also use scores from the spring of the 11th grade, when students took the tests for the first time, to avoid potential self-selection issues stemming from test retakes. As expected, the distribution of ACT scores shows no signs of lumpiness through the cutoff.</p> <p>Consistent with the integrity of the running variable, key baseline covariates are continuous at the cutoff. Using student gender, race/ethnicity, and FRL eligibility as the outcome, [<reflink idref="bib36" id="ref47">36</reflink>] estimated the DiRD model as described above and found no detectable discontinuity at the cutoff except for the percentage of female students, where the difference is equivalent to about 0.10 standard deviation. Controlling for gender (as well as other covariates) in the DiRD model produces no meaningful change in estimated TI effects on college outcomes. These results are reproduced as Table 2 here, where the estimated discontinuity at the cutoff is reported for each covariate (rows) by subject and sample (columns).</p> <p>Graph</p> <p>Table 2. Covariate Balance Check (Reproduced From Xu et al., 2022).</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center" colspan="4">Math</th><th align="center" colspan="4">English</th></tr><tr><th /><th align="center">All</th><th align="center">College</th><th align="center">2 year</th><th align="center">4 year</th><th align="center">All</th><th align="center">College</th><th align="center">2 year</th><th align="center">4 year</th></tr></thead><tbody><tr><td>Female</td><td>−0.04<xref ref-type="table-fn" rid="tfn1">***</xref>(0.02)</td><td>−0.05<xref ref-type="table-fn" rid="tfn1">***</xref>(0.02)</td><td>−0.07<xref ref-type="table-fn" rid="tfn1">**</xref>(0.03)</td><td>−0.04<xref ref-type="table-fn" rid="tfn1">*</xref>(0.03)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.04(0.03)</td><td>−0.02(0.03)</td></tr><tr><td>White</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>−0.02(0.02)</td><td>0.03<xref ref-type="table-fn" rid="tfn1">*</xref>(0.02)</td><td>0.00(0.01)</td><td>−0.00(0.02)</td><td>0.01(0.02)</td><td>−0.01(0.02)</td></tr><tr><td>Black</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>0.01(0.01)</td><td>−0.02(0.02)</td><td>−0.01(0.01)</td><td>−0.02(0.01)</td><td>−0.02(0.01)</td><td>−0.01(0.02)</td></tr><tr><td>Hispanic</td><td>0.00(0.01)</td><td>−0.00(0.01)</td><td>0.00(0.01)</td><td>−0.01(0.01)</td><td>−0.01<xref ref-type="table-fn" rid="tfn1">*</xref>(0.01)</td><td>−0.01(0.01)</td><td>−0.02(0.01)</td><td>0.00(0.01)</td></tr><tr><td>FRL</td><td>−0.01(0.02)</td><td>−0.01(0.02)</td><td>−0.01(0.03)</td><td>0.00(0.02)</td><td>−0.01(0.02)</td><td>−0.02(0.02)</td><td>−0.04(0.03)</td><td>−0.00(0.03)</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes with a bandwidth of three. Results are robust to other bandwidth choices. "All" column contains sample of all cohorts with ACT scores, "College" a sample of students who enrolled in college the year after graduating high school, "2 year" a sample of students who enrolled in a 2-year institution the year after graduating high school, and "4 year" a sample of students who enrolled in a 4-year institution the year after graduating high school.</p> <p>As a further test for the DiRD assumption that the college placement policy effect would have remained constant in the absence of TI, [<reflink idref="bib36" id="ref48">36</reflink>] conducted a falsification test by estimating TI "effects" away from the actual cutoff. Like the math cutoff of 19, a cutoff of 22 has also been used by colleges in Kentucky to place students in college math courses (albeit at a higher level). Unlike the cutoff of 19, the cutoff of 22 was unrelated to TI assignment. In the absence of TI, discontinuities in student outcomes around the cutoff of 22—due to college placement policies—remain constant before and after the implementation of TI, and the estimated DiRD "effects" are null.[<reflink idref="bib10" id="ref49">10</reflink>]</p> <hd id="AN0162243559-11">Findings</hd> <p>The estimated TI effects are presented in Table 3. The rows of the table represent different college outcomes, with Panel A consisting of early college outcomes and Panel B end-of-college outcomes. The estimated TI impact on each college outcome is reported by subject and college type as columns. For each unique regression, we show the estimated TI impact, its associated EHW standard error, and the control group mean. Full details on the optimal bandwidth and sample size for each regression can be found in AppendixTable A1.</p> <p>Graph</p> <p>Table 3. Estimated Effect of Targeted Interventions (TI) on College Outcomes.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" rowspan="3">Outcome</th><th align="center" colspan="2">Math</th><th align="center" colspan="2">English</th></tr><tr><th align="center">2 year</th><th align="center">4 year</th><th align="center">2 year</th><th align="center">4 year</th></tr><tr><th align="center">Coefficient(<italic>S.E.</italic>) [Control mean]</th><th align="center">Coefficient(<italic>S.E.</italic>) [Control mean]</th><th align="center">Coefficient(<italic>S.E.</italic>) [Control mean]</th><th align="center">Coefficient(<italic>S.E.</italic>) [control mean]</th></tr></thead><tbody><tr><td colspan="5"><italic>Panel A: Early college outcomes</italic></td></tr><tr><td>Started out in a non-degree program</td><td>−0.07<xref ref-type="table-fn" rid="tfn2">**</xref>(0.03) [0.10]</td><td /><td>0.01(0.02) [0.12]</td><td /></tr><tr><td>Started with undeclared program</td><td>0.02(0.04) [0.15]</td><td>−0.03<xref ref-type="table-fn" rid="tfn2">*</xref>(0.02) [0.04]</td><td>−0.00(0.02) [0.13]</td><td>0.00(0.02) [0.04]</td></tr><tr><td>Started in STEM</td><td>0.02<xref ref-type="table-fn" rid="tfn2">*</xref>(0.01) [0.06]</td><td>0.01(0.01) [0.09]</td><td>−0.00(0.01) [0.06]</td><td>0.03<xref ref-type="table-fn" rid="tfn2">**</xref>(0.02) [0.10]</td></tr><tr><td>Started in Health</td><td>−0.03<xref ref-type="table-fn" rid="tfn2">*</xref>(0.02) [0.07]</td><td>0.01(0.01) [0.06]</td><td>0.03<xref ref-type="table-fn" rid="tfn2">*</xref>(0.01) [0.08]</td><td>0.00(0.01) [0.06]</td></tr><tr><td>Program switch frequency within 1 year</td><td>0.01(0.01) [0.06]</td><td>−0.00(0.01) [0.07]</td><td>−0.00(0.01) [0.06]</td><td>−0.01(0.01) [0.08]</td></tr><tr><td>Program switch frequency within 3 years</td><td>0.00(0.04) [0.23]</td><td>−0.02(0.04) [0.37]</td><td>−0.00(0.03) [0.23]</td><td>0.03(0.04) [0.38]</td></tr><tr><td>Attempted ≥15 credits in first term</td><td>0.06<xref ref-type="table-fn" rid="tfn2">***</xref>(0.02) [0.18]</td><td>0.05<xref ref-type="table-fn" rid="tfn2">**</xref>(0.02) [0.52]</td><td>0.02(0.01) [0.14]</td><td>0.04<xref ref-type="table-fn" rid="tfn2">*</xref>(0.02) [0.47]</td></tr><tr><td colspan="5"><italic>Panel B: End-of-college outcomes</italic></td></tr><tr><td>Credits earned at the end of 3 years</td><td>1.08(1.81) [37.3]</td><td>2.33(1.75) [46.2]</td><td>−0.36(1.20) [32.1]</td><td>−1.37(1.62) [40.0]</td></tr><tr><td>Earned ≥ 60 credits by year 3</td><td>−0.00(0.02) [0.28]</td><td>0.03(0.02) [0.42]</td><td>0.01(0.02) [0.22]</td><td>−0.01(0.02) [0.35]</td></tr><tr><td>Transfer within 3 years</td><td>0.02(0.02) [0.14]</td><td /><td>0.02(0.01) [0.10]</td><td /></tr><tr><td>Credits earned at the end of 6 years</td><td /><td>2.34(2.73) [67.3]</td><td /><td>0.17(2.24) [57.9]</td></tr><tr><td>Earned ≥ 120 credits by year 6</td><td /><td>0.04(0.02) [0.26]</td><td /><td>−0.02(0.02) [0.22]</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Control group means in brackets. Difference-in-RD estimates assume linear association between the running variable and outcomes. More details about optimal bandwidth and sample sizes for each outcome and study sample can be found in AppendixTable A1.</p> <hd id="AN0162243559-12">Early College Outcomes</hd> <p>As noted in Section 3.2 above, there are reasons to believe that TI might affect various early college pursuits and progress toward a degree. Estimates reported in Table 3 suggest that TI has marginal effects on some types of program choices. These effects are not consistent across subject or institution type, and some are sensitive to bandwidth choices. Findings consistent across bandwidth choices (Table 4) are that TI in math reduces the likelihood that 2-year college students start in a health field by 3 percentage points, and that TI in English increases the likelihood that 4-year college students start in a STEM field by 3 percentage points. TI in either subject has no effect on the frequency of program switches within the first year or first 3 years in college.</p> <p>Graph</p> <p>Table 4. Robustness of Results to Alternate Bandwidths.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center" colspan="6">Math</th><th align="center" colspan="6">English</th></tr><tr><th /><th align="center" colspan="3">2 year</th><th align="center" colspan="3">4 year</th><th align="center" colspan="3">2 year</th><th align="center" colspan="3">4 year</th></tr><tr><th /><th align="center">BW</th><th align="center">BW + 1</th><th align="center">BW + 2</th><th align="center">BW</th><th align="center">BW + 1</th><th align="center">BW + 2</th><th align="center">BW</th><th align="center">BW + 1</th><th align="center">BW + 2</th><th align="center">BW</th><th align="center">BW + 1</th><th align="center">BW + 2</th></tr></thead><tbody><tr><td colspan="13"><italic>Attempted ≥15 credits in first term</italic></td></tr><tr><td>Disc. pre</td><td>−0.07<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.07<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.07<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>−0.04<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>−0.05<xref ref-type="table-fn" rid="tfn3">***</xref>(0.02)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.04<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.02(0.02)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td></tr><tr><td>Disc. post</td><td>−0.01<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>−0.02<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>−0.02<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>0.00(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>0.02(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td></tr><tr><td>Difference in RD</td><td>0.06<xref ref-type="table-fn" rid="tfn3">***</xref>(0.02)</td><td>0.05<xref ref-type="table-fn" rid="tfn3">***</xref>(0.02)</td><td>0.04<xref ref-type="table-fn" rid="tfn3">***</xref>(0.02)</td><td>0.05<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>0.05<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>0.05<xref ref-type="table-fn" rid="tfn3">***</xref>(0.02)</td><td>0.02(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">**</xref>(0.01)</td><td>0.04<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>0.04<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>0.04<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td></tr><tr><td colspan="13"><italic>Earned ≥ 60 credits by year 3</italic></td></tr><tr><td>Disc. pre</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>−0.03(0.02)</td><td>−0.02(0.02)</td><td>−0.02(0.02)</td><td>−0.02(0.02)</td><td>−0.02(0.01)</td><td>−0.02(0.01)</td><td>0.00(0.02)</td><td>−0.01(0.02)</td><td>−0.01(0.02)</td></tr><tr><td>Disc. post</td><td>0.01<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>−0.00(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.00(0.01)</td><td>0.01(0.01)</td><td>0.00(0.01)</td></tr><tr><td>Difference in RD</td><td>−0.00(0.02)</td><td>−0.00(0.02)</td><td>−0.00(0.02)</td><td>0.03(0.02)</td><td>0.02(0.02)</td><td>0.03(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>−0.01(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td></tr><tr><td colspan="13"><italic>Started with undeclared program</italic></td></tr><tr><td>Disc. pre</td><td>−0.03(0.03)</td><td>−0.02(0.03)</td><td>−0.03(0.03)</td><td>0.02(0.02)</td><td>0.02(0.02)</td><td>0.02(0.01)</td><td>−0.00(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.01(0.01)</td><td>0.02(0.01)</td></tr><tr><td>Disc. post</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.00(0.00)</td><td>−0.00(0.00)</td><td>−0.01(0.01)</td><td>−0.00(0.01)</td><td>−0.01(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td></tr><tr><td>Difference in RD</td><td>0.02(0.04)</td><td>0.01(0.03)</td><td>0.02(0.03)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>−0.02(0.02)</td><td>−0.02(0.02)</td><td>−0.00(0.02)</td><td>−0.01(0.02)</td><td>−0.01(0.02)</td><td>0.00(0.02)</td><td>−0.00(0.02)</td><td>−0.01(0.01)</td></tr><tr><td colspan="13"><italic>Start with STEM</italic></td></tr><tr><td>Disc. pre</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>−0.00(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td></tr><tr><td>Disc. post</td><td>0.01<xref ref-type="table-fn" rid="tfn3">**</xref>(0.00)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">***</xref>(0.00)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">***</xref>(0.00)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">**</xref>(0.01)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">***</xref>(0.00)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">***</xref>(0.00)</td><td>−0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">**</xref>(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td></tr><tr><td>Difference in RD</td><td>0.02<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.02(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>−0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">**</xref>(0.01)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">***</xref>(0.01)</td></tr><tr><td colspan="13"><italic>Start with health</italic></td></tr><tr><td>Disc. pre</td><td>0.03(0.02)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>0.02(0.02)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.02(0.01)</td><td>−0.01(0.01)</td><td>0.00(0.01)</td><td>−0.00(0.01)</td><td>−0.01(0.01)</td><td>−0.01<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td></tr><tr><td>Disc. post</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td>−0.00(0.01)</td><td>0.00(0.00)</td><td>0.00(0.00)</td><td>0.00(0.00)</td><td>0.01(0.01)</td><td>0.01<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.01(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td></tr><tr><td>Difference in RD</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>−0.04<xref ref-type="table-fn" rid="tfn3">**</xref>(0.02)</td><td>−0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.02)</td><td>0.01(0.01)</td><td>0.01(0.01)</td><td>0.02(0.01)</td><td>0.03<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td><td>0.02(0.01)</td><td>0.00(0.01)</td><td>0.00(0.01)</td><td>0.01(0.01)</td><td>0.02<xref ref-type="table-fn" rid="tfn3">*</xref>(0.01)</td></tr><tr><td colspan="13"><italic>Started out in a non-degree program</italic></td></tr><tr><td>Disc. pre</td><td>0.06<xref ref-type="table-fn" rid="tfn3">**</xref>(0.03)</td><td>0.03(0.02)</td><td>0.02(0.02)</td><td /><td /><td /><td>−0.01(0.02)</td><td>−0.01(0.02)</td><td>−0.01(0.01)</td><td /><td /><td /></tr><tr><td>Disc. post</td><td>−0.00(0.01)</td><td>−0.01(0.01)</td><td>−0.01(0.01)</td><td /><td /><td /><td>−0.00(0.01)</td><td>−0.00(0.01)</td><td>−0.01(0.01)</td><td /><td /><td /></tr><tr><td>Difference in RD</td><td>−0.07<xref ref-type="table-fn" rid="tfn3">**</xref>(0.03)</td><td>−0.04(0.02)</td><td>−0.03(0.02)</td><td /><td /><td /><td>0.01(0.02)</td><td>0.01(0.02)</td><td>0.00(0.02)</td><td /><td /><td /></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth selection. <emph>N</emph> is the number of students. BW + 1 and BW + 2 columns display results from wider choses of bandwidth. "Discontinuity pre" represents the estimated discontinuity in the pre period, "Discontinuity post" the different in the post period, and "Difference in RD" the difference between the two, which are the results displayed in other tables.</p> <p>During the first term, TI in math increases the likelihood of students enrolling in at least 15 credits by 6 percentage points in 2-year colleges and 5 percentage points in 4-year colleges (Table 3). This is consistent with the reduced need for DE reported in [<reflink idref="bib36" id="ref50">36</reflink>], suggesting that students used the freed-up time early in their college journeys to take more college-level courses. Figure 2, which plots select college outcomes against ACT scores for the pre-treatment (circles) and post-treatment (triangles) periods, shows that the positive impact is achieved by eliminating the gap between students scoring just above and below the ACT cutoff: Before the implementation of TI, due to the same TI cutoff also being used to place students into DE in college, students scoring just below the ACT cutoff were significantly less likely than students scoring just above the cutoff to enroll in at least 15 credits during the first term (estimates are presented in Table 3).</p> <p>Graph: Figure 2. College outcomes by ACT test scores, by subject, and institution type.</p> <p>In the post-implementation period, by contrast, the gap in first-term course load was mostly eliminated. Results from Table 2 also show that this finding is robust to alternate bandwidths. TI in English has weaker effects on first-term course load, increasing the likelihood of students attempting at least 15 credits by 4 percentage points in 4-year colleges but having no significant impact in 2-year colleges.[<reflink idref="bib11" id="ref51">11</reflink>]</p> <hd id="AN0162243559-13">Long Term Credit Accumulation and Transfer</hd> <p>To assess whether the above impacts of TI on early college choices and progression led to changes in longer term college outcomes, we examine total credits earned and the likelihood of earning at least 60 credits at the end of 3 years in college or at least 120 credits after 6 years in a 4-year college (Table 3, Panel B). TI has no detectable effect on any of these measures. Among 4-year college students, those who were subject to math TI appear to have completed close to one additional course (2.3 credits) more than students who were not subject to math TI by the end of 3 or 6 years in college, but these are imprecisely estimated. Results in Table 3 suggest that TI has no measurable impact on helping students meet the minimum credit requirement for graduation. TI also has no detectable effect on the likelihood that 2-year college students transfer to a 4-year institution.</p> <p>Figure 3 depicts the trajectory of credit accumulation by term. Cumulative credits are measured at the end of each fall, spring, and summer term for the first eight terms. Trajectories are plotted for students scoring within three points above and below the cutoff for cohorts before and after the implementation of TI separately. Gaps in credit accumulation between higher and lower scoring students emerge at the end of the first term. The higher scoring group also appears to earn credits at a faster rate, resulting in a steadily widening gap in completed credits over the course of 3 years. Notably, the gap between the higher and lower scoring groups at each point in time appears to remain unchanged before and after TI implementation. In other words, the lack of TI effect on accumulated credits at the end of college is not because early gains dissipated later. Rather, positive TI effects on early progress did not result in discernible gains in college progression early on.</p> <p>Graph: Figure 3. Cumulative credits earned by term.</p> <p>Finally, in Table 4, we show the sensitivity of the estimates provided in Table 3 to different bandwidths and present separate pre- and post-intervention RD estimates of the effect of failing ACT cutoffs.</p> <hd id="AN0162243559-14">Subgroup Analysis</hd> <p>We investigate whether TI has similar effects on college outcomes across three student subgroups that are thought to face unique challenges in college. FRL-eligible students, for instance, could feel more stigmatized by being labeled as not ready for college ([<reflink idref="bib25" id="ref52">25</reflink>]). Students who missed ACT cutoffs in more than one subject and therefore were subject to intervention in multiple subjects[<reflink idref="bib12" id="ref53">12</reflink>] could be overwhelmed by the burden of interventions, which in turn could undermine the effectiveness of TI. Finally, high schools with a high percentage of students who need transitional intervention[<reflink idref="bib13" id="ref54">13</reflink>] may be overburdened by the prevalent need for intervention; students enrolled in these high schools may receive interventions that are less effective.</p> <p>Estimated DiRD effects of TI on college outcomes largely mirror findings among the overall student population.[<reflink idref="bib14" id="ref55">14</reflink>] Math TI improves the likelihood of students taking a 15-credit course load in the first term by 6 to 13 percentage points in 2-year colleges for all student subgroups. English TI has a similar effect on first-term course load among FRL-eligible students and on students from schools with a high percentage of students who need TI who later enrolled in 4-year colleges. However, math TI's effect on early momentum is no longer significant among these student groups in 4-year colleges, which is partially due to smaller sample sizes and the loss of statistical power. TI also appears to reduce the likelihood that 2-year college students start with short-term diploma or certificate programs by 9 percentage points for FRL-eligible students and 14 percentage points for students from high-need schools. We find no detectable TI effect on any other college outcomes.</p> <hd id="AN0162243559-15">Further Exploration</hd> <p>The direct goal of TI and other types of transition interventions is to reduce the need for DE in college. [<reflink idref="bib36" id="ref56">36</reflink>] find evidence that TI reduced developmental course taking, and we show here that TI also affects other early college outcomes, including first-semester course load. However, we do not find evidence that these initial gains put students on an accelerated trajectory toward long-term credit accumulation or successful transfer from 2- to 4-year colleges. In this section, we explore two reasons that might explain the lack of statistical significance for long-term college outcomes despite evidence that TI decreases the likelihood that students are deemed to need development (non-credit-bearing) courses in college.</p> <p>The first potential explanation is that intervention activities were no more beneficial for long-run college success than the regular high school courses they crowded out. Under this hypothesis, TI is not expected to improve student skills associated with college progression, and the only channel through which TI would affect long-run college outcomes would be by moving some students out of developmental courses that were demonstrated to have significant negative impact on college outcomes ([<reflink idref="bib35" id="ref57">35</reflink>]). Tabulations using transcript data show that the most frequently taken math courses by students not subject to TI were Algebra II, Algebra III, and College Algebra. Taking Algebra II or higher is considered a critical precollegiate milestone for eventually earning a college degree ([<reflink idref="bib2" id="ref58">2</reflink>]). By contrast, student-level intervention data show that math TI most frequently focused on <emph>algebraic thinking</emph> and <emph>math reasoning</emph>. Although the definition of these content areas is unknown, similar terms were also used in course descriptions for developmental math courses in college.[<reflink idref="bib15" id="ref59">15</reflink>] This course substitution—in contrast to, for example, increasing total learning time—may impose an upper limit on the plausible long-run college impacts of TI. We explore this mechanism in three different ways: by measuring crowd-out of high school courses directly (Section 5.1), by examining college outcomes for a subset of successful TI completers (Section 5.2), and by examining a similar intervention earlier in secondary school (Section 5.3).</p> <p>The second explanation is that given the effect size on early college outcomes and the strength of the relationship between early college outcomes and later college outcomes estimated in previous studies, the expected long-run effects of TI are quite small. Thus, we do not have the power to detect the expected long-run effects of TI. We provide this discussion in Section 5.4.</p> <hd id="AN0162243559-16">Crowd-Out of Other High School Courses</hd> <p>We begin by investigating the extent to which TI may have crowded out regular high school courses. Although some TI students received interventions as extended school services, most interventions were delivered as transition courses during the normal school day. Time students spend on transition interventions during the normal school day implies less time spent on what students otherwise would have been doing.</p> <p>To investigate how course-taking changed after the introduction of TI for students below the cutoff relative to above the cutoff, we divide courses into four categories based on coding conventions stipulated by the Kentucky Uniform Academic Course Codes: transition courses in math or English, regular non-transition courses in math or English, career technical education (CTE) courses, and other non-transition courses.[<reflink idref="bib16" id="ref60">16</reflink>] Results are presented in Table 5 in the same format as in Tables 3 and 5, with each coefficient representing the DiRD estimate on a given outcome for a given subject. As expected, we see that the introduction of TI led students below college readiness benchmarks to take more transition math courses by 12 percentage points and more transition English courses by 2 percentage points. For math TI, we also see evidence that the increase in transition math course-taking largely came at the expense of regular math courses (an 8-percentage-point drop in enrollment) rather than CTE courses or other courses, although the latter is very imprecisely estimated. For English, we see smaller reallocation effects.</p> <p>Graph</p> <p>Table 5. Effect of Targeted Interventions (TI) on 12th Grade Course-Taking, by Intervention Subject.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center" colspan="2">Math</th><th align="center" colspan="2">English</th></tr><tr><th /><th align="center">Coefficient/<italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic>Control mean</th><th align="center">Coefficient/<italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic>Control mean</th></tr></thead><tbody><tr><td colspan="5"><italic>Within-subject substitution</italic></td></tr><tr><td>Transition courses in subject</td><td>0.12<xref ref-type="table-fn" rid="tfn4">***</xref>(0.02)</td><td>2/93,2430.20</td><td>0.02<xref ref-type="table-fn" rid="tfn4">***</xref>(0.01)</td><td>3/114,4100.07</td></tr><tr><td>Non-transition courses in subject</td><td>−0.08<xref ref-type="table-fn" rid="tfn4">***</xref>(0.03)</td><td>2/93,2430.71</td><td>−0.01(0.03)</td><td>4/157,3821.08</td></tr><tr><td colspan="5"><italic>Other course substitution</italic></td></tr><tr><td>CTE courses taken</td><td>−0.02(0.05)</td><td>3/168,2621.84</td><td>0.02(0.07)</td><td>3/114,4101.94</td></tr><tr><td>Other courses taken</td><td>−0.06(0.12)</td><td>2/93,2435.78</td><td>−0.02(0.12)</td><td>3/114,4105.65</td></tr></tbody></table> </ephtml> </p> <p>4 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth selection. <emph>N</emph> is the number of students.</p> <hd id="AN0162243559-17">Substitution of Curriculum</hd> <p>As another test of whether TI content was geared more toward the skills needed to avoid DE in college than toward the more advanced skills needed for later progression through college, we examine the relationship between successful TI completion and later college outcomes. In particular, we estimate the difference in college outcome <emph>Y</emph> between successful completers of TI (<emph>S_TI</emph>) and students who failed to complete TI (<emph>F_TI</emph>) relative to students who did not receive any intervention, the reference group, in the following OLS regression:</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>Y</mi><mi>i</mi></msub><mo>=</mo><msub><mi>β</mi><mn>0</mn></msub><mo>+</mo><msub><mi>β</mi><mn>1</mn></msub><mi>S</mi><mo>_</mo><mi>T</mi><msub><mi>I</mi><mi>i</mi></msub><mo>+</mo><msub><mi>β</mi><mn>2</mn></msub><mi>F</mi><mo>_</mo><mi>T</mi><msub><mi>I</mi><mi>i</mi></msub><mo>+</mo><mi>C</mi><mi>O</mi><mi>V</mi><mo>*</mo><mi>Θ</mi><mo>+</mo><msub><mi>S</mi><mi>s</mi></msub><mo>+</mo><msub><mi>u</mi><mi>i</mi></msub><mo>.</mo></mrow></math> </ephtml> </p> <p>Graph</p> <p>Equation 3 also controls for ACT score fixed effects <emph>S</emph> and a vector of covariates <emph>COV</emph> that include student race/ethnicity, gender, FRL eligibility, and whether a student was eligible for TI in more than one subject.</p> <p>Estimates reported in Table 6 show that among students with the same ACT score and similar background characteristics, successful completion of TI is associated with lower likelihood of DE enrollment, but not with other measures of progress later on (i.e., the likelihood of passing credit-bearing college courses during the first year, taking at least 15 credits in the first term, or completing at least 60 credits by the end of 3 years). Unsuccessful exit from TI, on the other hand, is associated with higher likelihood of DE enrollment and weaker college outcomes. These findings are not causal due to selection into TI and into completion of TI, but taken together, they suggest that the standards used by high schools to judge student progress toward college readiness are largely consistent with the skills needed to place out of developmental courses, but not sufficient to better prepare students for college-level instruction.</p> <p>Graph</p> <p>Table 6. Correlation Between College Outcomes and Intervention Completion Status.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="center">College type and subject</th><th align="center">Take developmental course</th><th align="center">Pass credit-bearing course in subject</th><th align="center">15 + credits in first term</th><th align="center">At least 60 credits in 3 years</th></tr></thead><tbody><tr><td colspan="5"><italic>2-year college, math</italic></td></tr><tr><td>No successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub></mrow></math></p>)</td><td>.083<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.038<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.023<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.021<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td></tr><tr><td>Successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>−.087<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>.01 (.01)</td><td>.01 (.01)</td><td>−.016<xref ref-type="table-fn" rid="tfn5">*</xref> (.01)</td></tr><tr><td><italic>p</italic>-Value (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub><mo>=</mo><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>.000</td><td>.000</td><td>.002</td><td>.541</td></tr><tr><td>Observations</td><td>11,292</td><td>11,292</td><td>11,292</td><td>11,292</td></tr><tr><td>R-squared</td><td>.054</td><td>.088</td><td>.038</td><td>.066</td></tr><tr><td colspan="5"><italic>4-year college, math</italic></td></tr><tr><td>No successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub></mrow></math></p>)</td><td>.083<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.071<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.053<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.060<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td></tr><tr><td>Successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>−.015 (.01)</td><td>−.051<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>.024<xref ref-type="table-fn" rid="tfn5">*</xref> (.01)</td><td>−.030<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td></tr><tr><td><italic>p</italic>-value (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub><mo>=</mo><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>.000</td><td>.098</td><td>.000</td><td>.019</td></tr><tr><td>Observations</td><td>9,296</td><td>9,296</td><td>9,296</td><td>9,296</td></tr><tr><td>R-squared</td><td>.032</td><td>.075</td><td>.041</td><td>.089</td></tr><tr><td colspan="5"><italic>2-year college, English</italic></td></tr><tr><td>No successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub></mrow></math></p>)</td><td>.041<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>−.007 (.01)</td><td>−.017<xref ref-type="table-fn" rid="tfn5">**</xref> (.01)</td><td>−.007 (.01)</td></tr><tr><td>Successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>−.082<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>.02 (.02)</td><td>.01 (.01)</td><td>−.009 (.01)</td></tr><tr><td><italic>p</italic>-value (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub><mo>=</mo><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>.000</td><td>.072</td><td>.051</td><td>.856</td></tr><tr><td>Observations</td><td>7,694</td><td>7,694</td><td>7,694</td><td>7,694</td></tr><tr><td>R-squared</td><td>.081</td><td>.091</td><td>.044</td><td>.057</td></tr><tr><td colspan="5"><italic>4-year college, English</italic></td></tr><tr><td>No successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub></mrow></math></p>)</td><td>.038<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>.030<xref ref-type="table-fn" rid="tfn5">*</xref> (.02)</td><td>−.064<xref ref-type="table-fn" rid="tfn5">***</xref> (.02)</td><td>−.023 (.02)</td></tr><tr><td>Successful exit (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>−.054<xref ref-type="table-fn" rid="tfn5">***</xref> (.01)</td><td>.01 (.02)</td><td>.02 (.02)</td><td>−.016 (.02)</td></tr><tr><td><italic>p</italic>-value (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mi>b</mi><mn>2</mn></msub><mo>=</mo><msub><mi>b</mi><mn>1</mn></msub></mrow></math></p>)</td><td>.000</td><td>.314</td><td>.000</td><td>.941</td></tr><tr><td>Observations</td><td>4,789</td><td>4,789</td><td>4,789</td><td>4,789</td></tr><tr><td>R-squared</td><td>.117</td><td>.046</td><td>.055</td><td>.072</td></tr></tbody></table> </ephtml> </p> <p>5 <emph>Notes</emph>: ACT test year 2014 and 2015. Fixed effect for ACT test score. Additional controls for cubic function of test scores in other ACT subjects, race, gender, FRL eligibility, and whether the student was below cut points in any other subject. Only below cut score students included. Robust standard errors. Omitted group is students not in intervention. <emph>p</emph>-Values indicate <emph>p</emph>-value of <emph>F</emph>-test of equality of the "No successful exit" and "Successful exit" coefficients. *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively.</p> <hd id="AN0162243559-18">Intervention and Student Skills</hd> <p>As an additional test for whether TI might be plausibly expected to lead to measurable changes in college-ready skills, we examine the relationship between falling below the Grade 8 cutoff (and thus being discontinuously likely to receive TI) and ACT scores in 11th grade. While this paper focuses on Grade 12 interventions because this is the grade that the TI program has been continuously focusing on, TI was implemented as early as Grade 8 for a short period between 2012 and 2013 and 2014 and 2015. Using these Grade 8 cohorts allows for a direct test of student knowledge gains due to TI that we cannot perform for Grade 12 interventions due to a lack of access to post-intervention test scores. If TI did not lead to an appreciable change in long-run student skills, we might expect to find little relationship between TI in Grades 8 and 9 and outcomes years later.</p> <p>Because reporting was not mandatory for interventions in the eighth and ninth grades, we are limited to examining the impact of TI among "complier" districts where reported TI participation is strongly associated with failing the cutoffs.[<reflink idref="bib17" id="ref61">17</reflink>] As shown in Table 7, students in these districts participated in TI discontinuously at the cutoffs, with students scoring just below the cutoffs 40 to 60 percentage points more likely than students scoring just above the cutoffs to receive interventions in a given subject.</p> <p>Graph</p> <p>Table 7. First-Stage Discontinuity in Complier Districts: Grade 8 to 9 Intervention.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center" colspan="3">Math</th><th align="center" colspan="3">English</th></tr><tr><th /><th align="center">BW = 2</th><th align="center">BW = 3</th><th align="center">BW = 4</th><th align="center">BW = 2</th><th align="center">BW = 3</th><th align="center">BW = 4</th></tr></thead><tbody><tr><td>Discontinuity in TI participation</td><td>0.37<xref ref-type="table-fn" rid="tfn6">***</xref> (0.06)</td><td>0.38<xref ref-type="table-fn" rid="tfn6">***</xref> (0.04)</td><td>0.36<xref ref-type="table-fn" rid="tfn6">***</xref> (0.04)</td><td>0.60<xref ref-type="table-fn" rid="tfn6">***</xref> (0.10)</td><td>0.57<xref ref-type="table-fn" rid="tfn6">***</xref> (0.07)</td><td>0.55<xref ref-type="table-fn" rid="tfn6">***</xref> (0.06)</td></tr><tr><td><italic>F</italic>-statistic</td><td>26.2</td><td>36.1</td><td>39.1</td><td>21.6</td><td>28.8</td><td>30.5</td></tr><tr><td><italic>N</italic></td><td>1,127</td><td>1,623</td><td>1,782</td><td>391</td><td>577</td><td>730</td></tr></tbody></table> </ephtml> </p> <p>6 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Discontinuity in TI participation is estimated by regression TI participation on EXPLORE score, an indicator of falling below cutoffs, and the interaction of the two terms. Complier districts defined as districts with an estimated district-specific discontinuity of 0.20 with at least 50 students within bandwidth. Columns show bandwidths of 2, 3, and 4 points around the cutoff in each subject. The <emph>F</emph>-statistic tests the joint significance of the indicator variable of falling below the cutoff and its interaction with the EXPLORE score. The critical value is 11.59 based on [<reflink idref="bib33" id="ref62">33</reflink>].</p> <p>We use five cohorts of eighth grade students to estimate the ITT effect using the same DiRD model as in our main analysis, with 2014 to 2015 representing the post-intervention period and 2009 to 2012 the pre-intervention period. Results are shown in Table 8. While estimates for some outcomes are imprecise due to limitations noted above, we do not find any evidence that falling just below the college-readiness threshold in eighth grade led to students "catching up" by meeting ACT benchmarks in 11th grade, or, further down the road, taking credit-bearing math or English courses. In particular, we are able to rule out a positive effect on ACT math scores any larger than 0.80 ACT points (−0.19 + 1.96 × 0.51), and the point estimates on ACT scores and the likelihood of meeting ACT benchmarks are negative. While this is consistent with TI not moving the needle on long-run student outcomes, possibly due to the lack of an increase in overall instruction time, there are two important caveats: First, it is possible that the "complier" districts are not representative of districts in the state. Second, this program was terminated only 3 years later, and it is possible it was not implemented with high fidelity.</p> <p>Graph</p> <p>Table 8. Estimated Effect of Targeted Interventions (TI) in Grades 8 and 9 on Later Outcomes.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th /><th align="center">Take ACT</th><th align="center">Above ACT threshold</th><th align="center">ACT score</th><th align="center">Graduate high school</th><th align="center">Attend college</th><th align="center">Take DE course in subject in first year in college</th><th align="center">Take credit-bearing college course in subject in first year in college</th></tr></thead><tbody><tr><td colspan="8"><italic>Math</italic></td></tr><tr><td>Discontinuity pre</td><td>−0.01(0.01)</td><td>−0.05(0.03)</td><td>−0.22(0.23)</td><td>−0.00(0.01)</td><td>−0.02(0.02)</td><td>−0.08<xref ref-type="table-fn" rid="tfn7">***</xref>(0.03)</td><td>0.05(0.03)</td></tr><tr><td>Discontinuity post</td><td>−0.05<xref ref-type="table-fn" rid="tfn7">*</xref>(0.03)</td><td>−0.06(0.07)</td><td>−0.42(0.47)</td><td>−0.06<xref ref-type="table-fn" rid="tfn7">*</xref>(0.03)</td><td>−0.07(0.04)</td><td>0.00(0.04)</td><td>−0.04(0.06)</td></tr><tr><td>Difference in RD</td><td>−0.05(0.03)</td><td>−0.01(0.08)</td><td>−0.19(0.51)</td><td>−0.05(0.03)</td><td>−0.05(0.05)</td><td>0.08<xref ref-type="table-fn" rid="tfn7">*</xref>(0.05)</td><td>−0.09(0.07)</td></tr><tr><td><italic>Bandwidth</italic></td><td>4</td><td>2</td><td>2</td><td>4</td><td>4</td><td>3</td><td>4</td></tr><tr><td><italic>N</italic></td><td>8,612</td><td>5,213</td><td>5,213</td><td>8,612</td><td>8,612</td><td>3,875</td><td>4,184</td></tr><tr><td colspan="8"><italic>English</italic></td></tr><tr><td>Discontinuity pre</td><td>0.03(0.03)</td><td>−0.07(0.04)</td><td>−0.33(0.30)</td><td>−0.00(0.03)</td><td>0.07<xref ref-type="table-fn" rid="tfn7">*</xref>(0.04)</td><td>−0.04(0.03)</td><td>0.04(0.05)</td></tr><tr><td>Discontinuity post</td><td>0.02(0.06)</td><td>−0.19<xref ref-type="table-fn" rid="tfn7">**</xref>(0.08)</td><td>−0.90(0.59)</td><td>−0.07(0.06)</td><td>0.06(0.07)</td><td>0.04(0.04)</td><td>−0.20<xref ref-type="table-fn" rid="tfn7">*</xref>(0.12)</td></tr><tr><td>Difference in RD</td><td>−0.02(0.07)</td><td>−0.12(0.09)</td><td>−0.57(0.62)</td><td>−0.07(0.06)</td><td>−0.01(0.08)</td><td>0.08(0.06)</td><td>−0.24(0.14)</td></tr><tr><td><italic>Bandwidth</italic></td><td>4</td><td>3</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td></tr><tr><td><italic>N</italic></td><td>3,415</td><td>2,304</td><td>2,927</td><td>3,415</td><td>3,415</td><td>1,377</td><td>1,377</td></tr></tbody></table> </ephtml> </p> <p>7 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth selection. <emph>N</emph> is the number of students.</p> <hd id="AN0162243559-19">Statistical Power</hd> <p>One possible explanation for our findings is that, although Kentucky's transition intervention has strengthened early college outcomes that are shown to be predictive of later attainment, the effects may be too small to produce statistically detectable changes in credit accumulation and transfer rates. Take first-term course load as an example. The empirical literature on early progress in college found that taking 15 or more credits during the first term increases the likelihood of college completion by 9 percentage points ([<reflink idref="bib3" id="ref63">3</reflink>]). Since TI in math is estimated to have increased the likelihood that students take at least 15 credits by 5 to 6 percentage points, a direct extrapolation suggests that we can expect the average graduation rate to increase by roughly half of a percentage point (i.e., 0.09 × 0.05 ≈ 0.005). Converting the percentage point change to an effect size using Cox index conversion ([<reflink idref="bib30" id="ref64">30</reflink>]):</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>d</mi><mrow><mi>C</mi><mi>o</mi><mi>x</mi></mrow></msub><mo>=</mo><mi>ω</mi><mfrac><mrow><mi>L</mi><mi>O</mi><mi>R</mi></mrow><mrow><mn>1</mn><mo>.</mo><mn>65</mn></mrow></mfrac><mo>,</mo></mrow></math> </ephtml> </p> <p>Graph</p> <p>where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>ω</mi><mo>=</mo><mo stretchy="false">[</mo><mn>1</mn><mo>−</mo><mfrac><mn>3</mn><mrow><mn>4</mn><mi>N</mi><mo>−</mo><mn>9</mn></mrow></mfrac><mo stretchy="false">]</mo></mrow></math> </ephtml> with <emph>N</emph> denoting total sample size, <emph>LOR</emph> = In (<emph>Odds</emph><subs><emph>i</emph></subs>) – In (<emph>Odds</emph><subs><emph>j</emph></subs>), and assuming 30% of students graduate before the intervention, we estimate the expected impact of TI on college graduation rate to be roughly equivalent to 0.01 standard deviations. Given a sample size of 54,000 students within a bandwidth of 3 around the cutoff who are about evenly split between the pre- and post-TI samples, a typical RD design is expected to have a minimum detectable effect size (MDES) of 0.05 standard deviations.[<reflink idref="bib18" id="ref65">18</reflink>] The estimated MDES for a DiRD design, assuming independence between the pre- and post-TI samples, would be <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msqrt><mn>2</mn></msqrt><mo>*</mo><mn>0</mn><mo>.</mo><mn>05</mn><mo>=</mo><mn>0</mn><mo>.</mo><mn>07</mn></mrow></math> </ephtml> standard deviations. Thus, the research design does not have sufficient power to detect a change of 0.01 standard deviations (i.e., a 0.5 percentage point change in graduation rate).</p> <p>Even if a study were powered to detect a 0.5 percentage point change, which would require at least 2 million observations split between the pre- and post-intervention periods, it would still raise the question of whether such a relatively small effect is substantively important. The Kentucky Council for Postsecondary Education (CPE) set a goal of raising the percentage of Kentuckians with a postsecondary degree from 45.5% in 2017 to 60% in 2030 ([<reflink idref="bib14" id="ref66">14</reflink>]). Thus, even if it led to a 0.5 percentage point increase in the rate at which students earned a postsecondary degree, TI alone would not be sufficient to move the needle toward this college completion goal.</p> <hd id="AN0162243559-20">Conclusion</hd> <p>In this study, we investigate the potential impact of Kentucky's high school transition intervention program on the likelihood of college completion and transfer as well as on some of the key predictors of college success. Consistent with [<reflink idref="bib36" id="ref67">36</reflink>] finding that the TI program significantly reduced the likelihood of students enrolling in DE in college, we find that TI also helped students build early momentum, as measured by the likelihood that a student takes 15 or more credits during the first term in college. However, the early benefits of the TI program did not lead to detectable improvements in measured proxies of college attainment.</p> <p>As discussed in Section 5, one possible explanation for the findings is that time constraints led to the substitution of regular high school courses with intervention activities and as a result, there was no increase in the time that students spent on academic pursuits in high school. Because the immediate mandate for TI was to get students ready to access credit-bearing coursework without the need for DE or supplemental courses in college, TI seems to have focused on skills required to place out of DE in college, explaining the changes in early college outcomes. However, while helping students avoid the need for college developmental courses, the transition curriculum did not appear to have helped a measurable share of students develop necessary skills to progress through college relative to what they would have otherwise taken.</p> <p>Kentucky's strategic agenda for postsecondary education set a goal of increasing the share of Kentuckians with a postsecondary degree from 44% in 2015 to 60% in 2030.[<reflink idref="bib19" id="ref68">19</reflink>] If much of this increase is going to come from students who currently do not appear prepared for college, our evidence suggests that TI may need to go beyond its immediate goal of reducing DE needs in college and improve the alignment between high school transition curriculum and skills required to complete college. Currently, high schools only report whether a student completed TI successfully or not. What assessment was used and assessment scores or ratings have not been systematically documented. A standardized post-intervention assessment designed with collaboration between high school and college educators can help inform policymakers the extent of adjustment that may be needed to better align TI with college expectations.</p> <p>Such an adjustment would likely require some level of integration of the transition curriculum with Kentucky's regular, Common Core-aligned curriculum. In other words, transition activities should supplement, not supplant regular high school coursework. More intensive intervention may be required with extra learning time. Although no direct evidence is available about the relationship between the dosage of transition intervention and college outcomes, empirical findings from elementary and middle school grades consistently suggest that increased instruction time has positive effects on student test scores (e.g., [<reflink idref="bib13" id="ref69">13</reflink>]; [<reflink idref="bib16" id="ref70">16</reflink>]; [<reflink idref="bib34" id="ref71">34</reflink>]). High school-to-college transition interventions that do not increase total instruction time are likely to have a limited scope for impact on long-run college success.</p> <p>In addition to expanded opportunities for students to acquire the requisite academic skills for college, another potential way to increase the impact of transition intervention programs is to add support for non-academic college readiness. As documented in [<reflink idref="bib28" id="ref72">28</reflink>], high schools in general do not provide sufficient guidance to help students make informed college choices. As a result, students often must rely on parents and friends for information about financial aid, the college application process, the quality of college, and the labor market value of college majors. Because academically underprepared students are disproportionately represented by families that tend to be less informed about college selection, informational barrier becomes an additional challenge to them ([<reflink idref="bib26" id="ref73">26</reflink>]). The difficulties of decision making for young adults when faced with complex options, coupled with a lack of good information, often lead students to make haphazard choices ([<reflink idref="bib26" id="ref74">26</reflink>]). Our analysis shows, for example, that many students at the margin of college readiness struggled to decide what to study in college. Many students started college with an undeclared program of study and switched programs frequently in subsequent years.</p> <p>Such non-academic barriers could be more serious to student success in college than academic weaknesses ([<reflink idref="bib4" id="ref75">4</reflink>]), and transition intervention programs in high school have an opportunity to help. For example, research evidence suggests that most high school students are unable to rank broad categories of majors accurately in terms of labor market prospects ([<reflink idref="bib5" id="ref76">5</reflink>]) and that students from lower socio-economic backgrounds are more likely to enroll in programs and colleges with lower labor market returns ([<reflink idref="bib22" id="ref77">22</reflink>]). Transition intervention programs could help fill in such informational gaps, provide personalized advising and career counseling, and ultimately help students progress through college more efficiently.</p> <hd id="AN0162243559-21">Appendix</hd> <p>Graph</p> <p>Table A1. Estimated Effect of Targeted Interventions (TI) on College Outcomes.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" rowspan="3">Outcome</th><th align="center" colspan="4">Math</th><th align="center" colspan="4">English</th></tr><tr><th align="center" colspan="2">2 years</th><th align="center" colspan="2">4 years</th><th align="center" colspan="2">2 years</th><th align="center" colspan="2">4 years</th></tr><tr><th align="center">Coefficient<break /><italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic><break />Control mean</th><th align="center">Coefficient<break /><italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic><break />Control mean</th><th align="center">Coefficient<break /><italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic><break />Control mean</th><th align="center">Coefficient<break /><italic>S.E.</italic></th><th align="center">Bandwidth/<italic>N</italic><break />Control mean</th></tr></thead><tbody><tr><td colspan="9">Panel A: Early college outcomes</td></tr><tr><td>Started out in a non-degree program</td><td>−0.07<xref ref-type="table-fn" rid="tfn8">**</xref><break />(0.03)</td><td>3/20,304<break />0.10</td><td /><td /><td>0.01<break />(0.02)</td><td>4/20,195<break />0.12</td><td /><td /></tr><tr><td>Started with undeclared program</td><td>0.02<break />(0.04)</td><td>4/27,473<break />0.15</td><td>−0.03<xref ref-type="table-fn" rid="tfn8">*</xref><break />(0.02)</td><td>4/32,380<break />0.04</td><td>−0.00<break />(0.02)</td><td>5/23,406<break />0.13</td><td>0.00<break />(0.02)</td><td>3/17,840<break />0.04</td></tr><tr><td>Started in STEM</td><td>0.02<xref ref-type="table-fn" rid="tfn8">*</xref><break />(0.01)</td><td>5/54,400<break />0.06</td><td>0.01<break />(0.01)</td><td>5/66,175<break />0.09</td><td>−0.00<break />(0.01)</td><td>4/35,707<break />0.06</td><td>0.03<xref ref-type="table-fn" rid="tfn8">**</xref><break />(0.02)</td><td>4/41,895<break />0.10</td></tr><tr><td>Started in Health</td><td>−0.03<xref ref-type="table-fn" rid="tfn8">*</xref><break />(0.02)</td><td>4/48,470<break />0.07</td><td>0.01<break />(0.01)</td><td>5/66,175<break />0.06</td><td>0.03<xref ref-type="table-fn" rid="tfn8">*</xref><break />(0.01)</td><td>5/41,648<break />0.08</td><td>0.00<break />(0.01)</td><td>4/41,895<break />0.06</td></tr><tr><td>Program switch frequency within 1 year</td><td>0.01<break />(0.01)</td><td>6/57,369<break />0.06</td><td>−0.00<break />(0.01)</td><td>5/66,175<break />0.07</td><td>−0.00<break />(0.01)</td><td>4/35,707<break />0.06</td><td>−0.01<break />(0.01)</td><td>4/41,895<break />0.08</td></tr><tr><td>Program switch frequency within 3 years</td><td>0.00<break />(0.04)</td><td>6/57,639<break />0.23</td><td>−0.02<break />(0.04)</td><td>5/66,175<break />0.37</td><td>−0.00<break />(0.03)</td><td>5/41,648<break />0.23</td><td>0.03<break />(0.04)</td><td>5/50,708<break />0.38</td></tr><tr><td>Attempted ≥15 credits in first term</td><td>0.06<xref ref-type="table-fn" rid="tfn8">***</xref><break />(0.02)</td><td>5/54,400<break />0.18</td><td>0.05<xref ref-type="table-fn" rid="tfn8">**</xref><break />(0.02)</td><td>4/57,524<break />0.52</td><td>0.02<break />(0.01)</td><td>5/41,648<break />0.14</td><td>0.04<xref ref-type="table-fn" rid="tfn8">*</xref><break />(0.02)</td><td>4/41,895<break />0.47</td></tr><tr><td colspan="9">Panel B: End-of-college outcomes</td></tr><tr><td>Credits earned at the end of 3 years</td><td>1.08<break />(1.81)</td><td>3/36,637<break />37.3</td><td>2.33<break />(1.75)</td><td>3/46,036<break />46.2</td><td>−0.36<break />(1.20)</td><td>5/41,648<break />32.1</td><td>−1.37<break />(1.62)</td><td>4/41,895<break />40.0</td></tr><tr><td>Earned ≥ 60 credits by year 3</td><td>−0.00<break />(0.02)</td><td>5/54,400<break />0.28</td><td>0.03<break />(0.02)</td><td>3/46,036<break />0.42</td><td>0.01<break />(0.02)</td><td>4/35,707<break />0.22</td><td>−0.01<break />(0.02)</td><td>4/41,895<break />0.35</td></tr><tr><td>Transfer within 3 years</td><td>0.02<break />(0.02)</td><td>5/54,398<break />0.14</td><td /><td /><td>0.02<break />(0.01)</td><td>5/41,646<break />0.10</td><td /><td /></tr><tr><td>Credits earned at the end of 6 years</td><td /><td /><td>2.34<break />(2.73)</td><td>3/46,036<break />67.3</td><td /><td /><td>0.17<break />(2.24)</td><td>5/50,708<break />57.9</td></tr><tr><td>Earned ≥ 120 credits by year 6</td><td /><td /><td>0.04<break />(0.02)</td><td>3/27,336<break />0.26</td><td /><td /><td>−0.02<break />(0.02)</td><td>5/29,889<break />0.22</td></tr></tbody></table> </ephtml> </p> <p>8 <emph>Note.</emph> *, **, and *** denote statistical significance at the.10,.05, and.01 levels, respectively. Eicker-Huber-White standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. Bandwidth represents the bandwidth for a given subject and outcome, chosen using optimal bandwidth selection. <emph>N</emph> is the number of students.</p> <p>We thank Karen Dodd, April Piper, Aaron Butler, and Hannah Poquette from the Kentucky Department of Education and Barrett Ross from the Kentucky Center for Statistics for their support. The opinions expressed are those of the authors and do not represent the views of the Institute, the U.S. Department of Education, or the Kentucky Department of Education.</p> <hd id="AN0162243559-22">Author Biographies</hd> <p> <bold>Zeyu Xu</bold> is a managing economist at the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research. His research focuses on the evaluation of education interventions and policies.</p> <p> <bold>Ben Backes</bold> is a senior economist at the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research. He uses administrative data to research education policy on topics such as teacher training and teacher performance along with students' transitions from high school to college.</p> <p> <bold>Dan Goldhaber</bold> is director of the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research and the director of the Center for Education Data & Research (CEDR) at the University of Washington. Each conducts research that informs decisions about policy and practice.</p> <ref id="AN0162243559-23"> <title> References </title> <blist> <bibl id="bib1" type="bt">1</bibl> <bibtext> ACT. (2019). The condition of college & career readiness 2019. ACT.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref36" type="bt">2</bibl> <bibtext> Adelman C. (2006). The toolbox revisited: Paths to degree completion from high school through college. U.S. Department of Education.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref10" type="bt">3</bibl> <bibtext> Attewell P., Monaghan D. (2016). How many credits should an undergraduate take? Research in Higher Education, 57(6), 682–713.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref1" type="bt">4</bibl> <bibtext> Bailey T., Jaggars S. (2016). When college students start behind. The Century Foundation.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref6" type="bt">5</bibl> <bibtext> Baker R., Bettinger E., Jacob B., Marinescu I. (2018). The effect of labor market information on community college students' major choice. Economics of Education Review, 65, 18–30.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref7" type="bt">6</bibl> <bibtext> Barnett E. A., Chavarín O., Griffin S. (2018). Math transition courses in context: Preparing students for college success (CCRC Research Brief). Community College Research Center, Teachers College, Columbia University.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref5" type="bt">7</bibl> <bibtext> Barnett E. A., Fay M. P., Pheatt L. (2016). Implementation of high school-to-college transition courses in four states. Columbia University, Teachers College, Community College Research Center.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref32" type="bt">8</bibl> <bibtext> Belfield C., Jenkins D., Lahr H. (2016). Momentum: The academic and economic value of a 15-credit first-semester course load for college students in Tennessee (CCRC Working Paper No. 88). Community College Research Center, Teachers College, Columbia University.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref2" type="bt">9</bibl> <bibtext> Bettinger E. P., Boatman A., Long B. T. (2013). Student supports: Developmental education and other academic programs. The Future of Children, 23(1), 93–115.</bibtext> </blist> <blist> <bibtext> Boatman A. (2012). Evaluating institutional efforts to streamline postsecondary remediation: The causal effects of the Tennessee Developmental Course Redesign Initiative on early student academic success. National Center for Postsecondary Research, Columbia University.</bibtext> </blist> <blist> <bibtext> Boatman A., Long B. T. (2018). Does remediation work for all students? How the effects of postsecondary remedial and developmental courses vary by level of academic preparation. Educational Evaluation and Policy Analysis, 40(1), 29–58.</bibtext> </blist> <blist> <bibtext> Calonico S., Cattaneo M. D., Farrell M. H. (2020). Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs. Econometrics Journal, 23(2), 192–210.</bibtext> </blist> <blist> <bibtext> Cortes K. E., Goodman J. S., Nomi T. (2015). Intensive math instruction and educational attainment long-run impacts of double-dose algebra. Journal of Human Resources, 50(1), 108–158.</bibtext> </blist> <blist> <bibtext> Council on Postsecondary Education (CPE). (2019). Progress report: Kentucky's progress toward postsecondary strategic agenda goals. <ulink href="http://cpe.ky.gov/data/reports/2019progressreport.pdf">http://cpe.ky.gov/data/reports/2019progressreport.pdf</ulink></bibtext> </blist> <blist> <bibtext> Fay M. P., Barnett E., Chavarín O. (2017). How states are implementing transition curricula: Results from a national scan. Columbia University, Teachers College, Community College Research Center.</bibtext> </blist> <blist> <bibtext> Figlio D., Holden K. L., Ozek U. (2018). Do students benefit from longer school days? Regression discontinuity evidence from Florida's additional hour of literacy instruction. Economics of Education Review, 67, 171–183.</bibtext> </blist> <blist> <bibtext> Holzer H. J., Xu Z. (2021). Community college pathways for disadvantaged students. Community College Review, 49, 351–388. https://doi.org/10.1177/00915521211002908</bibtext> </blist> <blist> <bibtext> Kane T. J., Boatman A., Kozakowski W., Bennett C., Hitch R., Weisenfeld D. (2021). Is college remediation a barrier or a boost? Evidence from the Tennessee SAILS program. Journal of Policy Analysis and Management, 40, 883–913. https://doi.org/10.1002/pam.22306</bibtext> </blist> <blist> <bibtext> Kolesár M., Rothe C. (2018). Inference in regression discontinuity designs with a discrete running variable. American Economic Review, 108(8), 2277–2304.</bibtext> </blist> <blist> <bibtext> K.Y. Legis. S. 09 Reg. Sess. (2009). https://apps.legislature.ky.gov/record/09rs/sb1.html</bibtext> </blist> <blist> <bibtext> Levin J., Carbuccia M., Adelman-Sil E., Danks A. (2020). Kentucky targeted intervention program cost study. American Institutes for Research.</bibtext> </blist> <blist> <bibtext> Lovenheim M. F., Smith J. (2022). Returns to different postsecondary investments: Institution type, academic programs, and credentials (Working Paper 29933). National Bureau of Economic Research. https://<ulink href="http://www.nber.org/papers/w29933">www.nber.org/papers/w29933</ulink></bibtext> </blist> <blist> <bibtext> McCaffrey D. F., Bell R. M. (2002). Bias reduction in standard errors for linear and generalized linear models with multi-stage samples [Conference session]. In: Statistics Canada Symposium.</bibtext> </blist> <blist> <bibtext> Michaels C., Liu L. (2020). Differences in academic achievements among high school graduates' from four career and technical education (CTE) program areas. International Journal of Technology in Teaching and Learning, 15(2), 109–125.</bibtext> </blist> <blist> <bibtext> Mokher C. G., Leeds D. M., Harris J. C. (2018). Adding it up: How the Florida College and career readiness initiative impacted developmental education. Educational Evaluation and Policy Analysis, 40(2), 219–242.</bibtext> </blist> <blist> <bibtext> Page L. C., Scott-Clayton J. (2016). Improving college access in the United States: Barriers and policy responses. Economics of Education Review, 51, 4–22.</bibtext> </blist> <blist> <bibtext> Pheatt L. E., Trimble M. J., Barnett E. (2016). Improving the transition to college: Estimating the impact of high school transition courses on short-term college outcomes. Community College Research Center, Teachers College, Columbia University.</bibtext> </blist> <blist> <bibtext> Radford A. W. (2013). Top student, top school? How social class shapes where valedictorians go to college. University of Chicago Press.</bibtext> </blist> <blist> <bibtext> Rosenbaum J. E., Ahearn C., Rosenbaum J. (2015). The new forgotten half and research directions to support them. William T. Grant Foundation.</bibtext> </blist> <blist> <bibtext> Sanchez-Meca J., Marin-Martinez F., Chacon-Moscoso S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8(4), 448–467.</bibtext> </blist> <blist> <bibtext> Schochet P. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87.</bibtext> </blist> <blist> <bibtext> Scott-Clayton J., Rodriguez O. (2015). Development, discouragement, or diversion? New evidence on the effects of college remediation policy. Education Finance and Policy, 10(1), 4–45.</bibtext> </blist> <blist> <bibtext> Stock J. H., Yogo M. (2005). Testing for weak instruments in Linear IV regression. In Andrews D. W. K., Stock J. H. (Eds.), Identification and inference for econometric models: Essays in honor of Thomas Rothenberg (pp. 80–108). Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Taylor E. (2014). Spending more of the school day in math class: Evidence from a regression discontinuity in middle school. Journal of Public Economics, 117, 162–181.</bibtext> </blist> <blist> <bibtext> Valentine J. C., Konstantopoulos S., Goldrick-Rab S. (2017). What happens to students placed into developmental education? A meta-analysis of regression discontinuity studies. Review of Educational Research, 87(4), 806–833.</bibtext> </blist> <blist> <bibtext> Xu Z., Backes B., Oliveira A., Goldhaber D. (2022). Ready for college? Examining the effectiveness of targeted interventions in high school. Educational Evaluation and Policy Analysis, 44, 183–209. https://doi.org/10.3102%2F01623737211036728</bibtext> </blist> </ref> <ref id="AN0162243559-24"> <title> Footnotes </title> <blist> <bibtext> Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160188 to the American Institutes for Research and the Kentucky Department of Education.</bibtext> </blist> <blist> <bibtext> ORCID iDs Ben Backes</bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext>https://orcid.org/0000-0003-3485-1586 Dan Goldhaber</bibtext> </blist> <blist> <bibtext>Graph https://orcid.org/0000-0003-4260-4040</bibtext> </blist> <blist> <bibtext> 1. See https://nces.ed.gov/programs/raceindicators/indicator_RED.asp.</bibtext> </blist> <blist> <bibtext> 2. Additional comments can be found in https://hechingerreport.org/solution-obvious-rare-making-high-school-graduates-ready-college/ and https://<ulink href="http://www.edweek.org/teaching-learning/transitional-courses-catch-on-as-college-prep-strategy/2014/02">www.edweek.org/teaching-learning/transitional-courses-catch-on-as-college-prep-strategy/2014/02</ulink>.</bibtext> </blist> <blist> <bibtext> 3. The Tennessee program automatically exempting students from DE in college after successful completion is a common practice across transition intervention programs ([15]). Kentucky's TI is one of only five (out of 17 statewide programs) in which successful completion of a transition intervention does not automatically place students out of DE in college.</bibtext> </blist> <blist> <bibtext> 4. Less than 3% of students with ACT scores 2 to 3 points around the cutoffs enrolled in out-of-state colleges, and there is no discontinuity in the likelihood of leaving the state at the cutoffs.</bibtext> </blist> <blist> <bibtext> 5.https://ccrc.tc.columbia.edu/Community-College-FAQs.html</bibtext> </blist> <blist> <bibtext> 6. Surveys and interviews suggest several explanations for noncompliance ([36]). Teachers could consider other performance metrics in addition to ACT scores when referring students to TI. Students also had the option in many schools to test out of TI. And in some cases, staff and parental resistance to remediation also played a role in under-participation in TI.</bibtext> </blist> <blist> <bibtext> 7. The results in this paper are primarily obtained from the sample of students who enrolled in college. Xu et al. (2022) use the DiRD model to examine the effect of TI on college enrollment and finds no significant effect on enrolling in either a 2- or 4-year college. This suggests that any TI effects on college outcomes are not achieved by altering the composition of college-going students.</bibtext> </blist> <blist> <bibtext> 8. In results available from the authors, we find that TI in math is associated with attempting more STEM credits in the first year, which is likely a consequence of being more likely to take a credit-bearing course in math instead of developmental math. However, the impact on total number of STEM credits accumulated by the end of the first year is small and not statistically significant.</bibtext> </blist> <blist> <bibtext> 9. Half of students who failed to meet the math benchmark also participated in English remediation, and 76% of students who failed to meet the English benchmark received remediation in math.</bibtext> </blist> <blist> <bibtext> 10. Schools in the top quarter have at least 69% of their students missing the ACT cutoff in math and 54% in English.</bibtext> </blist> <blist> <bibtext> 11. Because of the similarities in findings, these subgroup estimates are not presented here to save space, but they are available upon request.</bibtext> </blist> <blist> <bibtext> 12. See, for example, the course description for Mathematical Literacy (MAT 075) in the academic catalog of the Kentucky Community and Technical College System.</bibtext> </blist> <blist> <bibtext> 13. We examine CTE course-taking as a separate course category because low-achieving students are more likely to focus on CTE, so there is significant overlap between the target student population of transition interventions and students who are likely to benefit from CTE.</bibtext> </blist> <blist> <bibtext> 14. We use Equation 1 to estimate the discontinuity of reported TI participation around EXPLORE cutoffs district by district. The optimal bandwidth is 3 for math and 4 for English. Districts with an estimated discontinuity of 0.20 and at least 50 students within bandwidth were selected. This results in nine districts for math and six districts for English. We also used alternative criteria for district selection (i.e., a discontinuity of 0.15 and 0.25, with and without sample size restrictions, and alternating the bandwidth), but the findings on earlier intervention's impact on student outcomes did not materially change.</bibtext> </blist> <blist> <bibtext> 15. MDES is estimated following [31] for a two-tailed test with a significance level of.05 and statistical power of.80. We further assume that students within a bandwidth of 3 around the cutoff are evenly divided into treatment and control groups, and that covariates explain 15% of the variation in the outcome variable.</bibtext> </blist> <blist> <bibtext> 16.<ulink href="http://cpe.ky.gov/ourwork/60x30.html">http://cpe.ky.gov/ourwork/60x30.html</ulink></bibtext> </blist> </ref> <aug> <p>By Zeyu Xu; Ben Backes and Dan Goldhaber</p> <p>Reported by Author; Author; Author</p> <p></p> <p>Zeyu Xu is a managing economist at the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research. His research focuses on the evaluation of education interventions and policies.</p> <p>Ben Backes is a senior economist at the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research. He uses administrative data to research education policy on topics such as teacher training and teacher performance along with students' transitions from high school to college.</p> <p>Dan Goldhaber is director of the Center for Analysis of Longitudinal Data in Education Research (CALDER) at the American Institutes for Research and the director of the Center for Education Data & Research (CEDR) at the University of Washington. Each conducts research that informs decisions about policy and practice.</p> </aug> <nolink nlid="nl1" bibid="bib11" firstref="ref3"></nolink> <nolink nlid="nl2" bibid="bib35" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib15" firstref="ref8"></nolink> <nolink nlid="nl4" bibid="bib36" firstref="ref9"></nolink> <nolink nlid="nl5" bibid="bib29" firstref="ref11"></nolink> <nolink nlid="nl6" bibid="bib32" firstref="ref14"></nolink> <nolink nlid="nl7" bibid="bib24" firstref="ref16"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref17"></nolink> <nolink nlid="nl9" bibid="bib21" firstref="ref23"></nolink> <nolink nlid="nl10" bibid="bib27" firstref="ref24"></nolink> <nolink nlid="nl11" bibid="bib25" firstref="ref25"></nolink> <nolink nlid="nl12" bibid="bib18" firstref="ref26"></nolink> <nolink nlid="nl13" bibid="bib17" firstref="ref34"></nolink> <nolink nlid="nl14" bibid="bib23" firstref="ref42"></nolink> <nolink nlid="nl15" bibid="bib19" firstref="ref43"></nolink> <nolink nlid="nl16" bibid="bib12" firstref="ref44"></nolink> <nolink nlid="nl17" bibid="bib10" firstref="ref45"></nolink> <nolink nlid="nl18" bibid="bib13" firstref="ref54"></nolink> <nolink nlid="nl19" bibid="bib14" firstref="ref55"></nolink> <nolink nlid="nl20" bibid="bib16" firstref="ref60"></nolink> <nolink nlid="nl21" bibid="bib33" firstref="ref62"></nolink> <nolink nlid="nl22" bibid="bib30" firstref="ref64"></nolink> <nolink nlid="nl23" bibid="bib34" firstref="ref71"></nolink> <nolink nlid="nl24" bibid="bib28" firstref="ref72"></nolink> <nolink nlid="nl25" bibid="bib26" firstref="ref73"></nolink> <nolink nlid="nl26" bibid="bib22" firstref="ref77"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1369483
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Impact of Transition Intervention in High School on Pathways through College
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Xu%2C+Zeyu%22">Xu, Zeyu</searchLink><br /><searchLink fieldCode="AR" term="%22Backes%2C+Ben%22">Backes, Ben</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-3485-1586">0000-0003-3485-1586</externalLink>)<br /><searchLink fieldCode="AR" term="%22Goldhaber%2C+Dan%22">Goldhaber, Dan</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4260-4040">0000-0003-4260-4040</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Community+College+Review%22"><i>Community College Review</i></searchLink>. Apr 2023 51(2):216-245.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 30
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305A160188
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22College+Bound+Students%22">College Bound Students</searchLink><br /><searchLink fieldCode="DE" term="%22College+Readiness%22">College Readiness</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Transitional+Programs%22">Transitional Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22College+Credits%22">College Credits</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Remedial+Instruction%22">Remedial Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22College+Preparation%22">College Preparation</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Kentucky%22">Kentucky</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00915521221145325
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0091-5521<br />1940-2325
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objective: In 2009, the Kentucky General Assembly found unacceptable and costly the ongoing high numbers of high school students requiring remediation once they enter higher education. The state passed legislation to better align secondary and college education, establishing a diagnostic cycle that would become its Targeted Interventions (TI) program. Using 11 years of panel data, this study tracked college progression of seven cohorts of students in order to estimate the impacts of this program. Method: Using student-level administrative data from the state of Kentucky that tracks students from high school through college, a difference-in-regression-discontinuity design was used to compare how students just below college readiness benchmarks fared relative to those just above once TI was implemented. Results: The TI program significantly increased the likelihood that students took at least 15 credits during their first term, a key predictive measure for college completion. However, these early effects did not translate into detectable impacts on the likelihood of earning enough credits to graduate from college or likelihood of transfers from a 2-year to a 4-year college. One possible explanation for this pattern is that TI appears to have crowded out other core courses in high school, especially in math, without increasing total instructional time. Findings suggest that the standards used by high schools to judge student progress toward college readiness may be consistent with the skills needed to place out of developmental courses, but not sufficient to better prepare students for college-level instruction. Contributions: To our knowledge, this is the first study to explore how TI shapes longer term college outcomes. The transition curriculum, while helping students avoid the need for college developmental courses, did not help a measurable share of students develop necessary skills to progress through college relative to what they would have otherwise taken. A possible explanation for these findings is that high school-to-college transition interventions that do not increase total instruction time do not sufficiently move the needle on the college preparedness among high school graduates. For states concerned with the number of students entering college deemed not college ready, it appears that high school-to-college transition interventions that supplant instead of supplement regular high school curriculum have a limited scope for impact on long-run college success.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1369483
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1369483
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00915521221145325
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
        StartPage: 216
    Subjects:
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: College Bound Students
        Type: general
      – SubjectFull: College Readiness
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Transitional Programs
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: College Credits
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Remedial Instruction
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: College Preparation
        Type: general
      – SubjectFull: Kentucky
        Type: general
    Titles:
      – TitleFull: The Impact of Transition Intervention in High School on Pathways through College
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Xu, Zeyu
      – PersonEntity:
          Name:
            NameFull: Backes, Ben
      – PersonEntity:
          Name:
            NameFull: Goldhaber, Dan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 0091-5521
            – Type: issn-electronic
              Value: 1940-2325
          Numbering:
            – Type: volume
              Value: 51
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Community College Review
              Type: main
ResultId 1