Independent Evaluation of Lone Star AP CSP: Final Report
Saved in:
| Title: | Independent Evaluation of Lone Star AP CSP: Final Report |
|---|---|
| Language: | English |
| Authors: | Deborah J. Holtzman, Kathleen T. Jones, Charles Blankenship, American Institutes for Research (AIR), National Math and Science Initiative |
| Source: | American Institutes for Research. 2025. |
| Availability: | American Institutes for Research. 1400 Crystal Drive 10th Floor, Arlington, VA 22202. Tel: 202-403-5000; Fax: 202-403-5001; e-mail: inquiry@air.org; Web site: https://www.air.org/ |
| Peer Reviewed: | N |
| Page Count: | 221 |
| Publication Date: | 2025 |
| Contract Number: | U411C190037 |
| Document Type: | Reports - Research Numerical/Quantitative Data |
| Education Level: | High Schools Secondary Education |
| Descriptors: | Advanced Placement Programs, Computer Science Education, High School Students, High School Teachers, Program Effectiveness, Program Implementation, Fidelity, Teacher Improvement, Achievement Tests, Academic Achievement, Educational Practices, Student Participation |
| Geographic Terms: | Texas |
| Assessment and Survey Identifiers: | Advanced Placement Examinations (CEEB) |
| Abstract: | This report describes the evaluation of Lone Star AP CSP, a program developed by the National Mathematics + Science Initiative and funded by a 2019 Education Innovation and Research grant. The three-year program aimed to increase student access to and success in the Advanced Placement (AP) Computer Science Principles (CSP) course in 37 Texas high schools through teacher professional development, supports for students, and supports for schools. The American Institutes for Research conducted an independent evaluation of the program, including studies of implementation, impact on teachers' instruction, and impact on students' participation in and performance on the AP CSP exam. The implementation study examined whether the program was implemented with fidelity in each of its three years and the extent to which participants made use of the program and found it useful. Key findings were that there was no year in which all three components were implemented with fidelity and that teacher and student uptake levels were low but increased over time. The study of impact on teachers' instruction examined the effect of the program on three instructional practice outcomes that were derived from a survey of Lone Star AP CSP teachers along with a volunteer sample of other AP CSP teachers. Separately in each of the three years of the program, the instructional practices impact study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of Lone Star AP CSP on each of the three survey-based outcomes. The evaluation found that the program had positive effects on teachers' instructional practices in the first and second years of implementation but not in the third. Finally, the student impact study examined school-level outcomes in the analysis of the impact of Lone Star AP CSP on student AP CSP performance and participation. Separately in each of the three outcome years, the study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of the program on each outcome, followed by meta-analysis to combine the estimates from two samples of schools: those that had AP CSP exam takers in the baseline year and those that did not. Key findings were that the program did not have a significant positive impact on the percentage of AP CSP exam takers that received a qualifying score on the exam in any of the three years, nor did the program have a significant positive impact on schools' percentages of students in grades 9-12 that took the AP CSP exam in any of the three years. In fact, in Years 2 and 3 the program's effect on student participation in the AP CSP exam was significantly negative. The evaluators note that relatively low levels of program implementation and uptake probably help explain why the program did not achieve its hypothesized positive impacts on students. They also acknowledge that the comparison schools included in the analyses could have been implementing other interventions pertaining to AP CSP. [This is a report to the National Math + Science Initiative Regarding Grant U411C190037.] |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | ED675198 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED675198 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED675198 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Independent Evaluation of Lone Star AP CSP: Final Report – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Deborah+J%2E+Holtzman%22">Deborah J. Holtzman</searchLink><br /><searchLink fieldCode="AR" term="%22Kathleen+T%2E+Jones%22">Kathleen T. Jones</searchLink><br /><searchLink fieldCode="AR" term="%22Charles+Blankenship%22">Charles Blankenship</searchLink><br /><searchLink fieldCode="AR" term="%22American+Institutes+for+Research+%28AIR%29%22">American Institutes for Research (AIR)</searchLink><br /><searchLink fieldCode="AR" term="%22National+Math+and+Science+Initiative%22">National Math and Science Initiative</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22American+Institutes+for+Research%22"><i>American Institutes for Research</i></searchLink>. 2025. – Name: Avail Label: Availability Group: Avail Data: American Institutes for Research. 1400 Crystal Drive 10th Floor, Arlington, VA 22202. Tel: 202-403-5000; Fax: 202-403-5001; e-mail: inquiry@air.org; Web site: https://www.air.org/ – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: N – Name: Pages Label: Page Count Group: Src Data: 221 – Name: DatePubCY Label: Publication Date Group: Date Data: 2025 – Name: NumberContract Label: Contract Number Group: NumCntrct Data: U411C190037 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Reports - Research<br />Numerical/Quantitative Data – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Advanced+Placement+Programs%22">Advanced Placement Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+Education%22">Computer Science Education</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Teachers%22">High School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Improvement%22">Teacher Improvement</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Practices%22">Educational Practices</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Participation%22">Student Participation</searchLink> – Name: Subject Label: Geographic Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Texas%22">Texas</searchLink> – Name: SubjectThesaurus Label: Assessment and Survey Identifiers Group: Su Data: <searchLink fieldCode="SU" term="%22Advanced+Placement+Examinations+%28CEEB%29%22">Advanced Placement Examinations (CEEB)</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This report describes the evaluation of Lone Star AP CSP, a program developed by the National Mathematics + Science Initiative and funded by a 2019 Education Innovation and Research grant. The three-year program aimed to increase student access to and success in the Advanced Placement (AP) Computer Science Principles (CSP) course in 37 Texas high schools through teacher professional development, supports for students, and supports for schools. The American Institutes for Research conducted an independent evaluation of the program, including studies of implementation, impact on teachers' instruction, and impact on students' participation in and performance on the AP CSP exam. The implementation study examined whether the program was implemented with fidelity in each of its three years and the extent to which participants made use of the program and found it useful. Key findings were that there was no year in which all three components were implemented with fidelity and that teacher and student uptake levels were low but increased over time. The study of impact on teachers' instruction examined the effect of the program on three instructional practice outcomes that were derived from a survey of Lone Star AP CSP teachers along with a volunteer sample of other AP CSP teachers. Separately in each of the three years of the program, the instructional practices impact study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of Lone Star AP CSP on each of the three survey-based outcomes. The evaluation found that the program had positive effects on teachers' instructional practices in the first and second years of implementation but not in the third. Finally, the student impact study examined school-level outcomes in the analysis of the impact of Lone Star AP CSP on student AP CSP performance and participation. Separately in each of the three outcome years, the study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of the program on each outcome, followed by meta-analysis to combine the estimates from two samples of schools: those that had AP CSP exam takers in the baseline year and those that did not. Key findings were that the program did not have a significant positive impact on the percentage of AP CSP exam takers that received a qualifying score on the exam in any of the three years, nor did the program have a significant positive impact on schools' percentages of students in grades 9-12 that took the AP CSP exam in any of the three years. In fact, in Years 2 and 3 the program's effect on student participation in the AP CSP exam was significantly negative. The evaluators note that relatively low levels of program implementation and uptake probably help explain why the program did not achieve its hypothesized positive impacts on students. They also acknowledge that the comparison schools included in the analyses could have been implementing other interventions pertaining to AP CSP. [This is a report to the National Math + Science Initiative Regarding Grant U411C190037.] – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2025 – Name: AN Label: Accession Number Group: ID Data: ED675198 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED675198 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 221 Subjects: – SubjectFull: Advanced Placement Programs Type: general – SubjectFull: Computer Science Education Type: general – SubjectFull: High School Students Type: general – SubjectFull: High School Teachers Type: general – SubjectFull: Program Effectiveness Type: general – SubjectFull: Program Implementation Type: general – SubjectFull: Fidelity Type: general – SubjectFull: Teacher Improvement Type: general – SubjectFull: Achievement Tests Type: general – SubjectFull: Academic Achievement Type: general – SubjectFull: Educational Practices Type: general – SubjectFull: Student Participation Type: general – SubjectFull: Texas Type: general – SubjectFull: Advanced Placement Examinations (CEEB) Type: general Titles: – TitleFull: Independent Evaluation of Lone Star AP CSP: Final Report Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: American Institutes for Research (AIR) – PersonEntity: Name: NameFull: National Math and Science Initiative – PersonEntity: Name: NameFull: Deborah J. Holtzman – PersonEntity: Name: NameFull: Kathleen T. Jones – PersonEntity: Name: NameFull: Charles Blankenship IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 08 Type: published Y: 2025 Titles: – TitleFull: American Institutes for Research Type: main |
| ResultId | 1 |
