View in EDS

2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program

Saved in:
Bibliographic Details
Title: 2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
Language: English
Authors: Phelan, Julia, Egger, Jeffrey, Michiuye, Joanne K., Keum, Eunhee, Choi, Kilchan, Chung, Gregory K. W. K., Baker, Eva L., National Center for Research on Evaluation, Standards, and Student Testing (CRESST), National Math and Science Initiative
Source: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 2018.
Availability: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
Peer Reviewed: N
Page Count: 100
Publication Date: 2018
Sponsoring Agency: Department of Education (ED)
Document Type: Reports - Research
Tests/Questionnaires
Education Level: High Schools
Secondary Education
Descriptors: Program Implementation, Program Evaluation, College Readiness, STEM Education, High School Students, Advanced Placement Programs, Advanced Courses, Disproportionate Representation, Outcomes of Education, Fidelity, Scores, Course Selection (Students), Probability, Barriers, Mathematics Tests, Science Tests, Language Arts
Abstract: The National Math + Science Initiative's (NMSI's) College Readiness Program (CRP) is an established program whose goal is to promote science, technology, engineering, and mathematics education in high schools to improve students' readiness for college. The program provides teacher, student, and school supports to promote high school students' success in mathematics, science, and English Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 28 schools in the 2016-2017 school year. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 28 CRP schools and 24 control schools in 10 states. The evaluation of the CRP consisted of two parts: (1) assessment of the program's impact on selected student AP exam outcomes and (2) assessment of the fidelity of implementation of the CRP. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools The descriptive statistics showed that a higher percetange of students in the treatment schools took at least one AP course (30.7%) compared to those in the control schools (26.4%) by approximately 4.3%, however the difference was not statistically significant. In addition, students in the treatment schools were not more likely to achieve a score of 3 or higher, when compared to the delayed treatment schools. We further examined the effectiveness of the CRP using the prior year's school-level performance on the AP exam as a covariate. As with the above findings, the results indicated the probability of a student taking at least one AP course or scoring 3 or higher on at least one AP exam is not statistically different between students in the treatment schools and those in the control treatment schools. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. Overall results, however, indicated that 23 schools out of 28 treatment schools (82.1%) achieved 80% or better implementation fidelity, for an average fidelity score of 89.5%. Seven schools achieved a perfect 100% fidelity score. Looking at the different indicator groups (school, teacher and student), we found that all school support measures across all schools were implemented with fidelity. In over 80% of schools, not all teachers fulfilled their requirements for attending all training sessions, and so this component was not implemented with fidelity. Stipends and teacher awards were paid as expected as were student award payments. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports relating to helping increase student achievement in AP courses. Teacher incentives were chosen as the least important program component relating to increasing student performance by 16% of teachers and student incentives by 12% of teachers. Teachers did, however, view the student incentives as an important program component to encourage enrollment in AP courses. Likewise, students rated the financial incentives on average as somewhat important in encouraging them to participate in AP courses.
Abstractor: As Provided
What Works Clearinghouse (WWC) Reviewed: Meets Evidence Standards with Reservations
WWC Study Page: https://ies.ed.gov/ncee/wwc/Study/90296
Entry Date: 2021
Accession Number: ED615910
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED615910
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED615910
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: 2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Phelan%2C+Julia%22">Phelan, Julia</searchLink><br /><searchLink fieldCode="AR" term="%22Egger%2C+Jeffrey%22">Egger, Jeffrey</searchLink><br /><searchLink fieldCode="AR" term="%22Michiuye%2C+Joanne+K%2E%22">Michiuye, Joanne K.</searchLink><br /><searchLink fieldCode="AR" term="%22Keum%2C+Eunhee%22">Keum, Eunhee</searchLink><br /><searchLink fieldCode="AR" term="%22Choi%2C+Kilchan%22">Choi, Kilchan</searchLink><br /><searchLink fieldCode="AR" term="%22Chung%2C+Gregory+K%2E+W%2E+K%2E%22">Chung, Gregory K. W. K.</searchLink><br /><searchLink fieldCode="AR" term="%22Baker%2C+Eva+L%2E%22">Baker, Eva L.</searchLink><br /><searchLink fieldCode="AR" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22">National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</searchLink><br /><searchLink fieldCode="AR" term="%22National+Math+and+Science+Initiative%22">National Math and Science Initiative</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22"><i>National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</i></searchLink>. 2018.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 100
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2018
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Department of Education (ED)
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research<br />Tests/Questionnaires
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22College+Readiness%22">College Readiness</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Advanced+Placement+Programs%22">Advanced Placement Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Advanced+Courses%22">Advanced Courses</searchLink><br /><searchLink fieldCode="DE" term="%22Disproportionate+Representation%22">Disproportionate Representation</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Course+Selection+%28Students%29%22">Course Selection (Students)</searchLink><br /><searchLink fieldCode="DE" term="%22Probability%22">Probability</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Tests%22">Science Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Arts%22">Language Arts</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The National Math + Science Initiative's (NMSI's) College Readiness Program (CRP) is an established program whose goal is to promote science, technology, engineering, and mathematics education in high schools to improve students' readiness for college. The program provides teacher, student, and school supports to promote high school students' success in mathematics, science, and English Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 28 schools in the 2016-2017 school year. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 28 CRP schools and 24 control schools in 10 states. The evaluation of the CRP consisted of two parts: (1) assessment of the program's impact on selected student AP exam outcomes and (2) assessment of the fidelity of implementation of the CRP. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools The descriptive statistics showed that a higher percetange of students in the treatment schools took at least one AP course (30.7%) compared to those in the control schools (26.4%) by approximately 4.3%, however the difference was not statistically significant. In addition, students in the treatment schools were not more likely to achieve a score of 3 or higher, when compared to the delayed treatment schools. We further examined the effectiveness of the CRP using the prior year's school-level performance on the AP exam as a covariate. As with the above findings, the results indicated the probability of a student taking at least one AP course or scoring 3 or higher on at least one AP exam is not statistically different between students in the treatment schools and those in the control treatment schools. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. Overall results, however, indicated that 23 schools out of 28 treatment schools (82.1%) achieved 80% or better implementation fidelity, for an average fidelity score of 89.5%. Seven schools achieved a perfect 100% fidelity score. Looking at the different indicator groups (school, teacher and student), we found that all school support measures across all schools were implemented with fidelity. In over 80% of schools, not all teachers fulfilled their requirements for attending all training sessions, and so this component was not implemented with fidelity. Stipends and teacher awards were paid as expected as were student award payments. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports relating to helping increase student achievement in AP courses. Teacher incentives were chosen as the least important program component relating to increasing student performance by 16% of teachers and student incentives by 12% of teachers. Teachers did, however, view the student incentives as an important program component to encourage enrollment in AP courses. Likewise, students rated the financial incentives on average as somewhat important in encouraging them to participate in AP courses.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: What Works Clearinghouse (WWC) Reviewed
  Group: SrcInfo
  Data: Meets Evidence Standards with Reservations
– Name: URL
  Label: WWC Study Page
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://ies.ed.gov/ncee/wwc/Study/90296" linkWindow="_blank">https://ies.ed.gov/ncee/wwc/Study/90296</link>
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED615910
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED615910
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 100
    Subjects:
      – SubjectFull: Program Implementation
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: College Readiness
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Advanced Placement Programs
        Type: general
      – SubjectFull: Advanced Courses
        Type: general
      – SubjectFull: Disproportionate Representation
        Type: general
      – SubjectFull: Outcomes of Education
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Course Selection (Students)
        Type: general
      – SubjectFull: Probability
        Type: general
      – SubjectFull: Barriers
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Science Tests
        Type: general
      – SubjectFull: Language Arts
        Type: general
    Titles:
      – TitleFull: 2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
      – PersonEntity:
          Name:
            NameFull: National Math and Science Initiative
      – PersonEntity:
          Name:
            NameFull: Phelan, Julia
      – PersonEntity:
          Name:
            NameFull: Egger, Jeffrey
      – PersonEntity:
          Name:
            NameFull: Michiuye, Joanne K.
      – PersonEntity:
          Name:
            NameFull: Keum, Eunhee
      – PersonEntity:
          Name:
            NameFull: Choi, Kilchan
      – PersonEntity:
          Name:
            NameFull: Chung, Gregory K. W. K.
      – PersonEntity:
          Name:
            NameFull: Baker, Eva L.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2018
          Titles:
            – TitleFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
              Type: main
ResultId 1