View in EDS

2017-2019 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program

Saved in:
Bibliographic Details
Title: 2017-2019 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
Language: English
Authors: Phelan, Julia, Egger, Jeffrey, Kim, Junok, Choi, Kilchan, Keum, Eunhee, Chung, Gregory K. W. K., Baker, Eva L., National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
Source: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 2021.
Availability: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
Peer Reviewed: N
Page Count: 167
Publication Date: 2021
Sponsoring Agency: National Math and Science Initiative
Department of Education (ED)
Document Type: Reports - Research
Education Level: High Schools
Secondary Education
Descriptors: College Readiness, Nonprofit Organizations, Program Evaluation, STEM Education, High School Students, Advanced Placement, Comparative Analysis, Educational Innovation, Grants, Outcomes of Education, School Culture, Fidelity, Program Implementation, Probability, Scores, Faculty Development, Teacher Attitudes, Teacher Surveys, Science Achievement, Mathematics Achievement, Science Tests, Mathematics Tests, Mentors, Incentives, Educational Resources, Program Descriptions, English, Language Arts, School Size, Student Participation
Geographic Terms: California, Illinois, Georgia, Louisiana, Michigan, Missouri, Ohio, North Dakota, Pennsylvania, Texas
Abstract: The National Math + Science Initiative (NMSI) is a nonprofit organization committed to improving educational outcomes that traces its roots back to the early 1990s. NMSI's College Readiness Program (CRP) is a long-standing program with the goal of promoting STEM education in high schools to improve students' preparation for college. The three-year program provides teacher, student, and school supports to promote high school students' success in English, mathematics, and science Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 27 schools in the 2016-2017 school year (Treatment Schools) and in 21 schools in the 2017-2018 school year (Delayed Treatment Schools), collectively identified as Program Schools. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 48 CRP schools and 48 Comparison Schools in 10 states. The evaluation of the CRP consisted of three parts: (1) measuring the program's impact on selected student AP exam outcomes; (2) determining the impact of the program on school perspectives and culture; and (3) assessing of the fidelity of implementation of the CRP at the school level. AP exam data from 48 Treatment Schools, with a total of 8,778 exams in 2018 and 9,378 in 2019, and 48 matched control schools, with 7,505 exams in 2018 and 2019 in Year 3, were analyzed for this study. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools. The analyses revealed that in 2018 the probability of a student taking an AP exam in the Program Schools was, on average, 7% higher than the paired Comparison Schools, and the difference was statistically significant. And in 2019 the effect was even greater with the probability of taking an AP exam being significantly higher in Program Schools (18%) than in Comparison Schools (3%). When looking at the probability of an exam yielding a qualifying score, in 2018, the HGLM analyses found no significant difference between the two groups. In 2019, however, exams taken at the Program Schools had a significantly higher overall probability (2%) of receiving a qualifying score than the Comparison Schools (0%). These analyses compared results to the total school population. In the next analyses, we looked only at those students who took AP exams. In 2018, overall, the fitted probability of achieving a qualifying score among the exams taken was 8% in the Program Schools, compared to 22% in the Comparison Schools. In 2019, however, the difference between the Program Schools (7%) and the Comparison Schools (9%) was not statistically significant. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. In 2018, results indicated that 43 out of 48 schools (90%) achieved 80% or better implementation fidelity, for an average fidelity score of 89%. Four schools achieved a perfect 100% fidelity score. In 2019, 88% of schools achieved 80% or better implementation fidelity. Ten schools achieved a perfect 100% fidelity score. In 2018 in more than 75% of schools, not all teachers fulfilled their requirements for attending the required teacher training sessions, and so this component was not implemented with fidelity. In 2019, this picture improved a little with 15 schools (31%) meeting the 80% threshold. Teacher stipends, administrator bonuses, and student qualifying score awards were paid as expected. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports. Mentoring was chosen, across all years, as the least effective program component. When prompted for the second most effective CRP component, the same number of teachers selected the funding of classroom and lab supplies as did teacher training. [For "2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program," see ED615910.]
Abstractor: As Provided
What Works Clearinghouse (WWC) Reviewed: Does Not Meet Evidence Standards
WWC Study Page: https://ies.ed.gov/ncee/wwc/Study/90501
Entry Date: 2021
Accession Number: ED615919
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED615919
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED615919
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: 2017-2019 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Phelan%2C+Julia%22">Phelan, Julia</searchLink><br /><searchLink fieldCode="AR" term="%22Egger%2C+Jeffrey%22">Egger, Jeffrey</searchLink><br /><searchLink fieldCode="AR" term="%22Kim%2C+Junok%22">Kim, Junok</searchLink><br /><searchLink fieldCode="AR" term="%22Choi%2C+Kilchan%22">Choi, Kilchan</searchLink><br /><searchLink fieldCode="AR" term="%22Keum%2C+Eunhee%22">Keum, Eunhee</searchLink><br /><searchLink fieldCode="AR" term="%22Chung%2C+Gregory+K%2E+W%2E+K%2E%22">Chung, Gregory K. W. K.</searchLink><br /><searchLink fieldCode="AR" term="%22Baker%2C+Eva+L%2E%22">Baker, Eva L.</searchLink><br /><searchLink fieldCode="AR" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22">National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22National+Center+for+Research+on+Evaluation%2C+Standards%2C+and+Student+Testing+%28CRESST%29%22"><i>National Center for Research on Evaluation, Standards, and Student Testing (CRESST)</i></searchLink>. 2021.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 300 Charles E Young Drive N, GSE&IS Building 3rd Floor, Mailbox 951522, Los Angeles, CA 90095-1522. Tel: 310-206-1532; Fax: 310-825-3883; Web site: http://www.cresst.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 167
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Math and Science Initiative<br />Department of Education (ED)
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22College+Readiness%22">College Readiness</searchLink><br /><searchLink fieldCode="DE" term="%22Nonprofit+Organizations%22">Nonprofit Organizations</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Advanced+Placement%22">Advanced Placement</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Innovation%22">Educational Innovation</searchLink><br /><searchLink fieldCode="DE" term="%22Grants%22">Grants</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Education%22">Outcomes of Education</searchLink><br /><searchLink fieldCode="DE" term="%22School+Culture%22">School Culture</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Implementation%22">Program Implementation</searchLink><br /><searchLink fieldCode="DE" term="%22Probability%22">Probability</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Development%22">Faculty Development</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Attitudes%22">Teacher Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Surveys%22">Teacher Surveys</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Achievement%22">Science Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Tests%22">Science Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Mentors%22">Mentors</searchLink><br /><searchLink fieldCode="DE" term="%22Incentives%22">Incentives</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Resources%22">Educational Resources</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Descriptions%22">Program Descriptions</searchLink><br /><searchLink fieldCode="DE" term="%22English%22">English</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Arts%22">Language Arts</searchLink><br /><searchLink fieldCode="DE" term="%22School+Size%22">School Size</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Participation%22">Student Participation</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22California%22">California</searchLink><br /><searchLink fieldCode="DE" term="%22Illinois%22">Illinois</searchLink><br /><searchLink fieldCode="DE" term="%22Georgia%22">Georgia</searchLink><br /><searchLink fieldCode="DE" term="%22Louisiana%22">Louisiana</searchLink><br /><searchLink fieldCode="DE" term="%22Michigan%22">Michigan</searchLink><br /><searchLink fieldCode="DE" term="%22Missouri%22">Missouri</searchLink><br /><searchLink fieldCode="DE" term="%22Ohio%22">Ohio</searchLink><br /><searchLink fieldCode="DE" term="%22North+Dakota%22">North Dakota</searchLink><br /><searchLink fieldCode="DE" term="%22Pennsylvania%22">Pennsylvania</searchLink><br /><searchLink fieldCode="DE" term="%22Texas%22">Texas</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The National Math + Science Initiative (NMSI) is a nonprofit organization committed to improving educational outcomes that traces its roots back to the early 1990s. NMSI's College Readiness Program (CRP) is a long-standing program with the goal of promoting STEM education in high schools to improve students' preparation for college. The three-year program provides teacher, student, and school supports to promote high school students' success in English, mathematics, and science Advanced Placement (AP) courses, with a focus on students who are traditionally underrepresented in the targeted AP courses. Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, the CRP was implemented in 27 schools in the 2016-2017 school year (Treatment Schools) and in 21 schools in the 2017-2018 school year (Delayed Treatment Schools), collectively identified as Program Schools. CRESST conducted an independent evaluation of the impact of the CRP on students' AP outcomes using a randomized cluster trial with 48 CRP schools and 48 Comparison Schools in 10 states. The evaluation of the CRP consisted of three parts: (1) measuring the program's impact on selected student AP exam outcomes; (2) determining the impact of the program on school perspectives and culture; and (3) assessing of the fidelity of implementation of the CRP at the school level. AP exam data from 48 Treatment Schools, with a total of 8,778 exams in 2018 and 9,378 in 2019, and 48 matched control schools, with 7,505 exams in 2018 and 2019 in Year 3, were analyzed for this study. Program impact was evaluated using a 2-level hierarchical generalized linear model (HGLM) with students nested within schools. The analyses revealed that in 2018 the probability of a student taking an AP exam in the Program Schools was, on average, 7% higher than the paired Comparison Schools, and the difference was statistically significant. And in 2019 the effect was even greater with the probability of taking an AP exam being significantly higher in Program Schools (18%) than in Comparison Schools (3%). When looking at the probability of an exam yielding a qualifying score, in 2018, the HGLM analyses found no significant difference between the two groups. In 2019, however, exams taken at the Program Schools had a significantly higher overall probability (2%) of receiving a qualifying score than the Comparison Schools (0%). These analyses compared results to the total school population. In the next analyses, we looked only at those students who took AP exams. In 2018, overall, the fitted probability of achieving a qualifying score among the exams taken was 8% in the Program Schools, compared to 22% in the Comparison Schools. In 2019, however, the difference between the Program Schools (7%) and the Comparison Schools (9%) was not statistically significant. Fidelity of implementation was evaluated using a fidelity matrix approach (required as part of the evaluation of the i3 program), which showed that not all elements of the program were implemented with high fidelity. In 2018, results indicated that 43 out of 48 schools (90%) achieved 80% or better implementation fidelity, for an average fidelity score of 89%. Four schools achieved a perfect 100% fidelity score. In 2019, 88% of schools achieved 80% or better implementation fidelity. Ten schools achieved a perfect 100% fidelity score. In 2018 in more than 75% of schools, not all teachers fulfilled their requirements for attending the required teacher training sessions, and so this component was not implemented with fidelity. In 2019, this picture improved a little with 15 schools (31%) meeting the 80% threshold. Teacher stipends, administrator bonuses, and student qualifying score awards were paid as expected. Teacher survey data indicated that teachers found the training and professional development activities provided by the CRP to be the most beneficial program supports. Mentoring was chosen, across all years, as the least effective program component. When prompted for the second most effective CRP component, the same number of teachers selected the funding of classroom and lab supplies as did teacher training. [For "2016-2017 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program," see ED615910.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: What Works Clearinghouse (WWC) Reviewed
  Group: SrcInfo
  Data: Does Not Meet Evidence Standards
– Name: URL
  Label: WWC Study Page
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://ies.ed.gov/ncee/wwc/Study/90501" linkWindow="_blank">https://ies.ed.gov/ncee/wwc/Study/90501</link>
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED615919
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED615919
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 167
    Subjects:
      – SubjectFull: College Readiness
        Type: general
      – SubjectFull: Nonprofit Organizations
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Advanced Placement
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Educational Innovation
        Type: general
      – SubjectFull: Grants
        Type: general
      – SubjectFull: Outcomes of Education
        Type: general
      – SubjectFull: School Culture
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Program Implementation
        Type: general
      – SubjectFull: Probability
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Faculty Development
        Type: general
      – SubjectFull: Teacher Attitudes
        Type: general
      – SubjectFull: Teacher Surveys
        Type: general
      – SubjectFull: Science Achievement
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Science Tests
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Mentors
        Type: general
      – SubjectFull: Incentives
        Type: general
      – SubjectFull: Educational Resources
        Type: general
      – SubjectFull: Program Descriptions
        Type: general
      – SubjectFull: English
        Type: general
      – SubjectFull: Language Arts
        Type: general
      – SubjectFull: School Size
        Type: general
      – SubjectFull: Student Participation
        Type: general
      – SubjectFull: California
        Type: general
      – SubjectFull: Illinois
        Type: general
      – SubjectFull: Georgia
        Type: general
      – SubjectFull: Louisiana
        Type: general
      – SubjectFull: Michigan
        Type: general
      – SubjectFull: Missouri
        Type: general
      – SubjectFull: Ohio
        Type: general
      – SubjectFull: North Dakota
        Type: general
      – SubjectFull: Pennsylvania
        Type: general
      – SubjectFull: Texas
        Type: general
    Titles:
      – TitleFull: 2017-2019 Implementation Evaluation of the National Math and Science Initiative's College Readiness Program
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
      – PersonEntity:
          Name:
            NameFull: Phelan, Julia
      – PersonEntity:
          Name:
            NameFull: Egger, Jeffrey
      – PersonEntity:
          Name:
            NameFull: Kim, Junok
      – PersonEntity:
          Name:
            NameFull: Choi, Kilchan
      – PersonEntity:
          Name:
            NameFull: Keum, Eunhee
      – PersonEntity:
          Name:
            NameFull: Chung, Gregory K. W. K.
      – PersonEntity:
          Name:
            NameFull: Baker, Eva L.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2021
          Titles:
            – TitleFull: National Center for Research on Evaluation, Standards, and Student Testing (CRESST)
              Type: main
ResultId 1