View in EDS

A Protocol to Assess Contextual Factors during Program Impact Evaluation: A Case Study of a STEM Gender Equity Intervention in Higher Education

Saved in:
Bibliographic Details
Title: A Protocol to Assess Contextual Factors during Program Impact Evaluation: A Case Study of a STEM Gender Equity Intervention in Higher Education
Language: English
Authors: Suzanne Nobrega (ORCID 0000-0002-2354-0996), Kasper Edwards, Mazen El Ghaziri, Lauren Giacobbe, Serena Rice, Laura Punnett
Source: American Journal of Evaluation. 2024 45(4):592-609.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 18
Publication Date: 2024
Sponsoring Agency: National Institute for Occupational Safety and Health (NIOSH) (DHHS/CDC)
Contract Number: U19OH008857
U19OH012299
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: STEM Education, Gender Differences, Intervention, Program Evaluation, Outcome Measures, Evaluation Methods, Program Effectiveness, Influences, Context Effect, College Faculty, Diversity (Faculty), Work Environment, Case Studies, Teaching Experience, Sex Fairness
DOI: 10.1177/10982140231152281
ISSN: 1098-2140
1557-0878
Abstract: Program evaluations that lack experimental design often fail to produce evidence of impact because there is no available control group. Theory-based evaluations can generate evidence of a program's causal effects if evaluators collect evidence along the theorized causal chain and identify possible competing causes. However, few methods are available for assessing competing causes in the program environment. Effect Modifier Assessment (EMA) is a method previously used in smaller-scale studies to assess possible competing causes of observed changes following an intervention. In our case study of a university gender equity intervention, EMA generated useful evidence of competing causes to augment program evaluation. Top-down administrative culture, poor experiences with hiring and promotion, and workload were identified as impeding forces that might have reduced program benefits. The EMA addresses a methodological gap in theory-based evaluation and might be useful in a variety of program settings.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1452432
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1452432
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Protocol to Assess Contextual Factors during Program Impact Evaluation: A Case Study of a STEM Gender Equity Intervention in Higher Education
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Suzanne+Nobrega%22">Suzanne Nobrega</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2354-0996">0000-0002-2354-0996</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kasper+Edwards%22">Kasper Edwards</searchLink><br /><searchLink fieldCode="AR" term="%22Mazen+El+Ghaziri%22">Mazen El Ghaziri</searchLink><br /><searchLink fieldCode="AR" term="%22Lauren+Giacobbe%22">Lauren Giacobbe</searchLink><br /><searchLink fieldCode="AR" term="%22Serena+Rice%22">Serena Rice</searchLink><br /><searchLink fieldCode="AR" term="%22Laura+Punnett%22">Laura Punnett</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22American+Journal+of+Evaluation%22"><i>American Journal of Evaluation</i></searchLink>. 2024 45(4):592-609.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 18
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Institute for Occupational Safety and Health (NIOSH) (DHHS/CDC)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: U19OH008857<br />U19OH012299
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Gender+Differences%22">Gender Differences</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Outcome+Measures%22">Outcome Measures</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Influences%22">Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Context+Effect%22">Context Effect</searchLink><br /><searchLink fieldCode="DE" term="%22College+Faculty%22">College Faculty</searchLink><br /><searchLink fieldCode="DE" term="%22Diversity+%28Faculty%29%22">Diversity (Faculty)</searchLink><br /><searchLink fieldCode="DE" term="%22Work+Environment%22">Work Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Case+Studies%22">Case Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Experience%22">Teaching Experience</searchLink><br /><searchLink fieldCode="DE" term="%22Sex+Fairness%22">Sex Fairness</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/10982140231152281
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1098-2140<br />1557-0878
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Program evaluations that lack experimental design often fail to produce evidence of impact because there is no available control group. Theory-based evaluations can generate evidence of a program's causal effects if evaluators collect evidence along the theorized causal chain and identify possible competing causes. However, few methods are available for assessing competing causes in the program environment. Effect Modifier Assessment (EMA) is a method previously used in smaller-scale studies to assess possible competing causes of observed changes following an intervention. In our case study of a university gender equity intervention, EMA generated useful evidence of competing causes to augment program evaluation. Top-down administrative culture, poor experiences with hiring and promotion, and workload were identified as impeding forces that might have reduced program benefits. The EMA addresses a methodological gap in theory-based evaluation and might be useful in a variety of program settings.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1452432
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1452432
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/10982140231152281
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
        StartPage: 592
    Subjects:
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Gender Differences
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
      – SubjectFull: Outcome Measures
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Influences
        Type: general
      – SubjectFull: Context Effect
        Type: general
      – SubjectFull: College Faculty
        Type: general
      – SubjectFull: Diversity (Faculty)
        Type: general
      – SubjectFull: Work Environment
        Type: general
      – SubjectFull: Case Studies
        Type: general
      – SubjectFull: Teaching Experience
        Type: general
      – SubjectFull: Sex Fairness
        Type: general
    Titles:
      – TitleFull: A Protocol to Assess Contextual Factors during Program Impact Evaluation: A Case Study of a STEM Gender Equity Intervention in Higher Education
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Suzanne Nobrega
      – PersonEntity:
          Name:
            NameFull: Kasper Edwards
      – PersonEntity:
          Name:
            NameFull: Mazen El Ghaziri
      – PersonEntity:
          Name:
            NameFull: Lauren Giacobbe
      – PersonEntity:
          Name:
            NameFull: Serena Rice
      – PersonEntity:
          Name:
            NameFull: Laura Punnett
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1098-2140
            – Type: issn-electronic
              Value: 1557-0878
          Numbering:
            – Type: volume
              Value: 45
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: American Journal of Evaluation
              Type: main
ResultId 1