View in EDS

Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083

Saved in:
Bibliographic Details
Title: Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083
Language: English
Authors: Catherine Mata, Katharine Meyer, Lindsay Page, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2024.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: AISR_Info@brown.edu; Web site: http://www.annenberginstitute.org
Peer Reviewed: N
Page Count: 16
Publication Date: 2024
Sponsoring Agency: Ascendium Education Group, Inc.
Document Type: Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Chemistry, Science Instruction, Undergraduate Students, Large Group Instruction, Artificial Intelligence, Randomized Controlled Trials, Test Bias, Pilot Projects, Test Reliability, Inferences, Comparative Testing, Testing
Geographic Terms: Georgia (Atlanta)
Abstract: This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross over into the treatment group, thus biasing treatment effect estimates. This study provides empirical evidence from a pilot intervention in two sections of a college-level introductory chemistry course, where a course-specific chatbot was introduced. We tested two randomization strategies: simple student-level randomization and laboratory-level randomization. We hypothesized that the greatest risk for crossover would have occurred under the simple individual randomization approach, however, no crossover occurred in either condition. Survey responses and system usage data indicate that this was not due to a lack of interaction among students or disinterest in the chatbot. These findings suggest that student-level randomization, even in an in-person course setting, can proceed with minimal risk of contamination for testing our focal intervention.
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED663610
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED663610
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED663610
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Catherine+Mata%22">Catherine Mata</searchLink><br /><searchLink fieldCode="AR" term="%22Katharine+Meyer%22">Katharine Meyer</searchLink><br /><searchLink fieldCode="AR" term="%22Lindsay+Page%22">Lindsay Page</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: AISR_Info@brown.edu; Web site: http://www.annenberginstitute.org
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Ascendium Education Group, Inc.
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Chemistry%22">Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Instruction%22">Science Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Large+Group+Instruction%22">Large Group Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Pilot+Projects%22">Pilot Projects</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Inferences%22">Inferences</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Testing%22">Comparative Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Testing%22">Testing</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Georgia+%28Atlanta%29%22">Georgia (Atlanta)</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross over into the treatment group, thus biasing treatment effect estimates. This study provides empirical evidence from a pilot intervention in two sections of a college-level introductory chemistry course, where a course-specific chatbot was introduced. We tested two randomization strategies: simple student-level randomization and laboratory-level randomization. We hypothesized that the greatest risk for crossover would have occurred under the simple individual randomization approach, however, no crossover occurred in either condition. Survey responses and system usage data indicate that this was not due to a lack of interaction among students or disinterest in the chatbot. These findings suggest that student-level randomization, even in an in-person course setting, can proceed with minimal risk of contamination for testing our focal intervention.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED663610
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED663610
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
    Subjects:
      – SubjectFull: Chemistry
        Type: general
      – SubjectFull: Science Instruction
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Large Group Instruction
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Pilot Projects
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Inferences
        Type: general
      – SubjectFull: Comparative Testing
        Type: general
      – SubjectFull: Testing
        Type: general
      – SubjectFull: Georgia (Atlanta)
        Type: general
    Titles:
      – TitleFull: Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Catherine Mata
      – PersonEntity:
          Name:
            NameFull: Katharine Meyer
      – PersonEntity:
          Name:
            NameFull: Lindsay Page
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 11
              Type: published
              Y: 2024
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1