Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083
Saved in:
| Title: | Examining the Relationship between Randomization Strategies and Control Group Crossover in Higher Education Interventions. EdWorkingPaper No. 24-1083 |
|---|---|
| Language: | English |
| Authors: | Catherine Mata, Katharine Meyer, Lindsay Page, Annenberg Institute for School Reform at Brown University |
| Source: | Annenberg Institute for School Reform at Brown University. 2024. |
| Availability: | Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: AISR_Info@brown.edu; Web site: http://www.annenberginstitute.org |
| Peer Reviewed: | N |
| Page Count: | 16 |
| Publication Date: | 2024 |
| Sponsoring Agency: | Ascendium Education Group, Inc. |
| Document Type: | Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Chemistry, Science Instruction, Undergraduate Students, Large Group Instruction, Artificial Intelligence, Randomized Controlled Trials, Test Bias, Pilot Projects, Test Reliability, Inferences, Comparative Testing, Testing |
| Geographic Terms: | Georgia (Atlanta) |
| Abstract: | This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross over into the treatment group, thus biasing treatment effect estimates. This study provides empirical evidence from a pilot intervention in two sections of a college-level introductory chemistry course, where a course-specific chatbot was introduced. We tested two randomization strategies: simple student-level randomization and laboratory-level randomization. We hypothesized that the greatest risk for crossover would have occurred under the simple individual randomization approach, however, no crossover occurred in either condition. Survey responses and system usage data indicate that this was not due to a lack of interaction among students or disinterest in the chatbot. These findings suggest that student-level randomization, even in an in-person course setting, can proceed with minimal risk of contamination for testing our focal intervention. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | ED663610 |
| Database: | ERIC |
| Abstract: | This article examines the risk of crossover contamination in individual-level randomization, a common concern in experimental research, in the context of a large-enrollment college course. While individual-level randomization is more efficient for assessing program effectiveness, it also increases the potential for control group students to cross over into the treatment group, thus biasing treatment effect estimates. This study provides empirical evidence from a pilot intervention in two sections of a college-level introductory chemistry course, where a course-specific chatbot was introduced. We tested two randomization strategies: simple student-level randomization and laboratory-level randomization. We hypothesized that the greatest risk for crossover would have occurred under the simple individual randomization approach, however, no crossover occurred in either condition. Survey responses and system usage data indicate that this was not due to a lack of interaction among students or disinterest in the chatbot. These findings suggest that student-level randomization, even in an in-person course setting, can proceed with minimal risk of contamination for testing our focal intervention. |
|---|
