Quantifying 'Promising Trials Bias' in Randomized Controlled Trials in Education
Saved in:
| Title: | Quantifying 'Promising Trials Bias' in Randomized Controlled Trials in Education |
|---|---|
| Language: | English |
| Authors: | Sims, Sam (ORCID |
| Source: | Journal of Research on Educational Effectiveness. 2023 16(4):663-680. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 18 |
| Publication Date: | 2023 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Randomized Controlled Trials, Educational Research, Effect Size, Intervention, Statistical Bias, Research Problems, Statistical Analysis, Statistical Significance |
| DOI: | 10.1080/19345747.2022.2090470 |
| ISSN: | 1934-5747 1934-5739 |
| Abstract: | Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (p < [alpha]) We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. |
| Abstractor: | As Provided |
| Entry Date: | 2023 |
| Accession Number: | EJ1402265 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (p < [alpha]) We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. |
|---|---|
| ISSN: | 1934-5747 1934-5739 |
| DOI: | 10.1080/19345747.2022.2090470 |
