Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education.
Saved in:
| Title: | Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education. |
|---|---|
| Authors: | Sims, Sam1 (AUTHOR) s.sims@ucl.ac.uk, Anders, Jake1 (AUTHOR), Inglis, Matthew2 (AUTHOR), Lortie-Forgues, Hugues2 (AUTHOR) |
| Source: | Journal of Research on Educational Effectiveness. Oct-Dec2023, Vol. 16 Issue 4, p663-680. 18p. |
| Subject Terms: | *Computers in education, Randomized controlled trials, Crime & the press |
| Abstract: | Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( p < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Research on Educational Effectiveness is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( p < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 19345747 |
| DOI: | 10.1080/19345747.2022.2090470 |
