View in EDS HTML Full Text PDF Full Text

Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education.

Saved in:
Bibliographic Details
Title: Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education.
Authors: Sims, Sam1 (AUTHOR) s.sims@ucl.ac.uk, Anders, Jake1 (AUTHOR), Inglis, Matthew2 (AUTHOR), Lortie-Forgues, Hugues2 (AUTHOR)
Source: Journal of Research on Educational Effectiveness. Oct-Dec2023, Vol. 16 Issue 4, p663-680. 18p.
Subject Terms: *Computers in education, Randomized controlled trials, Crime & the press
Abstract: Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( p < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. [ABSTRACT FROM AUTHOR]
Copyright of Journal of Research on Educational Effectiveness is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Education Research Complete
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: ehh
DbLabel: Education Research Complete
An: 173116905
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Quantifying &quot;Promising Trials Bias&quot; in Randomized Controlled Trials in Education.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Sims%2C+Sam%22&quot;&gt;Sims, Sam&lt;/searchLink&gt;&lt;relatesTo&gt;1&lt;/relatesTo&gt; (AUTHOR)&lt;i&gt; s.sims@ucl.ac.uk&lt;/i&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Anders%2C+Jake%22&quot;&gt;Anders, Jake&lt;/searchLink&gt;&lt;relatesTo&gt;1&lt;/relatesTo&gt; (AUTHOR)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Inglis%2C+Matthew%22&quot;&gt;Inglis, Matthew&lt;/searchLink&gt;&lt;relatesTo&gt;2&lt;/relatesTo&gt; (AUTHOR)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Lortie-Forgues%2C+Hugues%22&quot;&gt;Lortie-Forgues, Hugues&lt;/searchLink&gt;&lt;relatesTo&gt;2&lt;/relatesTo&gt; (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%22Journal+of+Research+on+Educational+Effectiveness%22&quot;&gt;Journal of Research on Educational Effectiveness&lt;/searchLink&gt;. Oct-Dec2023, Vol. 16 Issue 4, p663-680. 18p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Computers+in+education%22&quot;&gt;Computers in education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Randomized+controlled+trials%22&quot;&gt;Randomized controlled trials&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Crime+%26+the+press%22&quot;&gt;Crime &amp; the press&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( p &lt; α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: &lt;i&gt;Copyright of Journal of Research on Educational Effectiveness is the property of Taylor &amp; Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder&#39;s express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.&lt;/i&gt; (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=173116905
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2022.2090470
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
        StartPage: 663
    Subjects:
      – SubjectFull: Computers in education
        Type: general
      – SubjectFull: Randomized controlled trials
        Type: general
      – SubjectFull: Crime & the press
        Type: general
    Titles:
      – TitleFull: Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sims, Sam
      – PersonEntity:
          Name:
            NameFull: Anders, Jake
      – PersonEntity:
          Name:
            NameFull: Inglis, Matthew
      – PersonEntity:
          Name:
            NameFull: Lortie-Forgues, Hugues
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Text: Oct-Dec2023
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 19345747
          Numbering:
            – Type: volume
              Value: 16
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1