P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods
Saved in:
| Title: | P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods |
|---|---|
| Language: | English |
| Authors: | Maya B. Mathur (ORCID |
| Source: | Research Synthesis Methods. 2024 15(3):483-499. |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 17 |
| Publication Date: | 2024 |
| Sponsoring Agency: | National Institutes of Health (NIH) (DHHS) |
| Contract Number: | P30CA124435 P30DK116074 R01LM013866 UL1TR003142 |
| Document Type: | Journal Articles Reports - Evaluative |
| Descriptors: | Meta Analysis, Research Reports, Research Methodology, Research Problems, Programming Languages, Web Sites, Robustness (Statistics) |
| DOI: | 10.1002/jrsm.1701 |
| ISSN: | 1759-2879 1759-2887 |
| Abstract: | As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results "within" their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Accession Number: | EJ1421897 |
| Database: | ERIC |
| Abstract: | As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results "within" their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS. |
|---|---|
| ISSN: | 1759-2879 1759-2887 |
| DOI: | 10.1002/jrsm.1701 |
