Estimating Publication Bias in Meta-Analyses of Peer-Reviewed Studies: A Meta-Meta-Analysis across Disciplines and Journal Tiers
Saved in:
| Title: | Estimating Publication Bias in Meta-Analyses of Peer-Reviewed Studies: A Meta-Meta-Analysis across Disciplines and Journal Tiers |
|---|---|
| Language: | English |
| Authors: | Mathur, Maya B. (ORCID |
| Source: | Research Synthesis Methods. Mar 2021 12(2):176-191. |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 16 |
| Publication Date: | 2021 |
| Sponsoring Agency: | National Institutes of Health (DHHS) |
| Contract Number: | R01CA222147 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Periodicals, Peer Evaluation, Bias, Meta Analysis, Research Reports, Intellectual Disciplines, Validity, Scientific Research |
| DOI: | 10.1002/jrsm.1464 |
| ISSN: | 1759-2879 |
| Abstract: | Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in "PLoS One," top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in "PLoS One," 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results. |
| Abstractor: | As Provided |
| Entry Date: | 2021 |
| Accession Number: | EJ1288277 |
| Database: | ERIC |
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFutflh-_4hYa6oA_K5PxuGAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLuhBs_uK6RUzYX2oQIBEICBmuop4OKK_f0NzJu6h6VzcR1KSKaHh12mJyZpoAcgHEaFs8cnbTL7l_rZ9i-2oONQJluhYCMmO28aCBv_fKUc6I1Q6lkRC50Ne4yV-S-JzccXlRJqBN7bsydnJVnY_3phufuimKXrxrtURgcqPbmjdzoQhjZ4BPUk8xXrKzUSI9xxsv3_lhxAUFrxPlv5XFkxB_D-UXQ6U-s-A2A= Text: Availability: 0 |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1288277 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Estimating Publication Bias in Meta-Analyses of Peer-Reviewed Studies: A Meta-Meta-Analysis across Disciplines and Journal Tiers – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Mathur%2C+Maya+B%2E%22">Mathur, Maya B.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6698-2607">0000-0001-6698-2607</externalLink>)<br /><searchLink fieldCode="AR" term="%22VanderWeele%2C+Tyler+J%2E%22">VanderWeele, Tyler J.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6112-0239">0000-0002-6112-0239</externalLink>) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. Mar 2021 12(2):176-191. – Name: Avail Label: Availability Group: Avail Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 16 – Name: DatePubCY Label: Publication Date Group: Date Data: 2021 – Name: SourceSuprt Label: Sponsoring Agency Group: SrcSuprt Data: National Institutes of Health (DHHS) – Name: NumberContract Label: Contract Number Group: NumCntrct Data: R01CA222147 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Periodicals%22">Periodicals</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+Evaluation%22">Peer Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Reports%22">Research Reports</searchLink><br /><searchLink fieldCode="DE" term="%22Intellectual+Disciplines%22">Intellectual Disciplines</searchLink><br /><searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Scientific+Research%22">Scientific Research</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1002/jrsm.1464 – Name: ISSN Label: ISSN Group: ISSN Data: 1759-2879 – Name: Abstract Label: Abstract Group: Ab Data: Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in "PLoS One," top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in "PLoS One," 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2021 – Name: AN Label: Accession Number Group: ID Data: EJ1288277 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1288277 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1002/jrsm.1464 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 16 StartPage: 176 Subjects: – SubjectFull: Periodicals Type: general – SubjectFull: Peer Evaluation Type: general – SubjectFull: Bias Type: general – SubjectFull: Meta Analysis Type: general – SubjectFull: Research Reports Type: general – SubjectFull: Intellectual Disciplines Type: general – SubjectFull: Validity Type: general – SubjectFull: Scientific Research Type: general Titles: – TitleFull: Estimating Publication Bias in Meta-Analyses of Peer-Reviewed Studies: A Meta-Meta-Analysis across Disciplines and Journal Tiers Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Mathur, Maya B. – PersonEntity: Name: NameFull: VanderWeele, Tyler J. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 03 Type: published Y: 2021 Identifiers: – Type: issn-print Value: 1759-2879 Numbering: – Type: volume Value: 12 – Type: issue Value: 2 Titles: – TitleFull: Research Synthesis Methods Type: main |
| ResultId | 1 |
