View in EDS PDF Full Text

P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods

Saved in:
Bibliographic Details
Title: P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods
Language: English
Authors: Maya B. Mathur (ORCID 0000-0001-6698-2607)
Source: Research Synthesis Methods. 2024 15(3):483-499.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 17
Publication Date: 2024
Sponsoring Agency: National Institutes of Health (NIH) (DHHS)
Contract Number: P30CA124435
P30DK116074
R01LM013866
UL1TR003142
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Meta Analysis, Research Reports, Research Methodology, Research Problems, Programming Languages, Web Sites, Robustness (Statistics)
DOI: 10.1002/jrsm.1701
ISSN: 1759-2879
1759-2887
Abstract: As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results "within" their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1421897
Database: ERIC
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwG78XnGwEguVUBWHLoXIkiUAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDE6ywUEEJTBNABna5QIBEICBm_frEj7i6giCT6SaaLtNucularKXKRm9ozDFVBBQmFOfHK56S59CJtzUGjW-ShpkZQuGxI3kuAfKqhiOm8taQTo5QcC8w_It-CCf_cjgOJZ_UJS1VQkr-k3p_lnkN0z9hKZaL6uxG3sua0dlXq1mEQlvpuu5ZhZBF3Nv7ZlwiQLhIJIyjGuLzvAYdWJkJD5QAJP_cqqkqDRFaFzF
Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1421897
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Maya+B%2E+Mathur%22">Maya B. Mathur</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6698-2607">0000-0001-6698-2607</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. 2024 15(3):483-499.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 17
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Institutes of Health (NIH) (DHHS)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: P30CA124435<br />P30DK116074<br />R01LM013866<br />UL1TR003142
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Reports%22">Research Reports</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Problems%22">Research Problems</searchLink><br /><searchLink fieldCode="DE" term="%22Programming+Languages%22">Programming Languages</searchLink><br /><searchLink fieldCode="DE" term="%22Web+Sites%22">Web Sites</searchLink><br /><searchLink fieldCode="DE" term="%22Robustness+%28Statistics%29%22">Robustness (Statistics)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1701
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879<br />1759-2887
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results "within" their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1421897
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1421897
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1701
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
        StartPage: 483
    Subjects:
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Research Reports
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Research Problems
        Type: general
      – SubjectFull: Programming Languages
        Type: general
      – SubjectFull: Web Sites
        Type: general
      – SubjectFull: Robustness (Statistics)
        Type: general
    Titles:
      – TitleFull: P-Hacking in Meta-Analyses: A Formalization and New Meta-Analytic Methods
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Maya B. Mathur
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
            – Type: issn-electronic
              Value: 1759-2887
          Numbering:
            – Type: volume
              Value: 15
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1