View in EDS HTML Full Text PDF Full Text

Sensitivity Analysis for the Interactive Effects of Internal Bias and Publication Bias in Meta-Analyses

Saved in:
Bibliographic Details
Title: Sensitivity Analysis for the Interactive Effects of Internal Bias and Publication Bias in Meta-Analyses
Language: English
Authors: Maya B. Mathur (ORCID 0000-0001-6698-2607)
Source: Research Synthesis Methods. 2024 15(1):21-43.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 23
Publication Date: 2024
Sponsoring Agency: National Institutes of Health (NIH) (DHHS)
Contract Number: P30CA124435
P30DK116074
R01LM013866
UL1TR003142
Document Type: Journal Articles
Reports - Research
Descriptors: Meta Analysis, Attribution Theory, Publications, Bias, Research Methodology, Programming Languages
DOI: 10.1002/jrsm.1667
ISSN: 1759-2879
1759-2887
Abstract: Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1405345
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwF1OKQlpeJTariRBXIZTRxUAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDcFYxYuurx7FvoX4wIBEICBmxitmLqYLairEns-P1CT3nQeaK1VHkYAPTIp2c0edY1BAt8nJyauY9YXLrp7Osq_it-y2uHyqmbg7edVC1Rj6HkPl41_6RL8DUGe8j6IcTMZRgoMN3UCwV67viyFeqynXY-ToV9Y9HkfRWWhzenOg5bNeC-eZAy0fnjiQ3rO-6DCfpIdHmFP_5a2UilzUYIcnjMQF9u_w4-kz_kQ
Text:
  Availability: 1
  Value: <anid>AN0174546109;[bdct]01jan.24;2024Jan03.05:07;v2.2.500</anid> <title id="AN0174546109-1">Sensitivity analysis for the interactive effects of internal bias and publication bias in meta‐analyses </title> <p>Meta‐analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (<reflink idref="bib1" id="ref1">1</reflink>) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (<reflink idref="bib2" id="ref2">2</reflink>) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst‐case" form of publication bias. Robust estimation methods accommodate non‐normal effects, small meta‐analyses, and clustered estimates. As we illustrate by re‐analyzing published meta‐analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).</p> <p>Keywords: bias analysis; file drawer; internal validity; selective reporting</p> <hd id="AN0174546109-2">INTRODUCTION</hd> <p>Meta‐analyses critically shape clinical guidelines and policy,[[<reflink idref="bib1" id="ref3">1</reflink>]] but several problems can undermine their credibility. Meta‐analyzed studies can be compromised by both internal bias (e.g., confounding in nonrandomized studies) and publication bias, producing meta‐analysis estimates that are too large, too small, or in the wrong direction. Citing these concerns, researchers in multiple disciplines have challenged the credibility of meta‐analyses.[[<reflink idref="bib3" id="ref4">3</reflink>], [<reflink idref="bib5" id="ref5">5</reflink>]] Although existing sensitivity analyses can help characterize how meta‐analytic estimates might be affected either by certain forms of internal bias or, alternatively, by publication bias, these methods do not assess the biases' <emph>combined</emph> effects. In fact, the biases generally do not operate additively, and so are not straightforward to intuit or model using existing methods. For example, if the severity or direction of internal bias differs across studies, then publication bias that selects for significant, positive results will also indirectly select for studies with more positive internal bias, creating a superadditive combined bias.</p> <p>Regarding publication bias, the numerous existing methods mostly fall into two categories.[[<reflink idref="bib6" id="ref6">6</reflink>]] First, classical methods based on the funnel plot assess whether small studies have systematically larger point estimates than large studies.[[<reflink idref="bib8" id="ref7">8</reflink>]] Second, selection models specify the underlying distribution of population effects (i.e., prior to publication bias) and specify a model for how a study's publication probability depends on, for example, its <emph>p</emph>‐value.[[<reflink idref="bib10" id="ref8">10</reflink>], [<reflink idref="bib12" id="ref9">12</reflink>]] The present work builds upon our previous sensitivity analyses for publication bias, which were conceptually related to selection models.[<reflink idref="bib14" id="ref10">14</reflink>] Those methods considered publication bias in which affirmative results (defined as those with positive point estimates and a two‐tailed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo><</mo><mn>0.05</mn></mrow></math> </ephtml> ) are more likely to be published than nonaffirmative results (defined as those with negative point estimates or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>≥</mo><mn>0.05</mn></mrow></math> </ephtml> ).[<reflink idref="bib14" id="ref11">14</reflink>] In this context, the severity of publication bias could be expressed as the "selection ratio" by which affirmative studies are more likely to be published than nonaffirmative studies. For a given selection ratio, we showed that a bias‐corrected estimate could be obtained by weighting each study inversely to its publication probability.[<reflink idref="bib14" id="ref12">14</reflink>] We further showed that a meta‐analytic point estimate corrected for hypothetical worst‐case publication bias of this form (i.e., such that affirmative studies are almost infinitely more likely to be published) could be obtained simply by conducting a standard meta‐analysis of only the nonaffirmative studies. This result arose as a special case of the bias‐corrected estimate in which the inverse‐probability weight for each nonaffirmative study approached infinity. For some meta‐analyses, this worst‐case method indicated that no amount of publication bias under the assumed model could "explain away" the results.[[<reflink idref="bib14" id="ref13">14</reflink>]]</p> <p>Regarding internal bias, some existing methods take a two‐stage approach, first adjusting each study's estimate and then meta‐analyzing these bias‐corrected estimates.[[<reflink idref="bib16" id="ref14">16</reflink>], [<reflink idref="bib18" id="ref15">18</reflink>]] Other methods take a one‐stage approach by specifying the distribution of bias across studies, rather than in each study individually.[[<reflink idref="bib19" id="ref16">19</reflink>], [<reflink idref="bib21" id="ref17">21</reflink>]] The present work is related to our previous one‐stage sensitivity analyses that considered internal bias due specifically to uncontrolled confounding.[[<reflink idref="bib19" id="ref18">19</reflink>], [<reflink idref="bib21" id="ref19">21</reflink>]] Those methods assessed the extent to which the results of a meta‐analysis could change due to a specified average amount (or distribution) of uncontrolled confounding across studies. The severity of uncontrolled confounding was characterized by sensitivity parameters corresponding to the strengths of associations, across studies, of uncontrolled confounder(s) with studies' exposures, with their outcomes, or both.[[<reflink idref="bib19" id="ref20">19</reflink>], [<reflink idref="bib23" id="ref21">23</reflink>]] Given these confounding associations, there exists an upper bound on the amount of bias that could be produced by uncontrolled confounder(s).[<reflink idref="bib23" id="ref22">23</reflink>] This bounding approach permitted the meta‐analytic sensitivity analyses to obviate certain assumptions on the structure of uncontrolled confounding. For example, some earlier methods assumed that there is single, discrete uncontrolled confounding variable[[<reflink idref="bib16" id="ref23">16</reflink>], [<reflink idref="bib22" id="ref24">22</reflink>]] that does not interact with the exposure and that is independent of any measured confounders, conditional on the exposure.[<reflink idref="bib22" id="ref25">22</reflink>] The latter assumption is highly problematic because it is always violated.[[<reflink idref="bib24" id="ref26">24</reflink>]] Recently, analytic bounds similar to those for confounding[<reflink idref="bib23" id="ref27">23</reflink>] have been developed for other types of bias in a causal estimate from a single study, such as differential misclassification or selection bias.[[<reflink idref="bib26" id="ref28">26</reflink>], [<reflink idref="bib28" id="ref29">28</reflink>]] The methods developed in this paper will also accommodate these other types of bias, or alternatively internal bias of unspecified origin.</p> <p>A different form of internal bias can arise from selective reporting of results <emph>within</emph> studies. For example, investigators may " <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hack" by analyzing multiple outcomes and reporting only affirmative results among these analyses, or by fitting multiple models to the same dataset in an attempt to obtain an affirmative estimate.[[<reflink idref="bib30" id="ref30">30</reflink>], [<reflink idref="bib32" id="ref31">32</reflink>]] Standard methods for publication bias typically assume that selection operates on a collection of point estimates that, individually, are unbiased for their corresponding population effects; that is, traditionally conceived publication bias involves selection of results <emph>across</emph> studies.[[<reflink idref="bib7" id="ref32">7</reflink>], [<reflink idref="bib33" id="ref33">33</reflink>]] In contrast, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking can distort studies' point estimates themselves.[<reflink idref="bib33" id="ref34">33</reflink>] As a result, standard methods for publication bias (i.e., conceived as selection of results <emph>across</emph>, but not within, studies) can be severely biased in either direction when there is <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref35">33</reflink>] Even when <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hackers select in a simple manner for affirmative results, corresponding models of publication bias can perform quite poorly because of the internal bias introduced by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref36">33</reflink>]</p> <p>To build upon existing methods for publication bias and for internal bias, we propose sensitivity analyses that capture the biases' interactive effects. To do so, the sensitivity analyses address two questions: (<reflink idref="bib1" id="ref37">1</reflink>) "For a given severity of average internal bias across studies and of publication bias, how much could the results change?"; and (<reflink idref="bib2" id="ref38">2</reflink>) "For a given severity of publication bias, how severe would the average internal bias across studies have to be, hypothetically, to attenuate the results to the null or by a given amount?" To do so, we will again consider publication bias that favors affirmative over nonaffirmative studies, an assumption that aligns well with empirical evidence on how researchers interpret and report <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐values.[[<reflink idref="bib15" id="ref39">15</reflink>], [<reflink idref="bib34" id="ref40">34</reflink>], [<reflink idref="bib36" id="ref41">36</reflink>]] For conservative sensitivity analyses, the same two questions could be asked for hypothetical worst‐case publication bias that favors affirmative studies infinitely more than nonaffirmative studies.</p> <p>In contrast to some existing methods for internal bias, the proposed methods will not require specifying the amount of bias in each study individually[[<reflink idref="bib18" id="ref42">18</reflink>], [<reflink idref="bib21" id="ref43">21</reflink>]]; instead, we will specify the average bias across studies. Some existing methods require access to certain studies that are assumed to be internally unbiased (e.g., randomized studies if one is specifically considering bias due to confounding; or studies using a perfectly measured outcome if one is considering outcome misclassification).[[<reflink idref="bib17" id="ref44">17</reflink>], [<reflink idref="bib37" id="ref45">37</reflink>]] Our proposed methods likewise allow one to assume that certain studies are internally unbiased, but also apply if all studies may be internally biased. Our proposed methods allow meta‐analysts to directly apply the growing suite of analytic bounds on various internal biases in individual studies; alternatively, the source of internal bias can be left unspecified. We will describe robust estimation methods that accommodate non‐normal population effects, small meta‐analyses, and dependence among the point estimates that can arise when, for example, some papers contribute multiple point estimates.</p> <p>We will show that the bias‐corrected meta‐analytic estimator has a simple closed form, which provides useful heuristics regarding when a meta‐analysis will be more or less robust to combined biases. Of course, all else equal, a meta‐analysis will be more robust if internal bias and publication bias are, individually, less severe. However, robustness also depends on how internal bias and publication bias interact in a given meta‐analysis, which can in fact be captured by relatively intuitive characteristics of the meta‐analysis. That is, for a <emph>given</emph> severity of publication bias and average internal bias across published studies, the bias‐corrected estimate will indicate better robustness (all else equal) when: (<reflink idref="bib1" id="ref46">1</reflink>) a random‐effects estimate in only the published nonaffirmative studies is large; and (<reflink idref="bib2" id="ref47">2</reflink>) the internal bias primarily affects the affirmative studies rather than the nonaffirmative studies. Heuristically, the random‐effects estimate and severity of internal bias in the published nonaffirmative studies affect the corrected meta‐analytic estimate disproportionately more than do the comparable quantities in the affirmative studies. Thus, while meta‐analysts often consider the proportion of internally unbiased studies when assessing robustness, an appropriately bias‐corrected estimate also depends on the distribution of the internal bias between affirmative and nonaffirmative studies.</p> <p>This paper is structured as follows. Section 2 develops notation, concepts, and assumptions for a random‐effects meta‐analysis that may have both internal bias and publication bias. Sections 3.1–3.3 develop the main results. Section 3.4 provides guidance on how to specify and interpret the sensitivity parameters regarding publication bias and internal bias. Section 3.5 addresses situations in which the meta‐analyst wishes to consider certain types of internal bias in a causal estimand. In such cases, we show that the sensitivity parameters regarding internal bias can be reparametrized to instead describe associations of latent variables (e.g., uncontrolled confounder(s)) with studies' exposures or outcomes. Section 4 applies the proposed methods to two previously published meta‐analyses. Section 5 establishes the methods' performance in a simulation study. An R package implementing the methods, multibiasmeta, is available along with a tutorial (https://cran.r-project.org/web/packages/multibiasmeta/vignettes/tutorial.html).</p> <hd id="AN0174546109-3">SETTING AND NOTATION</hd> <p></p> <hd id="AN0174546109-4">Random‐effects meta‐analysis of internally biased studies</hd> <p>Table 1 summarizes the key notation developed in this section and subsequent sections. Suppose that prior to selection due to publication bias, there are <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi>k</mi><mo>*</mo></msup></mrow></math> </ephtml> studies, whose mean population effect ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> ) is the meta‐analyst's estimand of interest. We use asterisks for quantities referring to these "underlying studies," and in later developments, will omit asterisks for quantities referring to published studies. Let <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>P</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> respectively denote the point estimate, standard error, and two‐tailed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐value of the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi>i</mi><mi mathvariant="italic">th</mi></msup></mrow></math> </ephtml> underlying study. If study <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>i</mi></mrow></math> </ephtml> is potentially subject to internal bias, denoted <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></math> </ephtml> , then <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> may be biased for the study's population effect. The subsequently developed methods apply even if there are no studies that are known to be internally unbiased. If the meta‐analyst is not sure whether certain studies are internally unbiased, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> can simply be set to 1 for those studies; for brevity, we will simply to studies with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></math> </ephtml> as "internally biased" rather than "potentially internally biased." We assume the underlying random‐effects model:1 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mi>μ</mi><mo>+</mo><msubsup><mi>γ</mi><mi>i</mi><mo>*</mo></msubsup><mo>+</mo><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup><mo>+</mo><msubsup><mi>ε</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> where the random intercept <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>γ</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> has mean 0 and variance (i.e., heterogeneity) <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi>τ</mi><mn>2</mn></msup></mrow></math> </ephtml> ; the error <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>ε</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> has mean 0 and variance <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mrow><mo>*</mo><mn>2</mn></mrow></msubsup></mrow></math> </ephtml> ; and the study‐specific internal bias <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> is equal to 0 for any studies that are known to be internally unbiased ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ). A standard assumption of meta‐analysis is that the point estimates and their standard errors are uncorrelated[<reflink idref="bib38" id="ref48">38</reflink>]; analogously, we assume that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0028" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced open="[" close="]" separators="|,"><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mi>c</mi></mrow></mfenced><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|"><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mi>c</mi></mrow></mfenced></mrow></math> </ephtml> for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0029" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>c</mi><mo>∈</mo><mfenced open="{" close="}"><mrow><mn>0</mn><mo>,</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> . Let <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0030" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|"><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> be a sensitivity parameter denoting the mean additive bias among internally biased, underlying studies, such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0031" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced open="[" close="]" separators="|"><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced><mo>=</mo><mi>μ</mi><mo>+</mo><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> . The average bias may be in either direction, such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0032" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>∈</mo><mfenced open="(" close=")" separators=","><mrow><mo>−</mo><mi>∞</mi></mrow><mi>∞</mi></mfenced></mrow></math> </ephtml> . Later developments will relate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0033" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> to more intuitively tractable sensitivity parameters regarding the internal bias in <emph>published</emph> studies (Section 3.1).</p> <p>1 TABLE Summary of notation.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Quantities describing each underlying study</th><th align="left" /></tr></thead><tbody valign="top"><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0034" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal" xmlns="">Δ</mi><mo xmlns="">̂</mo><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p>, <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0035" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">σ</mi><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p></td><td align="left">Point estimate of study <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0036" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">i</mi></math></p> (may be internally biased) and its standard error</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0037" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p></td><td align="left">Internal bias of study <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0038" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">i</mi></math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0039" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p></td><td align="left">Indicator that study <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0040" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">i</mi></math></p> is internally biased</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0041" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p></td><td align="left">Indicator that study <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0042" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">i</mi></math></p> is affirmative</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0043" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">p</mi><mi xmlns="">A</mi><mo xmlns="">=</mo><mi xmlns="">P</mi><mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>1<mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>1</math></p></td><td align="left">Proportion of published, internally biased studies that are affirmative</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0044" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">D</mi><mi xmlns="">i</mi><mo xmlns="">*</mo></math></p></td><td align="left">Indicator that study <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0045" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">i</mi></math></p> is published</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0046" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">w</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo><mi xmlns="">τ</mi><mo xmlns="">̂</mo>2<mo xmlns="">+</mo><mi xmlns="">σ</mi><mi xmlns="">i</mi><mo xmlns="">*</mo>2<mo xmlns="">−</mo>1</math></p></td><td align="left">Standard random‐effects inverse‐variance weight</td></tr><tr><td align="left">Sensitivity parameters</td><td align="left" /></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0047" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo><mi xmlns="">a</mi><mo xmlns="">=</mo><mi xmlns="">E</mi><mi xmlns="">B</mi><mi xmlns="">i</mi><mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>1<mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">=</mo><mi xmlns="">a</mi></math></p></td><td align="left">Mean additive bias among published, internally biased studies with affirmative status <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0048" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">a</mi><mo xmlns="">∈</mo>0<mo xmlns="">,</mo>1</math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0049" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mo xmlns="">=</mo><mi xmlns="">P</mi><mi xmlns="">D</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo>1<mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo>1<mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo><mi xmlns="">c</mi><mi xmlns="">P</mi><mi xmlns="">D</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo>1<mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo>0<mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo><mi xmlns="">c</mi></math></p></td><td align="left">Ratio by which publication bias favors affirmative studies</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0050" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">*</mo><mo xmlns="">=</mo><mi xmlns="">E</mi><mi xmlns="">B</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">*</mo><mo xmlns="">=</mo>1</math></p></td><td align="left">Mean bias among underlying, internally biased studies (can be calculated from above sensitivity parameters)</td></tr><tr><td align="left">Meta‐analytic quantities</td><td align="left" /></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0051" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script" xmlns="">C</mi></math></p></td><td align="left">Set of internally biased studies</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0052" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script" xmlns="">A</mi></math></p></td><td align="left">Set of affirmative studies</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0053" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">S</mi><mo xmlns="">=</mo><mo xmlns="">∑</mo><mi xmlns="">i</mi><mo xmlns="">∈</mo><mi mathvariant="script" xmlns="">S</mi><mi mathvariant="normal" xmlns="">Δ</mi><mo xmlns="">̂</mo><mi xmlns="">i</mi><mi xmlns="">w</mi><mi xmlns="">i</mi></math></p></td><td align="left">Inverse‐variance‐weighted sum of estimates for studies in arbitrary set <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0054" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script" xmlns="">S</mi></math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0055" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">S</mi><mo xmlns="">=</mo><mo xmlns="">∑</mo><mi xmlns="">i</mi><mo xmlns="">∈</mo><mi mathvariant="script" xmlns="">S</mi><mi xmlns="">w</mi><mi xmlns="">i</mi></math></p></td><td align="left">Total precision for studies in <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0056" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script" xmlns="">S</mi></math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0057" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">S</mi><mo xmlns="">=</mo><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">S</mi><mo xmlns="">/</mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">S</mi></math></p></td><td align="left">Uncorrected random‐effects estimate for studies in <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0058" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script" xmlns="">S</mi></math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0059" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">π</mi><mi xmlns="">i</mi><mo xmlns="">=</mo><mi xmlns="">η</mi><mi mathvariant="double-struck" xmlns="">1</mi><mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>0<mo xmlns="">+</mo><mi mathvariant="double-struck" xmlns="">1</mi><mi xmlns="">A</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>1</math></p></td><td align="left">Inverse‐probability‐of‐publication weight</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0060" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">θ</mi><mo xmlns="">̂</mo><mi xmlns="">i</mi><mo xmlns="">=</mo><mi mathvariant="normal" xmlns="">Δ</mi><mo xmlns="">̂</mo><mi xmlns="">i</mi><mo xmlns="">−</mo><mi mathvariant="double-struck" xmlns="">1</mi><mi xmlns="">C</mi><mi xmlns="">i</mi><mo xmlns="">=</mo>1<mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">*</mo></math></p></td><td align="left">Location‐shifted point estimate</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0061" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi></math></p> and <p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0062" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi></math></p></td><td align="left">Proportion of precision in published affirmative and published nonaffirmative studies, respectively, that is from internally biased studies</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0063" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">r</mi><mo xmlns="">=</mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi><mo xmlns="">/</mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi></math></p></td><td align="left">Ratio of total precision in published affirmative studies to that in published nonaffirmative studies</td></tr></tbody></table> </ephtml> </p> <hd id="AN0174546109-5">Assumed model of publication bias</hd> <p>As noted in the Introduction, we consider a mechanism of publication bias in which affirmative studies (defined by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0064" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><mo>></mo><mn>0</mn></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0065" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>P</mi><mi>i</mi><mo>*</mo></msubsup><mo><</mo><mn>0.05</mn></mrow></math> </ephtml> , and denoted <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0066" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></math> </ephtml> ) are more likely to be published than nonaffirmative studies (defined by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0067" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup><mo>≤</mo><mn>0</mn></mrow></math> </ephtml> or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0068" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>P</mi><mi>i</mi><mo>*</mo></msubsup><mo>≥</mo><mn>0.05</mn></mrow></math> </ephtml> , and denoted <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0069" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ). Without loss of generality, we assume that publication bias favors positive‐signed estimates and that the estimate from the uncorrected meta‐analysis, termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0070" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mtext>unadj</mtext></msub></mrow></math> </ephtml> , is positive. If publication bias is instead thought to favor results with negative estimates and if <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0071" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mtext>unadj</mtext></msub><mo><</mo><mn>0</mn></mrow></math> </ephtml> , one can simply reverse the sign of all point estimates before conducting our proposed analyses.</p> <p>This model of publication bias is clearly a simplification of reality, and its appropriateness will depend on the scientific context. Nevertheless, considerable empirical evidence[[<reflink idref="bib39" id="ref49">39</reflink>], [<reflink idref="bib41" id="ref50">41</reflink>]] suggests that publication bias does, in many contexts, operate strongly on statistical significance at <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0072" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>α</mi><mo>=</mo><mn>0.05</mn></mrow></math> </ephtml> . For these conceptual reasons and in light of estimation challenges seen with more flexible models of publication bias,[[<reflink idref="bib41" id="ref51">41</reflink>], [<reflink idref="bib43" id="ref52">43</reflink>]] models that assume simple forms of selection on statistical significance continue to be developed,[[<reflink idref="bib44" id="ref53">44</reflink>]] recommended,[<reflink idref="bib41" id="ref54">41</reflink>] and assessed in simulation studies.[[<reflink idref="bib42" id="ref55">42</reflink>], [<reflink idref="bib46" id="ref56">46</reflink>]] We adopt this model as well and explore its performance under model misspecification in an extensive simulation study (Section 5).</p> <p>For a given heterogeneity estimate, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0073" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> , let <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0074" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>w</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><msup><mfenced open="(" close=")"><mrow><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup><mo>+</mo><msubsup><mi>σ</mi><mi>i</mi><mrow><mo>*</mo><mn>2</mn></mrow></msubsup></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml> denote the usual random‐effects inverse‐variance weight for the <emph>i</emph>th study.[<reflink idref="bib38" id="ref57">38</reflink>] (We discuss heterogeneity estimation in Section 3.1.) Letting <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0075" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></math> </ephtml> indicate that the <emph>i</emph>th study is published, we assume the publication process arises as follows for any internally unbiased studies and for any internally biased studies (i.e., for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0076" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>∈</mo><mfenced open="{" close="}"><mrow><mn>0</mn><mo>,</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> ):A1 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0077" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mfenced open="(" close=")" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced><mo>∝</mo><msup><mi>η</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></mfenced><mo>+</mo><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced><mo>,</mo><mspace width="0.5em" /><mi>η</mi><mo>≥</mo><mn>1</mn></mrow></math> </ephtml> A2 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0078" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi><mfenced open="[" close="]" separators="|,,"><mrow><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn><mo>,</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced></math> </ephtml> A3 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0079" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><msubsup><mi>w</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>w</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced></mrow></math> </ephtml> A4 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0080" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><msubsup><mi>w</mi><mi>i</mi><mo>*</mo></msubsup><msup><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>*</mo></msup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced><mi>E</mi><mfenced open="[" close="]" separators="|,"><mrow><msubsup><mi>w</mi><mi>i</mi><mo>*</mo></msubsup><msup><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>*</mo></msup><mo>|</mo><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>,</mo><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup></mrow></mfenced></mrow></math> </ephtml></p> <p>Assumption A1 states that, for both internally unbiased and internally biased studies, affirmative studies are <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0081" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> times more likely to be published than nonaffirmative studies, where we will treat the selection ratio <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0082" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> as a sensitivity parameter. This assumption pertains to the selection <emph>ratio</emph>, and so does <emph>not</emph> require internally biased and internally unbiased studies to have the same absolute publication probabilities. That is, the assumption accommodates the possibility that the publication process favors internally unbiased studies over internally biased studies (conditional on whether studies are affirmative) such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0083" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mfenced open="(" close=")" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mi>a</mi></mrow><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></mfenced><mo>></mo><mi>P</mi><mfenced open="(" close=")" separators="|,"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mi>a</mi></mrow><mrow><msubsup><mi>C</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> , or vice versa. This might indeed be the case if a well‐functioning editorial and peer review system favors studies whose designs eliminate certain types of bias.[<reflink idref="bib1" id="ref58">1</reflink>]</p> <p>Assumption A2 states that conditional on whether an internally biased study is affirmative, publication bias does not select further based on the numerical value of the bias. This may be plausible if editors and reviewers are not aware of the precise amount of bias in a given study, even if they may discriminate between broad categories of internally biased versus internally unbiased studies (e.g., randomized studies versus nonrandomized studies in the case of uncontrolled confounding).</p> <p>Assumptions A3 and A4 state that conditional on whether a study is affirmative and whether it is internally biased, publication bias does not select further based on the inverse‐variance weights or their product with the point estimates. For example, these two assumptions exclude publication bias that favors larger point estimates, smaller standard errors, or smaller <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0084" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐values above and beyond its favoring of affirmative results. However, in the Supplement, we show that these assumptions can be relaxed somewhat to accommodate a publication process that operates on studies' standard errors (e.g., by favoring larger studies), as long as this form of selection operates in the same way regardless of whether a study is internally biased and whether it is affirmative. As we show in the Supplement, this additional form of selection can simply be ignored, requiring no modification to the sensitivity analyses we present below. More flexible models of publication bias have used additional parameters similar to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0085" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , or more flexible densities, to accommodate more complex selection mechanisms.[[<reflink idref="bib7" id="ref59">7</reflink>], [<reflink idref="bib10" id="ref60">10</reflink>], [<reflink idref="bib12" id="ref61">12</reflink>]] As discussed in the Introduction, selective reporting can also occur within studies, as in <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0086" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking. Although the resulting bias can be viewed as internal bias in that it distorts studies' point estimates, this form of internal bias may or may not conform to our assumptions depending on the form of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0087" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking. A formal treatment of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0088" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking and implications for correct model specification is beyond the scope of this paper, but is covered elsewhere.[<reflink idref="bib33" id="ref62">33</reflink>] However, in simulations, we include a number of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0089" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking scenarios to empirically assess performance.</p> <hd id="AN0174546109-6">METHODS</hd> <p>We first present consistent estimators, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0090" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0091" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mfenced></mrow></math> </ephtml> , that correct for the interactive effects of internal bias and publication bias, given fixed values of the respective sensitivity parameters <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0092" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0093" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> (Sections 3.1–3.2). Using those results, we then obtain sensitivity analyses that characterize, for a given severity of publication bias <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0094" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , the severity of average internal bias that would be required to attenuate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0095" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> to the null or to a specified non‐null value. Table 2 summarizes the results and interpretations developed below.</p> <p>2 TABLE Summary of results in Sections 3.1 and 3.3, organized by assumptions regarding publication bias severity and regarding C, the set of studies assumed to be internally biased. In the final column, "conservative" indicates that if the inequality does not hold this does not guarantee that μ̂adj can in fact be explained away (Section 3.4.3).</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Publication bias</th><th align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0099" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="script">C</mi></mrow></math></p></th><th align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0100" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math></p></th><th align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0101" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math></p></th><th align="left">Estimate cannot be explained away if ...</th></tr></thead><tbody valign="top"><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0102" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mo xmlns=""><</mo><mi xmlns="">∞</mi></math></p></td><td align="left">Known subset</td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0103" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">−</mo><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">*</mo><mo xmlns="">⋅</mo><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi>1<mo xmlns="">+</mo><mi xmlns="">r</mi><mo xmlns="">/</mo><mi xmlns="">η</mi><mo xmlns="">+</mo><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mo xmlns="">−</mo><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">*</mo><mo xmlns="">⋅</mo><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi>1<mo xmlns="">+</mo><mi xmlns="">η</mi><mo xmlns="">/</mo><mi xmlns="">r</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0104" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi mathvariant="italic" xmlns="">ηλ</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns=""> </mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi><mo xmlns=""> </mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0105" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>1<mo xmlns=""><</mo><mi xmlns="">B</mi><mi xmlns="">μ</mi><mo xmlns="">̂</mo><mi xmlns="">adj</mi><mi xmlns="">η</mi></math></p> (conservative)</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0106" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mo xmlns="">→</mo><mi xmlns="">∞</mi></math></p></td><td align="left">Known subset</td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0107" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">−</mo><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>0<mo xmlns="">⋅</mo><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0108" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">/</mo><mi xmlns="">λ</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0109" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>0<mo xmlns=""><</mo><mi xmlns="">lim</mi><mi xmlns="">n</mi><mo xmlns="">→</mo><mi xmlns="">∞</mi><mspace width="0.5em" xmlns="" /><mi xmlns="">B</mi><mi xmlns="">μ</mi><mo xmlns="">̂</mo><mi xmlns="">adj</mi><mo xmlns="">,</mo><mi xmlns="">η</mi></math></p></td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0111" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mo xmlns=""><</mo><mi xmlns="">∞</mi></math></p></td><td align="left">All</td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0112" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi mathvariant="italic" xmlns="">ην</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi><mo xmlns="">−</mo><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">*</mo></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0113" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">y</mi><mi mathvariant="script" xmlns="">A</mi><mi mathvariant="italic" xmlns="">ην</mi><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">+</mo><mi xmlns="">ν</mi><mi mathvariant="script" xmlns="">A</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0114" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>1<mo xmlns=""><</mo><mi xmlns="">B</mi><mi xmlns="">μ</mi><mo xmlns="">̂</mo><mi xmlns="">adj</mi><mi xmlns="">η</mi></math></p> (conservative)</td></tr><tr><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0115" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">η</mi><mo xmlns="">→</mo><mi xmlns="">∞</mi></math></p></td><td align="left">All</td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0116" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi><mo xmlns="">−</mo><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>0</math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0117" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mo xmlns="">˜</mo><mi mathvariant="script" xmlns="">A</mi><mi xmlns="">c</mi></math></p></td><td align="left"><p><math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0118" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">μ</mi><mi xmlns="">B</mi><mo xmlns="">∣</mo><mi xmlns="">A</mi><mo xmlns="">=</mo>0<mo xmlns=""><</mo><mi xmlns="">lim</mi><mi xmlns="">n</mi><mo xmlns="">→</mo><mi xmlns="">∞</mi><mspace width="0.5em" xmlns="" /><mi xmlns="">B</mi><mi xmlns="">μ</mi><mo xmlns="">̂</mo><mi xmlns="">adj</mi><mo xmlns="">,</mo><mi xmlns="">η</mi></math></p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0174546109-7">A meta‐analytic estimate corrected for combined biases</hd> <p>Let <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0120" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">A</mi><mo>=</mo><mfenced open="{" close="}"><mrow><mi>i</mi><mo>:</mo><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> be the set of published affirmative studies and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0121" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">C</mi><mo>=</mo><mfenced open="{" close="}"><mrow><mi>i</mi><mo>:</mo><msub><mi>C</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> the set of published internally biased studies. The sets' complements are <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0122" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0123" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi mathvariant="script">C</mi><mi>c</mi></msup></mrow></math> </ephtml> , respectively. We assume that the meta‐analysis contains at least one published affirmative and one published nonaffirmative study (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0124" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">A</mi></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0125" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></math> </ephtml> are non‐empty). Recall that for a given heterogeneity estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0126" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> , the usual random‐effects weight for study <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0127" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>i</mi></mrow></math> </ephtml> is <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0128" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>w</mi><mi>i</mi></msub><mo>=</mo><msup><mfenced open="(" close=")"><mrow><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup><mo>+</mo><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml> . For an arbitrary subset of studies <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0129" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">S</mi></mrow></math> </ephtml> , define <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0130" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>y</mi><mi mathvariant="script">S</mi></msub><mo>=</mo><msub><mo>∑</mo><mrow><mi>i</mi><mo>∈</mo><mi mathvariant="script">S</mi></mrow></msub><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><msub><mi>w</mi><mi>i</mi></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0131" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ν</mi><mi mathvariant="script">S</mi></msub><mo>=</mo><msub><mo>∑</mo><mrow><mi>i</mi><mo>∈</mo><mi mathvariant="script">S</mi></mrow></msub><msub><mi>w</mi><mi>i</mi></msub></mrow></math> </ephtml> ; the latter can be viewed as the total precision for studies in set <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0132" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">S</mi></mrow></math> </ephtml> . Thus, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0133" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mi mathvariant="script">S</mi></msub><mo>=</mo><msub><mi>y</mi><mi mathvariant="script">S</mi></msub><mo>/</mo><msub><mi>ν</mi><mi mathvariant="script">S</mi></msub></mrow></math> </ephtml> is a meta‐analytic random‐effects estimate for the subset <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0134" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">S</mi></mrow></math> </ephtml> .<sups>4</sups>[<reflink idref="bib2" id="ref63">2</reflink>] For the purpose of defining the weights <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0135" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>w</mi><mi>i</mi></msub></mrow></math> </ephtml> , we would suggest simply obtaining an uncorrected heterogeneity estimate, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0136" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup></mrow></math> </ephtml> , in an initial random‐effects meta‐analysis without any bias corrections.[[<reflink idref="bib38" id="ref64">38</reflink>], [<reflink idref="bib49" id="ref65">49</reflink>]] In our simulations and applied examples, we did so using the restricted maximum likelihood estimator[<reflink idref="bib49" id="ref66">49</reflink>] based on previous simulation results regarding closely related sensitivity analyses.[<reflink idref="bib14" id="ref67">14</reflink>] Bias in this heterogeneity estimate does not compromise our proposed methods for point estimation or inference on <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0137" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> , but rather may only somewhat reduce efficiency, because we will conduct inference using a robust method that does not require the heterogeneity to be correctly specified (Section 3.2).[<reflink idref="bib50" id="ref68">50</reflink>]</p> <p>To adjust for publication bias, let <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0138" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>π</mi><mi>i</mi></msub><mo>=</mo><mi>η</mi><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow></mfenced><mo>+</mo><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> be an additional weight that is inversely proportional to the probability that study <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0139" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>i</mi></mrow></math> </ephtml> is published, given whether it is affirmative (c.f. Assumption A1). Then, for a fixed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0140" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , define location‐shifted, published point estimates as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0141" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>−</mo><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msub><mi>C</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> . Note that these shifted estimates can be calculated without specifying each study's bias, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0142" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>B</mi><mi>i</mi></msub></mrow></math> </ephtml> . If we were to calculate the shifted estimates in the underlying studies prior to the occurrence of publication bias, these would be unbiased for the underlying mean (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0143" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mfenced open="[" close="]"><msubsup><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup></mfenced><mo>=</mo><mi>μ</mi></mrow></math> </ephtml> ). To additionally accommodate publication bias, under Assumptions A1–A4, a consistent estimate of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0144" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> can be obtained by weighting each published, location‐shifted estimate inversely to its publication probability (Theorem 1 in Data S1)[<reflink idref="bib3" id="ref69">3</reflink>]:2 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0145" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo linebreak="goodbreak">=</mo><munderover><mo movablelimits="false">∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><mfrac><mrow><msub><mi>π</mi><mi>i</mi></msub><msub><mi>w</mi><mi>i</mi></msub></mrow><mrow><munderover><mo movablelimits="false">∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><msub><mi>π</mi><mi>i</mi></msub><msub><mi>w</mi><mi>i</mi></msub></mrow></mfrac><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>,</mo></mrow></math> </ephtml> 3 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0146" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>=</mo><mfrac><mrow><mi>η</mi><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>y</mi><mi mathvariant="script">A</mi></msub></mrow><mrow><msub><mi mathvariant="italic">ην</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfrac><mo>−</mo><mfrac><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mfenced open="(" close=")"><mrow><msub><mi mathvariant="italic">ην</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><mi mathvariant="script">A</mi></mrow></msub></mrow></mfenced></mrow><mrow><msub><mi mathvariant="italic">ην</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfrac></mrow></math> </ephtml></p> <p>As an elucidative special case, if studies are on average unbiased ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0147" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ), the estimator simply becomes <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0148" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><mfenced open="(" close=")"><mrow><mi>η</mi><msub><mi>y</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>+</mo><msub><mi>y</mi><mi mathvariant="script">A</mi></msub></mrow></mfenced><msup><mfenced open="(" close=")"><mrow><msub><mi mathvariant="italic">ην</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>+</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml> . This coincides with our previous sensitivity analyses for publication bias, which simply upweighted by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0149" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> the contributions of the nonaffirmative studies (namely, the inverse‐variance‐weighted sum of their point estimates, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0150" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>y</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> , and their total precision, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0151" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ν</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> ).[<reflink idref="bib14" id="ref70">14</reflink>] For large <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0152" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , this <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0153" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> is more strongly influenced by the published nonaffirmative studies than by the affirmative studies. With the introduction of internal bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0154" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>≠</mo><mn>0</mn></mrow></math> </ephtml> ), the second fraction in Equation 3 further penalizes the point estimate in a manner that depends primarily on the amount of precision contributed by internally biased, nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0155" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></math> </ephtml> ). These observations motivate the following reparametrization:4 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0156" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><mfrac><mrow><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>−</mo><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>⋅</mo><msub><mi>λ</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow><mrow><mn>1</mn><mo>+</mo><mi>r</mi><mo>/</mo><mi>η</mi></mrow></mfrac><mo>+</mo><mfrac><mrow><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mi mathvariant="script">A</mi></msub><mo>−</mo><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>⋅</mo><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub></mrow><mrow><mn>1</mn><mo>+</mo><mi>η</mi><mo>/</mo><mi>r</mi></mrow></mfrac></mrow></math> </ephtml> where <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0157" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>=</mo><mfenced open="(" close=")"><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mfenced><mo>/</mo><msub><mi>ν</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0158" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub><mo>=</mo><mfenced open="(" close=")"><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><mi mathvariant="script">A</mi></mrow></msub></mfenced><mo>/</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></math> </ephtml> . These are the proportions of the precision in, respectively, the published nonaffirmative studies and in the published affirmative studies that is contributed by internally biased studies. Additionally, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0159" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo>=</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub><mo>/</mo><msub><mi>ν</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> is the ratio of total precision in affirmative studies to that in nonaffirmative studies. This reparameterization indicates that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0160" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> will tend to be large, and thus the meta‐analysis more robust (all else equal), when: (<reflink idref="bib1" id="ref71">1</reflink>) a random‐effects estimate in only the published nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0161" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> ) is large; and (<reflink idref="bib2" id="ref72">2</reflink>) among published nonaffirmative studies, much of the precision is contributed by internally <emph>unbiased</emph> studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0162" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> is small). Although the affirmative studies also affect <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0163" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> via their own random‐effects estimate ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0164" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mi mathvariant="script">A</mi></msub></mrow></math> </ephtml> ) and proportion of precision contributed by internally biased studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0165" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub></mrow></math> </ephtml> ), the affirmative studies contribute disproportionately less to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0166" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> than do the nonaffirmative studies, especially when <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0167" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> is large.</p> <p>To apply these sensitivity analyses in practice, one could in principle specify <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0168" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0169" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , calculate the location‐shifted estimates <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0170" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></math> </ephtml> , and obtain <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0171" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> from Equation 2. However, because <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0172" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> refers to the average amount of internal bias in <emph>underlying</emph> studies, it may be challenging to directly specify <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0173" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> by assessing risks of bias in the <emph>published</emph> studies. Indeed, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0174" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> will typically be less than the average internal bias in published studies, which have been selected for affirmative results and hence also for positive internal bias. Instead, one could specify the average amounts of internal bias in published nonaffirmative studies and in published affirmative studies, termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0175" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub><mo>=</mo><mi>E</mi><mfenced open="[" close="]" separators="|,"><msub><mi>B</mi><mi>i</mi></msub><mrow><msub><mi>C</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mi>a</mi></mrow></mfenced></mrow></math> </ephtml> for each <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0176" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>a</mi><mo>∈</mo><mfenced open="{" close="}"><mrow><mn>0</mn><mo>,</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> . Letting <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0177" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>p</mi><mi>A</mi></msub><mo>=</mo><mi>P</mi><mfenced open="(" close=")" separators="|"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow><mrow><msub><mi>C</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> denote the proportion of published, internally biased studies that are affirmative, the underlying <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0178" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> can be obtained as a weighted average of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0179" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0180" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> (Supplement, Lemma 1):5 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0181" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mfrac><mrow><mi>η</mi><mfenced open="(" close=")"><mrow><mn>1</mn><mo>−</mo><msub><mi>p</mi><mi>A</mi></msub></mrow></mfenced><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>+</mo><msub><mi>p</mi><mi>A</mi></msub><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow><mrow><mi>η</mi><mfenced open="(" close=")"><mrow><mn>1</mn><mo>−</mo><msub><mi>p</mi><mi>A</mi></msub></mrow></mfenced><mo>+</mo><msub><mi>p</mi><mi>A</mi></msub></mrow></mfrac></mrow></math> </ephtml></p> <p>In practice, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0182" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>p</mi><mi>A</mi></msub></mrow></math> </ephtml> would be replaced with its consistent sample estimate, namely the proportion of published affirmative studies in the sample. Thus, to conduct sensitivity analyses, one could specify <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0183" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0184" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> by examining risks of bias in the published studies, calculate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0185" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> from Equation 5, and then estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0186" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> using Equation 2. It can also be informative to plot <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0187" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> for different choices of the sensitivity parameters <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0188" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0189" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0190" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> , as we illustrate for the applied examples (Section 4).</p> <hd id="AN0174546109-8">Special cases</hd> <p>We now discuss two special cases of the estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0191" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> . First, to consider worst‐case publication bias that favors affirmative studies infinitely more than nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0192" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></math> </ephtml> ), note that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0193" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> as specified via Equation 5 is itself a function of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0194" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0195" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder><mi>lim</mi><mrow><mi>n</mi><mo>→</mo><mi>∞</mi></mrow></munder><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> . (Intuitively, as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0196" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></math> </ephtml> , one must upweight <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0197" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> infinitely compared with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0198" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> in order to recover the underlying <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0199" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> .) We therefore have:6 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0200" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder accentunder="false"><mi>lim</mi><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></munder><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>−</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>⋅</mo><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow><mrow><msub><mi>ν</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></mfrac><mo>=</mo><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>−</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>⋅</mo><msub><mi>λ</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></math> </ephtml></p> <p>Note that this estimate depends only on the published nonaffirmative studies, not the published affirmative studies. As noted in the Introduction, a worst‐case estimate that corrects only for publication bias coincides with a standard meta‐analysis of only the nonaffirmative studies[<reflink idref="bib14" id="ref73">14</reflink>]; the expression above is a generalization that additionally corrects the nonaffirmative studies for internal bias. Additionally, unlike standard selection models, this worst‐case estimate remains conservative under many (though not all) forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0201" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref74">33</reflink>] Informally, the estimate is conservative when investigators who attempt to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0202" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hack but do not succeed (i.e., they obtain multiple estimates in an attempt to obtain an affirmative result, but only obtain a series of nonaffirmative results) either: (<reflink idref="bib1" id="ref75">1</reflink>) do not submit their nonaffirmative results for publication; or (<reflink idref="bib2" id="ref76">2</reflink>) do submit one or more nonaffirmative results, but do not favor larger nonaffirmative point estimates over smaller ones. A more technically precise formalization of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0203" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking and of these conditions for conservatism were given elsewhere.[<reflink idref="bib33" id="ref77">33</reflink>]</p> <p>As a second special case, if all published studies are internally biased (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0204" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>C</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo> </mo><mo>∀</mo><mo> </mo><mi>i</mi></mrow></math> </ephtml> ), then in Equation 3, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0205" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub><mo>=</mo><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><mi mathvariant="script">A</mi></mrow></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0206" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ν</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>=</mo><msub><mi>ν</mi><mrow><mi mathvariant="script">C</mi><mo>∩</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></math> </ephtml> . In turn, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0207" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> is equivalent to first correcting for publication bias,[<reflink idref="bib14" id="ref78">14</reflink>] and then simply shifting the resulting estimate by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0208" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> . Thus, in this case, any interactive effects of internal bias and publication bias reflect the dependence of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0209" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> on <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0210" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> : for increasingly severe publication bias, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0211" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> will be increasingly weighted toward <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0212" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> rather than <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0213" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> . For the general case <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0214" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo><</mo><mi>∞</mi></mrow></math> </ephtml> and for the worst case <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0215" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></math> </ephtml> , respectively, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0216" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> is therefore:7 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0217" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><mfrac><mrow><mi>η</mi><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>y</mi><mi mathvariant="script">A</mi></msub></mrow><mrow><msub><mi mathvariant="italic">ην</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfrac><mo>−</mo><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> 8 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0218" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder accentunder="false"><mi>lim</mi><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></munder><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>−</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml></p> <hd id="AN0174546109-9">Robust inference for μ̂adj</hd> <p>For an uncorrected meta‐analysis of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0220" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>k</mi></mrow></math> </ephtml> studies, with estimated mean <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0221" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mtext>unadj</mtext></msub></mrow></math> </ephtml> and estimated heterogeneity <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0222" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> , the usual variance estimate for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0223" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mtext>unadj</mtext></msub></mrow></math> </ephtml> is simply the inverse of the summed inverse‐variance weights[<reflink idref="bib38" id="ref79">38</reflink>]:9 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0224" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mtext>unadj</mtext></msub></mrow></mfenced><mo>=</mo><mfrac><mn>1</mn><mrow><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msup><mfenced open="(" close=")"><mrow><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup><mo>+</mo><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></mfrac></mrow></math> </ephtml></p> <p>This estimate is consistent if each <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0225" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup><mo>+</mo><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup></mrow></math> </ephtml> is consistent for the marginal variance of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0226" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></math> </ephtml> . However, in our setting, the total weight for each study, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0227" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>π</mi><mi>i</mi></msub><msub><mi>w</mi><mi>i</mi></msub></mrow></math> </ephtml> , involves the sensitivity parameter <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0228" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> and an uncorrected heterogeneity estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0229" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> , so it cannot be assumed that the inverse of each weight is consistent for the variance of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0230" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></math> </ephtml> . Therefore, we will obtain <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0231" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mfenced></mrow></math> </ephtml> by robust variance estimation, an approach that is consistent even with arbitrarily specified weights.[<reflink idref="bib50" id="ref80">50</reflink>] This method was developed for meta‐analyses in which the point estimates are clustered (e.g., because some papers in the meta‐analysis contribute multiple estimates), but in which the covariance structure is unknown. Similarly to generalized estimating equations for regression, robust variance estimation uses a consistent, sandwich‐type variance estimator.[<reflink idref="bib50" id="ref81">50</reflink>] Additionally, whereas standard asymptotic inference for parametric random‐effects meta‐analysis can perform poorly for small meta‐analyses (e.g., yielding much lower than nominal confidence interval coverage), simple finite‐sample corrections allow the robust method to perform well in small samples, albeit with sometimes conservative inference.[<reflink idref="bib51" id="ref82">51</reflink>] Accommodating small meta‐analyses will become especially important when we consider sensitivity analyses for which the effective sample size is further reduced through inverse‐probability weighting.</p> <p>An asymptotic variance estimate for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0232" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> can be obtained as a special case of robust variance estimation in which the studies are independent and the weights are <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0233" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>π</mi><mi>i</mi></msub><msub><mi>w</mi><mi>i</mi></msub></mrow></math> </ephtml> . In this case, the sandwich estimator[<reflink idref="bib50" id="ref83">50</reflink>] simply involves the location‐shifted residuals <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0234" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>−</mo><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> , yielding:10 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0235" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi mathvariant="normal">Var</mi><mo>̂</mo></mover><mfenced><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi mathvariant="normal">adj</mi></msub></mfenced><mo linebreak="goodbreak">=</mo><mfrac><mi>k</mi><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow></mfrac><munderover><mo movablelimits="false">∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><msup><mfenced open="[" close="]"><mrow><mfenced><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo linebreak="goodbreak">−</mo><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi mathvariant="normal">adj</mi></msub></mrow></mfenced><msup><mi>η</mi><mrow><mn>1</mn><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><mn>0</mn></mrow></mfenced></mrow></msup><msup><mfenced><mrow><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup><mo linebreak="goodbreak">+</mo><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></mfenced><mn>2</mn></msup><msup><mfenced><mrow><munderover><mo movablelimits="false">∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><msup><mi>η</mi><mrow><mn>1</mn><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><mn>0</mn></mrow></mfenced></mrow></msup><msup><mfenced><mrow><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup><mo linebreak="goodbreak">+</mo><msup><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mn>2</mn></msup></mrow></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></mfenced><mrow><mo>−</mo><mn>2</mn></mrow></msup></math> </ephtml></p> <p>A robust confidence interval can then be calculated as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0236" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>±</mo><msub><mi>t</mi><mtext>crit</mtext></msub><msqrt><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mfenced></mrow></msqrt></mrow></math> </ephtml> , where <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0237" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>t</mi><mtext>crit</mtext></msub></mrow></math> </ephtml> is the critical value for a <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0238" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>t</mi></mrow></math> </ephtml> ‐distribution on <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0239" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow></math> </ephtml> degrees of freedom. Additionally, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0240" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mfenced></mrow></math> </ephtml> and the resulting confidence interval can be easily generalized to accommodate clustered point estimates (Data S1, Section 2.1) and meta‐regression.[<reflink idref="bib50" id="ref84">50</reflink>] In the main text, for brevity, we focus on the intercept‐only model with independent estimates. In practice, we recommend using Tipton's[<reflink idref="bib51" id="ref85">51</reflink>] finite‐sample correction to this variance estimator, which can be easily applied in R by fitting the model with the robumeta package[<reflink idref="bib52" id="ref86">52</reflink>] with the argument small = TRUE. We adopt this approach throughout the simulation study and applied examples.</p> <hd id="AN0174546109-10">Internal bias required to attenuate μ̂adj or its confidence interval to the null, given η</hd> <p>We now use <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0243" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> to obtain sensitivity analyses that characterize, for a given <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0244" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , the severity of average internal bias, termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0245" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> , that is required to attenuate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0246" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> to the null (i.e., to fully "explain away" the meta‐analytic estimate). A similar approach will characterize the severity of average internal bias that is required to attenuate the lower confidence interval limit for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0247" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> , termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0248" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup></mrow></math> </ephtml> , to the null. Regarding <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0249" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> , from Equation 3, setting <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0250" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>=</mo><mn>0</mn></mrow></math> </ephtml> and solving for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0251" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> yields the bias required to explain away the estimate for a fixed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0252" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> :11 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0253" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>,</mo><mi>η</mi></mrow></mfenced><mo>=</mo><mfrac><mrow><mi>η</mi><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>y</mi><mi mathvariant="script">A</mi></msub></mrow><mrow><msub><mi mathvariant="italic">ηλ</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><msub><mi>ν</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfrac></mrow></math> </ephtml></p> <p>For worst‐case publication bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0254" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></math> </ephtml> ), the internal bias required is:12 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0255" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder accentunder="false"><mi>lim</mi><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></munder><mi>B</mi><mfenced open="(" close=")" separators=","><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>,</mo><mi>η</mi></mrow></mfenced><mo>=</mo><mfrac><mrow><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow><mrow><msub><mi>λ</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><msub><mi>ν</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></mfrac><mo>=</mo><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>/</mo><msub><mi>λ</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></math> </ephtml></p> <p>If all studies are internally biased, then we simply have:13 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0256" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>,</mo><mi>η</mi></mrow></mfenced><mo>=</mo><mfrac><mrow><mi>η</mi><msub><mi>y</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>y</mi><mi mathvariant="script">A</mi></msub></mrow><mrow><msub><mi mathvariant="italic">ην</mi><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub><mo>+</mo><msub><mi>ν</mi><mi mathvariant="script">A</mi></msub></mrow></mfrac></mrow></math> </ephtml> 14 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0257" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder accentunder="false"><mi>lim</mi><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></munder><mi>B</mi><mfenced open="(" close=")" separators=","><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mo>,</mo><mi>η</mi></mrow></mfenced><mo>=</mo><msub><mover accent="true"><mi>μ</mi><mo>˜</mo></mover><mrow><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></msub></mrow></math> </ephtml></p> <p>The value of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0258" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> required to attenuate the lower confidence limit, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0259" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup></mrow></math> </ephtml> , to the null is termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0260" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mi>η</mi></mfenced></mrow></math> </ephtml> and can be obtained by a numerical grid search. That is, one can simply evaluate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0261" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0262" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Var</mi><mo>̂</mo></mover><mfenced open="(" close=")"><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mfenced></mrow></math> </ephtml> over a grid of values of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0263" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> and set <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0264" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mi>η</mi></mfenced></mrow></math> </ephtml> to the smallest value of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0265" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0266" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mo>≤</mo><mn>0</mn></mrow></math> </ephtml> . The R package multibiasmeta automates this approach.</p> <p>More generally, one can conduct sensitivity analyses that characterize the amount of internal bias required to shift <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0267" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0268" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup></mrow></math> </ephtml> to a specified non‐null value. Methods for this case appear in the Supplement (Section 3.1) and are implemented in the R package. Additionally, instead of specifying a value of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0269" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> and obtaining the severity of average internal bias required to attenuate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0270" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> to the null, one could conversely use Equation 3 or Equation 4 to specify the average internal bias and obtain the severity of publication bias required. While these expressions do not have closed‐form solutions in <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0271" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , one could conduct a simple grid search over values of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0272" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> .</p> <hd id="AN0174546109-11">Guidance on specifying and interpreting the sensitivity parameters</hd> <p></p> <hd id="AN0174546109-12">Specifying η</hd> <p>By default, we would suggest conducting sensitivity analyses for at least one fixed, finite value of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0274" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> as well as for worst‐case publication bias, an approach we illustrate in the applied examples (Section 4). One could also create plots of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0275" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> for a range of the sensitivity parameters <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0276" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0277" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0278" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> , as in Figure 1. To help empirically ground the specification of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0279" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , one could consider the results of a meta‐meta‐analysis in which selection models[<reflink idref="bib13" id="ref87">13</reflink>] were used to estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0280" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> in an objectively sampled corpus of 58 large meta‐analyses across several scientific disciplines.[<reflink idref="bib15" id="ref88">15</reflink>] The overall estimated average selection ratio was 1.17 (95% CI: [0.93, 1.47]). However, selection ratios varied across meta‐analyses, with an estimated 95th quantile across meta‐analyses of 3.51.[<reflink idref="bib15" id="ref89">15</reflink>] Estimated selection ratios for different disciplines, journal tiers, and other meta‐analysis characteristics were also reported.[<reflink idref="bib15" id="ref90">15</reflink>] Existing meta‐analyses illustrate using these empirical benchmarks to interpret sensitivity analyses that consider publication bias without internal bias.[<reflink idref="bib53" id="ref91">53</reflink>] This meta‐meta‐analysis had limitations[<reflink idref="bib15" id="ref92">15</reflink>]; for example, only meta‐analyses comprising at least 40 point estimates were analyzed because selection models can perform poorly in smaller meta‐analyses.[[<reflink idref="bib42" id="ref93">42</reflink>], [<reflink idref="bib54" id="ref94">54</reflink>]] Other estimates of selection ratios are available. Some studies have followed cohorts of study protocols submitted to specific ethics committees or funded by specific granting agencies, yielding estimated selection ratios ranging from approximately 0.73 to 3.51.[[<reflink idref="bib15" id="ref95">15</reflink>], [<reflink idref="bib39" id="ref96">39</reflink>]] However, these estimates refer to individual studies rather than meta‐analyses. Other meta‐meta‐analyses have also assessed publication bias, but have not reported estimated selection ratios.[[<reflink idref="bib56" id="ref97">56</reflink>]]</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01jan24/jrsm1667-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1667-fig-0001.jpg" title="1 For the meat‐consumption meta‐analysis, μ̂adj (RR scale) for varying assumptions on η and on the distribution of confounding bias between affirmative and nonaffirmative studies. Panel (a): Assumes μB∣A=1=μB∣A=0=log1.25. Panel (b): Assumes μB∣A=1=log2.5 but μB∣A=0=0. The y‐axis is on a log scale. [Colour figure can be viewed at wileyonlinelibrary.com]" /> </p> <p></p> <hd id="AN0174546109-14">Interpreting the worst‐case estimate limn→∞Bμ̂adjη</hd> <p>The estimate under worst‐case publication bias (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0289" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder><mi>lim</mi><mrow><mi>n</mi><mo>→</mo><mi>∞</mi></mrow></munder><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> ) is, by design, mathematically conservative in that, for any finite amount of publication bias and under the assumptions given in Section 2, the worst‐case estimate will be biased toward the null. Whether this mathematical conservatism property yields an estimate that is scientifically informative will depend on the scientific question and on whether the meta‐analyst is primarily interested in average effects in a particular direction. Like other mathematically conservative sensitivity analyses and bounds,[[<reflink idref="bib58" id="ref98">58</reflink>]] the estimate under worst‐case publication bias is scientifically informative primarily when it suggests robustness (e.g., the worst‐case estimate remains meaningfully large in scientific context). In this case, one could conclude that under our assumed model of biases, any amount of internal bias such that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0290" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo><</mo><munder><mi>lim</mi><mrow><mi>n</mi><mo>→</mo><mi>∞</mi></mrow></munder><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> would not suffice to explain away the point estimate, even if publication bias could be extremely severe. Such results have been informative for real meta‐analyses that provoked sustained controversy over the plausible severity of publication bias and its potential to explain away the results.[<reflink idref="bib60" id="ref99">60</reflink>] If instead the worst‐case estimate is not meaningfully large or is on the opposite side of the null from the uncorrected estimate, this does not indicate that the meta‐analysis is not robust to potential biases, but rather that the worst‐case analysis is inconclusive. In this situation, estimates for fixed values of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0291" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> may be more scientifically informative.</p> <hd id="AN0174546109-15">Interpreting Bμ̂adjη and Bμ̂adjlbη</hd> <p>As noted above, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0294" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0295" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mi>η</mi></mfenced></mrow></math> </ephtml> refer to the average amount of internal bias in <emph>underlying</emph> studies. However, one can instead interpret them in a manner that can be more directly assessed in the published studies. That is, when specifying a fixed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0297" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , one can interpret <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0298" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> as the <emph>minimum</emph> amount of internal bias that must be present in published, internally biased affirmative studies (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0299" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> ) in order to explain away the point estimate, and similarly for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0300" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mi>η</mi></mfenced></mrow></math> </ephtml> . This interpretation is conservative in the following sense. If, in reality, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0301" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo><</mo><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> , this implies that there is not enough internal bias to explain away the point estimate. However, if instead <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0302" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>≥</mo><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> , this does not guarantee that there is enough internal bias to explain away the point estimate. In particular, if the amount of bias in <emph>nonaffirmative</emph> studies is small, the actual <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0303" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> may remain less than the required <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0304" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> . This conservative interpretation of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0305" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> in terms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0306" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> holds as long as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0307" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>≥</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> , which will generally hold because more positively biased underlying studies are more likely to be affirmative. In turn, if <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0308" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo><</mo><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> , this implies that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0309" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> is less than <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0310" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> as well (see Equation 5 and Data S1, Lemma 1).</p> <p>For the special case of worst‐case publication bias, this conservative interpretation is not needed because, in the limit, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0311" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> approaches <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0312" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> (Equation 5). Therefore, in this case, one can directly interpret <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0313" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><munder><mi>lim</mi><mrow><mi>n</mi><mo>→</mo><mi>∞</mi></mrow></munder><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> as the amount of internal bias that must be present in published, internally biased <emph>nonaffirmative</emph> studies (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0314" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> ) in order to explain away the point estimate. Table 2 summarizes these interpretations of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0315" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> for worst‐case and for fixed publication bias.</p> <p>In practice, to assess whether it is plausible that the meta‐analyzed studies are subject to internal bias that could explain away the point estimate, one should consider the design characteristics of the relevant subset of studies (the affirmative studies in the case <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0316" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo><</mo><mi>∞</mi></mrow></math> </ephtml> and the nonaffirmative studies in the case <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0317" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>→</mo><mi>∞</mi></mrow></math> </ephtml> ), ideally using risk‐of‐bias tools.[<reflink idref="bib61" id="ref100">61</reflink>] For example, when considering uncontrolled confounding as the key source of internal bias, more uncontrolled confounding would be plausible in a meta‐analysis of cross‐sectional studies than in an otherwise comparable meta‐analysis of longitudinal studies that control for an ample set of baseline confounders, including baseline values of the exposure and outcome.[[<reflink idref="bib21" id="ref101">21</reflink>], [<reflink idref="bib62" id="ref102">62</reflink>]] Existing meta‐analyses provide good examples of using risk‐of‐bias assessments to interpret quantitative sensitivity analyses.[[<reflink idref="bib63" id="ref103">63</reflink>], [<reflink idref="bib65" id="ref104">65</reflink>]]</p> <hd id="AN0174546109-16">Expressing internal bias in terms of latent‐variable associations</hd> <p>Thus far, we have worked with sensitivity parameters, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0318" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0319" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> , that simply summarize the additive internal bias across studies without making assumptions on its source. However, for certain types of bias in estimates of a causal parameter (e.g., a treatment effect), the bias can be reparameterized to more directly describe structural associations that involve latent variables, such as uncontrolled confounders, that are responsible for the bias. These reparameterizations apply if studies' unadjusted estimates, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0320" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></math> </ephtml> , are on the log risk ratio scale or have been approximately transformed to this scale via the usual approaches in meta‐analysis.[<reflink idref="bib66" id="ref105">66</reflink>] Then, the sensitivity parameters regarding the bias on the risk ratio ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0321" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> ) scale (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0322" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced></mrow></math> </ephtml> ) can be expressed in terms of structural associations involving latent variables.</p> <p>For example, for uncontrolled confounding, the bias on the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0323" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale can be expressed in terms of the minimum strength of "confounding associations" that would need to be present on average across studies in order to produce the amount of bias indicated by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0324" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced></mrow></math> </ephtml> .[[<reflink idref="bib20" id="ref106">20</reflink>], [<reflink idref="bib23" id="ref107">23</reflink>], [<reflink idref="bib67" id="ref108">67</reflink>]] For a given study, the two relevant confounding associations are defined as: (<reflink idref="bib1" id="ref109">1</reflink>) the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0325" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> by which uncontrolled confounder(s) are associated with the exposure; and (<reflink idref="bib2" id="ref110">2</reflink>) the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0326" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> by which uncontrolled confounder(s) are associated with the outcome. Both confounding associations are conditional on any controlled covariates. (Figure S1a depicts these confounding associations graphically.) These are the same sensitivity parameters used in the E‐value, a sensitivity analysis for uncontrolled confounding in a single study.[[<reflink idref="bib23" id="ref111">23</reflink>], [<reflink idref="bib67" id="ref112">67</reflink>]] The E‐value represents the minimum strength of at least one of these two confounding associations that would be required to fully explain the study's estimate.[[<reflink idref="bib23" id="ref113">23</reflink>], [<reflink idref="bib67" id="ref114">67</reflink>]] In our setting, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0327" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mrow></math> </ephtml> (on the log‐ <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0328" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale) can be transformed into a "meta‐analytic E‐value" as follows[<reflink idref="bib20" id="ref115">20</reflink>]:15 <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0329" xmlns="http://www.w3.org/1998/Math/MathML"><mi>g</mi><mfenced open="(" close=")"><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced></mrow></mfenced><mo linebreak="goodbreak">=</mo><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced><mo linebreak="goodbreak">+</mo><msqrt><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced><mfenced open="(" close=")"><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced><mo linebreak="goodbreak">−</mo><mn>1</mn></mrow></mfenced></mrow></msqrt></math> </ephtml></p> <p>This meta‐analytic E‐value represents the average strengths of association across studies, on the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0330" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale, that uncontrolled confounder(s) would need to have with studies' exposures, with their outcomes, or with both in order to produce the amount of bias indicated by <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0331" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>exp</mi><mo>{</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub><mo>}</mo></mrow></math> </ephtml> . Therefore, for any of the above sensitivity analyses characterizing the amount of bias required to shift the meta‐analytic estimate or its confidence interval limit to the null (or to a non‐null value), one has the option to transform the relevant sensitivity parameter <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0332" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0333" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> using Equation 15 to allow interpretation in terms of confounding associations.</p> <p>Sensitivity analyses similar to the E‐value for uncontrolled confounding have been developed for several other biases in a single causal estimate, including selection bias,[<reflink idref="bib28" id="ref116">28</reflink>] differential misclassification of the exposure or outcome,[<reflink idref="bib29" id="ref117">29</reflink>] uncontrolled confounding of effect modification estimates,[<reflink idref="bib26" id="ref118">26</reflink>] and combinations of these biases.[<reflink idref="bib27" id="ref119">27</reflink>] To assess whether each of these biases may be present for a given study, one can represent the study's presumed causal structure using a graphical model, such as a directed acyclic graph (DAG), and then can apply existing d‐separation criteria to assess whether the bias may be present.[[<reflink idref="bib68" id="ref120">68</reflink>], [<reflink idref="bib70" id="ref121">70</reflink>]] For certain biases, there exist graphical criteria that can be applied even with only partial knowledge of the underlying DAG (e.g., for confounding, if one knows only which variables might affect both the exposure and the outcome[<reflink idref="bib70" id="ref122">70</reflink>]; for selection bias, if one knows which variables might affect both selection into analysis and the outcome[<reflink idref="bib71" id="ref123">71</reflink>]). For each of the aforementioned biases, the bound on the resulting bias is a simple transformation analogous to Equation 15.</p> <p>Example DAGs that depict the relevant latent‐variable associations for three illustrative types of bias appear in Figure S1. For example, for differential outcome misclassification, the relevant transformation is simply the identity (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0334" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>g</mi><mfenced open="(" close=")"><mrow><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced></mrow></mfenced><mo>=</mo><mi>exp</mi><mfenced open="{" close="}"><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mi>a</mi></mrow></msub></mfenced></mrow></math> </ephtml> ), and the relevant latent associations are the associations of the exposure with the mismeasured outcome, conditional on each level of the true outcome (Figure S1b).[<reflink idref="bib29" id="ref124">29</reflink>]</p> <hd id="AN0174546109-17">APPLIED EXAMPLES</hd> <p>We now illustrate applying and interpreting the proposed sensitivity analyses using two published meta‐analyses in which some or all studies were nonrandomized. For each meta‐analysis, we specifically considered internal bias arising from uncontrolled confounding, and we conducted sensitivity analyses for two scenarios regarding publication bias. First, we considered publication bias in which affirmative studies are <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0335" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>4</mn></mrow></math> </ephtml> times more likely to be published than nonaffirmative studies, a selection ratio that exceeds the estimated 95th quantile ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0336" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>3.51</mn></mrow></math> </ephtml> ) in the aforementioned meta‐meta‐analysis (Section 3.4.1).[<reflink idref="bib15" id="ref125">15</reflink>] Second, we considered worst‐case publication bias. For each of these scenarios regarding publication bias, we first estimated <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0337" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> without correction for internal bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0338" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ). Then, we assessed the amount of internal bias required to shift <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0339" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> to the null and to shift its lower confidence interval limit, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0340" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup></mrow></math> </ephtml> , to the null. We transformed the required amounts of internal bias, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0341" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0342" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi><mfenced open="(" close=")" separators=","><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup><mi>η</mi></mfenced></mrow></math> </ephtml> , to characterize the minimum strengths of confounding associations required to explain away <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0343" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0344" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi><mi>lb</mi></msubsup></mrow></math> </ephtml> (Equation 15). We fit all uncorrected and corrected meta‐analyses using robust variance estimation as described in Section 3.2. For one meta‐analysis,[<reflink idref="bib72" id="ref126">72</reflink>] articles could contribute multiple point estimates, so we accounted for potential clustering of point estimates within articles. For each meta‐analysis, we meta‐analyzed studies' point estimates on the log‐ <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0345" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale and report results transformed back to the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0346" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale.[<reflink idref="bib4" id="ref127">4</reflink>]</p> <hd id="AN0174546109-18">A meta‐analysis of nonrandomized studies</hd> <p>The first meta‐analysis[<reflink idref="bib73" id="ref128">73</reflink>] assessed the association of atrial fibrillation with cognitive impairment in 14 nonrandomized studies, of which 5 were cross‐sectional and 9 were longitudinal. For our re‐analysis, we assumed that all studies were potentially confounded. The uncorrected pooled estimate is <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0347" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.40</mn></mrow></math> </ephtml> (95% CI: [1.18, 1.66]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0348" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.002</mn></mrow></math> </ephtml> ; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0349" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo>=</mo><mn>0.20</mn></mrow></math> </ephtml> ). The authors noted that studies often did not control adequately for confounding by comorbidities such as hypertension.[<reflink idref="bib73" id="ref129">73</reflink>] They assessed publication bias using Egger's regression, but correctly noted this method's limited power for small meta‐analyses.[<reflink idref="bib73" id="ref130">73</reflink>] This suggests the importance of additionally conducting sensitivity analyses that perform well for small meta‐analyses (as established in the simulation study, Section 5).</p> <p>If we consider publication bias with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0350" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>4</mn></mrow></math> </ephtml> , then prior to correction for confounding, the estimate is somewhat reduced to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0351" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.23</mn></mrow></math> </ephtml> (95% CI: [1.04, 1.45]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0352" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.03</mn></mrow></math> </ephtml> ). The meta‐analytic E‐value for this estimate is 1.75, indicating that this estimate could potentially be explained away if, on average across the published affirmative studies, there were uncontrolled confounder(s) that were associated with atrial fibrillation, with cognitive impairment, or with both by risk ratios of at least 1.75 (conditional on any controlled covariates). This interpretation refers to only the published affirmative studies because, as noted in Section 3.4.3, if <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0353" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo><</mo><mi>B</mi><mfenced open="(" close=")" separators=","><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub><mi>η</mi></mfenced></mrow></math> </ephtml> , this implies that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0354" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> cannot be explained away. The corresponding meta‐analytic E‐value for shifting the confidence interval to include the null is 1.23.</p> <p>If we instead consider worst‐case publication bias, then prior to correcting for confounding, the estimate is reduced to 1.07 (95% CI: [0.97, 1.17]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0355" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.11</mn></mrow></math> </ephtml> ). In this scenario, the meta‐analytic E‐value for the point estimate is reduced to 1.34, and the confidence interval already includes the null even without correcting for any internal bias. This meta‐analytic E‐value under worst‐case publication bias can be interpreted as the average strengths of confounding associations that would need to be present in the published <emph>nonaffirmative</emph> studies in order to explain away the estimate. As noted in Section 3.4.3, this is because the worst‐case estimate involves only the published nonaffirmative studies, so <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0356" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> is equal to the amount of internal bias in the underlying studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0357" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> ).</p> <hd id="AN0174546109-19">A meta‐analysis of randomized and nonrandomized studies</hd> <p>The second meta‐analysis assessed the effectiveness of educational behavior interventions that attempt to reduce meat consumption by appealing to animal welfare.[<reflink idref="bib72" id="ref131">72</reflink>] The meta‐analysis included 100 studies (from 34 articles, treated as clusters in analysis) that measured behavioral or self‐reported outcomes related to meat consumption or purchasing. The uncorrected pooled estimate is <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0358" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.18</mn></mrow></math> </ephtml> (95% CI: [1.09, 1.28]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0359" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.0003</mn></mrow></math> </ephtml> ; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0360" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo>=</mo><mn>0.17</mn></mrow></math> </ephtml> ). The meta‐analysts' risk‐of‐bias assessments categorized 52% of studies as being at low risk of bias due to confounding or other threats to exchangeability. Relevant to interpreting <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0361" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0362" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> , this percentage was comparable between nonaffirmative studies (53%) and affirmative studies (48%). For the present re‐analysis, we assumed that the 23 nonrandomized studies were potentially biased due to uncontrolled confounding, but that the 77 randomized studies were not internally biased. We assumed that point estimates could be clustered within articles (Data S1, Section 2.1), a situation not readily accommodated by other sensitivity analysis methods.</p> <p>If we consider publication bias with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0363" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>4</mn></mrow></math> </ephtml> , then prior to correction for confounding, the estimate is slightly reduced to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0364" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.14</mn></mrow></math> </ephtml> (95% CI: [1.06, 1.22]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0365" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.0008</mn></mrow></math> </ephtml> ). The meta‐analytic E‐value for this estimate is 2.82, indicating that the estimate could potentially be explained away if, on average across published affirmative studies, there were uncontrolled confounder(s) that were associated with intervention receipt, with meat consumption, or with both by risk ratios of at least 2.82 (conditional on any controlled covariates). The corresponding meta‐analytic E‐value for the confidence interval limit is 1.56. Given the risk‐of‐bias assessments noted above, it seems somewhat implausible that this much‐uncontrolled confounding was actually present. If we instead consider worst‐case publication bias, then prior to correcting for confounding, the estimate is 1.07 (95% CI: [1.02, 1.12]; <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0366" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>=</mo><mn>0.01</mn></mrow></math> </ephtml> ). In this scenario, the meta‐analytic E‐values for the point estimate and confidence interval limit are respectively reduced to 1.35 and 1.17.</p> <p>Figure 1 shows the dependence of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0367" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> on different choices of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0368" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0369" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0370" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> , which demonstrates that the corrections for internal bias and publication bias may or may not operate additively. In Figure 1a, we have assumed that the published affirmative studies and published nonaffirmative studies are both, on average, moderately confounded to the same degree ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0371" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>=</mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>=</mo><mi>log</mi><mfenced open="(" close=")"><mn>1.25</mn></mfenced></mrow></math> </ephtml> ). For this meta‐analysis, under this assumption, the correction for internal bias is approximately additive with the correction for publication bias. Specifically, the additive penalty on the log scale that is incurred by correcting for internal bias is almost the same regardless of whether one also corrects for publication bias, even when the publication is assumed to be severe (i.e., the difference between the red and blue lines is almost constant in <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0372" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> ). In contrast, in Figure 1b, we have assumed that the published affirmative studies are, on average, subject to fairly severe internal bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0373" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>=</mo><mi>log</mi><mfenced open="(" close=")"><mn>2.5</mn></mfenced></mrow></math> </ephtml> ) but that the published nonaffirmative studies are, on average, not internally biased ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0374" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ). In this case, the bias correction for confounding becomes subadditive with the correction for publication bias. That is, the penalty incurred by correcting for internal bias is reduced once one has already corrected for publication bias, especially when the publication bias is assumed to be severe (i.e., the red and blue lines begin to converge as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0375" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> increases).</p> <p>A heuristic explanation is as follows. If published affirmative studies are considerably more internally biased than published nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0376" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>></mo><msub><mi>μ</mi><mrow><mi>B</mi><mo>∣</mo><mi>A</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> ), then the penalty for publication bias indirectly also reduces internal bias by downweighting the affirmative studies; this in turn reduces the additional penalty required to fully correct for internal bias (Figure 1b). In contrast, if the severity of internal bias is comparable between published affirmative and published nonaffirmative studies (Figure 1a), then the penalty for publication bias alone may not reduce internal bias, so the penalty for internal bias may operate additively. Indeed, the latter case may be more accurate for this meta‐analysis given the comparable risk‐of‐bias ratings for nonaffirmative and for affirmative studies.</p> <hd id="AN0174546109-20">Interpreting and comparing the results for the two meta‐analyses</hd> <p>Although the uncorrected estimate in the atrial‐fibrillation meta‐analysis[<reflink idref="bib73" id="ref132">73</reflink>] exceeds that of the meat‐consumption meta‐analysis[<reflink idref="bib72" id="ref133">72</reflink>] (i.e., <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0377" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.40</mn></mrow></math> </ephtml> versus <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0378" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi><mo>=</mo><mn>1.18</mn></mrow></math> </ephtml> ), the former estimate is more substantially attenuated when considering publication bias. For example, for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0379" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>4</mn></mrow></math> </ephtml> without considering internal bias, the estimates in the atrial‐fibrillation and meat‐consumption meta‐analyses are attenuated by 12% and 13%, respectively. The parameterization of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0380" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> in Equation 4 provides an intuitive explanation. Namely, the ratio of precision contributed by affirmative studies versus nonaffirmative studies, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0381" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi></mrow></math> </ephtml> , is substantially higher for the atrial‐fibrillation meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0382" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo>=</mo><mn>2.18</mn></mrow></math> </ephtml> ), in which 57% of studies were affirmative, than for the meat‐consumption meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0383" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo>=</mo><mn>0.53</mn></mrow></math> </ephtml> ), in which only 25% of studies were affirmative.</p> <p>Additionally, for a given amount of publication bias, considerably weaker confounding associations would be required to explain away the estimate in the atrial‐fibrillation meta‐analysis than the meat‐consumption meta‐analysis (e.g., for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0384" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>=</mo><mn>4</mn></mrow></math> </ephtml> , the meta‐analytic E‐values for the point estimates were 1.75 and 2.82, respectively). Equation 4 again provides intuition: the proportions of precision in nonaffirmative and in affirmative studies that is contributed by potentially confounded studies is considerably larger for the atrial‐fibrillation meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0385" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub><mo>=</mo><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>=</mo><mn>1</mn></mrow></math> </ephtml> ), in which all studies were nonrandomized, than for the meat‐consumption meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0386" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub><mo>=</mo><mn>0.36</mn></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0387" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub><mo>=</mo><mn>0.28</mn></mrow></math> </ephtml> ), in which only a minority of studies were nonrandomized.</p> <p>These findings illustrate how applying quantitative sensitivity analyses can characterize sensitivity to combined biases in a manner that cannot easily be gleaned from the size of the uncorrected point estimate alone, nor from the number of affirmative studies. Nevertheless, the results of the sensitivity analyses can be understood in terms of other relatively intuitive characteristics of the meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0388" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0389" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0390" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub></mrow></math> </ephtml> ). It is important to note, however, that these findings do not necessarily mean that the atrial‐fibrillation meta‐analysis is less robust than the meat‐consumption meta‐analysis. As noted in Section 3.4.3, the amount of internal bias that would be considered implausible for a given meta‐analysis must be determined with attention to the design quality of the synthesized studies: a large amount of internal bias may be plausible for a set of poorly designed studies (e.g., with poor confounding control), but not for a set of better‐designed studies (e.g., in which the measured covariates already provide good control of confounding).[<reflink idref="bib21" id="ref134">21</reflink>] Additionally, the present re‐analyses focused only on uncontrolled confounding for illustrative purposes, but other biases could compromise both meta‐analyses. For example, both meta‐analysts noted potential outcome misclassification due to imperfect clinical ascertainment[<reflink idref="bib73" id="ref135">73</reflink>] or social desirability bias.[<reflink idref="bib72" id="ref136">72</reflink>] This form of bias could be assessed using the aforementioned sensitivity analyses for misclassification (Section 3.5).[<reflink idref="bib29" id="ref137">29</reflink>]</p> <hd id="AN0174546109-21">SIMULATION STUDY</hd> <p></p> <hd id="AN0174546109-22">Data generation</hd> <p>We conducted a simulation study to assess the performance of our proposed point estimate, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0391" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> , and inference in 12,480 scenarios, which varied by the following characteristics:</p> <p></p> <ulist> <item> Distribution of population effects (mean, heterogeneity, and skewness)</item> <p></p> <item> Distribution of within‐study standard errors</item> <p></p> <item> Number of meta‐analyzed nonaffirmative studies</item> <p></p> <item> Severity of internal bias and proportion of studies that were internally biased</item> <p></p> <item> Presence, severity, and type of publication bias</item> <p></p> <item> Presence and type of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0392" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking</item> </ulist> <p>For each of 1,000 simulation iterates per scenario, we generated a meta‐analysis whose underlying population effects were either normal or exponential. Normal population effects were generated as <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0393" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>N</mi><mfenced open="(" close=")" separators=","><mi>μ</mi><msup><mi>τ</mi><mn>2</mn></msup></mfenced></mrow></math> </ephtml> , where we varied <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0394" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi><mo>∈</mo><mfenced open="{" close="}"><mn>0,0.5</mn></mfenced></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0395" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo>∈</mo><mfenced open="{" close="}"><mn>0,0.25,0.5</mn></mfenced></mrow></math> </ephtml> . Exponential population effects were generated from an appropriately scaled and shifted distribution to achieve the desired population moments, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0396" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0397" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi>τ</mi><mn>2</mn></msup></mrow></math> </ephtml> . We generated studies' population standard errors, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0398" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , from one of two moderately right‐skewed distributions: one chosen to resemble empirical distributions[<reflink idref="bib15" id="ref138">15</reflink>] ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0399" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup><mo>∼</mo><mn>0.02</mn><mo>+</mo><mi>Exp</mi><mfenced open="(" close=")"><mn>3</mn></mfenced></mrow></math> </ephtml> ), and the other chosen to result in larger standard errors overall ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0400" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup><mo>∼</mo><mn>0.02</mn><mo>+</mo><mi>Exp</mi><mfenced open="(" close=")"><mn>1</mn></mfenced></mrow></math> </ephtml> ).</p> <p>Among underlying studies, either 50% or 100% had internal bias that manifested as a simple location shift in studies' point estimates (e.g., as would arise from confounding or similar biases), such that among internally biased studies, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0401" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup><mo>∼</mo><mi>N</mi><mfenced open="(" close=")" separators=","><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><msubsup><mi>σ</mi><mi>B</mi><mrow><mo>*</mo><mn>2</mn></mrow></msubsup></mfenced></mrow></math> </ephtml> , where we varied <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0402" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>∈</mo><mfenced open="{" close="}"><mn>0.1,0.25,0.5</mn></mfenced></mrow></math> </ephtml> and fixed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0403" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mn>0.5</mn></mrow></math> </ephtml> . For brevity, we will refer this internal bias as "confounding." We additionally included some scenarios in which all studies were subject to one of two forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0404" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking because, as noted in Section 2.2 and elsewhere,[<reflink idref="bib33" id="ref139">33</reflink>] <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0405" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking can result in pernicious forms of internal bias that violate the assumptions of standard selection models as well as our proposed methods. We simulated forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0406" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking resembling those in recent simulation studies,[[<reflink idref="bib33" id="ref140">33</reflink>], [<reflink idref="bib43" id="ref141">43</reflink>], [<reflink idref="bib54" id="ref142">54</reflink>], [<reflink idref="bib74" id="ref143">74</reflink>]] and we designed them to be pernicious to SM‐step and our proposed method based on theoretical results regarding <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0407" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref144">33</reflink>]</p> <p>In both types of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0408" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, investigators of each study obtained up to five point estimates but selected only one estimate to submit for publication. This single "favored"[<reflink idref="bib33" id="ref145">33</reflink>] estimate for each study was then subjected to publication bias (when present), as described below. In the first type of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0409" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, termed "<bold>favor‐lowest‐</bold><ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0410" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ," investigators obtained exactly five‐point estimates and favored the estimate with the lowest <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0411" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐value,[<reflink idref="bib33" id="ref146">33</reflink>] regardless of the sign of the estimate. In the second type of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0412" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, termed "<bold>favor‐first‐affirmative</bold>," investigators of each study obtained <emph>up to</emph> five estimates in an attempt to obtain an affirmative result.[<reflink idref="bib33" id="ref147">33</reflink>] If they obtained an affirmative result within five attempts, they favored this result and stopped obtaining more results. If they did not obtain an affirmative result by the fifth attempt, they gave up and favored the last (nonaffirmative) result obtained.</p> <p>Based on <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0413" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0414" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0415" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>B</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0416" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , we generated underlying studies from the hierarchical model in Equation 1. To obtain <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0417" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0418" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup></mrow></math> </ephtml> , we generated continuous subject‐level outcome data, calculated the sample mean, and conducted a one‐sample <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0419" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>t</mi></mrow></math> </ephtml> ‐test. In scenarios with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0420" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, we repeated this process within each study, as described above, to obtain multiple point estimates, exactly one of which investigators selected as the favored estimate. Thus, estimates were clustered within studies, but were not autocorrelated within a study. We chose to simulate independent estimates within a study because autocorrelation would reduce the effective number of estimates within a study, thus reducing bias due to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0421" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref148">33</reflink>] The type of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0422" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking we simulated could arise in practice if, for example, investigators fit models to disjoint subsets of the data.[<reflink idref="bib32" id="ref149">32</reflink>] Many other forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0423" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking are possible[[<reflink idref="bib33" id="ref150">33</reflink>], [<reflink idref="bib75" id="ref151">75</reflink>]]; we simply chose one mechanism for this non‐comprehensive simulation study.</p> <p>We then introduced one of two types of publication bias. First, "<bold>stepwise publication bias</bold>" conformed to the assumed model in Section 2, in which we varied <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0424" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> from <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0425" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>1</mn></mrow></math> </ephtml> (no publication bias) to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0426" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>10</mn></mrow></math> </ephtml> . Similar mechanisms of publication bias have been studied in several previous simulation studies.[[<reflink idref="bib14" id="ref152">14</reflink>], [<reflink idref="bib42" id="ref153">42</reflink>], [<reflink idref="bib46" id="ref154">46</reflink>]] Thus, for scenarios with stepwise publication bias but no <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0427" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, our method was correctly specified. Second, "<bold>fuzzy publication bias</bold>," was implemented as in two previous simulation studies[[<reflink idref="bib43" id="ref155">43</reflink>], [<reflink idref="bib54" id="ref156">54</reflink>]] and involved a complex selection model in which studies with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0428" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo><</mo><mn>0.05</mn></mrow></math> </ephtml> were always published regardless of direction, and in which the probability of publication declined nonlinearly in <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0429" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> for studies with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0430" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo>≥</mo><mn>0.05</mn></mrow></math> </ephtml> (Figure 2).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01jan24/jrsm1667-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1667-fig-0002.jpg" title="2 Under fuzzy publication bias, the publication probability as a function of a study's p‐value." /> </p> <p></p> <p>Because the performance of our proposed methods depends more on the number of published, nonaffirmative estimates in a meta‐analysis (denoted <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0432" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>#</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></math> </ephtml> ) than on the total number of published estimates, we varied <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0433" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>#</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></math> </ephtml> between 5 and 50 across scenarios. We generated underlying studies until we had reached the target <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0434" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>#</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></mrow></math> </ephtml> . Across scenarios, the median number of published affirmative studies was 34 (25th percentile: 14; 75th percentile: 75), and the median total number of published studies was 56 (25th percentile: 25; 25th percentile: 96).</p> <hd id="AN0174546109-24">Estimation methods</hd> <p>To implement our proposed method, we estimated <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0435" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> per Equation 2, using the correctly‐specified <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0436" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> and after estimating <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0437" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> from Equation 5. Because we chose mechanisms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0438" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking that violate our modeling assumptions, we implemented our method using values of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0439" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0440" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup></mrow></math> </ephtml> that ignored the presence of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0441" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking. For comparison, we also fit an uncorrected, robust random‐effects meta‐analysis[[<reflink idref="bib50" id="ref157">50</reflink>], [<reflink idref="bib52" id="ref158">52</reflink>]] and two different selection models, both of which attempt to correct for publication bias but not for internal bias. First, we fit a two‐parameter selection model ("<bold>SM‐step</bold>"), a well‐established method to correct for publication bias that favors affirmative studies.[[<reflink idref="bib13" id="ref159">13</reflink>], [<reflink idref="bib76" id="ref160">76</reflink>]] We fit a widely used version of this model in which, for some selection ratio <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0442" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> , the publication probability given a study's affirmative status is assumed to be <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0443" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mfenced open="(" close=")" separators="|"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup></mfenced><mo>∝</mo><msup><mi>η</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>0</mn></mrow></mfenced><mo>+</mo><mi mathvariant="double-struck">1</mi><mfenced open="{" close="}"><mrow><msubsup><mi>A</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn></mrow></mfenced></mrow></math> </ephtml> . Because this model does not accommodate internal bias and is typically fit under the assumption that the underlying population effects are normal, the published estimates are assumed to have the following likelihood[<reflink idref="bib13" id="ref161">13</reflink>]: <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0444" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>f</mi><mrow><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></msub><mfenced open="(" close=")"><mi>t</mi></mfenced><mo>=</mo><mfrac><mrow><msup><mi>η</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mn mathvariant="double-struck">1</mn><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow></mfenced><mo>+</mo><mn mathvariant="double-struck">1</mn><mfenced open="{" close="}"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mfenced></mrow><mrow><msup><mi>η</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mi mathvariant="normal">Φ</mi><mfenced open="(" close=")"><mfrac><mrow><mi>c</mi><msub><mi>σ</mi><mi>i</mi></msub><mo>−</mo><mi>μ</mi></mrow><mrow><msub><mi>S</mi><mi>i</mi></msub></mrow></mfrac></mfenced><mo>+</mo><mfenced open="[" close="]"><mrow><mn>1</mn><mo>−</mo><mi mathvariant="normal">Φ</mi><mfenced open="(" close=")"><mfrac><mrow><mi>c</mi><msub><mi>σ</mi><mi>i</mi></msub><mo>−</mo><mi>μ</mi></mrow><mrow><msub><mi>S</mi><mi>i</mi></msub></mrow></mfrac></mfenced></mrow></mfenced></mrow></mfrac><mo>⋅</mo><msup><mi>f</mi><mo>*</mo></msup><mfenced open="(" close=")"><mi>t</mi></mfenced></mrow></math> </ephtml> where <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0445" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>c</mi><mo>≈</mo><mn>1.96</mn></mrow></math> </ephtml> is the critical value on the <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0446" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Z</mi></mrow></math> </ephtml> ‐score scale defining affirmative status, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0447" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>S</mi><mi>i</mi></msub><mo>=</mo><msqrt><mrow><msup><mi>τ</mi><mn>2</mn></msup><mo>+</mo><msubsup><mi>σ</mi><mi>i</mi><mn>2</mn></msubsup></mrow></msqrt></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0448" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="normal">Φ</mi></mrow></math> </ephtml> is the cumulative distribution function of the standard normal distribution. The underlying likelihood for a study whose standard error is equal to the observed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0449" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>σ</mi><mi>i</mi></msub></mrow></math> </ephtml> is termed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0450" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mi>f</mi><mo>*</mo></msup><mfenced open="(" close=")"><mi>t</mi></mfenced></mrow></math> </ephtml> and is usually taken to be normal. As usual, we fit the model by maximum likelihood estimation to jointly estimate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0451" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>μ</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0452" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0453" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi></mrow></math> </ephtml> .[[<reflink idref="bib12" id="ref162">12</reflink>]] SM‐step is nested within the model we propose; that is, SM‐step makes stronger assumptions.</p> <p>Second, we fit a more recently proposed selection model ("<bold>SM‐beta</bold>") that uses the beta density to model a study's probability of publication given its <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0454" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐value, thus accommodating certain forms of selection that favor, for example, smaller <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0455" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐values or larger point estimates even conditional on affirmative status.[[<reflink idref="bib10" id="ref163">10</reflink>], [<reflink idref="bib77" id="ref164">77</reflink>]] This model is similar to SM‐step except that a study's publication probability is assumed to be a continuous function of the study's one‐tailed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0456" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐value, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0457" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>1</mn><mo>−</mo><mi mathvariant="normal">Φ</mi><mfenced open="(" close=")"><mrow><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>/</mo><msub><mi>σ</mi><mi>i</mi></msub></mrow></mfenced></mrow></math> </ephtml> . That is, for two unknown shape parameters, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0458" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>δ</mi><mn>1</mn></msub><mo>></mo><mn>0</mn></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0459" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>δ</mi><mn>2</mn></msub><mo>></mo><mn>0</mn></mrow></math> </ephtml> , SM‐beta assumes that: <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0460" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mfenced open="(" close=")" separators="|"><mrow><msubsup><mi>D</mi><mi>i</mi><mo>*</mo></msubsup><mo>=</mo><mn>1</mn><mo>|</mo><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></mfenced><mo>∝</mo><msup><mfenced open="[" close="]"><mrow><mn>1</mn><mo>−</mo><mi mathvariant="normal">Φ</mi><mfenced open="(" close=")"><mrow><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>/</mo><msub><mi>σ</mi><mi>i</mi></msub></mrow></mfenced></mrow></mfenced><mrow><msub><mi>δ</mi><mn>1</mn></msub><mo>−</mo><mn>1</mn></mrow></msup><mo>⋅</mo><msup><mfenced open="[" close="]"><mrow><mi mathvariant="normal">Φ</mi><mfenced open="(" close=")"><mrow><msub><mover accent="true"><mi mathvariant="normal">Δ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>/</mo><msub><mi>σ</mi><mi>i</mi></msub></mrow></mfenced></mrow></mfenced><mrow><msub><mi>δ</mi><mn>2</mn></msub><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml></p> <p>The resulting likelihood takes an analogous form to that of SM‐step.[<reflink idref="bib10" id="ref165">10</reflink>]</p> <hd id="AN0174546109-25">Model misspecification</hd> <p>The uncorrected meta‐analysis was misspecified in the presence of confounding, publication bias, or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0461" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking (i.e., in all scenarios). Both selection models were misspecified in the presence of confounding or <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0462" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking.[<reflink idref="bib33" id="ref166">33</reflink>] SM‐step was additionally misspecified in the presence of fuzzy publication bias, which was selected for smaller <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0463" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐values (or equivalently, for larger effect sizes) in a continuous rather than stepwise manner and which also operated symmetrically with respect to estimates' signs. SM‐beta was misspecified under either type of publication bias, though only mildly so for fuzzy publication bias. Our proposed method was misspecified in the presence of either form of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0464" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking or in the presence of fuzzy publication bias.</p> <hd id="AN0174546109-26">Metrics of estimators' performance</hd> <p>For each scenario, we assessed the point estimators' performance and variability in terms of their mean bias, mean absolute error (MAE), and root‐mean‐square error (RMSE), defined as follows for a generic parameter <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0465" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>ω</mi><mi>r</mi></msub></mrow></math> </ephtml> that varies across simulation iterates, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0466" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi></mrow></math> </ephtml> : <ephtml> <math display="block" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0467" xmlns="http://www.w3.org/1998/Math/MathML"><mrow>Bias=11000∑r=11000ω̂r−ωrMAE=11000∑r=11000∣ω̂r−ωr∣RMSE=11000∑r=11000ω̂r−ωr21/2</mrow></math> </ephtml></p> <p>For each scenario, we assessed inference in terms of the coverage and width of 95% confidence intervals. We also report convergence, defined as the proportion of simulation iterates for each scenario for which a method produced a point estimate and confidence interval. When summarizing results across scenarios, we report medians because the metrics were often skewed across scenarios. To help characterize variability in results across scenarios, we also report on each method's performance in the 10% of scenarios in which the method performed the worst.[<reflink idref="bib78" id="ref167">78</reflink>] For each respective performance metric, we defined "worse" performance as larger values of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0468" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>∣</mo><mtext>bias</mtext><mo>∣</mo></mrow></math> </ephtml> , MAE, RMSE, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0469" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>∣</mo><mtext>coverage</mtext><mo>−</mo><mn>0.95</mn><mo>∣</mo></mrow></math> </ephtml> , confidence interval width, and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0470" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mfenced open="(" close=")"><mrow><mn>1</mn><mo>−</mo><mtext>convergence</mtext></mrow></mfenced></mrow></math> </ephtml> .</p> <hd id="AN0174546109-27">Results</hd> <p>Given the very large number of simulation scenarios, we focus on the following subsets of the simulation scenarios. We chose these subsets based on predictions from statistical theory and from peer reviewers about which characteristics would likely affect methods' performances.</p> <p></p> <ulist> <item> All 12,480 scenarios</item> <p></p> <item> The 2,154 scenarios in which our method was correctly specified (i.e., excluding scenarios with fuzzy publication bias or with <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0471" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking)</item> <p></p> <item> The 10,326 scenarios in which our method was incorrectly specified</item> <p></p> <item> The 2,592 scenarios with very few published nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0472" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>#</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup><mo>=</mo><mn>5</mn></mrow></math> </ephtml> )</item> <p></p> <item> The 4,160 scenarios with relatively little internal bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0473" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>μ</mi><mi>B</mi></msub><mo>=</mo><mn>0.1</mn></mrow></math> </ephtml> )</item> <p></p> <item> The 6,420 scenarios with larger within‐study standard errors ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0474" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup><mo>∼</mo><mn>0.02</mn><mo>+</mo><mi>Exp</mi><mfenced open="(" close=")"><mn>1</mn></mfenced></mrow></math> </ephtml> )</item> <p></p> <item> The 6,225 scenarios in which population effects were skewed</item> </ulist> <p>For the subsets that most affected methods' performances, we present tables showing median and worst‐10% performances of each method, ordering the methods in descending order of performance.[<reflink idref="bib43" id="ref168">43</reflink>] We qualitatively discuss results for the other subsets, and comprehensive results for each individual simulation scenario are publicly available as a dataset (https://osf.io/vr7y6/). As discussed below, the proposed method generally had better numerical performance metrics than the other methods, but importantly, this does not mean that the proposed method "outperformed" existing methods, which serve a different purpose. That is, our method is a sensitivity analysis, which involves specifying or solving for parameters regarding the severity of biases, whereas the existing methods directly estimate such parameters. Thus, the proposed method is not a replacement for existing methods but rather is complementary.</p> <p>Additionally, although most methods had convergence rates near 100% across scenarios, SM‐beta had a median convergence rate of only 70% across scenarios. Therefore, results for this method are not directly comparable to those of other methods due to missing data. These low convergence rates of SM‐beta were consistent with previous results for this model,[<reflink idref="bib10" id="ref169">10</reflink>] as well as other results that generally suggest convergence challenges for flexible selection models.[<reflink idref="bib43" id="ref170">43</reflink>]</p> <hd id="AN0174546109-28">All scenarios</hd> <p>Tables 3 and 4 display results across all scenarios, including numerous scenarios in which our proposed method was misspecified (83% of all scenarios). At a high level, the proposed method achieved the best numerical performances across all metrics, except that its confidence intervals were sometimes wider than those of certain other methods whose coverage was substantially compromised. We now discuss each method in more detail.</p> <p>3 TABLE All scenarios; median performances.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.01</td><td align="left">Proposed</td><td align="char" char=".">0.15</td><td align="left">Proposed</td><td align="char" char=".">0.18</td><td align="left">Proposed</td><td align="char" char=".">0.95</td><td align="left">SM‐beta</td><td align="char" char=".">0.54</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.05</td><td align="left">SM‐step</td><td align="char" char=".">0.29</td><td align="left">SM‐step</td><td align="char" char=".">0.35</td><td align="left">SM‐step</td><td align="char" char=".">0.85</td><td align="left">Uncorrected</td><td align="char" char=".">0.55</td><td align="left">SM‐step</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.31</td><td align="left">SM‐beta</td><td align="char" char=".">0.34</td><td align="left">SM‐beta</td><td align="char" char=".">0.40</td><td align="left">SM‐beta</td><td align="char" char=".">0.43</td><td align="left">Proposed</td><td align="char" char=".">0.63</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.47</td><td align="left">Uncorrected</td><td align="char" char=".">0.48</td><td align="left">Uncorrected</td><td align="char" char=".">0.52</td><td align="left">Uncorrected</td><td align="char" char=".">0.13</td><td align="left">SM‐step</td><td align="char" char=".">0.96</td><td align="left">SM‐beta</td><td align="char" char=".">0.70</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</p> <ulist> <item>2 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>4 TABLE All scenarios; worst 10% of performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.23</td><td align="left">Proposed</td><td align="char" char=".">0.33</td><td align="left">Proposed</td><td align="char" char=".">0.39</td><td align="left">Proposed</td><td align="char" char=".">0.40</td><td align="left">SM‐beta</td><td align="char" char=".">1.17</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.42</td><td align="left">SM‐step</td><td align="char" char=".">0.53</td><td align="left">SM‐step</td><td align="char" char=".">0.69</td><td align="left">SM‐step</td><td align="char" char=".">0.38</td><td align="left">Uncorrected</td><td align="char" char=".">1.36</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.65</td><td align="left">SM‐beta</td><td align="char" char=".">0.66</td><td align="left">SM‐beta</td><td align="char" char=".">0.73</td><td align="left">SM‐beta</td><td align="char" char=".">0.01</td><td align="left">Proposed</td><td align="char" char=".">1.54</td><td align="left">SM‐step</td><td align="char" char=".">0.99</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.91</td><td align="left">Uncorrected</td><td align="char" char=".">0.91</td><td align="left">Uncorrected</td><td align="char" char=".">0.92</td><td align="left">Uncorrected</td><td align="char" char=".">0.00</td><td align="left">SM‐step</td><td align="char" char=".">1.83</td><td align="left">SM‐beta</td><td align="char" char=".">0.54</td></tr></tbody></table> </ephtml> </p> <ulist> <item>3 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>4 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> </ulist> <p>As expected, the uncorrected meta‐analysis performed poorly on average across all performance metrics, showing substantial positive bias, MAE, RMSE, and under‐coverage. Our proposed <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0475" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> and SM‐step had relatively little bias on average across scenarios (Table 3), but it is important to note that when misspecified, each of these methods could be biased in either direction. This may be counterintuitive given that the forms of bias we simulated yielded a positively biased uncorrected estimate. Negative bias in our <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0476" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> and SM‐step typically arose in scenarios with fuzzy publication bias, which selected for estimates in either direction, whereas these two methods consider selection for positive (affirmative) estimates.[<reflink idref="bib14" id="ref171">14</reflink>] Additionally, SM‐step was typically negatively biased when there was <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0477" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking, for reasons detailed elsewhere.[<reflink idref="bib33" id="ref172">33</reflink>][<reflink idref="bib5" id="ref173">5</reflink>] Thus, despite the minimal average bias of these methods, their worst‐10% biases were substantial (Table 4). However, the worst‐10% bias for our <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0478" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> was considerably lower than for SM‐step. Both on average and when considering worst‐10% performances, our proposed method had substantially better MAE, RMSE, coverage, and CI width than SM‐step. Both on average and when considering worst‐10% performances, SM‐beta appeared to perform better than the uncorrected meta‐analysis, but considerably worse than the proposed method and SM‐step. Again, we note the difficulty of comparing this method to the others given its low convergence rate.</p> <hd id="AN0174546109-29">Scenarios in which our method was correctly specified</hd> <p>These scenarios yielded a similar pattern of relative performances (Tables 5 and 6) to those for all scenarios, but with larger differences between the performances of our proposed methods and the existing ones. As expected, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0479" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> was unbiased on average and was minimally biased in its worst‐10% performances. Whereas each existing method achieved less than 80% coverage on average and less than 25% in their worst‐10% performance, the proposed method exhibited over‐coverage (98% coverage on average and 96% in its worst‐10% performance). This over‐coverage is attributable to our use of a small‐sample correction to the robust variance estimates, which can lead to conservative inference even without bias corrections.[<reflink idref="bib51" id="ref174">51</reflink>] However, as discussed in Section 3.2, using the small‐sample correction is important to avoid substantial under‐coverage for small meta‐analyses, which frequently occurs even in standard meta‐analyses without bias corrections[<reflink idref="bib51" id="ref175">51</reflink>] as well as in meta‐analyses that correct for publication bias alone.[<reflink idref="bib14" id="ref176">14</reflink>] Despite the over‐coverage, the confidence intervals for <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0480" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> were on average narrower than those of both selection models (which, again, had far below nominal coverage), though were wider than those of the severely biased uncorrected meta‐analysis.</p> <p>5 TABLE Scenarios in which proposed method was correctly specified; median performances.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.00</td><td align="left">Proposed</td><td align="char" char=".">0.08</td><td align="left">Proposed</td><td align="char" char=".">0.10</td><td align="left">Proposed</td><td align="char" char=".">0.98</td><td align="left">Uncorrected</td><td align="char" char=".">0.47</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">0.24</td><td align="left">SM‐step</td><td align="char" char=".">0.29</td><td align="left">SM‐step</td><td align="char" char=".">0.34</td><td align="left">SM‐step</td><td align="char" char=".">0.79</td><td align="left">Proposed</td><td align="char" char=".">0.55</td><td align="left">SM‐step</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.42</td><td align="left">SM‐beta</td><td align="char" char=".">0.44</td><td align="left">SM‐beta</td><td align="char" char=".">0.49</td><td align="left">SM‐beta</td><td align="char" char=".">0.21</td><td align="left">SM‐beta</td><td align="char" char=".">0.57</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.49</td><td align="left">Uncorrected</td><td align="char" char=".">0.50</td><td align="left">Uncorrected</td><td align="char" char=".">0.52</td><td align="left">Uncorrected</td><td align="char" char=".">0.05</td><td align="left">SM‐step</td><td align="char" char=".">0.81</td><td align="left">SM‐beta</td><td align="char" char=".">0.70</td></tr></tbody></table> </ephtml> </p> <ulist> <item>5 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>6 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>6 TABLE Scenarios in which proposed method was correctly specified; worst 10% of performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">−0.01</td><td align="left">Proposed</td><td align="char" char=".">0.18</td><td align="left">Proposed</td><td align="char" char=".">0.23</td><td align="left">Proposed</td><td align="char" char=".">0.96</td><td align="left">Uncorrected</td><td align="char" char=".">1.09</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">0.51</td><td align="left">SM‐step</td><td align="char" char=".">0.52</td><td align="left">SM‐step</td><td align="char" char=".">0.62</td><td align="left">SM‐step</td><td align="char" char=".">0.20</td><td align="left">Proposed</td><td align="char" char=".">1.29</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.70</td><td align="left">SM‐beta</td><td align="char" char=".">0.71</td><td align="left">SM‐beta</td><td align="char" char=".">0.78</td><td align="left">SM‐beta</td><td align="char" char=".">0.00</td><td align="left">SM‐beta</td><td align="char" char=".">1.29</td><td align="left">SM‐step</td><td align="char" char=".">0.97</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.84</td><td align="left">Uncorrected</td><td align="char" char=".">0.84</td><td align="left">Uncorrected</td><td align="char" char=".">0.85</td><td align="left">Uncorrected</td><td align="char" char=".">0.00</td><td align="left">SM‐step</td><td align="char" char=".">1.54</td><td align="left">SM‐beta</td><td align="char" char=".">0.52</td></tr></tbody></table> </ephtml> </p> <ulist> <item>7 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>8 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> </ulist> <hd id="AN0174546109-30">Scenarios in which our method was incorrectly specified</hd> <p>Tables 7 and 8 show results for these scenarios. The methods' relative performances remained similar to those in all scenarios, and numerical results were fairly similar to those seen in all scenarios. This is expected given that this subset represented 83% of all scenarios.</p> <p>7 TABLE Scenarios in which proposed method was incorrectly specified; median performances.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.06</td><td align="left">Proposed</td><td align="char" char=".">0.18</td><td align="left">Proposed</td><td align="char" char=".">0.20</td><td align="left">Proposed</td><td align="char" char=".">0.93</td><td align="left">SM‐beta</td><td align="char" char=".">0.54</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.11</td><td align="left">SM‐step</td><td align="char" char=".">0.29</td><td align="left">SM‐step</td><td align="char" char=".">0.36</td><td align="left">SM‐step</td><td align="char" char=".">0.86</td><td align="left">Uncorrected</td><td align="char" char=".">0.57</td><td align="left">SM‐step</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.28</td><td align="left">SM‐beta</td><td align="char" char=".">0.32</td><td align="left">SM‐beta</td><td align="char" char=".">0.38</td><td align="left">SM‐beta</td><td align="char" char=".">0.47</td><td align="left">Proposed</td><td align="char" char=".">0.64</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.46</td><td align="left">Uncorrected</td><td align="char" char=".">0.48</td><td align="left">Uncorrected</td><td align="char" char=".">0.52</td><td align="left">Uncorrected</td><td align="char" char=".">0.15</td><td align="left">SM‐step</td><td align="char" char=".">0.99</td><td align="left">SM‐beta</td><td align="char" char=".">0.70</td></tr></tbody></table> </ephtml> </p> <ulist> <item>9 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>10 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>8 TABLE Scenarios in which proposed method was incorrectly specified; worst 10% of performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.25</td><td align="left">Proposed</td><td align="char" char=".">0.35</td><td align="left">Proposed</td><td align="char" char=".">0.40</td><td align="left">SM‐step</td><td align="char" char=".">0.43</td><td align="left">SM‐beta</td><td align="char" char=".">1.15</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.44</td><td align="left">SM‐step</td><td align="char" char=".">0.54</td><td align="left">SM‐step</td><td align="char" char=".">0.70</td><td align="left">Proposed</td><td align="char" char=".">0.32</td><td align="left">Uncorrected</td><td align="char" char=".">1.42</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.63</td><td align="left">SM‐beta</td><td align="char" char=".">0.64</td><td align="left">SM‐beta</td><td align="char" char=".">0.72</td><td align="left">SM‐beta</td><td align="char" char=".">0.01</td><td align="left">Proposed</td><td align="char" char=".">1.58</td><td align="left">SM‐step</td><td align="char" char=".">0.99</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.92</td><td align="left">Uncorrected</td><td align="char" char=".">0.92</td><td align="left">Uncorrected</td><td align="char" char=".">0.94</td><td align="left">Uncorrected</td><td align="char" char=".">0.00</td><td align="left">SM‐step</td><td align="char" char=".">1.87</td><td align="left">SM‐beta</td><td align="char" char=".">0.54</td></tr></tbody></table> </ephtml> </p> <ulist> <item>11 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>12 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> </ulist> <hd id="AN0174546109-31">Other scenario characteristics</hd> <p>For each subset described below, our proposed method's performance relative to the others remained the same as in all scenarios. That is, our proposed method had better numerical performances on all metrics except confidence interval width (and again, it was the only method that did not display under‐coverage). The other methods' relative rankings were mostly unchanged, with one exception noted below. Tables 9 and 10 show results for scenarios with only five published, nonaffirmative studies ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0481" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>#</mo><msup><mi mathvariant="script">A</mi><mi>c</mi></msup><mo>=</mo><mn>5</mn></mrow></math> </ephtml> ). As expected, all methods became less precise. The average coverage rates of SM‐beta and the uncorrected meta‐analysis improved somewhat (presumably due to the increased confidence interval width), though the coverage rates of the proposed method and SM‐step remained essentially the same. Tables 11 and 12 show results for scenarios with relatively little internal bias ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0482" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>μ</mi><mi>B</mi><mo>*</mo></msubsup><mo>=</mo><mn>0.1</mn></mrow></math> </ephtml> ). In this subset, the relative rankings of the existing methods changed in that SM‐beta appeared to outperform SM‐step on most metrics, again with the caveat that SM‐beta had low convergence rates. In scenarios with larger within‐study standard errors ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0483" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>σ</mi><mi>i</mi><mo>*</mo></msubsup><mo>∼</mo><mn>0.02</mn><mo>+</mo><mi>Exp</mi><mfenced open="(" close=")"><mn>1</mn></mfenced></mrow></math> </ephtml> ), all methods became less precise, as expected, but results otherwise remained similar. In scenarios with skewed population effects, results also resembled those for all scenarios.</p> <p>9 TABLE Scenarios with #Ac=5; median performances.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.02</td><td align="left">Proposed</td><td align="char" char=".">0.21</td><td align="left">Proposed</td><td align="char" char=".">0.27</td><td align="left">Proposed</td><td align="char" char=".">0.97</td><td align="left">SM‐beta</td><td align="char" char=".">0.99</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.02</td><td align="left">SM‐beta</td><td align="char" char=".">0.40</td><td align="left">SM‐beta</td><td align="char" char=".">0.53</td><td align="left">SM‐step</td><td align="char" char=".">0.85</td><td align="left">Uncorrected</td><td align="char" char=".">1.13</td><td align="left">SM‐step</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.30</td><td align="left">SM‐step</td><td align="char" char=".">0.41</td><td align="left">SM‐step</td><td align="char" char=".">0.55</td><td align="left">SM‐beta</td><td align="char" char=".">0.75</td><td align="left">Proposed</td><td align="char" char=".">1.36</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.44</td><td align="left">Uncorrected</td><td align="char" char=".">0.50</td><td align="left">Uncorrected</td><td align="char" char=".">0.57</td><td align="left">Uncorrected</td><td align="char" char=".">0.53</td><td align="left">SM‐step</td><td align="char" char=".">1.71</td><td align="left">SM‐beta</td><td align="char" char=".">0.76</td></tr></tbody></table> </ephtml> </p> <ulist> <item>13 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>14 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>10 TABLE Scenarios with #Ac=5; worst 10% of performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.22</td><td align="left">Proposed</td><td align="char" char=".">0.38</td><td align="left">Proposed</td><td align="char" char=".">0.47</td><td align="left">Proposed</td><td align="char" char=".">0.81</td><td align="left">Uncorrected</td><td align="char" char=".">2.03</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.38</td><td align="left">SM‐beta</td><td align="char" char=".">0.68</td><td align="left">Uncorrected</td><td align="char" char=".">0.93</td><td align="left">SM‐step</td><td align="char" char=".">0.73</td><td align="left">SM‐beta</td><td align="char" char=".">2.05</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.64</td><td align="left">SM‐step</td><td align="char" char=".">0.68</td><td align="left">SM‐beta</td><td align="char" char=".">0.99</td><td align="left">SM‐beta</td><td align="char" char=".">0.11</td><td align="left">Proposed</td><td align="char" char=".">2.47</td><td align="left">SM‐step</td><td align="char" char=".">0.96</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.88</td><td align="left">Uncorrected</td><td align="char" char=".">0.89</td><td align="left">SM‐step</td><td align="char" char=".">1.49</td><td align="left">Uncorrected</td><td align="char" char=".">0.02</td><td align="left">SM‐step</td><td align="char" char=".">2.79</td><td align="left">SM‐beta</td><td align="char" char=".">0.67</td></tr></tbody></table> </ephtml> </p> <ulist> <item>15 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>16 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>11 TABLE Scenarios with μB*=0.1; median performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">−0.23</td><td align="left">Proposed</td><td align="char" char=".">0.33</td><td align="left">Proposed</td><td align="char" char=".">0.39</td><td align="left">Proposed</td><td align="char" char=".">0.49</td><td align="left">SM‐beta</td><td align="char" char=".">1.19</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">−0.50</td><td align="left">SM‐beta</td><td align="char" char=".">0.52</td><td align="left">SM‐beta</td><td align="char" char=".">0.61</td><td align="left">SM‐step</td><td align="char" char=".">0.36</td><td align="left">Uncorrected</td><td align="char" char=".">1.48</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.51</td><td align="left">SM‐step</td><td align="char" char=".">0.58</td><td align="left">Uncorrected</td><td align="char" char=".">0.74</td><td align="left">SM‐beta</td><td align="char" char=".">0.02</td><td align="left">Proposed</td><td align="char" char=".">1.65</td><td align="left">SM‐step</td><td align="char" char=".">0.98</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.73</td><td align="left">Uncorrected</td><td align="char" char=".">0.73</td><td align="left">SM‐step</td><td align="char" char=".">0.82</td><td align="left">Uncorrected</td><td align="char" char=".">0.00</td><td align="left">SM‐step</td><td align="char" char=".">1.89</td><td align="left">SM‐beta</td><td align="char" char=".">0.55</td></tr></tbody></table> </ephtml> </p> <ulist> <item>17 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>18 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> <item>12 TABLE Scenarios with μB*=0.1; worst 10% of performances.</item> </ulist> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Bias</th><th align="left">MAE</th><th align="left">RMSE</th><th align="left">CI coverage</th><th align="left">CI width</th><th align="left">Convergence</th></tr></thead><tbody valign="top"><tr><td align="left">Proposed</td><td align="char" char=".">0.30</td><td align="left">Proposed</td><td align="char" char=".">0.33</td><td align="left">Proposed</td><td align="char" char=".">0.39</td><td align="left">Proposed</td><td align="char" char=".">0.49</td><td align="left">SM‐beta</td><td align="char" char=".">1.19</td><td align="left">Proposed</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐beta</td><td align="char" char=".">0.51</td><td align="left">SM‐beta</td><td align="char" char=".">0.52</td><td align="left">SM‐beta</td><td align="char" char=".">0.61</td><td align="left">SM‐step</td><td align="char" char=".">0.36</td><td align="left">Uncorrected</td><td align="char" char=".">1.48</td><td align="left">Uncorrected</td><td align="char" char=".">1.00</td></tr><tr><td align="left">SM‐step</td><td align="char" char=".">0.51</td><td align="left">SM‐step</td><td align="char" char=".">0.58</td><td align="left">Uncorrected</td><td align="char" char=".">0.74</td><td align="left">SM‐beta</td><td align="char" char=".">0.02</td><td align="left">Proposed</td><td align="char" char=".">1.65</td><td align="left">SM‐step</td><td align="char" char=".">0.98</td></tr><tr><td align="left">Uncorrected</td><td align="char" char=".">0.73</td><td align="left">Uncorrected</td><td align="char" char=".">0.73</td><td align="left">SM‐step</td><td align="char" char=".">0.82</td><td align="left">Uncorrected</td><td align="char" char=".">0.00</td><td align="left">SM‐step</td><td align="char" char=".">1.89</td><td align="left">SM‐beta</td><td align="char" char=".">0.55</td></tr></tbody></table> </ephtml> </p> <ulist> <item>19 <emph>Note</emph>: Methods are sorted from best to worst performance within each column. Convergence: Proportion of simulation iterates for which the method produced a point estimate and confidence interval.</item> <item>20 Abbreviations: CI, 95% confidence interval; MAE, mean absolute error; RMSE, root‐mean‐square error.</item> </ulist> <hd id="AN0174546109-32">Summary</hd> <p>In summary, across all scenarios and in each subset we considered, the proposed method achieved the best numerical performances across all metrics, except that its confidence intervals were sometimes wider than those of certain other methods whose coverage was substantially compromised. Additionally, as noted, our proposed method sometimes exhibited over‐coverage, which is a limitation. However, we reiterate that these relative performances between our proposed sensitivity analysis and existing estimation methods should not be interpreted as true rankings given the methods' differing objectives. Additionally, although our simulation study was extensive, it was not exhaustive, and there may be other scenarios we did not explore in which methods' relative performances would change. Not exhaustively, such scenarios could include those with different forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0488" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking (e.g., different numbers of estimates produced per study, or different strengths of correlation between estimates) or with different forms of publication bias (e.g., step functions with different numbers of cut points).</p> <hd id="AN0174546109-33">DISCUSSION</hd> <p>We have proposed two types of sensitivity analyses: (<reflink idref="bib1" id="ref177">1</reflink>) a consistent meta‐analytic estimate, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0489" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> , and variance estimate when correcting for specified amounts of internal bias and publication bias; and (<reflink idref="bib2" id="ref178">2</reflink>) for a given severity of publication bias, the severity of average internal bias that would be required to attenuate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0490" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> (or its confidence interval) to the null or to a chosen non‐null value. The methods accommodate the possibility that some studies are known to be internally unbiased, though this is not required; and they allow consideration of a fixed severity of publication bias or of worst‐case publication bias. The methods can be easily applied using the R package multibiasmeta.</p> <p>These methods can provide insights about robustness that cannot be obtained simply by considering the size of the uncorrected meta‐analytic estimate, the proportion of studies that are affirmative, and the typical amount of internal bias in the constituent studies. As the applied examples illustrated, a meta‐analysis with a larger point estimate or more affirmative studies may not be more robust to combined internal bias and publication bias than a meta‐analysis with a smaller estimate or fewer affirmative studies. Nevertheless, the form of the bias‐corrected estimator, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0491" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> , indicates that the results of the sensitivity analyses can be understood in terms of relatively intuitive characteristics of the meta‐analysis ( <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0492" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0493" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><msup><mi mathvariant="script">A</mi><mi>c</mi></msup></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0494" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>λ</mi><mi mathvariant="script">A</mi></msub></mrow></math> </ephtml> ). A meta‐analysis will, of course, be more robust if publication bias and the average internal bias are less severe. But less intuitively, for a <emph>given</emph> severity of publication bias and average internal bias across published studies, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0495" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> is larger (indicating better robustness, all else equal) when: (<reflink idref="bib1" id="ref179">1</reflink>) a random‐effects estimate in only the published nonaffirmative studies is large; and (<reflink idref="bib2" id="ref180">2</reflink>) the internal bias primarily affects the affirmative studies rather than the nonaffirmative studies. The random‐effects estimate and severity of internal bias in the published nonaffirmative studies matter more to <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0496" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mover accent="true"><mi>μ</mi><mo>̂</mo></mover><mi>adj</mi></msub></mrow></math> </ephtml> than the comparable quantities in the affirmative studies.</p> <p>Stated otherwise, one might intuitively think that if two meta‐analyses are comparably affected by publication bias, and their constituent studies have the same average amount of internal bias, the meta‐analyses would be comparable in their degree of robustness to the combined biases. However, because the biases interact, this intuition may fail if the distribution of internal bias between the affirmative and nonaffirmative studies differs between the two meta‐analyses. Suppose that, in one meta‐analysis, the studies at highest risk of bias are almost always affirmative whereas in the other, these high‐risk studies are sometimes affirmative and sometimes nonaffirmative. In that case, our proposed methods would likely indicate that, all else equal, the former meta‐analysis is more robust to the combined biases. In fact, the former case may occur more often in practice, because positive internal bias will tend to produce affirmative results.</p> <p>These methods have limitations. The methods assume a particular model of publication bias in which affirmative studies are more likely to be published. Although this model aligns well with empirical evidence on how researchers interpret <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0497" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐values,[[<reflink idref="bib15" id="ref181">15</reflink>], [<reflink idref="bib34" id="ref182">34</reflink>], [<reflink idref="bib36" id="ref183">36</reflink>]] publication bias may operate differently in certain scientific contexts. Elsewhere, we suggested some simple diagnostics to assess the plausibility of some of these assumptions.[<reflink idref="bib14" id="ref184">14</reflink>] If these diagnostics or substantive knowledge suggests that publication bias is likely to depart from this model, the proposed analyses should be interpreted cautiously. As one plausible violation, publication bias might select for significant results in either direction, rather than positive‐signed significant results. However, we showed previously that the corrections for publication bias we extend in the present paper are often conservative (biased toward the null) for this violation of the assumptions.[<reflink idref="bib14" id="ref185">14</reflink>] As noted, many forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0498" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking compromise our proposed methods as well as almost all existing methods for publication bias, although our simulations indicated that under two forms of pernicious <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0499" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking and other forms of model misspecification, our proposed method generally still achieved better numerical performances than the existing methods (Section 5). Additionally, we recently showed that conducting a standard meta‐analysis of only the nonaffirmative studies is robust to many forms of <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0500" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐hacking that favor affirmative over nonaffirmative results.[<reflink idref="bib33" id="ref186">33</reflink>] The present worst‐case estimators that additionally accommodate internal bias (Equation 6) would have similar properties because they, too, involve only the nonaffirmative studies.[<reflink idref="bib33" id="ref187">33</reflink>]</p> <p>Our proposed methods use sensitivity parameters that characterize the <emph>average</emph> internal bias across studies, obviating specification of the bias for each study individually.[[<reflink idref="bib16" id="ref188">16</reflink>], [<reflink idref="bib18" id="ref189">18</reflink>]] Working with the average internal bias greatly reduces the number of sensitivity parameters and may therefore reduce the possibility of misspecifying the sensitivity parameters. On the other hand, our approach does entail assumptions on how internal bias and publication bias interact. In particular, our assumptions accommodate the possibility that, conditional on whether a study is affirmative, publication bias favors internally unbiased studies over internally biased studies. However, we did need to assume that, <emph>among internally biased studies</emph> and conditional on whether a study is affirmative, publication bias does not select further on the bias itself (Assumption A2). For example, if the internal bias under consideration is uncontrolled confounding, our methods accommodate the possibility that the editorial or peer‐review system favors randomized over nonrandomized studies. However, our methods could be compromised if, among nonrandomized studies, designs at lower risk of bias are strongly favored as well (conditional on whether the results are affirmative). Meta‐analysts will need to consider the plausibility of our assumptions on a case‐by‐case basis. Nevertheless, these assumptions are strictly more general than those of SM‐step, a commonly used selection model that assumes there is no internal bias, assumes normal population effects, and relies on asymptotic inference. Indeed, there are very few existing methods to address the joint contributions of internal bias and publication bias in meta‐analyses. A key contribution of this paper is in demonstrating how the biases can operate interactively, which can severely compromise existing methods (as shown in our simulation study). Future research could extend the conceptual framework of the present paper to accommodate other assumptions on the various forms of bias.</p> <p>Our proposed sensitivity analyses characterize evidence strength using the standard meta‐analytic point estimate and its confidence interval, but these metrics alone do not fully characterize evidence strength in a potentially heterogeneous distribution of effects.[[<reflink idref="bib17" id="ref190">17</reflink>], [<reflink idref="bib79" id="ref191">79</reflink>]] It can be informative to conduct sensitivity analyses using metrics of evidence strength that more holistically summarize the distribution of effects, such as the percentage of population effects that are stronger than a threshold chosen to represent a meaningfully strong effect size.[[<reflink idref="bib19" id="ref192">19</reflink>], [<reflink idref="bib79" id="ref193">79</reflink>]] The aforementioned methods for internal bias without publication bias, which the present methods extend, can indeed be conducted for this percentage metric.[[<reflink idref="bib20" id="ref194">20</reflink>]] However, publication bias can severely compromise estimation of the heterogeneity, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0501" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> , in meta‐analyses of realistic size, in turn compromising estimation of such metrics that rely on <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0502" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow></math> </ephtml> to characterize the distribution of effects across studies.[<reflink idref="bib14" id="ref195">14</reflink>]</p> <p>The proposed methods can be applied when the source of internal bias is unknown; alternatively, in the context of causal estimation with certain known sources of bias, the internal bias can be reparameterized to describe the minimum strength of relevant latent‐variable associations required to produce a given amount of internal bias. In the latter case, any strengths and limitations associated with the relevant analytic bound will of course be inherited by the proposed sensitivity analyses. Such topics have been covered extensively elsewhere for each analytic bound that has been developed.[[<reflink idref="bib26" id="ref196">26</reflink>], [<reflink idref="bib28" id="ref197">28</reflink>], [<reflink idref="bib80" id="ref198">80</reflink>], [<reflink idref="bib82" id="ref199">82</reflink>], [<reflink idref="bib84" id="ref200">84</reflink>]]</p> <p>In summary, we have proposed sensitivity analyses for the interactive effects of internal bias and publication bias in meta‐analyses. These methods generalize existing methods that consider only one source of bias, indicating that robustness to combined biases often cannot be assessed by simply considering each source of bias in turn. Reporting these sensitivity analyses could help calibrate confidence in meta‐analyses that may be subject to multiple sources of bias.</p> <hd id="AN0174546109-34">AUTHOR CONTRIBUTIONS</hd> <p> <bold>Maya Mathur:</bold> Conceptualization; data curation; formal analysis; funding acquisition; investigation; methodology; project administration; resources; software; validation; visualization; writing – original draft; writing – review and editing.</p> <hd id="AN0174546109-35">FUNDING INFORMATION</hd> <p>This research was supported by NIH grants R01 LM013866, UL1TR003142, P30CA124435, and P30DK116074. The funders had no role in the design, conduct, or reporting of this research.</p> <hd id="AN0174546109-36">CONFLICT OF INTEREST STATEMENT</hd> <p>The authors declare no conflict of interest.</p> <hd id="AN0174546109-37">DATA AVAILABILITY STATEMENT</hd> <p>All code and data required to reproduce the simulation study and applied examples are publicly available and documented (https://osf.io/vr7y6). The codebase for the R package multibiasmeta is public (https://github.com/cran/multibiasmeta).</p> <p>GRAPH: Data S1: Supporting Information.</p> <ref id="AN0174546109-38"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> In contrast, if one fits any standard selection model (i.e., that does not incorporate <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0504" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>C</mi><mi>i</mi></msub></mrow></math> </ephtml> ) to a meta‐analysis that does contain some internally biased studies, then the selection model would, in fact, implicitly invoke the considerably stronger assumption that publication bias does not select on whether a study is internally unbiased, conditional on study characteristics that do enter the selection function (e.g., a study's <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0505" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi></mrow></math> </ephtml> ‐value).</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Our framework and notation apply to meta‐analyses conducted using inverse‐variance weighting, the most common approach. However, other meta‐analysis frameworks, such as generalized linear mixed models with individual participant data (IPD),[48] can perform better when IPD are available, especially for non‐continuous outcomes. Generalizing our proposed methods to these other frameworks would be an interesting direction for future research.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref4" type="bt">3</bibl> <bibtext> This consistency result also requires a mild regularity condition that is often plausible by a Central Limit Theorem (Data S1, Assumption A5).</bibtext> </blist> <blist> <bibl id="bib4" idref="ref127" type="bt">4</bibl> <bibtext> However, we report heterogeneity estimates on the original log‐ <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1667:jrsm1667-math-0506" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">RR</mi></mrow></math> </ephtml> scale since they cannot be directly transformed.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref5" type="bt">5</bibl> <bibtext> Briefly, when there is heterogeneity, favored estimates (prior to the introduction of publication bias) more likely to be affirmative than under publication bias alone, yet the distributions conditional on either affirmative status overrepresent small population effects, yielding a negative bias.[33]</bibtext> </blist> </ref> <ref id="AN0174546109-39"> <title> REFERENCES </title> <blist> <bibtext> Grossman DC, Bibbins‐Domingo K, Curry SJ, et al. Screening for obesity in children and adolescents: US preventive services task force recommendation statement. JAMA. 2017 ; 317 (23): 2417 ‐ 2426.</bibtext> </blist> <blist> <bibtext> Shaneyfelt T. Pyramids are guides not rules: the evolution of the evidence pyramid. BMJ Evid Based Med. 2016 ; 21 (4): 121 ‐ 122.</bibtext> </blist> <blist> <bibtext> Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013 ; 14 (5): 365.</bibtext> </blist> <blist> <bibtext> de Vrieze J. The metawars. Science. 2018 ; 361 : 1184 ‐ 1188.</bibtext> </blist> <blist> <bibtext> Nelson LD, Simmons J, Simonsohn U. Psychology's renaissance. Annu Rev Psychol. 2018 ; 69 : 511 ‐ 534.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref6" type="bt">6</bibl> <bibtext> Jin Z‐C, Zhou X‐H, He J. Statistical methods for dealing with publication bias in meta‐analysis. Stat Med. 2015 ; 34 (2): 343 ‐ 360.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref32" type="bt">7</bibl> <bibtext> Marks‐Anglin A, Chen Y. A historical review of publication bias. Res Synth Methods. 2020 ; 11 (6): 725 ‐ 742.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref7" type="bt">8</bibl> <bibtext> Duval S, Tweedie R. Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis. Biometrics. 2000 ; 56 (2): 455 ‐ 463.</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Egger M, Smith GD, Schneider M, Minder C. Bias in meta‐analysis detected by a simple, graphical test. BMJ. 1997 ; 315 (7109): 629 ‐ 634.</bibtext> </blist> <blist> <bibtext> Citkowicz M, Vevea JL. A parsimonious weight function for modeling publication bias. Psychol Methods. 2017 ; 22(1):28 : 28 ‐ 41.</bibtext> </blist> <blist> <bibtext> Dear KBG, Begg CB. An approach for assessing publication bias prior to performing a meta‐analysis. Stat Sci. 1992 ; 7 : 237 ‐ 245.</bibtext> </blist> <blist> <bibtext> Hedges LV. Modeling publication selection effects in meta‐analysis. Stat Sci. 1992 ; 7 : 246 ‐ 255.</bibtext> </blist> <blist> <bibtext> Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995 ; 60 (3): 419 ‐ 435.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Sensitivity analysis for publication bias in meta‐analyses. J R Stat Soc Ser C Appl Stat. 2020 ; 69 (5): 1091 ‐ 1119.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Estimating publication bias in meta‐analyses of peer‐reviewed studies: a meta‐meta‐analysis across disciplines and journal tiers. Res Synth Methods. 2021 ; 12 (2): 176 ‐ 191.</bibtext> </blist> <blist> <bibtext> Goto A, Arah OA, Goto M, Terauchi Y, Noda M. Severe hypoglycaemia and cardiovascular disease: systematic review and meta‐analysis with bias analysis. BMJ. 2013 ; 347. https://<ulink href="http://www.bmj.com/content/347/bmj.f4533">www.bmj.com/content/347/bmj.f4533</ulink></bibtext> </blist> <blist> <bibtext> Greenland S, O'Rourke K. Meta‐analysis. Modern epidemiology. 3rd ed. Lippincott Williams & Wilkins ; 2008.</bibtext> </blist> <blist> <bibtext> Turner RM, Spiegelhalter DJ, Smith G, Thompson SG. Bias modelling in evidence synthesis. J R Stat Soc A Stat Soc. 2009 ; 172 (1): 21 ‐ 47.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Robust metrics and sensitivity analyses for meta‐analyses of heterogeneous effects. Epidemiology. 2020 ; 31 (3): 356 ‐ 358.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Sensitivity analysis for unmeasured confounding in meta‐analyses. J Am Stat Assoc. 2020 ; 115 (529): 163 ‐ 172.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Methods to address confounding and other biases in meta‐analyses: review and recommendations. Annu Rev Public Health. 2021 ; 43 (1): 19 ‐ 35.</bibtext> </blist> <blist> <bibtext> McCandless LC. Meta‐analysis of observational studies with unmeasured confounders. Int J Biostat. 2012 ; 8 (2): 331 ‐ 368.</bibtext> </blist> <blist> <bibtext> Ding P, VanderWeele TJ. Sensitivity analysis without assumptions. Epidemiology. 2016 ; 27 (3): 368 ‐ 377.</bibtext> </blist> <blist> <bibtext> Hernan MA, Robins JM. Letter to the editor of Biometrics. Biometrics. 1999 ; 55 (4): 1316‐1317.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ. Sensitivity analysis: distributional assumptions and confounding assumptions. Biometrics. 2008 ; 64 (2): 645 ‐ 649.</bibtext> </blist> <blist> <bibtext> Mathur MB, Smith LH, Yoshida K, Ding P, VanderWeele TJ. E‐values for effect heterogeneity and approximations for causal interaction. Int J Epidemiol. 2022 ; 51 (4): 1268 ‐ 1275.</bibtext> </blist> <blist> <bibtext> Smith LH, Mathur MB, VanderWeele TJ. Multiple‐bias sensitivity analysis using bounds. Epidemiology. 2021 ; 32 (5): 625 ‐ 634.</bibtext> </blist> <blist> <bibtext> Smith LH, VanderWeele TJ. Bounding bias due to selection. Epidemiology. 2019 ; 30 (4): 509 ‐ 516.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ, Li Y. Simple sensitivity analysis for differential measurement error. Am J Epidemiol. 2019 ; 188 (10): 1823 ‐ 1829.</bibtext> </blist> <blist> <bibtext> Brodeur A, Lé M, Sangnier M, Zylberberg Y. Star wars: the empirics strike back. Am Econ J Appl Econ. 2016 ; 8 (1): 1 ‐ 32.</bibtext> </blist> <blist> <bibtext> Jager LR, Leek JT. An estimate of the science‐wise false discovery rate and application to the top medical literature. Biostatistics. 2014 ; 15 (1): 1 ‐ 12.</bibtext> </blist> <blist> <bibtext> John LK, Loewenstein G, Prelec D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci. 2012 ; 23 (5): 524 ‐ 532.</bibtext> </blist> <blist> <bibtext> Mathur M. p ‐hacking in meta‐analyses: a formalization and new meta‐analytic methods. 2022. Retrieved from https://osf.io/ezjsx/</bibtext> </blist> <blist> <bibtext> Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p‐hacking in science. PLoS Biol. 2015 ; 13 (3): e1002106.</bibtext> </blist> <blist> <bibtext> Masicampo EJ, Lalande DR. A peculiar prevalence of p values just below.05. Q J Exp Psychol. 2012 ; 65 (11): 2271 ‐ 2279.</bibtext> </blist> <blist> <bibtext> McShane BB, Gal D. Statistical significance and the dichotomization of evidence. J Am Stat Assoc. 2017 ; 112 (519): 885 ‐ 895.</bibtext> </blist> <blist> <bibtext> Resche‐Rigon M, White IR, Bartlett JW, Peters SAE, Thompson SG, PROG‐IMT Study Group. Multiple imputation for handling systematically missing confounders in meta‐analysis of individual participant data. Stat Med. 2013 ; 32 (28): 4890 ‐ 4905.</bibtext> </blist> <blist> <bibtext> Brockwell SE, Gordon IR. A comparison of statistical methods for meta‐analysis. Stat Med. 2001 ; 20 (6): 825 ‐ 840.</bibtext> </blist> <blist> <bibtext> Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One. 2013 ; 8 (7): e66844.</bibtext> </blist> <blist> <bibtext> Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014 ; 345 (6203): 1502 ‐ 1505.</bibtext> </blist> <blist> <bibtext> Hedges LV. Plausibility and influence in selection models: a comment on Citkowicz and Vevea (2017). Psychol Methods. 2017 ; 22 (1): 42 ‐ 46.</bibtext> </blist> <blist> <bibtext> McShane BB, Böckenholt U, Hansen KT. Adjusting for publication bias in meta‐analysis: an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci. 2016 ; 11 (5): 730 ‐ 749.</bibtext> </blist> <blist> <bibtext> Hong S, Robert Reed W. Using Monte Carlo experiments to select meta‐analytic estimators. Res Synth Methods. 2021 ; 12 (2): 192 ‐ 215.</bibtext> </blist> <blist> <bibtext> Andrews I, Kasy M. Identification of and correction for publication bias. Am Econ Rev. 2019 ; 109 (8): 2766 ‐ 2794.</bibtext> </blist> <blist> <bibtext> Stanley TD, Doucouliagos H, Ioannidis JPA, Carter EC. Detecting publication selection bias through excess statistical significance. Res Synth Methods. 2021 ; 12 (6): 776 ‐ 795.</bibtext> </blist> <blist> <bibtext> Moreno SG, Sutton AJ, Ades AE, et al. Assessment of regression‐based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol. 2009 ; 9 : 1 ‐ 17.</bibtext> </blist> <blist> <bibtext> Rodgers MA, Pustejovsky JE. Evaluating meta‐analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychol Methods. 2021 ; 26 (2): 141 ‐ 160.</bibtext> </blist> <blist> <bibtext> Turner RM, Omar RZ, Yang M, Goldstein H, Thompson SG. A multilevel model framework for meta‐analysis of clinical trials with binary outcomes. Stat Med. 2000 ; 19 (24): 3417 ‐ 3432.</bibtext> </blist> <blist> <bibtext> Viechtbauer W. Bias and efficiency of meta‐analytic variance estimators in the random‐effects model. J Educ Behav Stat. 2005 ; 30 (3): 261 ‐ 293.</bibtext> </blist> <blist> <bibtext> Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta‐regression with dependent effect size estimates. Res Synth Methods. 2010 ; 1 (1): 39 ‐ 65.</bibtext> </blist> <blist> <bibtext> Tipton E. Small‐sample adjustments for robust variance estimation with meta‐regression. Psychol Methods. 2015 ; 20 (3): 375 ‐ 393.</bibtext> </blist> <blist> <bibtext> Fisher Z, Tipton E. Robumeta: an R‐package for robust variance estimation in meta‐analysis. arXiv Preprint arXiv:1503.02220. 2015.</bibtext> </blist> <blist> <bibtext> Roberts B, MacLeod CM, Fernandes M. The enactment effect: a systematic review and meta‐analysis of behavioral, neuroimaging, and patient studies. Psychol Bull. 2022 ; 132 : 29 ‐ 32.</bibtext> </blist> <blist> <bibtext> Carter EC, Schönbrodt FD, Gervais WM, Hilgard J. Correcting for bias in psychology: a comparison of meta‐analytic methods. Adv Methods Pract Psychol Sci. 2019 ; 2 (2): 115 ‐ 144.</bibtext> </blist> <blist> <bibtext> Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003 ; 22 (13): 2113 ‐ 2126.</bibtext> </blist> <blist> <bibtext> Bartoš F, Maier M, Wagenmakers E‐J, et al. Footprint of publication selection bias on meta‐analyses in medicine, economics, and psychology. arXiv:2208.12334. 2022.</bibtext> </blist> <blist> <bibtext> Schwab S, Kreiliger G, Held L. Assessing treatment effects and publication bias across different specialties in medicine: a meta‐epidemiological study. BMJ Open. 2021 ; 11 (9): e045942.</bibtext> </blist> <blist> <bibtext> Lash TL, VanderWeele TJ, Haneuse S, Rothman KJ. Modern Epidemiology. 4th ed. Wolters Kluwer ; 2021.</bibtext> </blist> <blist> <bibtext> Manski CF. Nonparametric bounds on treatment effects. Am Econ Rev. 1990 ; 80 (2): 319 ‐ 323.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Finding common ground in meta‐analysis "wars" on violent video games. Perspect Psychol Sci. 2019 ; 14 (4): 705 ‐ 708.</bibtext> </blist> <blist> <bibtext> Sterne JAC, Hernán MA, Reeves BC, et al. ROBINS‐I: a tool for assessing risk of bias in non‐randomised studies of interventions. BMJ. 2016 ; 355 : i4919.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ. Chapter 25, causal inference with time‐varying exposures. Modern Epidemiology. 4th ed. Wolters Kluwer ; 2021 : 605 ‐ 618.</bibtext> </blist> <blist> <bibtext> Baumeister SE, Leitzmann MF, Linseisen J, Schlesinger S. Physical activity and the risk of liver cancer: a systematic review and meta‐analysis of prospective studies and a bias analysis. J Natl Cancer Inst. 2019 ; 111 (11): 1142 ‐ 1151.</bibtext> </blist> <blist> <bibtext> Rui F, Sekercioglu N, Mathur MB, Couban R, Coyte PC. Dialysis initiation and all‐cause mortality among incident adult patients with advanced ckd: a meta‐analysis with bias analysis. Kidney Med. 2020 ; 3 : 64 ‐ 75.e1.</bibtext> </blist> <blist> <bibtext> Ling S, Brown K, Miksza JK, et al. Association of type 2 diabetes with cancer: a meta‐analysis with bias analysis for unmeasured confounding in 151 cohorts comprising 32 million people. Diabetes Care. 2020 ; 43 (9): 2313 ‐ 2322.</bibtext> </blist> <blist> <bibtext> Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons ; 2019.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ, Ding P. Sensitivity analysis in observational research: introducing the E‐value. Ann Intern Med. 2017 ; 167 (4): 268 ‐ 274.</bibtext> </blist> <blist> <bibtext> Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology. 1999 ; 10 : 37 ‐ 48.</bibtext> </blist> <blist> <bibtext> Pearl J. Causal diagrams for empirical research. Biometrika. 1995 ; 82 (4): 669 ‐ 688.</bibtext> </blist> <blist> <bibtext> Pearl J. Causality. Cambridge University Press ; 2009.</bibtext> </blist> <blist> <bibtext> Mathur MB, Shpitser I, VanderWeele TJ. A common‐cause principle for eliminating selection bias in causal estimands through covariate adjustment. 2023. Retrieved from https://osf.io/ths4e/</bibtext> </blist> <blist> <bibtext> Mathur MB, Peacock J, Reichling DB, et al. Interventions to reduce meat consumption by appealing to animal welfare: meta‐analysis and evidence‐based recommendations. Appetite. 2021 ; 164 : 105277.</bibtext> </blist> <blist> <bibtext> Kalantarian S, Stern TA, Mansour M, Ruskin JN. Cognitive impairment associated with atrial fibrillation: a meta‐analysis. Ann Intern Med. 2013 ; 158 (5): 338 ‐ 346.</bibtext> </blist> <blist> <bibtext> Bruns SB, Ioannidis JPA. P‐curve and p‐hacking in observational research. PLoS One. 2016 ; 11 (2): e0149144.</bibtext> </blist> <blist> <bibtext> Stefan AM, Schönbrodt FD. Big little lies: a compendium and simulation of p‐hacking strategies. R Soc Open Sci. 2023 ; 10 (2): 220346.</bibtext> </blist> <blist> <bibtext> Coburn KM, Vevea JL, Coburn MKM. weightr: Estimating weight‐function models for publication bias. R package version 2.0.2. 2019 https://CRAN.R-project.org/package=weightr</bibtext> </blist> <blist> <bibtext> Viechtbauer W. Conducting meta‐analyses in R with the metafor package. J Stat Softw. 2010 ; 36 (3): 1 ‐ 48.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. Meta‐regression methods to characterize evidence strength using meaningful‐effect percentages conditional on study characteristics. Res Synth Methods. 2021 ; 12 (6): 731 ‐ 749.</bibtext> </blist> <blist> <bibtext> Mathur MB, VanderWeele TJ. New metrics for meta‐analyses of heterogeneous effects. Stat Med. 2019 ; 38 (8): 1336 ‐ 1342.</bibtext> </blist> <blist> <bibtext> Greenland S. Commentary: an argument against E‐values for assessing the plausibility that an association could be explained away by residual confounding. Int J Epidemiol. 2020 ; 49 (5): 1501 ‐ 1503.</bibtext> </blist> <blist> <bibtext> Ioannidis JPA, Tan YJ, Blum MR. Limitations and misinterpretations of E‐values for sensitivity analyses of observational studies. Ann Intern Med. 2019 ; 170 (2): 108 ‐ 111.</bibtext> </blist> <blist> <bibtext> Poole C. Commentary: continuing the E‐value's post‐publication peer review. Int J Epidemiol. 2020 ; 49 (5): 1497 ‐ 1500.</bibtext> </blist> <blist> <bibtext> Sjölander A, Greenland S. Are E‐values too optimistic or too pessimistic? Both and neither! Int J Epidemiol. 2022 ; 51 (2): 355 ‐ 363.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ, Mathur MB, Ding P. Correcting misinterpretations of the E‐value. Ann Intern Med. 2019 ; 170 (2): 131 ‐ 132.</bibtext> </blist> <blist> <bibtext> VanderWeele TJ. Are Greenland, Ioannidis, and Poole opposed to the Cornfield conditions? A defense of the E‐value. Int J Epidemiol. 2021 ; 51 : 364 ‐ 371.</bibtext> </blist> </ref> <aug> <p>By Maya B. Mathur</p> <p>Reported by Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref8"></nolink> <nolink nlid="nl2" bibid="bib12" firstref="ref9"></nolink> <nolink nlid="nl3" bibid="bib14" firstref="ref10"></nolink> <nolink nlid="nl4" bibid="bib16" firstref="ref14"></nolink> <nolink nlid="nl5" bibid="bib18" firstref="ref15"></nolink> <nolink nlid="nl6" bibid="bib19" firstref="ref16"></nolink> <nolink nlid="nl7" bibid="bib21" firstref="ref17"></nolink> <nolink nlid="nl8" bibid="bib23" firstref="ref21"></nolink> <nolink nlid="nl9" bibid="bib22" firstref="ref24"></nolink> <nolink nlid="nl10" bibid="bib24" firstref="ref26"></nolink> <nolink nlid="nl11" bibid="bib26" firstref="ref28"></nolink> <nolink nlid="nl12" bibid="bib28" firstref="ref29"></nolink> <nolink nlid="nl13" bibid="bib30" firstref="ref30"></nolink> <nolink nlid="nl14" bibid="bib32" firstref="ref31"></nolink> <nolink nlid="nl15" bibid="bib33" firstref="ref33"></nolink> <nolink nlid="nl16" bibid="bib15" firstref="ref39"></nolink> <nolink nlid="nl17" bibid="bib34" firstref="ref40"></nolink> <nolink nlid="nl18" bibid="bib36" firstref="ref41"></nolink> <nolink nlid="nl19" bibid="bib17" firstref="ref44"></nolink> <nolink nlid="nl20" bibid="bib37" firstref="ref45"></nolink> <nolink nlid="nl21" bibid="bib38" firstref="ref48"></nolink> <nolink nlid="nl22" bibid="bib39" firstref="ref49"></nolink> <nolink nlid="nl23" bibid="bib41" firstref="ref50"></nolink> <nolink nlid="nl24" bibid="bib43" firstref="ref52"></nolink> <nolink nlid="nl25" bibid="bib44" firstref="ref53"></nolink> <nolink nlid="nl26" bibid="bib42" firstref="ref55"></nolink> <nolink nlid="nl27" bibid="bib46" firstref="ref56"></nolink> <nolink nlid="nl28" bibid="bib49" firstref="ref65"></nolink> <nolink nlid="nl29" bibid="bib50" firstref="ref68"></nolink> <nolink nlid="nl30" bibid="bib51" firstref="ref82"></nolink> <nolink nlid="nl31" bibid="bib52" firstref="ref86"></nolink> <nolink nlid="nl32" bibid="bib13" firstref="ref87"></nolink> <nolink nlid="nl33" bibid="bib53" firstref="ref91"></nolink> <nolink nlid="nl34" bibid="bib54" firstref="ref94"></nolink> <nolink nlid="nl35" bibid="bib56" firstref="ref97"></nolink> <nolink nlid="nl36" bibid="bib58" firstref="ref98"></nolink> <nolink nlid="nl37" bibid="bib60" firstref="ref99"></nolink> <nolink nlid="nl38" bibid="bib61" firstref="ref100"></nolink> <nolink nlid="nl39" bibid="bib62" firstref="ref102"></nolink> <nolink nlid="nl40" bibid="bib63" firstref="ref103"></nolink> <nolink nlid="nl41" bibid="bib65" firstref="ref104"></nolink> <nolink nlid="nl42" bibid="bib66" firstref="ref105"></nolink> <nolink nlid="nl43" bibid="bib20" firstref="ref106"></nolink> <nolink nlid="nl44" bibid="bib67" firstref="ref108"></nolink> <nolink nlid="nl45" bibid="bib29" firstref="ref117"></nolink> <nolink nlid="nl46" bibid="bib27" firstref="ref119"></nolink> <nolink nlid="nl47" bibid="bib68" firstref="ref120"></nolink> <nolink nlid="nl48" bibid="bib70" firstref="ref121"></nolink> <nolink nlid="nl49" bibid="bib71" firstref="ref123"></nolink> <nolink nlid="nl50" bibid="bib72" firstref="ref126"></nolink> <nolink nlid="nl51" bibid="bib73" firstref="ref128"></nolink> <nolink nlid="nl52" bibid="bib74" firstref="ref143"></nolink> <nolink nlid="nl53" bibid="bib75" firstref="ref151"></nolink> <nolink nlid="nl54" bibid="bib76" firstref="ref160"></nolink> <nolink nlid="nl55" bibid="bib77" firstref="ref164"></nolink> <nolink nlid="nl56" bibid="bib78" firstref="ref167"></nolink> <nolink nlid="nl57" bibid="bib79" firstref="ref191"></nolink> <nolink nlid="nl58" bibid="bib80" firstref="ref198"></nolink> <nolink nlid="nl59" bibid="bib82" firstref="ref199"></nolink> <nolink nlid="nl60" bibid="bib84" firstref="ref200"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1405345
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Sensitivity Analysis for the Interactive Effects of Internal Bias and Publication Bias in Meta-Analyses
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Maya+B%2E+Mathur%22">Maya B. Mathur</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6698-2607">0000-0001-6698-2607</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. 2024 15(1):21-43.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 23
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Institutes of Health (NIH) (DHHS)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: P30CA124435<br />P30DK116074<br />R01LM013866<br />UL1TR003142
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Attribution+Theory%22">Attribution Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Publications%22">Publications</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Programming+Languages%22">Programming Languages</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1667
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879<br />1759-2887
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1405345
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1405345
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1667
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 23
        StartPage: 21
    Subjects:
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Attribution Theory
        Type: general
      – SubjectFull: Publications
        Type: general
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Programming Languages
        Type: general
    Titles:
      – TitleFull: Sensitivity Analysis for the Interactive Effects of Internal Bias and Publication Bias in Meta-Analyses
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Maya B. Mathur
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
            – Type: issn-electronic
              Value: 1759-2887
          Numbering:
            – Type: volume
              Value: 15
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1