View in EDS HTML Full Text PDF Full Text

Visualizing the Assumptions of Network Meta-Analysis

Saved in:
Bibliographic Details
Title: Visualizing the Assumptions of Network Meta-Analysis
Language: English
Authors: Yu-Kang Tu (ORCID 0000-0002-2461-474X), Pei-Chun Lai, Yen-Ta Huang, James Hodges
Source: Research Synthesis Methods. 2024 15(6):1175-1182.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 8
Publication Date: 2024
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Visualization, Meta Analysis, Comparative Analysis, Statistical Studies, Research and Development, Evaluation Methods, Health Services, Outcomes of Treatment
DOI: 10.1002/jrsm.1760
ISSN: 1759-2879
1759-2887
Abstract: Network meta-analysis (NMA) incorporates all available evidence into a general statistical framework for comparing multiple treatments. Standard NMAs make three major assumptions, namely homogeneity, similarity, and consistency, and violating these assumptions threatens an NMA's validity. In this article, we suggest a graphical approach to assessing these assumptions and distinguishing between qualitative and quantitative versions of these assumptions. In our plot, the absolute effect of each treatment arm is plotted against the level of effect modifiers, and the three assumptions of NMA can then be visually evaluated. We use four hypothetical scenarios to show how violating these assumptions can lead to different consequences and difficulties in interpreting an NMA. We present an example of an NMA evaluating steroid use to treat septic shock patients to demonstrate how to use our graphical approach to assess an NMA's assumptions and how this approach can help with interpreting the results. We also show that all three assumptions of NMA can be summarized as an exchangeability assumption. Finally, we discuss how reporting of NMAs can be improved to increase transparency of the analysis and interpretability of the results.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1447378
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFiH5kdshn7ZArEQ0mzG5JGAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDH170L8p0ZLX0L5SiAIBEICBm9TdnM-mxh3yWkLA31abIT_AEdlJzpDRVFVg80sCvcT0iwufr3ud09DIrl89q1f6moG7B2_zDngiYNpzUyTIOytpjwfgSwEMNtTm2jYbEuFQWGESPqPAIjtFE1ykDwDxQQrNCVE9vZbwwROgxV1O8fBMpPADcTSPBKPtBaEbznIjoN7RJ2BTD3-maVjaH1MBkmdMqJNXlm6Rjdby
Text:
  Availability: 1
  Value: <anid>AN0180736619;[bdct]01nov.24;2024Nov11.04:50;v2.2.500</anid> <title id="AN0180736619-1">Visualizing the assumptions of network meta‐analysis </title> <p>Network meta‐analysis (NMA) incorporates all available evidence into a general statistical framework for comparing multiple treatments. Standard NMAs make three major assumptions, namely homogeneity, similarity, and consistency, and violating these assumptions threatens an NMA's validity. In this article, we suggest a graphical approach to assessing these assumptions and distinguishing between qualitative and quantitative versions of these assumptions. In our plot, the absolute effect of each treatment arm is plotted against the level of effect modifiers, and the three assumptions of NMA can then be visually evaluated. We use four hypothetical scenarios to show how violating these assumptions can lead to different consequences and difficulties in interpreting an NMA. We present an example of an NMA evaluating steroid use to treat septic shock patients to demonstrate how to use our graphical approach to assess an NMA's assumptions and how this approach can help with interpreting the results. We also show that all three assumptions of NMA can be summarized as an exchangeability assumption. Finally, we discuss how reporting of NMAs can be improved to increase transparency of the analysis and interpretability of the results.</p> <p>Keywords: consistency; exchangeability; homogeneity; network meta‐analysis; similarity; transitivity</p> <hd id="AN0180736619-2">Highlights</hd> <p></p> <hd id="AN0180736619-3">What is already known</hd> <p>The validity of a network meta‐analysis (NMA) depends on three assumptions: homogeneity, similarity, and consistency. A related assumption, transitivity, is analogous to similarity and consistency requires that the distribution of effect modifiers is similar across all trials included in an NMA. Although statistical models have been developed to detect inconsistency between direct and indirect evidence within an NMA, it is not clear how to evaluate the transitivity assumption or the potential consequences of violating it.</p> <hd id="AN0180736619-4">What is new?</hd> <p>We suggest a graphical approach to assessing these assumptions and distinguishing between qualitative and quantitative versions of them. In our graph, the absolute outcome for each treatment arm is plotted against the level of an effect modifier, which allows NMA's three assumptions to be visually evaluated. We use four hypothetical scenarios to show how violating these assumptions can lead to different consequences and difficulties in interpreting an NMA. We also show that all three assumptions of NMA can be summarized as the exchangeability assumption.</p> <hd id="AN0180736619-5">Potential impact for Research Synthesis Methods readers</hd> <p>NMA has been hailed as the highest level of evidence, but authors and readers of NMAs do not always appreciate the complexity of conducting and interpreting an NMA. Our graphical approach provides a simple, intuitive tool for visualizing the assumptions of NMA and assessing the consequences of violating these assumptions.</p> <hd id="AN0180736619-6">INTRODUCTION</hd> <p>Network meta‐analysis (NMA) was proposed about 20 years ago to compare the effects of multiple treatments.[<reflink idref="bib1" id="ref1">1</reflink>] A recent article by Ades and colleagues reviews NMA's history, assumptions, and continuing controversies.[<reflink idref="bib2" id="ref2">2</reflink>] Standard NMA makes three major assumptions, violation of which may threaten validity. The first is homogeneity: studies comparing the same treatments are comparable.[[<reflink idref="bib3" id="ref3">3</reflink>]] The second is similarity,[[<reflink idref="bib5" id="ref4">5</reflink>]] which extends homogeneity to the network: all included studies are comparable. The third assumption is consistency: estimates using direct and indirect evidence are similar.[[<reflink idref="bib7" id="ref5">7</reflink>], [<reflink idref="bib9" id="ref6">9</reflink>]] Ades et al. also made a distinction between qualitative and quantitative versions of these assumptions because they do not always agree: An NMA may include studies with similar eligibility criteria, treatment protocols, and outcome assessment methods, while statistical methods still detect inconsistency in some loops of the network. Another assumption, transitivity, requires that effect modifiers be similarly distributed in an NMA's studies.[<reflink idref="bib11" id="ref7">11</reflink>] Because no specific statistical method assesses transitivity, it may be considered the qualitative version of similarity and consistency.[<reflink idref="bib2" id="ref8">2</reflink>]</p> <p>This article uses graphics to illustrate NMA's assumptions and to enhance understanding of these assumptions in practice. This method has been used to illustrate Simpson's paradox and transitivity of randomized controlled trials.[[<reflink idref="bib12" id="ref9">12</reflink>]] We use four hypothetical scenarios to show how violating NMA's assumptions produces different consequences and difficulties in interpretation. We then use a real example to demonstrate the graphical approach and show how it can aid interpretation.</p> <hd id="AN0180736619-7">CONSEQUENCES OF VIOLATING NMA ASSUMPTIONS</hd> <p>Ades et al. reduced NMA's three assumptions to exchangeability of treatment contrasts[<reflink idref="bib2" id="ref10">2</reflink>]: if all treatment contrasts are exchangeable across studies, then all studies are qualitatively homogeneous and similar, and direct and indirect comparisons are consistent. Suppose baseline disease severity is the most important prognostic factor and effect modifier. This could be a single measure or a score combining measures. (Constructing the latter is outside of this paper's scope.) We might say homogeneity and similarity hold if the studies' average baseline severity scores are within a narrow range. Nonetheless, this is a strong assumption and hard to verify when only aggregate data is available.</p> <p>We use four scenarios with different relationships between treatment effects and baseline severity to demonstrate the relationship between NMA assumptions and the consequences of violating them. Each is an NMA of randomized trials comparing treatments A, B, and C, with trials having different designs[[<reflink idref="bib7" id="ref11">7</reflink>], [<reflink idref="bib14" id="ref12">14</reflink>]]: design AB, comparing A with B; design AC, comparing A with C; and design BC, comparing B with C.</p> <hd id="AN0180736619-8">Scenario 1: Treatment effects and baseline severity are unrelated</hd> <p>In this scenario, the absolute treatment effects of A, B, and C do not depend on baseline severity, so all treatment contrasts are exchangeable. In Figure 1, each plot's horizontal axis is baseline severity, with vertical axis the absolute treatment effect on the appropriate scale. For example, if the outcome is mortality and the risk ratio is the relative efficacy measure, the Y‐axis is the log risk of mortality. Each letter A, B, or C represents a treatment arm in a trial. A letter's color represents the trial's design: designs AB, AC, and BC are black, blue, and red, respectively. Each plot has three horizontal lines representing the unobserved "true" relationships between baseline severity and each treatment's absolute effect.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0001.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0001.jpg" title="1 Scatterplot for Scenario 1: No relationship between treatment effects and baseline severity. The horizontal axis is the baseline severity score and the vertical axis is mortality." /> </p> <p></p> <p>Figure 1a shows an "ideal" NMA in which all included trials have similar, small ranges of baseline severity, qualitatively satisfying homogeneity, similarity, and transitivity. Because this NMA has only one closed loop, we can examine consistency for any treatment pair.[[<reflink idref="bib7" id="ref13">7</reflink>], [<reflink idref="bib14" id="ref14">14</reflink>]] For consistency of contrast BC, we check whether the estimated difference B vs. C given by trials of design BC, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> , is similar to the indirect comparison, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>ind</mi></msubsup><mo>=</mo><msubsup><mi>θ</mi><mi>AC</mi><mi>dir</mi></msubsup><mo>−</mo><msubsup><mi>θ</mi><mi>AB</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> , estimated from trials of designs AB and AC. In Figure 1a, we measure the distance between lines B and C, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>BC</mi></msub></mrow></math> </ephtml> , describing the arms of trials with design BC (B and C letters in red), and analogously the distances between lines A and B, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>AB</mi></msub></mrow></math> </ephtml> , and between lines A and C, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>AC</mi></msub></mrow></math> </ephtml> . If <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>BC</mi></msub><mo>=</mo><msub><mi>θ</mi><mi>AC</mi></msub><mo>−</mo><msub><mi>θ</mi><mi>AB</mi></msub></mrow></math> </ephtml> , consistency holds, as in Figure 1a, where the three lines are parallel.</p> <p>Figure 1b violates transitivity qualitatively because trials with different designs have different baseline severity. Although absolute and relative treatment effects are independent of baseline severity, it is unclear whether transitivity is violated. Thus, Figure 1b's NMA satisfies homogeneity and consistency quantitatively but may violate similarity/transitivity.</p> <hd id="AN0180736619-10">Scenario 2: Homogeneous relationships between treatment effects and baseline severity</hd> <p>Here mortality after treatment is higher for populations with higher baseline severity, but <emph>relative</emph> treatment effects are independent of baseline severity, reflected in Figure 2's three parallel lines. All assumptions hold qualitatively and quantitatively, and estimates of the <emph>relative</emph> treatment effects are generalizable or <emph>transportable</emph> to patient populations with different baseline severity.[<reflink idref="bib15" id="ref15">15</reflink>]</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0002.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0002.jpg" title="2 Scatterplot for Scenario 2: Homogeneous relationship between treatment effects and baseline severity. The horizontal axis is the baseline severity score and the vertical axis is mortality." /> </p> <p></p> <p>Figure 2b violates transitivity qualitatively because trials with different designs have different baseline severity. Homogeneity and consistency still hold because relative treatment effects remain constant on the appropriate scale. Although transitivity is violated, the relative effects are still considered exchangeable across patient populations.[[<reflink idref="bib15" id="ref16">15</reflink>]] In practice, the three lines are unknown, so even though there is no statistical inconsistency, we may be cautious in interpreting the results and hesitant to transport the estimated effects to another patient population because the included trials mix patients of widely varying baseline severity.[<reflink idref="bib16" id="ref17">16</reflink>]</p> <hd id="AN0180736619-12">Scenario 3: Heterogenous relationships between treatment effects and baseline severity, homog...</hd> <p>Here, as baseline severity increases, mortality becomes worse at different rates for the three treatments, so relative treatment effects also change with baseline severity. This is often called an interaction between relative treatment effect and baseline severity, shown in Figure 3 by non‐parallel lines. The distance between lines A and B increases with baseline severity, but the distances between lines A and C and between lines B and C decrease as baseline severity increases. In Figure 3a, homogeneity and transitivity hold qualitatively, and homogeneity and consistency hold quantitatively. However, relative treatment‐effect estimates are unlikely to be generalizable or transportable to populations with different baseline severity, although the treatment ranking, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="normal">C</mi><mo>></mo><mi mathvariant="normal">B</mi><mo>></mo><mi mathvariant="normal">A</mi></mrow></math> </ephtml> , is preserved across populations with different baseline severity.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0003.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0003.jpg" title="3 Scatterplot for Scenario 3: Heterogenous relationships between treatment effects and baseline severity but with the same treatment ranking." /> </p> <p></p> <p>In Figure 3b, transitivity is violated qualitatively because trials with different designs have different baseline severity. Absolute treatment effects differ in trials of different designs, and relative effects change differentially with baseline severity. Because lines A, B, and C have different slopes, the distances <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>AB</mi></msub></mrow></math> </ephtml> , <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>AC</mi></msub></mrow></math> </ephtml> , and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>θ</mi><mi>BC</mi></msub></mrow></math> </ephtml> change with baseline severity. Consistency is violated because <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>ind</mi></msubsup><mo>=</mo><msubsup><mi>θ</mi><mi>AC</mi><mi>dir</mi></msubsup><mo>−</mo><msubsup><mi>θ</mi><mi>AB</mi><mi>dir</mi></msubsup><mo>></mo><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> . However, if trials of design AB recruit patients with high baseline severity, trials of design AC recruit patients with low severity, and trials of design BC recruit patients with intermediate severity, the indirect and direct estimates, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>ind</mi></msubsup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> , may be similar. Transitivity is also violated because trials with different designs have different baseline severity. We are uncertain about how or whether estimated relative effects can be transported to a population with a different range of baseline severity. Although treatment ranking appears to be preserved in Figure 3b, this may not always hold, for example, if trials of design AB recruit patients with low baseline severity, while trials of designs AC and BC recruit patients with high baseline severity.</p> <hd id="AN0180736619-14">Scenario 4: Heterogenous relationships between treatment effects and baseline severity, treat...</hd> <p>Scenario 4 differs from Scenario 3 in that the absolute effects change so differently with baseline severity that treatment ranking also changes, a more extreme interaction between baseline severity and relative treatment effects. Therefore, in Figure 4a, neither the relative treatment effects nor treatment ranking of our NMA is transportable to populations with different baseline severity even though all of NMA's assumptions hold.[[<reflink idref="bib17" id="ref18">17</reflink>]]</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0004.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0004.jpg" title="4 Scatterplot for Scenario 3: Heterogenous relationships between treatment effects and baseline severity with treatment ranking depending on baseline severity." /> </p> <p></p> <p>Figure 4b's NMA violates transitivity and consistency. The direct and indirect estimates for B versus C, <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>ind</mi></msubsup></mrow></math> </ephtml> , differ in magnitude and in direction. Direct evidence shows B is slightly better than C, while indirect evidence shows B is rather worse than C. The final result depends on whether direct or indirect evidence dominates the combined effect estimate. Although Figure 4b seems to suggest direct evidence is always superior to indirect evidence, they are derived from trials with different patient populations and are not comparable. The substantial difference between <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> and <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>ind</mi></msubsup></mrow></math> </ephtml> indicates that <ephtml> <math display="inline" overflow="scroll" altimg="urn:x-wiley:17592879:media:jrsm1760:jrsm1760-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mi>θ</mi><mi>BC</mi><mi>dir</mi></msubsup></mrow></math> </ephtml> may not generalize to other populations.</p> <hd id="AN0180736619-16">PRACTICAL EXAMPLE: STEROIDS FOR PATIENTS WITH SEPTIC SHOCK</hd> <p>We use a recent NMA to show how the graphical approach can help evaluate an NMA's assumptions and interpret its results.[<reflink idref="bib19" id="ref19">19</reflink>] This NMA evaluated efficacy and safety of hydrocortisone alone (H) or with fludrocortisone (H + F) compared to placebo (P) in adults with septic shock. The primary outcome was short‐term mortality (within 28–30 days) in‐hospital or intensive care unit. Nineteen trials were included: 1 compared all three treatments, 2 compared H with H + F, 3 compared H + F with P, and 13 compared H with P. Appendix Figure 1 in the online supplement shows mortality for the 39 treatment groups in the 19 trials; the vertical axis is logarithmic, so the vertical distance between treatments is log relative risk. Mortality was consistently lower for H + F than P, while the relative risks of mortality for H + F versus H and for H versus P varied.</p> <p>We re‐analyzed the data using the frequentist random‐effects model in the "netmeta" package for the R system.[<reflink idref="bib20" id="ref20">20</reflink>] R code to reproduce the analyses and graphs is in the online supplement. Appendix Table 1 in the online supplement gives the data. The risk ratios for H versus P, H + F versus P, and H + F versus H are 0.96 (95% confidence interval: 0.89–1.02), 0.88 (0.82–0.95), and 0.92 (0.88–0.96), respectively. Heterogeneity was small (Appendix Figure 2's forest plot), and no inconsistency was significant quantitatively. These results suggest H + F is better than H and P. The original NMA made similar findings but concluded they were not definitive due to limited certainty of evidence.[<reflink idref="bib19" id="ref21">19</reflink>]</p> <p>The Sequential Organ Failure Assessment (SOFA) score[<reflink idref="bib21" id="ref22">21</reflink>] rates the performance of several organ systems (neurological, blood, liver, kidney, and blood pressure/hemodynamics), assigning a score from 0 to 24 based on data for each system. Higher SOFA scores imply a higher likelihood of mortality. Most included trials selected patients based on their SOFA score; 11 trials reported average SOFA score for each treatment group, while fewer reported APACHE (Acute Physiology And Chronic Health Evaluation) or SAPS (Simplified Acute Physiology Score) scores. Figure 5 shows the relationship between SOFA score and mortality in the 11 studies, showing no apparent relationship between SOFA score and mortality except that the study with the highest SOFA scores (P and H at the plot's right) also had the highest mortality. Restricting the analysis to these 11 trials, the risk ratios for H versus P, H + F versus P, and H + F versus H are 1.03 (0.92–1.16), 0.90 (0.79–1.02), and 0.87 (0.76–1.00), respectively, which still suggests H + F is better than H and P. We may, therefore, conclude that H + F seems to show a small short‐term mortality benefit for patients with moderate to high SOFA scores.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0005.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0005.jpg" title="5 Scatterplot for the absolute effects of treatment arms in each study. The horizontal axis is SOFA score, and the vertical axis is the mortality on the logarithmic scale. Each letter is a treatment arm: H: Hydrocortisone; H + F: Hydrocortisone plus fludrocortisone; P: Placebo. The superscripted number is the study identification number listed in the variable study in Appendix Table 1." /> </p> <p></p> <p>We also considered serum lactate level, a better prognostic factor than SOFA score,[[<reflink idref="bib22" id="ref23">22</reflink>]] to evaluate transitivity. Twelve studies reported average baseline lactate for each treatment group. Despite considerable heterogeneity within treatments, Figure 6 shows no obvious relationship between mortality and average lactate. Restricting the analysis to these 12 trials, the risk ratios for H versus P, H + F versus P, and H + F versus H are 0.97 (0.89–1.05), 0.88 (0.80–0.98), and 0.91 (0.81–1.03), respectively, which again suggests H + F is slightly better than H and P for patients with moderate to high baseline lactate.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/BDCT/01nov24/jrsm1760-fig-0006.jpg?ephost1=dGJyMNHX8kSepq84v%2bvlOLCmsE6epq5Srqa4SK6WxWXS" alt="jrsm1760-fig-0006.jpg" title="6 Scatterplot for the absolute effects of treatment arms in each study. The horizontal axis is the average baseline lactate level in mmole/L, and the vertical axis is the mortality. Each letter is a treatment arm: H: Hydrocortisone; H + F: Hydrocortisone plus fludrocortisone; P: Placebo. The superscripted number is the study identification number listed in the variable study in Appendix Table 1." /> </p> <p></p> <p>Although both SOFA score and lactate level are important predictors of mortality, they were not strongly correlated (<emph>r</emph> = 0.31 in Appendix Figure 3), and removing one study reverses this positive correlation (<emph>r</emph> = −0.58). As less than one‐third of the included studies reported both predictors, we need to be cautious in interpreting this relationship. The popular approach to assessing the transitivity assumption is to evaluate the distribution of an effect modifier across different designs of studies comparing the same treatments, for example, the violin plots in Appendix Figure 4 showing the distributions of SOFA scores and lactate levels in different direct comparisons. However, violin plots do not show the relationship between the treatment outcome and the effect modifier, so the distribution of an effect modifier cannot be used to evaluate NMA assumptions. Compared with these violin plots, our plots concentrate on the relationship between effect modifiers and treatment effects, and we have demonstrated that this relationship is useful for assessing NMA assumptions.</p> <hd id="AN0180736619-19">DISCUSSION</hd> <p>Our study demonstrates how the transitivity assumption relates to the consistency and homogeneity assumptions and suggests a graphical approach to visualizing and evaluating these assumptions. Our four scenarios assumed homogeneity so we could focus on similarity and consistency, or, as Ades et al.[<reflink idref="bib2" id="ref24">2</reflink>] suggested, transitivity. Our graphical approach has been tested in a recent NMAs by two of the authors (PCL and YTH) on the benefits of various blood purification modalities for adult patients with severe infection or sepsis.[<reflink idref="bib24" id="ref25">24</reflink>] Although polymyxin‐B hemoperfusion was shown to reduce mortality significantly, our plot found high mortality rates in the standard care group in early trials assessing the effectiveness of polymyxin‐B hemoperfusion (see Supplementary Figure 17 in Reference [[<reflink idref="bib24" id="ref26">24</reflink>]]). Discovering this intransitivity led to a more cautious conclusion that the evidence was inconclusive regarding the benefits of polymyxin‐B hemoperfusion.</p> <p>In practice, uncertainty in assessing these assumptions and interpreting the results is more challenging, as most NMAs compare more than 3 treatments and include trials with different effect‐modifier distributions. As the distribution of evidence in an NMA is usually uneven, users could include in the graph treatments that are compared in (say) at least 4 or 5 trials to learn how treatment effects and effect modifiers are related. We encourage researchers to use our graphics to explore the relationships between treatment outcomes and potential effect modifiers, giving a more transparent assessment of transitivity and exchangeability. Our graphs may also be incorporated into the evaluation of evidence certainty systems, such as GRADE[<reflink idref="bib25" id="ref27">25</reflink>] and CINeMA.[<reflink idref="bib26" id="ref28">26</reflink>] A further extension may be development of an interactive tool allowing users to explore various aspects of the graphs to assess different NMA assumptions.</p> <p>Unconditional exchangeability, that is, not conditional on potential or actual effect modifiers, is the most critical assumption for an NMA with heterogeneous patient populations. It is difficult to assess this assumption visually for an NMA with many treatments and trials due to heterogeneity in absolute and relative treatment effects. In contrast, our graphs are of limited use when an NMA has very few trials per treatment. A single effect modifier is unlikely to explain heterogeneity in treatment effects across a broad patient population, and combining multiple modifiers can be challenging as shown in our septic shock example. Further research is required to develop methods for assessing the relationship between treatment outcomes and multiple effect modifiers, and potential interactions between effect modifiers.</p> <hd id="AN0180736619-20">CONCLUSION</hd> <p>Our graphical method demonstrates the importance of assumptions underlying NMA and potential consequences of violating them. Previous NMA research has focused mainly on statistical theory and model building. We hope our graphical method will encourage development of data exploration and visualization tools, allowing robust interpretation of results.</p> <hd id="AN0180736619-21">AUTHOR CONTRIBUTIONS</hd> <p> <bold>Yu‐Kang Tu:</bold> Conceptualization; methodology; software; investigation; formal analysis; funding acquisition; visualization; project administration; writing – original draft. <bold>Pei‐Chun Lai:</bold> Data curation; writing – review and editing; validation; investigation. <bold>Yen‐Ta Huang:</bold> Investigation; validation; writing – review and editing; data curation. <bold>James Hodges:</bold> Methodology; investigation; writing – review and editing; visualization.</p> <hd id="AN0180736619-22">ACKNOWLEDGMENTS</hd> <p>This study is partly supported by a grant from the National Science & Technology Council in Taiwan (grant no. NSTC 112‐2314‐B‐002‐210‐MY3).</p> <hd id="AN0180736619-23">CONFLICT OF INTEREST STATEMENT</hd> <p>The authors declare no conflicts of interest.</p> <hd id="AN0180736619-24">DATA AVAILABILITY STATEMENT</hd> <p>The example data used in this study are available in online Appendix Table 1.</p> <p>GRAPH: Appendix S1: Online Appendix_20240909</p> <p>GRAPH: Data S1: Sepsis_20240909</p> <p>GRAPH: Data S2: Septic</p> <ref id="AN0180736619-25"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Lumley T. Network meta‐analysis for indirect treatment comparisons. Stat Med. 2002 ; 21 (16): 2313 ‐ 2324. doi: 10.1002/sim.1201</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Ades AE, Welton NJ, Dias S, Phillippo DM, Caldwell DM. Twenty years of network meta‐analysis: continuing controversies and recent developments. Res synth. Methods. 2024 ; 15 (5): 702 ‐ 727. doi: 10.1002/jrsm.1700</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta‐analyses. BMJ. 2003 ; 326 (7387): 472. doi: 10.1136/bmj.326.7387.472</bibtext> </blist> <blist> <bibl id="bib4" type="bt">4</bibl> <bibtext> Salanti G. Indirect and mixed‐treatment comparison, network, or multiple‐treatments meta‐analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods. 2012 ; 3 (2): 80 ‐ 97. doi: 10.1002/jrsm.1037</bibtext> </blist> <blist> <bibl id="bib5" idref="ref4" type="bt">5</bibl> <bibtext> Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta‐analysis of randomized controlled trials. Med Decis Making. 2013 ; 33 (5): 607 ‐ 617. doi: 10.1177/0272989X12458724</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Dias S, Sutton AJ, Welton NJ, Ades AE. Evidence synthesis for decision making 3: heterogeneity—subgroups, meta‐regression, bias, and bias‐adjustment. Med Decis Making. 2013 ; 33 (5): 618 ‐ 640. doi: 10.1177/0272989X13485157</bibtext> </blist> <blist> <bibl id="bib7" idref="ref5" type="bt">7</bibl> <bibtext> White IR, Barrett JK, Jackson D, Higgins JP. Consistency and inconsistency in network meta‐analysis: model estimation using multivariate meta‐regression. Res Synth Methods. 2012 ; 3 (2): 111 ‐ 125. doi: 10.1002/jrsm.1045</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Shih M‐C, Tu Y‐K. An evidence‐splitting approach to evaluation of direct‐indirect evidence inconsistency in network meta‐analysis. Res Synth Methods. 2021 ; 12 (2): 226 ‐ 238. doi: 10.1002/jrsm.1480</bibtext> </blist> <blist> <bibl id="bib9" idref="ref6" type="bt">9</bibl> <bibtext> Tu Y‐K. Node‐splitting generalized linear mixed models for evaluation of inconsistency in network meta‐analysis. Value Health. 2016 ; 19 (8): 957 ‐ 963. doi: 10.1016/j.jval.2016.07.005</bibtext> </blist> <blist> <bibtext> Tu YK. Using generalized linear mixed models to evaluate inconsistency within a network meta‐analysis. Value Health. 2015 ; 18 (8): 1120 ‐ 1125. doi: 10.1016/j.jval.2015.10.002</bibtext> </blist> <blist> <bibtext> Chaimani A, Caldwell DM, Li T, Higgins JP, Salanti G. Undertaking network meta‐analyses. Cochrane handbook for systematic reviews of interventions. Wiley ; 2019 : 285 ‐ 320.</bibtext> </blist> <blist> <bibtext> Baker SG, Kramer BS. The transitive fallacy for randomized trials: if a bests B and B bests C in separate trials, is a better than C? BMC Med Res Methodol. 2002 ; 2 (1): 13. doi: 10.1186/1471‐2288‐2‐13</bibtext> </blist> <blist> <bibtext> Baker SG, Kramer BS. Good for women, good for men, bad for people: Simpson's paradox and the importance of sex‐specific analysis in observational studies. J Womens Health Gend Based Med. 2001 ; 10 (9): 867 ‐ 872. doi: 10.1089/152460901753285769</bibtext> </blist> <blist> <bibtext> Higgins JPT, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconsistency in network meta‐analysis: concepts and models for multi‐arm studies. Res Synth Methods. 2012 ; 3 (2): 98 ‐ 110. doi: 10.1002/jrsm.1044</bibtext> </blist> <blist> <bibtext> Rott KW, Bronfort G, Chu H, et al. Causally interpretable meta‐analysis: clearly defined causal effects and two case studies. Res Synth Methods. 2024 ; 15 (1): 61 ‐ 72. doi: 10.1002/jrsm.1671</bibtext> </blist> <blist> <bibtext> Dahabreh IJ, Petito LC, Robertson SE, Hernán MA, Steingrimsson JA. Toward causally interpretable meta‐analysis: transporting inferences from multiple randomized trials to a new target population. Epidemiology. 2020 ; 31 (3): 334 ‐ 344. doi: 10.1097/ede.0000000000001177</bibtext> </blist> <blist> <bibtext> Degtiar I, Rose S. A review of generalizability and transportability. Annu Revi Stat Appl. 2023 ; 10 (1): 501 ‐ 524. doi: 10.1146/annurev‐statistics‐042522‐103837</bibtext> </blist> <blist> <bibtext> Schwartz AL, Alsan M, Morris AA, Halpern SD. Why diverse clinical trial participation matters. New Engl J Med. 2023 ; 388 (14): 1252 ‐ 1254. doi: 10.1056/NEJMp2215609</bibtext> </blist> <blist> <bibtext> Lai P‐C, Lai C‐H, Lai EC‐C, Huang Y‐T. Do we need to administer fludrocortisone in addition to hydrocortisone in adult patients with septic shock? An updated systematic review with Bayesian network meta‐analysis of randomized controlled trials and an observational study with target trial emulation. Crit Care Med. 2024 ; 52 (4): e193 ‐ e202. doi: 10.1097/ccm.0000000000006161</bibtext> </blist> <blist> <bibtext> Rücker G, Krahn U, König J, Efthimiou O, Davies A, Papakonstantinou TS. G. netmeta: Network Meta‐Analysis using Frequentist Methods. R package version 2.1–0. 2022 https://CRAN.R-project.org/package=netmeta</bibtext> </blist> <blist> <bibtext> Moreno R, Rhodes A, Piquilloud L, et al. The sequential organ failure assessment (SOFA) score: has the time come for an update? Crit Care. 2023 ; 27 (1): 15. doi: 10.1186/s13054‐022‐04290‐9</bibtext> </blist> <blist> <bibtext> Gattinoni L, Vasques F, Camporota L, et al. Understanding lactatemia in human sepsis. Potential impact for early management. Am J Respir Crit Care Med. 2019 ; 200 (5): 582 ‐ 589. doi: 10.1164/rccm.201812‐2342OC</bibtext> </blist> <blist> <bibtext> He L, Yang D, Ding Q, Su Y, Ding N. Association between lactate and 28‐day mortality in elderly patients with sepsis: results from MIMIC‐IV database. Infect Dis Ther. 2023 ; 12 (2): 459 ‐ 472. doi: 10.1007/s40121‐022‐00736‐3</bibtext> </blist> <blist> <bibtext> Chen J‐J, Lai P‐C, Lee T‐H, Huang Y‐T. Blood purification for adult patients with severe infection or sepsis/septic shock: a network meta‐analysis of randomized controlled trials. Crit Care Med. 2023 ; 51 (12): 1777 ‐ 1789. doi: 10.1097/ccm.0000000000005991</bibtext> </blist> <blist> <bibtext> Brignardello‐Petersen R, Tomlinson G, Florez I, et al. Grading of recommendations assessment, development, and evaluation concept article 5: addressing intransitivity in a network meta‐analysis. J Clin Epidemiol. 2023 ; 160 : 151 ‐ 159. doi: 10.1016/j.jclinepi.2023.06.010</bibtext> </blist> <blist> <bibtext> Nikolakopoulou A, Higgins JPT, Papakonstantinou T, et al. CINeMA: an approach for assessing confidence in the results of a network meta‐analysis. PLoS Med. 2020 ; 17 (4): e1003082. doi: 10.1371/journal.pmed.1003082</bibtext> </blist> </ref> <aug> <p>By Yu‐Kang Tu; Pei‐Chun Lai; Yen‐Ta Huang and James Hodges</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib11" firstref="ref7"></nolink> <nolink nlid="nl2" bibid="bib12" firstref="ref9"></nolink> <nolink nlid="nl3" bibid="bib14" firstref="ref12"></nolink> <nolink nlid="nl4" bibid="bib15" firstref="ref15"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref17"></nolink> <nolink nlid="nl6" bibid="bib17" firstref="ref18"></nolink> <nolink nlid="nl7" bibid="bib19" firstref="ref19"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref20"></nolink> <nolink nlid="nl9" bibid="bib21" firstref="ref22"></nolink> <nolink nlid="nl10" bibid="bib22" firstref="ref23"></nolink> <nolink nlid="nl11" bibid="bib24" firstref="ref25"></nolink> <nolink nlid="nl12" bibid="bib25" firstref="ref27"></nolink> <nolink nlid="nl13" bibid="bib26" firstref="ref28"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1447378
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Visualizing the Assumptions of Network Meta-Analysis
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Yu-Kang+Tu%22">Yu-Kang Tu</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2461-474X">0000-0002-2461-474X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Pei-Chun+Lai%22">Pei-Chun Lai</searchLink><br /><searchLink fieldCode="AR" term="%22Yen-Ta+Huang%22">Yen-Ta Huang</searchLink><br /><searchLink fieldCode="AR" term="%22James+Hodges%22">James Hodges</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Research+Synthesis+Methods%22"><i>Research Synthesis Methods</i></searchLink>. 2024 15(6):1175-1182.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 8
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Visualization%22">Visualization</searchLink><br /><searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Studies%22">Statistical Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Research+and+Development%22">Research and Development</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Health+Services%22">Health Services</searchLink><br /><searchLink fieldCode="DE" term="%22Outcomes+of+Treatment%22">Outcomes of Treatment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jrsm.1760
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1759-2879<br />1759-2887
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Network meta-analysis (NMA) incorporates all available evidence into a general statistical framework for comparing multiple treatments. Standard NMAs make three major assumptions, namely homogeneity, similarity, and consistency, and violating these assumptions threatens an NMA's validity. In this article, we suggest a graphical approach to assessing these assumptions and distinguishing between qualitative and quantitative versions of these assumptions. In our plot, the absolute effect of each treatment arm is plotted against the level of effect modifiers, and the three assumptions of NMA can then be visually evaluated. We use four hypothetical scenarios to show how violating these assumptions can lead to different consequences and difficulties in interpreting an NMA. We present an example of an NMA evaluating steroid use to treat septic shock patients to demonstrate how to use our graphical approach to assess an NMA's assumptions and how this approach can help with interpreting the results. We also show that all three assumptions of NMA can be summarized as an exchangeability assumption. Finally, we discuss how reporting of NMAs can be improved to increase transparency of the analysis and interpretability of the results.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1447378
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1447378
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jrsm.1760
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 8
        StartPage: 1175
    Subjects:
      – SubjectFull: Visualization
        Type: general
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Statistical Studies
        Type: general
      – SubjectFull: Research and Development
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Health Services
        Type: general
      – SubjectFull: Outcomes of Treatment
        Type: general
    Titles:
      – TitleFull: Visualizing the Assumptions of Network Meta-Analysis
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Yu-Kang Tu
      – PersonEntity:
          Name:
            NameFull: Pei-Chun Lai
      – PersonEntity:
          Name:
            NameFull: Yen-Ta Huang
      – PersonEntity:
          Name:
            NameFull: James Hodges
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 11
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1759-2879
            – Type: issn-electronic
              Value: 1759-2887
          Numbering:
            – Type: volume
              Value: 15
            – Type: issue
              Value: 6
          Titles:
            – TitleFull: Research Synthesis Methods
              Type: main
ResultId 1