View in EDS HTML Full Text PDF Full Text

Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables

Saved in:
Bibliographic Details
Title: Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables
Language: English
Authors: Chris Felton (ORCID 0000-0001-9214-9985), Brandon M. Stewart (ORCID 0000-0002-7657-3089)
Source: Sociological Methods & Research. 2026 55(1):3-50.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 48
Publication Date: 2026
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Social Science Research, Sociology, Statistical Inference, Statistical Bias, Computation, Research Methodology
DOI: 10.1177/00491241241235900
ISSN: 0049-1241
1552-8294
Abstract: Instrumental variables (IV) analysis is a powerful, but fragile, tool for drawing causal inferences from observational data. Sociologists increasingly turn to this strategy in settings where unmeasured confounding between the treatment and outcome is likely. This paper reviews the assumptions required for IV and the consequences of violating them, focusing on sociological applications. We highlight three methodological problems IV faces: (i) identification bias, an asymptotic bias from assumption violations; (ii) estimation bias, a finite-sample bias that persists even when assumptions hold; and (iii) type-M error, the exaggeration of effect size given statistical significance. In each case, we emphasize how weak instruments exacerbate these problems and make results sensitive to minor violations of assumptions. We survey IV papers from top sociology journals, finding that assumptions often go unstated and robust uncertainty measures are rarely used. We provide a practical checklist to show how IV, despite its fragility, can still be useful when handled with care.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1495936
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGPl5tGViPqmFRY1mxo8NX5AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDK1ATfbBd_4CUnDvEgIBEICBmxKO7TyADtgxCJxk7A4HWN2Xpd_Jhia0q7_9YR12J-RwjOL-JrNKCcCFDzBH7NIhIrS1HGKCO-MXfM60bOSNFgQIWnPwgJcMUbwGMxOcKLSbWpmoVfHlVL_cVWCQZVPDikcjzpezuwYMt8H_ZPXetA0EEcjP5kA_0vYpg18Mg5u5KDlSKiePGuttsTP_WMsBFCoNHqbCZKhsVCb7
Text:
  Availability: 1
  Value: <anid>AN0190929145;som01feb.26;2026Jan20.00:54;v2.2.500</anid> <title id="AN0190929145-1">Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables </title> <p>Instrumental variables (IV) analysis is a powerful, but fragile, tool for drawing causal inferences from observational data. Sociologists increasingly turn to this strategy in settings where unmeasured confounding between the treatment and outcome is likely. This paper reviews the assumptions required for IV and the consequences of violating them, focusing on sociological applications. We highlight three methodological problems IV faces: (i) identification bias, an asymptotic bias from assumption violations; (ii) estimation bias, a finite-sample bias that persists even when assumptions hold; and (iii) type-M error, the exaggeration of effect size given statistical significance. In each case, we emphasize how weak instruments exacerbate these problems and make results sensitive to minor violations of assumptions. We survey IV papers from top sociology journals, finding that assumptions often go unstated and robust uncertainty measures are rarely used. We provide a practical checklist to show how IV, despite its fragility, can still be useful when handled with care.</p> <p>Keywords: instrumental variables; causal inference; quantitative methods; selection on observables; sensitivity analysis; robust inference</p> <hd id="AN0190929145-2">Introduction</hd> <p>Sociology is full of difficult but important questions. In many cases—for instance, studying the effect of incarceration on future employment—we seek causal evidence where randomized controlled trials would be impractical or unethical. The most common strategy for identifying causal effects with observational data is <emph>selection on observables</emph>. This approach involves conditioning on a set of observed confounders—typically through regression, weighting, or matching—that help to block non-causal associations between the treatment and the outcome. Under strong assumptions, the remaining correlation is causation.</p> <p>Selection on observables demands we block <emph>all</emph> confounding between the treatment and outcome. Part of what makes many sociological questions difficult is that we can never measure enough variables to make this no-unmeasured-confounding assumption completely plausible. Following economics, sociologists have increasingly turned to instrumental variables analysis (IV) as an alternative—or complementary—strategy for estimating causal effects. IV exploits the presence of an <emph>instrument</emph>, an unconfounded source of variation that affects the treatment but only influences the outcome through its effect on the treatment. IV isolates the variation in treatment caused by the instrument and uses only this variation to estimate the causal effect. The promise of IV is that it allows us to credibly estimate causal effects even when unmeasured confounding plagues the treatment–outcome association.</p> <p>IV is powerful. But it is also more fragile than many researchers realize. IV rests on strong assumptions, and it can be highly sensitive to violations of these assumptions. In this paper, we walk through the conditions IV requires and explain why IV estimators can be brittle under even modest departures from these conditions. To clarify what the assumptions require, we discuss violations in the context of published empirical examples and illustrate these violations graphically when possible. Our hope is that a better understanding of IV's assumptions will help researchers select more plausible instruments, while a better understanding of its fragility will enable researchers and readers to make appropriate comparisons between IV and selection on observables.</p> <p>IV can still be a useful tool if we handle it with care. To this end, we provide concrete guidance—in the form of a checklist—on how to improve the use of IV. Of particular importance is <emph>bias analysis</emph>. In practice, IV assumptions will often be violated. Bias analysis allows us to assess how severe the violations would have to be to change the conclusions of our study. In other words, IV is fragile, but bias analysis helps us address its fragility in a constructive way. In addition to bias analysis, we review robust diagnostics and uncertainty measures for IV that are seldom used in sociology.</p> <p>Our discussion of IV assumptions and our guidelines for using IV are grounded in a survey of 34 IV papers published in the <emph>American Journal of Sociology</emph> (<emph>AJS</emph>) and the <emph>American Sociology Review</emph> (<emph>ASR</emph>) between 2004 and 2022. Our survey revealed that authors rarely stated or defended IV assumptions. For instance, out of 34 papers, only one stated the "monotonicity" assumption, an important condition we review in "Monotonicity." Furthermore, researchers seldom reported key diagnostics that are considered standard in other fields. For instance, only 18 of 34 papers reported a first-stage <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic, a diagnostic we review in "Estimation Bias," and no study reported a formal bias analysis, which we describe in "Identification Bias." Finally, like reviews in other fields, we found that IV estimates were typically larger in magnitude than selection-on-observables estimates ([<reflink idref="bib44" id="ref1">44</reflink>]; [<reflink idref="bib49" id="ref2">49</reflink>]). This finding is concerning—we often resort to IV because we suspect selection-on-observables estimates are biased <emph>upward</emph> in magnitude—and we consider two explanations for the trend in "Type-M Error." By collecting guidance from different literatures and organizing it in one place, our checklist can provide researchers with an easy reference to improve the use of IV in sociology and other fields.</p> <hd id="AN0190929145-3">Related Work</hd> <p>IV is a popular approach in the social sciences, and there are consequently many resources on the approach. In sociology, [<reflink idref="bib10" id="ref3">10</reflink>] reviews IV primarily in the context of linear structural equation models. In contrast, we discuss IV in the potential outcomes framework, avoiding constant effects assumptions and allowing us to describe identification assumptions in explicitly causal terms more akin to the coverage of [<reflink idref="bib59" id="ref4">59</reflink>]. Compared with [<reflink idref="bib59" id="ref5">59</reflink>], our review places more emphasis on how IV analyses are reported and conducted in practice, and we highlight more recently developed diagnostics and inferential tools for improving these analyses. We also pay special attention to commonly used instruments and clarify how they may violate critical assumptions.</p> <p>Reader's guides of this sort have appeared in numerous fields. [<reflink idref="bib72" id="ref6">72</reflink>] offer a useful reader's guide to IV use in political science. In economics, [<reflink idref="bib6" id="ref7">6</reflink>] review identification assumptions for IV and explain the interpretation of IV estimates across a range of settings. [<reflink idref="bib9" id="ref8">9</reflink>] give a thorough introduction to using IV in medical research, focusing on instruments that are widely used in medicine and epidemiology. With the exception of [<reflink idref="bib68" id="ref9">68</reflink>]—who critically surveys IV use in marketing—our review provides a more skeptical perspective on IV than other reviews. Furthermore, we clarify the conditions under which instruments can be used to address "simultaneity bias," an oft-cited motivation for IV that receives little coverage in IV reviews.</p> <p>We also discuss weak instruments differently from most reviews. We clarify that an instrument can be "weak" in three different ways, each exacerbating a distinct methodological problem. Review articles typically mention only two of these problems and provide diagnostics for only one. Furthermore, methodologists sometimes describe these diagnostics as "weak-instrument tests" even though they diagnose only one form of weakness. To improve both understanding and practice, we provide new names for the distinct problems caused by different types of weak instruments and suggest diagnostics for all three.</p> <p>More recently, [<reflink idref="bib83" id="ref10">83</reflink>] and [<reflink idref="bib49" id="ref11">49</reflink>] use mass replications of published IV papers to study statistical inference in economics and political science, respectively. [<reflink idref="bib83" id="ref12">83</reflink>] demonstrates that published IV results in economics are extremely sensitive to outliers and shows that even robust confidence intervals typically fail to achieve nominal coverage, particularly when clustered or panel data is used. [<reflink idref="bib49" id="ref13">49</reflink>] find that similar inferential problems plague the political science literature, and further document that IV estimates are often much larger in magnitude than selection-on-observables estimates. We take inspiration from [<reflink idref="bib49" id="ref14">49</reflink>] in comparing the magnitude of IV and selection-on-observables estimates in sociology and providing a checklist for best practices. While these surveys also emphasize the fragility of IV, we primarily focus on research design and identification assumptions rather than statistical inference—issues that would be hard to address through mass replications. Additionally, we highlight the problem of <emph>type-M</emph> errors, a concern that has received less attention in the IV setting.</p> <hd id="AN0190929145-4">Structure of the Paper</hd> <p>In "What is an Instrumental Variable, and Why Would We Use One?," we introduce the basic logic of IV and provide several running examples from the sociological literature. "Identification and Estimation of Treatment Effects with IV" reviews the assumptions used to identify the <emph>Local Average Treatment Effect (LATE)</emph>, a common effect of interest in IV studies. We highlight subtleties of these assumptions that often go unstated and review how to estimate treatment effects with IVs. "Weakness Exacerbates Three Methodological Problems" describes three methodological challenges IV faces and provide guidelines on how to address each one. In "A Checklist for Conducting and Reporting Instrumental Variables Analysis," we compile a thorough checklist for both authors and readers of IV studies.</p> <hd id="AN0190929145-5">What is an IV, and Why Would We Use One?</hd> <p>If a recently released parolee were to move to a new neighborhood—rather than return home—would they be less likely to reoffend? This is the question asked by [<reflink idref="bib46" id="ref15">46</reflink>]. In observational data, the association between moving and recidivism is almost certainly confounded—for instance, by the resources that enable such moves—and we may lack the information necessary to produce a credible selection-on-observables estimate. If we could randomly assign some parolees to move and others to return home, we could produce a dataset free of confounding, but such an experiment would likely be infeasible.</p> <p>[<reflink idref="bib46" id="ref16">46</reflink>] approaches this problem by using the timing of release—before or after Hurricane Katrina—as an IV. The logic is that many parolees in Louisiana were unable to return home following Katrina. Assuming the timing of the release is random, it is as though those who were released after Katrina were "assigned" to a condition where they were unable to return home in a hypothetical randomized experiment.</p> <p>Importantly, randomness is not enough. IV also requires that release timing affects recidivism <emph>only through</emph> preventing parolees from returning home. If this condition holds, any association between release timing (the instrument) and recidivism (the outcome) must reflect the effects of moving (the treatment), barring association from chance alone. Under the necessary assumptions—which we detail more explicitly below—IV does an amazing thing: it enables us to identify causal effects even if unmeasured factors cause parolees to both change neighborhoods and reoffend. In practice, though, our estimates can be sensitive to seemingly minor violations of these assumptions.</p> <p>Table 1 summarizes the research design and assumptions employed by [<reflink idref="bib46" id="ref17">46</reflink>] along with three other IV studies in sociology that we use throughout this paper. We will also make reference throughout to our survey of all 34 papers using IV in AJS and ASR published between 2004 and 2022. Complete details on this survey can be found in Online Appendix A.</p> <p>Table 1. Example IV Studies in Sociology.</p> <p>Graph</p> <hd id="AN0190929145-6">Identification and Estimation of Treatment Effects with IV</hd> <p>We begin by reviewing the assumptions under which IV can provide asymptotically unbiased treatment effect estimates. In particular, we focus on identifying the LATE. This is the average treatment effect among <emph>compliers</emph>—people for whom the instrument encourages treatment uptake ([<reflink idref="bib41" id="ref18">41</reflink>]). We include an annotated proof of the identification result in Online Appendix D. We focus on average effects rather than quantile effects because researchers are typically more interested in the former, and we focus on the LATE because identifying other average effects requires more stringent assumptions (Angrist and Imbens, 1999; [<reflink idref="bib38" id="ref19">38</reflink>]).[<reflink idref="bib7" id="ref20">7</reflink>]</p> <p>Our summary of identification assumptions serves three purposes. First, we aim to clarify what exactly the assumptions require so that researchers can select more plausible instruments. Second, we demonstrate how to explain these assumptions in a non-technical fashion so that researchers can better convey them to readers. Finally, we illustrate why it is so difficult to find instruments that meet all of these assumptions. In "Identification Bias," we show how to proceed with IV analysis when our assumptions fail to hold exactly.</p> <p>Before turning to identification assumptions, we briefly review the notation we use throughout the paper, although we describe each assumption in non-mathematical terms as well. Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Z</mi><mi>i</mi></msub></math> </ephtml> represent the instrument. For [<reflink idref="bib46" id="ref21">46</reflink>], <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Z</mi><mi>i</mi></msub></math> </ephtml> captures whether parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> was released before ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Z</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></math> </ephtml> ) or after ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Z</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></math> </ephtml> ) Hurricane Katrina. Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>i</mi></msub></math> </ephtml> represent the treatment—whether parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> returned to their home parish ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn></math> </ephtml> ) or a different parish ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn></math> </ephtml> ). Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub></math> </ephtml> represent the outcome—whether parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> was re-arrested within one year following release. [<reflink idref="bib46" id="ref22">46</reflink>] also includes a set of pre-instrument controls, which we call <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">X</mi></mrow><mi>i</mi></msub></math> </ephtml> .</p> <p>We employ potential outcomes notation to specify causal assumptions ([<reflink idref="bib70" id="ref23">70</reflink>]). Understanding the difference between observed and potential outcomes is crucial to following the formal descriptions of assumptions. Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></math> </ephtml> represent the outcome we would observe for parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> if—possibly contrary to fact—she had been assigned the treatment. If parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> had been assigned to the control condition, we would instead observe <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></math> </ephtml> , and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></math> </ephtml> would be an unobserved, counterfactual outcome. The treatment effect for parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> is the difference between her potential outcomes under treatment and control: <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></math> </ephtml> . The fundamental problem of causal inference is that we can only observe one of these potential outcomes ([<reflink idref="bib37" id="ref24">37</reflink>]).</p> <p>Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>Z</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></math> </ephtml> represent the potential treatment parolee <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> would have received had she been randomly assigned to be released post-Katrina rather than pre-Katrina—again, possibly contrary to fact. Let <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>D</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></math> </ephtml> represent the potential outcome we would observe for person <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> had she received both the treatment.[<reflink idref="bib8" id="ref25">8</reflink>] We can now define the LATE (Definition 1)—the quantity we aim to identify.</p> <p>Local Average Treatment Effect.</p> <p>The LATE is</p> <p>Graph</p> <p>"Monotonicity" clarifies the interpretation of the estimand in non-mathematical language.</p> <p>This section outlines six identification assumptions that, together, can be used to identify the LATE. For each assumption, we provide concrete examples of how it might be violated. Table 2 displays the percentage of papers in our survey that state each assumption in causal terms.</p> <p>Table 2. Proportions of Papers Stating Identification Assumptions in Causal Terms.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="right" /></colgroup><thead><tr><th align="left">Identification assumption</th><th align="left">% of papers that state assumption</th></tr></thead><tbody><tr><td>Relevance</td><td>82</td></tr><tr><td>Unconfoundedness</td><td>21</td></tr><tr><td>Exclusion restriction</td><td>62</td></tr><tr><td>Monotonicity</td><td>3</td></tr><tr><td>Stable-unit-treatment-value assumption (SUTVA)</td><td>0</td></tr><tr><td>Positivity</td><td>0</td></tr></tbody></table> </ephtml> </p> <p>1 For instance, we can describe the unconfoundedness assumption as the assumption that the instrument shares no common causes with the treatment or outcome. In contrast, papers that merely state that the instrument must be uncorrelated with an error term are describing the assumption in terms of model-based associations rather than causal effects. Stating assumptions in causal terms helps both author and audience reason about them more clearly. We round percentages to the nearest whole number.</p> <p>Throughout this section, we focus on the most straightforward setting: The case with a binary instrument and binary treatment. We discuss more general settings and alternative estimands in Online Appendix F.</p> <hd id="AN0190929145-7">Relevance</hd> <p>In [<reflink idref="bib46" id="ref26">46</reflink>], the relevance assumption requires that being released after Hurricane Katrina has a causal effect on whether a parolee returned to his home parish or not. Kirk shows that roughly 75% of parolees released pre-Katrina returned to their home parishes, compared with only 50% of parolees released post-Katrina. Such a large and abrupt difference in return rates likely indicates a causal effect of the instrument on the treatment. We state this assumption formally below.</p> <p>Relevance.</p> <p>Graph</p> <p>More generally, this assumption states that the instrument (post-Katrina release) has a non-zero average causal effect on treatment uptake (moving neighborhoods). This is an assumption about causation—not association. The instrument causes the change in the treatment. An instrument that is caused by the treatment, in contrast, cannot be used to identify treatment effects.[<reflink idref="bib9" id="ref27">9</reflink>]</p> <p>It is also possible to estimate treatment effects using a <emph>proxy</emph> instrument—something that has no causal effect on the treatment but shares an unmeasured common cause with the treatment. Even in the proxy case, an instrument that causes the treatment must exist—although it is unobserved—and key assumptions apply to the true causal instrument. Because proxy-instrument designs require different and more nuanced assumptions, we discuss them separately in Online Appendix B. Note also that a <emph>relevant</emph> instrument can still be a <emph>weak</emph> instrument, a topic we return to in "Weakness Exacerbates Three Methodological Problems."</p> <hd id="AN0190929145-8">Unconfounded Instrument</hd> <p>[<reflink idref="bib71" id="ref28">71</reflink>] examine the effect of lead exposure on deliquency. As an instrument, they use a person's distance from a smelting plant. Smelting plants contaminate soil, exposing nearby residents to lead and making distance a relevant instrument for lead exposure. The unconfoundedness assumption has two parts.[<reflink idref="bib10" id="ref29">10</reflink>] First, the instrument (distance to a smelting plant) can share no unmeasured common causes with the outcome (deliquency).[<reflink idref="bib11" id="ref30">11</reflink>] Second, the instrument can share no unmeasured common causes with the treatment (lead exposure).[<reflink idref="bib12" id="ref31">12</reflink>] We state the assumption in potential outcomes notation below.</p> <p>Unconfounded Instrument.</p> <p>Graph</p> <p>and</p> <p>Graph</p> <p>Assumption 2 <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow /><mo>′</mo></msup></math> </ephtml> . Conditionally Unconfounded Instrument.</p> <p>Graph</p> <p>The reason we resort to IV in the first place is that we believe the <emph>treatment</emph> is counfounded. As such, the unconfoundedness of the <emph>instrument</emph> plays a central role in the analysis. Yet, of 34 papers in our survey of IV papers, only seven stated this central assumption in causal terms. Most authors instead describe a broader assumption about the correlation between the instrument and an error term, a common practice in the classical structural-equation-model setting (see e.g. [<reflink idref="bib10" id="ref32">10</reflink>]). We avoid this description because it combines both an unconfoundedness assumption and the <emph>exclusion restriction</emph>, a condition we describe in "The Exclusion Restriction." While these two assumptions can be collapsed mathematically, it is easier to reason about them separately.</p> <p>How plausible is the unconfoundedness assumption in [<reflink idref="bib71" id="ref33">71</reflink>]? To answer this question, we refer to a figure from their online supplement reproduced here as Figure 1. The map shows that location of smelting plants is strongly associated with neighborhood poverty composition. Just as neighborhood disadvantage may confound the relationship between the treatment and the outcome, it may confound the relationship between the instrument and the outcome. We depict this potential violation graphically in Figure 2 using red dashed arrows.</p> <p>Graph: Figure 1. Locations of smelting plants relative to neighborhood poverty composition. The figure is taken from [<reflink idref="bib71" id="ref34">71</reflink>], Appendix D. The locations are strongly associated with neighborhood poverty composition, raising concerns about the unconfoundedness assumption.</p> <p>Graph: Figure 2. Directed acyclic graph (DAG) depicting potential unconfoundedness violations in Sampson and Winter (2018). Red, dashed arrows represent violations. Neighborhood disadvantage generates both instrument—outcome confounding and instrument—treatment confounding. The bi-directed arrow between Neighborhood Disadvantage and Proximity to Smelting Plant indicates they are themselves related through some unmeasured confounder.</p> <p>[<reflink idref="bib71" id="ref35">71</reflink>] acknowledge that distance from smelting plants is likely confounded with deliquency, but they argue that the instrument should be unconfounded conditional on measured confounders such as the percentage of a neighborhood living in poverty. The motivation for IV in this setting relies on the idea that these measured covariates are sufficient for blocking instrument–treatment and instrument–outcome confounding while also being insufficient for blocking treatment–outcome confounding. If residents select into neighborhoods on the basis of unmeasured characteristics that are associated with the instrument (e.g., school quality), and these characteristics also affect the outcome, the assumption will be violated.</p> <p>Distance-based instruments are common: [<reflink idref="bib12" id="ref36">12</reflink>] uses distance from a college as an instrument for attending college, and [<reflink idref="bib57" id="ref37">57</reflink>] use differential distance to alternative hospitals as an instrument for intensive heart attack treatment. Distance-based instruments raise concerns about the unconfoundedness assumption, since where someone lives tends to be associated with many other individual- and neighborhood-level characteristics. While instruments may often seem <emph>less</emph> confounded than their corresponding treatments, in "Weakness Exacerbates Three Methodological Problems" we explain why IV estimates can still be more biased than selection-on-observables estimates in such settings. In "Identification Bias" and Online Appendix C, we discuss tools for addressing potential violations of unconfoundedness.</p> <hd id="AN0190929145-9">The Exclusion Restriction</hd> <p>Returning to [<reflink idref="bib46" id="ref38">46</reflink>], the exclusion restriction requires that being released after Hurricane Katrina (the instrument) has no effect on recidivism (the outcome) <emph>except through residential change</emph> (the treatment). More generally, the exclusion restriction requires that the instrument has no effect on the outcome except through the treatment. We present this assumption in potential outcomes notation below.</p> <p>Exclusion Restriction.</p> <p>Graph</p> <p>Of the 34 papers we reviewed, 21 describe what the exclusion restriction entails in causal terms. While the exclusion restriction is the most widely stated assumption in papers exploiting IVs, we also believe it may be the most misunderstood by casual readers. We emphasize three subtleties. First, while the exclusion restriction is often described as the instrument having "no direct effect" on the outcome, it is clearer to say that there are no effects <emph>unmediated by the treatment</emph>. The reason is that effects mediated by unmeasured variables other than the treatment still violate the assumption. Second, while we can block exclusion restriction violations by conditioning on post-instrument covariates, doing so may induce violations of unconfoundedness ([<reflink idref="bib27" id="ref39">27</reflink>]). Third, coarsely measured treatments can generate exclusion restriction violations ([<reflink idref="bib55" id="ref40">55</reflink>]). We expand on each of these points in the three subsections below.</p> <hd id="AN0190929145-10">What "No Direct Effect" Means in the Context of the Exclusion Restriction</hd> <p>In the case considered by [<reflink idref="bib46" id="ref41">46</reflink>], suppose that Hurricane Katrina prevents a hypothetical parolee from getting a job. Suppose further that his unemployment causes him to re-offend, which in turn leads to his arrest. This causal chain—from the hurricane (instrument), to employment, to crime, to re-arrest (outcome)—would violate the exclusion restriction. Another potential exclusion restriction might occur through strain on police resources. We depict both in Figure 3.</p> <p>Graph: Figure 3. DAG depicting potential exclusion-restriction violations in Kirk (2009). Violations of the exclusion restriction drawn using red, dashed arrows. The causal pathways Post-Katrina Release → Police Resources → Re-arrest and Post-Katrina Release → Employment → Crime → Re-arrest each violate the exclusion restriction because they do not operate through the treatment. Note that we can also think of the former causal pathway as differential measurement error rather than a violation of the exclusion restriction, where being re-arrested is a mismeasure of recidivism, and the error in measurement is affected by the instrument. To simplify the DAG, we opted to use re-arrest (whether the parolee was arrested) rather than recidivism (whether the parolee committed a crime) as the outcome.</p> <p>One case where we should be particularly cautious of exclusion restriction violations through indirect causal chains is when the same instrument is used for many different treatments. [<reflink idref="bib58" id="ref42">58</reflink>] found that weather has been used as an instrument in at least 159 studies across the social sciences. The fact that weather appears to affect so many different treatments raises concerns about its validity in any given application. [<reflink idref="bib48" id="ref43">48</reflink>], for instance, study the effects of outdoor leisure on math test scores using sunlight as an instrument for outdoor activity. As the authors themselves highlight, one potential violation of the exclusion restriction occurs through mood. If sunlight improves mood through pathways other than outdoor activity, and mood itself affects math performance, the exclusion restriction could be violated. Indeed, [<reflink idref="bib58" id="ref44">58</reflink>] locates a number of studies showing that weather affects mood and that mood in turn influences a wide range of other outcomes. [<reflink idref="bib48" id="ref45">48</reflink>] argue that effects of weather on mood are likely trivial in magnitude. However, as we emphasize in "Identification Bias," even small violations can yield counterintuitively large bias in IV.</p> <p>Historical instruments—those that occurred long before the treatment—present another complicated case for exclusion restriction assumptions. Consider Rothwell and Massey's ([<reflink idref="bib69" id="ref46">69</reflink>]) study using population density in 1910 as an instrument for density zoning in 2000 to assess its effect on segregation in the year 2000. While it is hard to imagine a "completely unmediated" effect of population density in 1910 on segregation in 2000, it is easy to imagine effects <emph>unmediated by the treatment</emph> (density zoning in 2000). Population density affects a lot of things about cities—economic growth, housing prices, wages—and it is difficult to believe that none of these things have any downstream effects on segregation ([<reflink idref="bib1" id="ref47">1</reflink>]).</p> <hd id="AN0190929145-11">Why Blocking Exclusion Restriction Violations Can Induce Unconfoundedness Violations</hd> <p>To block potential violations of the exclusion restriction, [<reflink idref="bib46" id="ref48">46</reflink>] conditions on the parish-level unemployment rate and other post-instrument variables, a common practice in IV studies. The idea is that we can block causal pathways between release timing and re-arrest by identifying mediators on these pathways and conditioning on them. Unfortunately, if there are any unmeasured common causes between unemployment (the mediator) and re-arrest, conditioning on the unemployment rate will violate conditional unconfoundedness.</p> <p>When we condition on the unemployment rate, we compare <emph>pre</emph>-Katrina parishes suffering from high unemployment with <emph>post</emph>-Katrina parishes suffering from high unemployment in order to estimate the effects of the instrument. But this is not an "apples-to-apples" comparison. Pre-Katrina, high unemployment can be a mark of long-term disadvantage. But in the immediate aftermath of Katrina, high unemployment was largely a function of living near the coast. When we compare pre-Katrina parishes and post-Katrina parishes with the same levels of unemployment, we end up comparing more disadvantaged parishes with less disadvantaged parishes. We have inadvertently induced an association between being released post-Katrina and living in a more advantaged parish. We depict this violation graphically in Figure 4.</p> <p>Graph: Figure 4. Directed acyclic graph (DAG) depicting potential post-instrument bias in Kirk (2009). Conditioning on Employment opens up the path Post-Katrina Release → Employment ← Neighborhood Disadvantage → Re-arrest , illustrated with red-dashed arrows. The rectangle drawn around Employment indicates it has been conditioned on. The conditioning induces an association between the instrument and potential outcomes, violating conditional unconfoundedness.</p> <p>More generally, we can say that conditioning on post-instrument variables induces unconfoundedness violations in IV settings. In the language of DAGs, post-instrument variables are often "colliders" between the instrument and an unmeasured confounder, and conditioning on this collider will induce an association between the instrument and the unmeasured confounder ([<reflink idref="bib24" id="ref49">24</reflink>]). Following [<reflink idref="bib27" id="ref50">27</reflink>], we call this <emph>post-instrument bias</emph>. Post-instrument bias parallels post-treatment bias in selection-on-observables settings.</p> <hd id="AN0190929145-12">Why Coarse or Coarsened Treatments Induce Exclusion Restriction Violations</hd> <p>[<reflink idref="bib46" id="ref51">46</reflink>] measures residential change using a parolee's parish of residence. Parishes measure neighborhoods coarsely: In 2010, there were 204,447 census blocks and 3,471 block groups in Louisiana, but only 64 parishes. Within any parish, then, there is substantial variability in neighborhood conditions. Suppose that Hurricane Katrina induced a parolee to move to a different neighborhood in the same parish, and that moving to this different neighborhood prevented him from reoffending. Because his move will not be captured in our measure of residential change (i.e., he returned to the same parish), the exclusion restriction will be violated: The instrument has an effect on the outcome that is not captured by treatment <emph>as we have measured it</emph>.[<reflink idref="bib13" id="ref52">13</reflink>] More generally, coarsely measured treatments often bias the IV estimator upward in magnitude ([<reflink idref="bib55" id="ref53">55</reflink>]).[<reflink idref="bib14" id="ref54">14</reflink>]</p> <hd id="AN0190929145-13">Monotonicity</hd> <p>A common instrument for incarceration is judge leniency. To give a toy example, suppose a courthouse has two judges—Judge Alice and Judge Bill—and that defendants are randomly assigned to one for sentencing. If Bill is more likely to sentence someone to prison than Alice, being assigned to Bill could serve as an instrument for incarceration. Being assigned to Bill increases your probability of incarceration, and it is unconfounded since judges are randomly assigned. We can use variation in incarceration caused by judge leniency to identify the effects of incarceration on a range of outcomes, such as future employment or earnings.</p> <p>To illustrate the monotonicity assumption, we define four possible "compliance types" from [<reflink idref="bib5" id="ref55">5</reflink>] as shown in Table 3. Consider a hypothetical defendant Carol. Carol is a <emph>never-taker</emph> if, regardless of the judge she is assigned to, she is never incarcerated. Carol is an <emph>always-taker</emph> if she is always incarcerated regardless of her judge assignment. Carol is a <emph>complier</emph> if the harsher judge (Bill) would incarcerate her but the more lenient judge (Alice) would not. Finally, Carol is a <emph>defier</emph> if Alice would incarcerate her but Bill would not. The monotonicity assumption requires that there are <emph>no defiers</emph> in our sample. Because this is an assumption about individual-level effects rather than average effects, it is fundamentally untestable. Of the 34 sociology papers we reviewed, only one stated the monotonicity assumption. We define the assumption formally below.[<reflink idref="bib15" id="ref56">15</reflink>]</p> <p>Table 3. IV Compliance Classes for the toy Judge Example.</p> <p>Graph</p> <p>Monotonicity.</p> <p>Graph</p> <p>How might monotonicity be violated? Suppose that Bill orders harsh sentences for violent offenses but lenient sentences for drug offenses. In contrast, Alice is harsh on drug offenders but lenient with violent offenders. If the courthouse sees more violent offenders than drug offenders, Bill will appear to be harsher overall. But if our defendant, Carol, is a drug offender, she might be a defier: She has a higher probability of being incarcerated if assigned to Alice rather than Bill. The monotonicity assumption prohibits defiers like Carol. If present in our sample, monotonicity will be violated. More generally, the monotonicity assumption holds that an instrument that encourages treatment <emph>can never discourage treatment for any unit</emph>, although it may fail to encourage treatment for many. When treatment effects differ between compliers and other compliance classes, the monotonicity assumption allows us to identify the average treatment effect <emph>just for compliers</emph>—i.e., the LATE (Definition 1).[<reflink idref="bib16" id="ref57">16</reflink>]</p> <p>To give a concrete example, when we use judge leniency as an instrument for incarceration, this means that we identify the effects of incarceration only for people who were right on the cusp of being incarcerated. If Carol had been found guilty of first-degree murder, for instance, she would be sentenced to prison regardless of her judge. She would be an always-taker, and we would be unable to identify treatment effects for people like her.</p> <p>Of the 34 IV papers we surveyed, only eight clarified that they could only identify the LATE. As [<reflink idref="bib52" id="ref58">52</reflink>] point out, the population of compliers can be extremely small and may be unrelated to the theoretical motivation for the study. Following them, we urge researchers to not only state the estimand but also make clear why compliers are a population of interest.[<reflink idref="bib17" id="ref59">17</reflink>]</p> <p>The judge instrument, and instruments like it, are common in the social sciences. [<reflink idref="bib30" id="ref60">30</reflink>] uses judges as instruments for incarceration, and [<reflink idref="bib21" id="ref61">21</reflink>] uses doctors' differential preferences for prescribing opioids as an instrument for taking opioids. [<reflink idref="bib75" id="ref62">75</reflink>] call these "preference-based" instruments and provide strong evidence that monotonicity is violated in the case of doctors' preferences for prescribing certain drugs. One way to address monotonicity violations is by constructing a more complicated set of instruments rather than using a unidimensional measure of preference. For instance, [<reflink idref="bib30" id="ref63">30</reflink>] interact individual judges with indicators for the type of crime committed, leading to a set of dozens of instruments. This approach has its own challenges (as we discuss in "Estimation Bias").</p> <hd id="AN0190929145-14">SUTVA and Positivity</hd> <p>[<reflink idref="bib5" id="ref64">5</reflink>] use two additional assumptions to identify the LATE that we discuss only briefly. The stable-unit-treatment-value assumption (SUTVA) requires no "hidden versions" of the instrument or treatment and rules out interference across units. [<reflink idref="bib78" id="ref65">78</reflink>] and [<reflink idref="bib40" id="ref66">40</reflink>] ([<reflink idref="bib40" id="ref67">40</reflink>]: p. 9–11) elucidate the nuances of SUTVA in selection-on-observables settings, but they are still useful for understanding SUTVA in the IV setting.</p> <p>SUTVA.</p> <p>Graph</p> <p>Positivity requires that, in every stratum of measured confounders, at least some units receive the instrument ([<reflink idref="bib8" id="ref68">8</reflink>]; [<reflink idref="bib35" id="ref69">35</reflink>]). Positivity is only required for confounders that are necessary for (conditional) unconfoundedness to hold and for values of the confounders that actually occur in the population of interest.</p> <p>Positivity.</p> <p>Graph</p> <p>where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">X</mi></mrow><mi>i</mi></msub></math> </ephtml> is a vector of observed confounders for unit <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> .</p> <hd id="AN0190929145-15">Simultaneity and Over-Identification Tests</hd> <p>Substantial minorities of the surveyed papers mentioned two concerns we largely omit from this paper: Simultaneity and over-identification tests.</p> <p>Of the 34 papers we reviewed, 14 cited "simultaneity" or "reverse causation" as part of their justification for using IV. While IV can be used to identify parameters in so-called "simultaneous equation models," such models typically capture states of <emph>equilibrium</emph>. The canonical case is the identification of supply and demand functions, where we expect the quantity supplied and quantity demanded to be in equilibrium at a given price ([<reflink idref="bib82" id="ref70">82</reflink>]; [<reflink idref="bib39" id="ref71">39</reflink>]). To adequately approximate recursive relationships, equilibrium must be achieved very fast relative to the interval at which we measure time averages of the treatment and outcome ([<reflink idref="bib25" id="ref72">25</reflink>]; [<reflink idref="bib66" id="ref73">66</reflink>]). When simultaneity is the concern, we only advise using IV in cases where we have strong theoretical reasons to believe the treatment and outcome achieve equilibrium relatively quickly—a condition we suspect is frequently violated.</p> <p>Of the 34 papers we reviewed, seven report over-identification tests to support the validity of their instruments, as recommended by [<reflink idref="bib10" id="ref74">10</reflink>]. In the case where we have multiple instruments for a given treatment, over-identification tests assess the validity of a subset of the instruments under two strong conditions. First, we must assume that at least some of the instruments are valid. Given how difficult it is to find even a single valid instrument, this assumption often strains credibility. Second, over-identification tests assume constant treatment effects, but in practice it is likely that different instruments have different LATEs ([<reflink idref="bib81" id="ref75">81</reflink>]; [<reflink idref="bib6" id="ref76">6</reflink>]). In Online Appendix C, we review more compelling ways to investigate an instrument's validity.</p> <hd id="AN0190929145-16">Estimation</hd> <p>Having established the assumptions used to identify the LATE, we briefly review estimation with IVs. While we can use many estimators for IV, we focus on the most common one: two-stage least-squares (2SLS) regression. As the name suggests, there are two stages to this procedure. First, we regress the treatment on the instrument using a linear regression. This is called the <emph>first-stage</emph> regression. In the second stage, we regress the outcome on the fitted values of the treatment from the first-stage regression. To build intuition for why this estimator works, notice that some of the variation in the treatment is caused by confounders (confounded) and some is caused by the instrument (unconfounded). When the instrument's signal is strong enough, the first-stage fitted values will largely isolate the unconfounded variation in the treatment. Another way to understand the estimator is that the first-stage regression represents a model of compliance. Since we use the predicted values from this regression, the second-stage provides us with an estimated treatment effect only for those who comply with the instrument. We caution that performing 2SLS manually will produce invalid confidence intervals for the estimated treatment effect, but standard statistical software for 2SLS will report corrected intervals. See "Estimation Bias" for more guidance on confidence intervals.</p> <p>The regression of the outcome directly on the instrument (without the treatment) is called the <emph>reduced-form</emph> or <emph>intention-to-treat</emph> (ITT) regression. Under unconfoundedness, the ITT effect captures the total effect of the instrument on the outcome. When the exclusion restriction holds, this effect will operate entirely through the treatment. The 2SLS treatment coefficient is numerically equivalent to the ITT coefficient on the instrument divided by the first-stage coefficient on the instrument. Table 4 reviews 2SLS estimation terminology.</p> <p>Table 4. Terminology for Conducting IV Analysis with 2SLS Regression.</p> <p>Graph</p> <hd id="AN0190929145-17">Weakness Exacerbates Three Methodological Problems</hd> <p>Understanding the identification assumptions for IV can help us select better instruments. But perfectly valid instruments are rare, and even valid instruments can lead us astray if we fail to handle them with care. In this section, we describe three methodological problems that arise with IV—only one of which is regularly addressed in published IV papers—and provide guidance on managing each one. We summarize the problems and our recommendations in Table 5.</p> <p>Table 5. Three Methodological Problems Exacerbated by Weakness.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Problem</th><th align="left">What is it?</th><th align="left">When is it most severe?</th><th align="left">How do I address it?</th></tr></thead><tbody><tr><td><italic>Identification bias</italic></td><td>An asymptotic bias that can be large under seemingly trivial violations of unconfoundedness, monotonicity, or the exclusion restriction.</td><td>When the first-stage effect is small relative to violations of identification assumptions.</td><td>Conduct a bias analysis (see "Identification Bias"), and run <italic>well-powered</italic> placebo tests if possible.</td></tr><tr><td><italic>Estimation bias</italic></td><td>A finite-sample bias that pulls 2SLS estimates toward the probability limit of the OLS estimator (i.e., the OLS model that uses the same specification apart from the instrumented treatment).</td><td>When the first-stage effect is small relative to the sampling variability of the first-stage estimator.</td><td>Report the robust partial <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math></p>-statistic for the instrument in the first stage. Construct <italic>VtF</italic>, Anderson–Rubin, and bootstrapped confidence sets for the estimated treatment effect. Assess how sensitive results are to the removal of outliers.</td></tr><tr><td><italic>Type-M error</italic></td><td>For a given hypothetical effect size, how exaggerated the estimate has to be to achieve statistical significance.</td><td>When the first-stage effect is small relative to the conditional outcome variance among compliers.</td><td>Report the expected type-M error across a range of plausible effect sizes.</td></tr></tbody></table> </ephtml> </p> <p>2 Each problem requires its own diagnostics. Identification bias can be severe even when the population first-stage <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi></math> </ephtml> -statistic is large, and estimation bias can be severe even when identification assumptions hold exactly. Our survey of IV papers revealed that researchers rarely address identification bias or type-M error.</p> <p>What ties these distinct problems together is that they are all exacerbated by <emph>weak</emph> instruments. Roughly speaking, an instrument is weak when its effect on the treatment—the first-stage effect—is small. But researchers must recognize two subtleties of weak instruments. First, we can distinguish between different types of weakness, and an instrument can simultaneously be strong in one sense and weak in another. So-called "weak-instrument tests," which we review in "Estimation Bias," diagnose only one form of weakness and fail to address two of the three problems we review. Second, weakness is a continuum: a very weak instrument is more worrisome than a slightly weak instrument, all else equal. These nuances will become more apparent by the section's end, and we will specify a more precise meaning of "weak" for each subsection.</p> <p>Each form of weakness complicates comparisons between IV and selection on observables in its own way. We pay special attention to how a small first stage amplifies the bias from seemingly trivial violations of identification assumptions. Violations of IV assumptions often seem negligible, leading many scholars to view IV as more credible than selection on observables. But when we compare the two approaches, we have to consider not just how large the violations might be but also how severely those violations might bias the estimators. With a weak instrument, credibly small violations of IV assumptions can produce more bias than credibly large violations of selection-on-observables assumptions. Understanding how weakness magnifies the bias from such violations—and exacerbates two other methodological problems—can enable researchers to make more prudent choices between the strategies and empower readers to become more critical consumers of IV studies.</p> <hd id="AN0190929145-18">Identification Bias</hd> <p>As we emphasized above, the identification assumptions for IV are strong, and we suspect they often fail to hold perfectly in the real world. We might nonetheless prefer an imperfect IV analysis over an imperfect selection-on-observables design, suspecting the former to be <emph>less</emph> biased than the latter. For example, a researcher might believe that while both the instrument and the treatment are confounded with the outcome, the instrument is "less" confounded than the treatment.</p> <p>Unfortunately, such comparisons are complicated by weak instruments. Bias from violations of unconfoundedness, monotonicity, and the exclusion restriction are inflated by the inverse of the first-stage effect. As a result, seemingly "small" violations of unconfoundedness or the exclusion restriction can generate large biases. We call this <emph>identification bias</emph> because it stems from violations of the identification assumptions. Identification bias leads us to the first definition of "weak instrument": One whose first-stage effect is small relative to seemingly trivial violations of identification assumptions. We include a stylized example of the surprising severity of identification bias in Figure 5.</p> <p>Graph: Figure 5. Linear structural equation model illustrating the potentially extreme sensitivity of IV to unmeasured confounding. For simplicity, assume effects are constant and each variable is continuous. Numbers represent hypothetical marginal effects indicated by the corresponding arrows. Y represents the outcome, D the treatment, Z the instrument, UDY an unmeasured treatment–outcome confounder, and UZY an unmeasured instrument–outcome confounder. OLS refers to an OLS regression of Y on D, while IV refers to a 2SLS regression using Z as an instrument for D. Despite the fact that the effects of UDY (0.3 and 0.4) are more than 4 times larger, respectively, than the effects of UZY (0.07 and 0.09), the IV estimator exhibits larger (approximate) asymptotic bias. In practice, we typically run more complicated models than this and avoid constant-effects assumptions. This is simply a toy example meant to illustrate a general principle ([<reflink idref="bib65" id="ref77">65</reflink>]).</p> <p>Researchers can address identification bias through <emph>bias analysis</emph>—often called "sensitivity analysis" in the causal inference literature. Rather than asking <emph>whether</emph> assumptions are violated, this procedure asks <emph>how severe</emph> the violations would have to be in order to change the conclusions of our study. Of the 34 papers we surveyed, none employed a bias analysis, and 18 (60%) failed to even report the strength of the first-stage effect, which would give the reader a sense of how serious the bias amplification might be. In what follows, we introduce a running example from [<reflink idref="bib17" id="ref78">17</reflink>] and illustrate three forms of bias analysis.</p> <hd id="AN0190929145-19">Running Example for Bias Analysis</hd> <p>[<reflink idref="bib17" id="ref79">17</reflink>] study the effects of sibship size on private school attendance. The treatment is whether a family has three or more children; the outcome is whether a child attends private school; and the instrument is whether the first two children are the same sex. The logic of the instrument is that parents prefer having at least one boy and one girl, so having two children of the same sex might encourage some parents to have a third child.</p> <p>[<reflink idref="bib17" id="ref80">17</reflink>] focus on effects for second-born boys. In doing so, however, they alter the instrument. By analyzing effects only for second-born sons, the instrument becomes the sex of the first-born child since the sex of the second is fixed. Using the sex of the first-born child as an instrument likely violates the exclusion restriction. The sex of the first-born affects a wide range of outcomes, including the speed of the transition to marriage, the probability of divorce, and father's wages and work hours ([<reflink idref="bib53" id="ref81">53</reflink>], [<reflink idref="bib54" id="ref82">54</reflink>]; [<reflink idref="bib20" id="ref83">20</reflink>]). Both marriage and father's wages plausibly affect household finances, which could in turn influence private school attendance.[<reflink idref="bib18" id="ref84">18</reflink>]</p> <p>There is also some evidence that unconfoundedness is violated. External stressors reduce the proportion of male births in a population ([<reflink idref="bib29" id="ref85">29</reflink>]; [<reflink idref="bib13" id="ref86">13</reflink>]; [<reflink idref="bib43" id="ref87">43</reflink>]). If such external stressors are more common among materially disadvantaged families, and material disadvantage in turn affects private school attendance, unconfoundedness may be violated. We illustrate these potential violations of the exclusion restriction and unconfoundedness in the Figure 6.</p> <p>Graph: Figure 6. Potential violations of identification assumptions in Conley and Glauber (2006). The exclusion restriction violation First Born's Sex → Post-Birth Income → Private School Attendance is induced by only examining treatment effects for second-born boys. The path First Born's Sex ← Prenatal Stressors ← Pre-Birth Income → Post-Birth Income → Private School Attendance represents unmeasured instrument–outcome confounding.</p> <p>Graph: Figure 7. Bias analysis plot for Conley and Glauber (2006) replication. The plot shows adjusted treatment effect estimates for each value of θ with 95% Anderson–Rubin confidence intervals. θ represents the effect of a first-born son on the probability of private school attendance that does not occur through sibship size. An effect of −0.15 percentage points would render the treatment effect statistically insignificant at the α=0.05 level. Replication code for conducting the analysis and generating the plot can be found at https://github.com/cmfelton/iv_checklist.</p> <hd id="AN0190929145-20">A Simple Bias Analysis for the Exclusion Restriction</hd> <p>One easy-to-implement bias analysis consists of simply reporting a single additional regression—the ITT regression. Recall that the ITT regression estimates the total effect of the instrument on the outcome. If we assume a linear, constant-effects model, then the ITT estimate tells us what the direct effect of the instrument would have to be to explain away our estimated treatment effect entirely—that is, to reduce our estimated treatment effect to 0. In [<reflink idref="bib17" id="ref88">17</reflink>], the estimated ITT is <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>−</mo></math> </ephtml> 0.00368. This means that a direct effect of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>−</mo></math> </ephtml> 0.00368—less than <emph>four-tenths of a percentage point</emph>—could <emph>entirely</emph> explain away the estimated treatment effect.</p> <p>The literature suggests that a first-born boy should increase, rather than decrease, the probability that a second-born boy attends private school, so it is unlikely that an exclusion restriction violation alone could explain away the findings in [<reflink idref="bib17" id="ref89">17</reflink>]. But the analysis illustrates that seemingly trivial violations of identification assumptions can generate counterintuitively large biases in practical IV applications.</p> <p>While we distrust linear, constant-effects models, using one in this way simplifies the bias analysis. If the analysis is unfavorable under the linear, constant-effects setting, we consider it unlikely to be favorable under a heterogeneous-effects model. The simplification also enables us to perform the analysis without the original data. For example, even when the ITT is not explicitly reported, as in [<reflink idref="bib17" id="ref90">17</reflink>], we can calculate it by multiplying together the first-stage estimate (0.08) and the LATE estimate ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>−</mo><mn>0.046</mn></math> </ephtml> ).</p> <hd id="AN0190929145-21">Bias Analysis Plots for the Exclusion Restriction</hd> <p>[<reflink idref="bib18" id="ref91">18</reflink>] and [<reflink idref="bib80" id="ref92">80</reflink>] introduce bias analysis methods that adjust confidence intervals to incorporate hypothetical violations of the exclusion restriction. We present a modified version of these analyses that shows how our estimated treatment effects would change under a range of potential exclusion restriction violations. The procedure works as follows:</p> <p></p> <ulist> <item> Specify a hypothetical value <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>θ</mi></math> </ephtml> that represents the direct effect of the instrument <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Z</mi></math> </ephtml> on the outcome <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> .</item> <p></p> <item> Calculate an adjusted outcome <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mrow><mi mathvariant="normal">adj</mi></mrow></msub><mo>=</mo><mi>Y</mi><mo>−</mo><mi>θ</mi><mi>Z</mi></math> </ephtml> .</item> <p></p> <item> Estimate treatment effects with 2SLS using <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mrow><mi mathvariant="normal">adj</mi></mrow></msub></math> </ephtml> instead of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> , reporting 95% confidence intervals.</item> <p></p> <item> Repeat Steps 1–3 for a range of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>θ</mi></math> </ephtml> values, making sure to include values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>θ</mi></math> </ephtml> sufficient to explain away the point estimate entirely.</item> <p></p> <item> Plot the results.</item> </ulist> <p>We include a plot with the results from this sensitivity analysis below.[<reflink idref="bib19" id="ref93">19</reflink>] Following [<reflink idref="bib80" id="ref94">80</reflink>], we use Anderson–Rubin confidence intervals, although we do not standardize <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>θ</mi></math> </ephtml> as they do. Our results show that if a first-born son reduced the probability of private school attendance by 0.15 percentage points, then 95% confidence intervals on the estimated treatment effect would overlap with 0. We include replication R code for this analysis and plot at https://github.com/cmfelton/iv_checklist.</p> <hd id="AN0190929145-22">Cinelli and Hazlett's Bias Analysis</hd> <p>[<reflink idref="bib15" id="ref95">15</reflink>] offer a useful bias analysis for unmeasured confounding in 2SLS models. One key difference in interpretation is that sensitivity is assessed in terms of predictive power rather than causal effects. This method assesses sensitivity in terms of how much of the residual variation in the instrument or outcome can be <emph>predicted</emph> by an unmeasured confounder. In particular, their <emph>robustness value</emph> indicates how strongly, in terms of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>R</mi><mn>2</mn></msup></math> </ephtml> , the unmeasured confounder must predict residual variation in both the instrument and the outcome in order to reduce the estimated ITT to 0.</p> <p>To perform this analysis, we use the authors' sensemakr package for R to calculate two robustness values. For Conley and Glauber's ([<reflink idref="bib17" id="ref96">17</reflink>]) results, we find that if an unmeasured confounder had a partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>R</mi><mn>2</mn></msup></math> </ephtml> of 0.0067 on both the instrument and outcome, it would reduce the effect estimate to 0. An unmeasured confounder need only predict 0.67% of the residual variation in the instrument and outcome in order to reduce the estimated treatment effect to 0. This finding is not damning: Figure 8 shows that the partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>R</mi><mn>2</mn></msup></math> </ephtml> values for average parental education—an observed confounder—are even smaller than 0.0067. The plot shows that if the partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>R</mi><mn>2</mn></msup></math> </ephtml> values on the unmeasured confounder were similar in magnitude to the partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>R</mi><mn>2</mn></msup></math> </ephtml> values on parental education, the adjusted [<reflink idref="bib17" id="ref97">17</reflink>] estimate would remain virtually unchanged. We include replication code for this analysis at https://github.com/cmfelton/iv%5fchecklist. See [<reflink idref="bib15" id="ref98">15</reflink>] for additional sensitivity statistics.</p> <p>Graph: Figure 8. Contour plot for Cinelli and Hazlett's (2022) bias analysis. A single contour represents pairs of R2 values that would reduce the point estimate to the specified value. For instance, the red, dashed contour represents pairs of R2 values that would reduce the estimated treatment effect to 0; the contour immediately below it represents R2 pairs that would reduce the estimate to −0.001. The benchmark-adjusted estimate uses R2 values estimated from a covariate included in the model. In this case, we used the parents' average education. The benchmark-adjusted estimate indicates that an unmeasured confounder as strongly associated with the instrument and outcome as parental average education would change the estimated treatment effect very little. Researchers can use such benchmarks to reason about how much unmeasured confounding biases their effect estimates. The plot was made using the sensemakr package in R, and replication code can be found at https://github.com/cmfelton/iv_checklist.</p> <p>The bias analyses presented above demonstrate how a small ITT effect—a common occurrence with weak instruments—can turn seemingly minor violations of identification assumptions into large biases. Because the instrument has such a small effect on the treatment in [<reflink idref="bib17" id="ref99">17</reflink>], the ITT is also small, and only tiny violations of unconfoundedness or the exclusion restriction are necessary to explain away the reported treatment effect.</p> <hd id="AN0190929145-23">Estimation Bias</hd> <p>We now turn to a second type of weak instrument—one whose first-stage effect is small relative to the sampling variability of the first-stage estimator. This kind of weakness aggravates what we call <emph>estimation bias</emph>. In finite samples, the 2SLS estimator is biased toward the OLS regression of the outcome on the treatment (including any covariates used in the 2SLS model).[<reflink idref="bib20" id="ref100">20</reflink>] To see why, recall that the second stage in 2SLS is an OLS regression that replaces <emph>observed</emph> values of the treatment with <emph>predicted</emph> values of the treatment. When the first-stage effect is small relative to sampling variability, the first-stage regression will do a poor job of predicting the true "instrumented" treatment values—i.e., the predicted values we would obtain if we knew the true first-stage effect. The first-stage regression will overfit the data, and first-stage estimates will reflect variation in the treatment that is not actually caused by the instrument. The smaller the effect of the instrument, the more these predicted values simply reflect this non-instrumented treatment variation—the same variation OLS uses to estimate treatment effects. The weaker the first stage, then, the closer the 2SLS sampling distribution moves toward the probability limit of OLS.</p> <p>We now see how an instrument can be weak in one sense but strong in another. For instance, the first stage in [<reflink idref="bib17" id="ref101">17</reflink>] is small relative to tiny violations of identification assumptions, producing potentially large identification bias. But because the authors use close to 200,000 observations, the first stage is large relative to the first-stage sampling variability. Consequently, estimation bias is likely small.</p> <p>Estimation bias is always present—even when the identification assumptions hold perfectly—but it becomes negligible when the first stage is large enough relative to the sampling variability of the first-stage estimator. More precisely, 2SLS is approximately median-unbiased when the first stage is large enough relative to the first-stage sampling variability ([<reflink idref="bib6" id="ref102">6</reflink>]).</p> <p>How do we know when the first stage is large enough? We can gain some insight by examining the partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic on the instrument in the first-stage regression. In particular, when <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> is larger than some <emph>critical value</emph>, the bias of 2SLS should be relatively small (but not necessarily zero). A conventional rule of thumb uses 10 as this critical value, but the rule is most justified in settings with homoskedastic errors and multiple instruments ([<reflink idref="bib74" id="ref103">74</reflink>]). More recently, [<reflink idref="bib62" id="ref104">62</reflink>] suggest an <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> of at least 23.11 in the single-instrument setting when using a robust <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi></math> </ephtml> -test, which allows for heteroskedastic errors.[<reflink idref="bib21" id="ref105">21</reflink>] They also provide a set of critical values for their effective <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi></math> </ephtml> -test that can be used in the multi-instrument setting. Of the 34 sociology papers we surveyed, only 18 (53%) reported results from any <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi></math> </ephtml> -test. Furthermore, we identified only two papers that explicitly reported an <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic robust to heteroskedasticity or clustering. For context, 26 papers used either panel data or clustered data, indicating that reporting robust partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistics should be far more common. We recommend sociologists regularly report robust partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistics when using a single instrument and the [<reflink idref="bib62" id="ref106">62</reflink>] effective <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> when using multiple instruments ([<reflink idref="bib3" id="ref107">3</reflink>]).</p> <p>Estimation bias can be particularly severe when using multiple weak instruments. For instance, [<reflink idref="bib11" id="ref108">11</reflink>] find strong evidence of estimation bias in a study with over 300,000 observations that uses 180 instruments. Similarly, [<reflink idref="bib30" id="ref109">30</reflink>] report partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistics well below 10 when using judge–covariate interactions as instruments in a sample with over 100,000 observations. Notably, estimates from this specification are very close to the OLS estimates, which is consistent with the direction of estimation bias. Furthermore, the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> thresholds for inference grow rapidly with the number of instruments, so the confidence intervals reported in [<reflink idref="bib30" id="ref110">30</reflink>] may be too narrow ([<reflink idref="bib74" id="ref111">74</reflink>]).[<reflink idref="bib22" id="ref112">22</reflink>]</p> <p>Even when the first-stage <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> suggests non-trivial estimation bias, valid inference is possible. Anderson–Rubin (AR) confidence sets can provide valid coverage even when the first-stage effect is very small relative to the first-stage sampling variability, as they make no assumptions about the strength of the first stage ([<reflink idref="bib2" id="ref113">2</reflink>]). (They are called <emph>sets</emph> rather than intervals because they can consist of two disconnected intervals.) [<reflink idref="bib51" id="ref114">51</reflink>] introduce <emph>VtF</emph> confidence intervals for 2SLS, which are also robust to weak instruments but typically much shorter than AR intervals. These intervals modify standard <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>t</mi></math> </ephtml> -ratio-based inference using the first-stage <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic and <emph>v</emph>ariance information.</p> <p>Importantly, both <emph>VtF</emph> intervals and AR confidence sets rely on asymptotic approximations that may break down in finite samples ([<reflink idref="bib3" id="ref115">3</reflink>]). [<reflink idref="bib83" id="ref116">83</reflink>] provides evidence that these approximations can be poor in practice. Re-analyzing 2SLS regressions from 30 economics papers, he finds that removing just a single observation or cluster of observations can substantially alter results, suggesting typical sample sizes may be too small for the asymptotic approximation to be useful ([<reflink idref="bib83" id="ref117">83</reflink>]; [<reflink idref="bib3" id="ref118">3</reflink>]). Simulations by [<reflink idref="bib83" id="ref119">83</reflink>] suggest the bootstrap may work better in settings with highly influential observations or clusters, but [<reflink idref="bib3" id="ref120">3</reflink>] note that it lacks theoretical guarantees in the weak-instrument setting. In large samples without clustering or dependence between units, we recommend reporting <emph>VtF</emph> or AR confidence sets. In small samples or samples with clustering, we recommend reporting bootstrapped confidence intervals in addition to <emph>VtF</emph> or AR confidence sets.[<reflink idref="bib23" id="ref121">23</reflink>] We also encourage researchers to assess how sensitive results are to the removal of single observations or clusters of observations. Of the 34 papers we reviewed, none reported AR or bootstrapped confidence sets. Online Appendix E provides names of software packages in both R and Stata for conducting the diagnostics we recommend.</p> <hd id="AN0190929145-24">Type-M Error</hd> <p>Finally, the first-stage effect might be small relative to the variance of the outcome, increasing the variance of the IV estimator.[<reflink idref="bib24" id="ref122">24</reflink>] Instruments that are weak in this way can increase the expected type-M error, where <emph>M</emph> stands for the <emph>m</emph>agnitude of effect estimates. When treatment effects are small, using high-variance estimators can systematically exaggerate the magnitude of effect sizes that make it into published work ([<reflink idref="bib26" id="ref123">26</reflink>]). The reason is that achieving statistical "significance" with a high-variance estimator requires an unusually large effect estimate, and statistically "<emph>in</emph>significant" results often go unreported. As a result, published findings may systematically overestimate the magnitude of causal effects.</p> <p>Reviews of published IV findings are consistent with systematic exaggeration of effect sizes. In political science, [<reflink idref="bib49" id="ref124">49</reflink>] find that IV estimates are almost always larger in magnitude than OLS-based selection-on-observables estimates. This is concerning: we typically resort to IV because we suspect that selection-on-observables estimates are biased away from zero and expect IV to be less biased. But if IV really were less biased—and if selection on observables really were biased away from zero—we should expect IV to produce <emph>smaller</emph> estimates than OLS, at least on average.[<reflink idref="bib25" id="ref125">25</reflink>] Alarmingly, they also find that in 32.8% of papers, the IV estimates are at least <emph>five times larger</emph> than OLS estimates. While the authors primarily attribute this finding to assumption violations, it is also consistent with widespread type-M errors. Similarly, in a review of finance studies using IV, [<reflink idref="bib44" id="ref126">44</reflink>] finds that IV estimates are, on average, 9 times larger in magnitude than their corresponding OLS estimates in cases where we would expect OLS to be biased upward in magnitude.</p> <p>Of the 17 papers in our sample that report both OLS and 2SLS results, the 2SLS estimates were larger in 15 (88.2%).[<reflink idref="bib26" id="ref127">26</reflink>] In 11 (64.7%) of the papers, 2SLS estimates were more than twice as large, and in five (29.4%), they were more than five times larger. We emphasize that this is not evidence of wrongdoing on the part of researchers. Rather, it is evidence that publishing only statistically "significant" findings can lead to systematically exaggerated effect sizes when using high-variance estimators like 2SLS.</p> <p>Type-M error is difficult to address because it involves a complex interplay between the variance of the estimator and the practices of scientific publishing. Following [<reflink idref="bib26" id="ref128">26</reflink>], we can calculate expected type-M error for IV estimates if we are willing to specify a hypothetical "true" effect size based on prior research. Alternatively, researchers can calculate and plot the expected type-M error across a range of plausible effect sizes. Since the expected type-M error is a function of the estimator's variance, we also recommend plotting the IV estimate with confidence intervals, which conveys uncertainty more straightforwardly than printing an estimated standard error in parentheses.</p> <hd id="AN0190929145-25">Weakness Complicates the Comparison of IV and Selection on Observables</hd> <p>Most causal work in sociology follows a selection-on-observables design, whether via regression, matching, weighting or some other estimation strategy. Authors turn to IV because the identification assumptions often seem more plausible than assuming the treatment is conditionally unconfounded. But comparisons between these approaches are more complicated than they appear. As we have seen, weak instruments create challenges for reasoning about the credibility of the research design, estimating treatment effects, and interpreting reported results in the literature. When instruments are even modestly weak, in one sense or another, we must handle IV with care. We recommend researchers err on the side of caution by including diagnostics for all three potential problems that IV faces.</p> <hd id="AN0190929145-26">A Checklist for Conducting and Reporting IV Analysis</hd> <p>Taking inspiration from [<reflink idref="bib72" id="ref129">72</reflink>] and [<reflink idref="bib49" id="ref130">49</reflink>], we have compiled a checklist summarizing our recommendations to aid researchers in conducting and reporting IV analysis. Like [<reflink idref="bib72" id="ref131">72</reflink>], we emphasize the importance of clarifying the estimand and defending assumptions. We particularly underscore the value of communicating those assumptions to readers in non-technical language rooted in the substantive problem at hand. Like [<reflink idref="bib49" id="ref132">49</reflink>], we encourage using the bootstrap and AR confidence intervals for inference. They also encourage routine use of bias analysis, along with placebo tests to set the bias analysis parameters. While this is highly effective when a placebo sub-population is available, we expect such populations to be difficult to locate in many sociological settings (but see Online Appendix C for more on placebo tests and balance checks). For this reason, we place more emphasis on simpler forms of bias analysis.</p> <p>Our checklist focuses on identification of the LATE, although most items still apply if the researcher is willing to invoke alternative assumptions to target the ATE or other causal estimands. We show an abridged version of the checklist in Figure 9.</p> <hd id="AN0190929145-27">Item 1. State the Theoretical Estimand</hd> <p> <emph>1a. State the theoretical estimand: A LATE ("Identification and Estimation of Treatment Effects with IV") or weighted average of LATEs (Online Appendix F).</emph> Following [<reflink idref="bib52" id="ref133">52</reflink>], we urge researchers to explicitly state what they term the <emph>theoretical estimand</emph> and clarify how it connects to the argument of the paper. Without imposing strong restrictions on treatment effect heterogeneity, IV analysis identifies the average treatment effect only for <emph>compliers</emph>—those induced into treatment by the instrument. This can be phrased substantively—for instance, [<reflink idref="bib46" id="ref134">46</reflink>] clearly states that he is estimating a treatment effect only "for parolees who would not have moved had it not been for Hurricane Katrina" (<reflink idref="bib495" id="ref135">495</reflink>). We also recommend discussing what can and cannot be learned from such a subpopulation. When non-dichotomous treatments or instruments are used, or when covariates are included in the model, 2SLS will identify a weighted average of LATEs, a nuance we discuss in Online Appendix F.</p> <p>Graph: Figure 9. Checklist for conducting IV analysis. We expand on each item in the main text.</p> <hd id="AN0190929145-28">Item 2. Explain Assumptions</hd> <p> <emph>2a. State the identification assumptions: Unconfoundedness, exclusion restriction, monotonicity, instrument relevance, SUTVA, and positivity ("Identification and Estimation of Treatment Effects with IV").</emph> Researchers should state these assumptions in substantive terms for the reader. Consider the unconfoundedness assumption in the context of [<reflink idref="bib46" id="ref136">46</reflink>]. This assumption requires that being released after Hurricane Katrina shares no unmeasured common causes with being re-arrested. Compare this explanation with the more opaque statement that the instrument must be uncorrelated with the error term in some regression model, a common description in the applied literature.</p> <p> <emph>2b. Discuss plausible violations of any identification assumptions.</emph> In practice, it is rare that no plausible violations exist. Indeed, even the Vietnam draft lottery instrument, which helped popularize IV in economics and launch the "credibility revolution," likely violates the exclusion restriction for many outcomes. The instrument is whether a person received a high or low lottery number for the Vietnam draft, and the treatment is whether the person served in the military. But the lottery numbers caused some people to change their educational plans in order to avoid the draft, and education affects a wide range of outcomes ([<reflink idref="bib5" id="ref137">5</reflink>]). Help the reader understand that the instrument can have no "direct" effect on the outcome in the sense that it can cause the outcome <emph>only through the treatment</emph> ("What 'No Direct Effect' Means in the Context of the Exclusion Restriction"). Weather-based instruments, large-scale disasters, and historical instruments may need special care in this regard. Remember that while conditioning on post-instrument covariates may block exclusion restriction violations, they can induce violations of unconfoundedness ("Why Blocking Exclusion Restriction Violations Can Induce Unconfoundedness Violations"). Finally, note that coarsely measured treatments often violate the exclusion restriction ("Why Coarse or Coarsened Treatments Induce Exclusion Restriction Violations").</p> <p> <emph>2c. If using a <emph>proxy</emph> instrument, be sure to state the <emph>causal</emph> instrument (Online Appendix B). The estimand and identification assumptions are defined with respect to the causal instrument—<emph>not</emph> the proxy instrument.</emph> A proxy instrument is one that does not cause the treatment, but rather shares an unmeasured common cause with the treatment. The unmeasured common cause is the causal instrument. Crucially, identification assumptions must hold with respect to the causal instrument. Furthermore, compliers are those who are induced into treatment by the causal instrument—not the proxy instrument, which exerts no causal influence on the treatment. We discuss examples in Online Appendix B.</p> <hd id="AN0190929145-29">Item 3. Report Results with Weak-Instrument-Robust Confidence Intervals</hd> <p> <emph>3a. Report the estimated first-stage effect, "reduced-form" or ITT effect, and treatment effect estimates with confidence intervals ("Weakness Exacerbates Three Methodological Problems").</emph> The estimated first-stage effect represents how strongly the instrument affects the treatment. The smaller it is, the greater the potential for both identification bias and estimation bias. Under certain assumptions, the ITT effect estimate represents how strong the direct effect of the instrument would have to be to explain away the effect estimate ("Identification Bias").</p> <p> <emph>3b. Report Anderson–Rubin, VtF, and bootstrapped confidence sets for the treatment effect ("Estimation Bias").</emph> In the case with a single instrument and treatment, Anderson–Rubin and <emph>VtF</emph> confidence sets will be robust to weak instruments under the assumptions described in [<reflink idref="bib3" id="ref138">3</reflink>] and [<reflink idref="bib51" id="ref139">51</reflink>], respectively. [<reflink idref="bib83" id="ref140">83</reflink>] argues that these asymptotic approximations may be poor in practice, particularly in settings with clustering. His simulations suggest that bootstrapped confidence intervals nonetheless work well in such settings. In the multi-instrument setting, researchers can use Moreira's (2003) conditional likelihood ratio test, although this is not robust to heteroskedasticity.</p> <hd id="AN0190929145-30">Item 4. Assess Bias and Type-M Error</hd> <p> <emph>4a. Conduct bias analysis for violations of key assumptions ("Identification Bias").</emph> When using IV in observational settings, some of our identification assumptions will likely be violated. Bias analysis allows us to assess <emph>how badly</emph> our assumptions need to be violated in order to change the conclusions of our study. We describe several different bias analyses in "Identification Bias" and provide example code for conducting these analyses in Online Appendix E.</p> <p> <emph>4b. Report the results of placebo tests and scaled balance checks if possible.</emph> Placebo tests can sometimes provide evidence in support of the exclusion restriction and unconfoundedness assumption ([<reflink idref="bib23" id="ref141">23</reflink>]). Balance checks, when scaled by the inverse of the first-stage, can provide evidence in favor of unconfoundedness ([<reflink idref="bib42" id="ref142">42</reflink>]). We describe these procedures and their limitations in Online Appendix C.</p> <p>4c. Report a heteroskedasticity-robust partial <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic for the instrument in the first-stage regression ("Estimation Bias"). In the multi-instrument setting, researchers can use the [<reflink idref="bib62" id="ref143">62</reflink>] effective <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi></math> </ephtml> -test. An <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow><mo>></mo><mn>23.11</mn></math> </ephtml> will indicate small relative bias of 2SLS in the single-instrument setting, but see [<reflink idref="bib62" id="ref144">62</reflink>] for different sets of critical values. With many instruments, the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover><mi>F</mi><mo stretchy="false">^</mo></mover></mrow></math> </ephtml> -statistic should be much larger to ensure small undercoverage of conventional confidence intervals ([<reflink idref="bib74" id="ref145">74</reflink>]).</p> <p> <emph>4d. Report the expected type-M error ("Type-M Error").</emph> The expected type-M error quantifies how exaggerated an effect estimate would have to be in order to achieve statistical significance. This quantity depends on a hypothetical effect size, which can be provided by the existing literature, but researchers may prefer to specify a broad range of effect sizes and show the expected type-M error at each value.</p> <hd id="AN0190929145-31">Conclusion</hd> <p>In this paper, we provided an introduction to IV identification assumptions and outlined various tools for improving empirical practice. We documented ways to improve the communication of these assumptions to readers. We also identified underutilized diagnostics and approaches to constructing confidence intervals. We compiled our recommendations into a simple checklist.</p> <p>Researchers typically employ IV to make causal claims when they suspect that a selection-on-observables strategy will be biased due to unmeasured confounding. Our discussion highlighted potential sources of bias in IV, focusing on sensitivity to even minor violations in the weak-instrument setting. We hope this helps analysts and researchers improve their intuitions about which estimators may be more biased. More generally, we hope that researchers spend less time thinking about <emph>whether</emph> causal estimates are biased and more time reasoning about <emph>how severe</emph> the bias might be. Bias analysis, balance checks, and placebo tests are some of the tools we can use to help assess this bias—for both IV and selection on observables ([<reflink idref="bib42" id="ref146">42</reflink>]; [<reflink idref="bib67" id="ref147">67</reflink>]; [<reflink idref="bib23" id="ref148">23</reflink>]).</p> <p>Triangulating different pieces of evidence can also help build support for a causal claim, even when each individual piece is imperfect ([<reflink idref="bib16" id="ref149">16</reflink>]; [<reflink idref="bib56" id="ref150">56</reflink>]; [<reflink idref="bib45" id="ref151">45</reflink>]; [<reflink idref="bib73" id="ref152">73</reflink>]). In practice, researchers often use multiple identification strategies and consider the combination of results together. We have focused on interpretation of IV in isolation, but in our sample of papers, we found that IV results were often presented alongside selection-on-observables results. Thinking about how to combine different results effectively is an important area for future work.</p> <p>Sociologists have understandably been attracted to IV—it is a powerful strategy that can accomplish the miraculous task of causal identification in the presence of unmeasured confounding. We simply emphasize here that it is also a fragile technique that we must handle with care.</p> <hd id="AN0190929145-32">Supplemental Material</hd> <p>Graph: Supplemental material, sj-pdf-1-smr-10.1177_00491241241235900 for Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables by Chris Felton and Brandon M. Stewart in Sociological Methods & Research</p> <hd id="AN0190929145-33">Acknowledgments</hd> <p>For helpful discussions and feedback relevant to this project, we thank (in reverse-alphabetical order) Simone Zhang, Yiqing Xu, Patrick Sharkey, Hannah Postel, Ian Lundberg, Angela Li, Felix Elwert, Katie Donnelly Moran, Dalton Conley, members of the Stewart Lab, and three anonymous reviewers.</p> <ref id="AN0190929145-34"> <title> References </title> <blist> <bibl id="bib1" idref="ref47" type="bt">1</bibl> <bibtext> Ahlfeldt Gabriel M, Pietrostefani Elisabetta. 2019. " The Economic Effects of Density: A Synthesis." Journal of Urban Economics. 111:93-107.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref113" type="bt">2</bibl> <bibtext> Anderson Theodore W, Rubin Herman. 1949. " Estimation of the Parameters of a Single Equation in a Complete System of Stochastic Equations." The Annals of Mathematical Statistics. 20(1): 46-63.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref107" type="bt">3</bibl> <bibtext> Andrews Isaiah, Stock James H, Sun Liyang. 2019. " Weak Instruments in Instrumental Variables Regression: Theory and Practice." Annual Review of Economics. 11: 727-53.</bibtext> </blist> <blist> <bibl id="bib4" type="bt">4</bibl> <bibtext> Angrist Joshua D, Imbens Guido W. 1999. " Comment on James J. Heckman, 'Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations'." Journal of Human Resources 34(4):823-7.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref55" type="bt">5</bibl> <bibtext> Angrist Joshua D, Imbens Guido W, Rubin Donald B. 1996. " Identification of Causal Effects Using Instrumental Variables." Journal of the American Statistical Association. 91(434): 444-55.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref7" type="bt">6</bibl> <bibtext> Angrist Joshua D, Pischke Jörn-Steffen. 2008. Mostly Harmless Econometrics. Princeton, NJ: Princeton university press.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref20" type="bt">7</bibl> <bibtext> Angrist Joshua, Kolesár Michal. 2023. " One Instrument to Rule them All: The Bias and Coverage of Just-ID IV." Journal of Econometrics 240(2).</bibtext> </blist> <blist> <bibl id="bib8" idref="ref25" type="bt">8</bibl> <bibtext> Aronow P.M., Miller Benjamin T. 2019. Foundations of Agnostic Statistics. New York, NY: Cambridge University Press.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref8" type="bt">9</bibl> <bibtext> Baiocchi Michael, Cheng Jing, Small Dylan S. 2014. " Instrumental Variable Methods for Causal Inference." Statistics in Medicine. 33(13): 2297-340.</bibtext> </blist> <blist> <bibtext> Bollen Kenneth A. 2012. " Instrumental Variables in Sociology and the Social Sciences." Annual Review of Sociology. 38: 37-72.</bibtext> </blist> <blist> <bibtext> Bound John, Jaeger David A, Baker Regina M. 1995. " Problems with Instrumental Variables Estimation when the Correlation Between the Instruments and the Endogenous Explanatory Variable is Weak." Journal of the American Statistical Association. 90(430): 443-50.</bibtext> </blist> <blist> <bibtext> Card David. (1995) "Using Geographic Variation in College Proximity to Estimate the Return to Schooling." In Aspects of Labor Market Behaviour: Essays in Honour of John Vanderkamp, ed. EK Christofides, LN abd Grant and R Swidinsky. Toronto: University of Toronto Press.</bibtext> </blist> <blist> <bibtext> Catalano Ralph, Bruckner Tim, Gould Jeff, Eskenazi Brenda, Anderson Elizabeth. 2005. " Sex Ratios in California Following the Terrorist Attacks of September 11, 2001." Human Reproduction. 20(5): 1221-7.</bibtext> </blist> <blist> <bibtext> Chernozhukov Victor, Hansen Christian. 2005. " An IV Model of Quantile Treatment Effects." Econometrica. 73(1): 245-61.</bibtext> </blist> <blist> <bibtext> Cinelli Carlos, Hazlett Chad. 2022. "An omitted variable bias framework for sensitivity analysis of instrumental variables." Available at SSRN 4217915.</bibtext> </blist> <blist> <bibtext> Cochran William G, Chambers S Paul. 1965. " The Planning of Observational Studies of Human Populations." Journal of the Royal Statistical Society. Series A (General). 128(2): 234-66.</bibtext> </blist> <blist> <bibtext> Conley Dalton, Glauber Rebecca. 2006. " Parental Educational Investment and Children's Academic Risk Estimates of the Impact of Sibship Size and Birth Order From Exogenous Variation in Fertility." Journal of Human Resources. 41(4): 722-37.</bibtext> </blist> <blist> <bibtext> Conley Timothy G, Hansen Christian B, Rossi Peter E. 2012. " Plausibly Exogenous." Review of Economics and Statistics. 94(1): 260-72.</bibtext> </blist> <blist> <bibtext> Cui Yifan, Tchetgen Eric Tchetgen. 2021. " A Semiparametric Instrumental Variable Approach to Optimal Treatment Regimes Under Endogeneity." Journal of the American Statistical Association. 116(533): 162-73.</bibtext> </blist> <blist> <bibtext> Dahl Gordon B, Moretti Enrico. 2008. " The Demand for Sons." The Review of Economic Studies. 75(4): 1085-120.</bibtext> </blist> <blist> <bibtext> de Vaan Mathijs, Stuart Toby. 2019. " Does Intra-household Contagion Cause An Increase in Prescription Opioid Use? " American Sociological Review. 84(4): 577-608.</bibtext> </blist> <blist> <bibtext> Deaton Angus. (2009) "Instruments of Development: Randomisation in the Tropics, and the Search for the Elusive Keys to Economic Development." In Proceedings of the British Academy. Vol. 162 pp. 123–160.</bibtext> </blist> <blist> <bibtext> Eggers Andrew C, Tuñón Guadalupe, Dafoe Allan. 2023. " Placebo Tests for Causal Inference." American Journal of Political Science forthcoming.</bibtext> </blist> <blist> <bibtext> Elwert Felix, Winship Christopher. 2014. " Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable." Annual Review of Sociology. 40: 31-53.</bibtext> </blist> <blist> <bibtext> Fisher Franklin M. 1970. " A Correspondence Principle for Simultaneous Equation Models." Econometrica: Journal of the Econometric Society 38(1):73-92.</bibtext> </blist> <blist> <bibtext> Gelman Andrew, Carlin John. 2014. " Beyond Power Calculations: Assessing Type S (sign) and Type M (magnitude) Errors." Perspectives on Psychological Science. 9(6): 641-51.</bibtext> </blist> <blist> <bibtext> Glynn Adam N, Rueda Miguel R, Schuessler Julian. In Press. " Post-Instrument Bias in Linear Models." Sociological Methods & Research.</bibtext> </blist> <blist> <bibtext> Greene William H. 2008. Econometric Analysis. 6 ed. Upper Saddle River, NJ: Pearson Education.</bibtext> </blist> <blist> <bibtext> Hansen Dorthe, Møller Henrik, Olsen Jørn. 1999. " Severe Periconceptional Life Events and the Sex Ratio in Offspring: Follow Up Study Based on Five National Registers." BMJ (Clinical research ed.). 319(7209): 548-9.</bibtext> </blist> <blist> <bibtext> Harding David J, Morenoff Jeffrey D, Nguyen Anh P, Bushway Shawn D. 2018. " Imprisonment and Labor Market Outcomes: Evidence From a Natural Experiment." American Journal of Sociology. 124(1): 49-110.</bibtext> </blist> <blist> <bibtext> Hartwig Fernando Pires, Wang Linbo, Smith George Davey, Davies Neil Martin. 2022. "Homogeneity in the instrument-treatment association is not sufficient for the Wald estimand to equal the average causal effect when the exposure is continuous." arXiv preprint arXiv:2107.01070.</bibtext> </blist> <blist> <bibtext> Heckman James. 1997. " Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations." Journal of Human Resources 32(3):441-62.</bibtext> </blist> <blist> <bibtext> Heckman James J, Vytlacil Edward. 2001. " Policy-relevant Treatment Effects." American Economic Review. 91(2): 107-11.</bibtext> </blist> <blist> <bibtext> Heckman James J, Urzua Sergio. 2010. " Comparing IV with Structural Models: What Simple IV Can and Cannot Identify." Journal of Econometrics. 156(1): 27-37.</bibtext> </blist> <blist> <bibtext> Hernán Miguel A, Robins James M. (2023) "Causal Inference: What If.".</bibtext> </blist> <blist> <bibtext> Hipp John R. 2007. " Block, Tract, and Levels of Aggregation: Neighborhood Structure and Crime and Disorder As a Case in Point." American Sociological Review. 72(5): 659-80.</bibtext> </blist> <blist> <bibtext> Holland Paul W. 1986. " Statistics and Causal Inference." Journal of the American Statistical Association. 81(396): 945-60.</bibtext> </blist> <blist> <bibtext> Imbens Guido W. 2010. " Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)." Journal of Economic Literature. 48(2): 399-423.</bibtext> </blist> <blist> <bibtext> Imbens Guido W. 2014. " Instrumental Variables: An Econometrician's Perspective." Statistical Science. 29(3): 323-58.</bibtext> </blist> <blist> <bibtext> Imbens Guido W, Rubin Donald B. 2015. Causal Inference in Statistics, Social, and Biomedical Sciences. New York, NY: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Imbens Guido W, Angrist Joshua D. 1994. " Identification and Estimation of Local Average Treatment Effects." Econometrica: Journal of the Econometric Society 62(2):467-75.</bibtext> </blist> <blist> <bibtext> Jackson John W, Swanson Sonja A. 2015. " Toward a Clearer Portrayal of Confounding Bias in Instrumental Variable Applications." Epidemiology (Cambridge, Mass.). 26(4): 498.</bibtext> </blist> <blist> <bibtext> James William H. 2009. " The Variations of Human Sex Ratio At Birth During and After Wars, and Their Potential Explanations." Journal of Theoretical Biology. 257(1): 116-23.</bibtext> </blist> <blist> <bibtext> Jiang Wei. 2017. " Have Instrumental Variables Brought Us Closer to the Truth." The Review of Corporate Finance Studies. 6(2): 127-40.</bibtext> </blist> <blist> <bibtext> Karmakar Bikram, French Benjamin, Small Dylan. 2019. " Integrating the Evidence From Evidence Factors in Observational Studies." Biometrika. 106(2): 353-67.</bibtext> </blist> <blist> <bibtext> Kirk David S. 2009. " A Natural Experiment on Residential Change and Recidivism: Lessons From Hurricane Katrina." American Sociological Review. 74(3): 484-505.</bibtext> </blist> <blist> <bibtext> Kolesár Michal. 2013. "Estimation in an instrumental variables model with treatment effect heterogeneity." Unpublished Manuscript.</bibtext> </blist> <blist> <bibtext> Laidley Thomas, Conley Dalton. 2018. " The Effects of Active and Passive Leisure on Cognition in Children: Evidence From Exogenous Variation in Weather." Social Forces. 97(1): 129-56.</bibtext> </blist> <blist> <bibtext> Lal Apoorva, Lockhart Mackenzie William, Xu Yiqing, Zu Ziwen. (2023) How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies. Working paper.</bibtext> </blist> <blist> <bibtext> Lee David S, McCrary Justin, Moreira Marcelo J, Porter Jack. 2022. " Valid T-ratio Inference for IV." American Economic Review. 112(10): 3260-90.</bibtext> </blist> <blist> <bibtext> Lee David S, McCrary Justin, Moreira Marcelo J, Porter Jack R, Yap Luther. 2023. What to do when you can't use '1.96' Confidence Intervals for IV. Technical report National Bureau of Economic Research.</bibtext> </blist> <blist> <bibtext> Lundberg Ian, Johnson Rebecca, Stewart Brandon M. 2021. " What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory." American Sociological Review. 86(3): 532-65.</bibtext> </blist> <blist> <bibtext> Lundberg Shelly, Rose Elaina. 2002. " The Effects of Sons and Daughters on Men's Labor Supply and Wages." Review of Economics and Statistics. 84(2): 251-68.</bibtext> </blist> <blist> <bibtext> Lundberg Shelly, Rose Elaina. 2003. " Child Gender and the Transition to Marriage." Demography. 40(2): 333-49.</bibtext> </blist> <blist> <bibtext> Marshall John. 2016. " Coarsening Bias: How Coarse Treatment Measurement Upwardly Biases Instrumental Variable Estimates." Political Analysis. 24(2): 157-71.</bibtext> </blist> <blist> <bibtext> Martin John Levi. 2017. Thinking Through Methods: A Social Science Primer. Chicago:University of Chicago Press.</bibtext> </blist> <blist> <bibtext> McClellan Mark, McNeil Barbara J, Newhouse Joseph P. 1994. " Does More Intensive Treatment of Acute Myocardial Infarction in the Elderly Reduce Mortality?: Analysis Using Instrumental Variables." JAMA. 272(11): 859-66.</bibtext> </blist> <blist> <bibtext> Mellon Jonathan. (2023) "Rain, Rain, Go Away: 195 Potential Exclusion-Restriction Violations for Studies Using Weather as an Instrumental Variable." Working paper.</bibtext> </blist> <blist> <bibtext> Morgan Stephen L, Winship Christopher. 2015. Counterfactuals and Causal Inference. New York, NY:Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Nagar Anirudh L. 1959. " The Bias and Moment Matrix of the General K-class Estimators of the Parameters in Simultaneous Equations." Econometrica: Journal of the Econometric Society 27(4):575-95.</bibtext> </blist> <blist> <bibtext> Nelson Charles R, Startz Richard. 1990. " Some Further Results on the Exact Small Sample Properties of the Instrumental Variable Estimator." Econometrica: Journal of the Econometric Society 58(4):967-76.</bibtext> </blist> <blist> <bibtext> Olea José Luis, Pflueger Montiel Carolin. 2013. " A Robust Test for Weak Instruments." Journal of Business & Economic Statistics. 31(3): 358-69.</bibtext> </blist> <blist> <bibtext> Pearl Judea. 1995. " Causal Diagrams for Empirical Research." Biometrika. 82(4): 669-88.</bibtext> </blist> <blist> <bibtext> Pearl Judea. 2009. Causality. New York, NY: Cambridge university press.</bibtext> </blist> <blist> <bibtext> Pearl Judea. 2013. " Linear Models: A Useful 'microscope' for Causal Analysis." Journal of Causal Inference. 1(1): 155-70.</bibtext> </blist> <blist> <bibtext> Richardson Thomas S, Robins James M. 2014. " ACE Bounds; SEMs with Equilibrium Conditions." Statistical Science. 29(3): 363-6.</bibtext> </blist> <blist> <bibtext> Rosenbaum Paul R. 2015. " How to See More in Observational Studies: Some New Quasi-experimental Devices." Annual Review of Statistics and Its Application. 2: 21-48.</bibtext> </blist> <blist> <bibtext> Rossi Peter E. 2014. " Even the Rich Can Make Themselves Poor: A Critical Examination of IV Methods in Marketing Applications." Marketing Science. 33(5): 655-72.</bibtext> </blist> <blist> <bibtext> Rothwell Jonathan T, Massey Douglas S. 2010. " Density Zoning and Class Segregation in US Metropolitan Areas." Social Science Quarterly. 91(5): 1123-43.</bibtext> </blist> <blist> <bibtext> Rubin Donald B. 1974. " Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies." Journal of Educational Psychology. 66(5): 688.</bibtext> </blist> <blist> <bibtext> Sampson Robert J, Winter Alix S. 2018. " Poisoned Development: Assessing Childhood Lead Exposure As a Cause of Crime in a Birth Cohort Followed Through Adolescence." Criminology; an interdisciplinary journal. 56(2): 269-301.</bibtext> </blist> <blist> <bibtext> Sovey Allison J, Green Donald P. 2011. " Instrumental Variables Estimation in Political Science: A Readers' Guide." American Journal of Political Science. 55(1): 188-200.</bibtext> </blist> <blist> <bibtext> Spirling Arthur, Stewart Brandon M. (2022) What good is a regression. Technical report Technical report.</bibtext> </blist> <blist> <bibtext> Stock James H, Yogo Motohiro. 2005. " Testing for Weak Instruments in Linear IV Regression." Pp. 80-108 in Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg, edited by Donald W.K. Andrews and James H. Stock. New York, NY: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Swanson Sonja A, Miller Matthew, Robins James M, Hernán Miguel A. 2015. " Definition and evaluation of the monotonicity condition for preference-based instruments." Epidemiology (Cambridge, Mass.). 26(3): 414.</bibtext> </blist> <blist> <bibtext> Swanson Sonja A, Hernán Miguel A. 2018. " The challenging interpretation of instrumental variable estimates under monotonicity." International Journal of Epidemiology. 47(4): 1289-97.</bibtext> </blist> <blist> <bibtext> Swanson Sonja A, Hernán Miguel A, Miller Matthew, Robins James M, Richardson Thomas S. 2018. " Partial identification of the average treatment effect using instrumental variables: review of methods for binary instruments, treatments, and outcomes." Journal of the American Statistical Association. 113(522): 933-47.</bibtext> </blist> <blist> <bibtext> VanderWeele Tyler J. 2009. " Concerning the consistency assumption in causal inference." Epidemiology (Cambridge, Mass.). 20(6): 880-3.</bibtext> </blist> <blist> <bibtext> VanderWeele Tyler J, Shpitser Ilya. 2011. " A new criterion for confounder selection." Biometrics. 67(4): 1406-13.</bibtext> </blist> <blist> <bibtext> Wang Xuran, Jiang Yang, Zhang Nancy R, Small Dylan S. 2018. " Sensitivity analysis and power for instrumental variable studies." Biometrics. 74(4): 1150-60.</bibtext> </blist> <blist> <bibtext> Wooldridge Jeffrey M. 2010. Econometric Analysis of Cross Section and Panel Data. Cambridge, MA MIT press.</bibtext> </blist> <blist> <bibtext> Wright Philip G. 1928. Tariff on Animal and Vegetable Oils. Macmillan Company, New York, NY New York.</bibtext> </blist> <blist> <bibtext> Young Alwyn. 2022. " Consistency without Inference: Instrumental Variables in Practical Application." European Economic Review. 147: 104112.</bibtext> </blist> </ref> <ref id="AN0190929145-35"> <title> Footnotes </title> <blist> <bibtext> Replication code, along with instructions for obtaining replication data from IPUMS, can be found at https://github.com/cmfelton/iv%5fchecklist/. The color version of this article will be available online.</bibtext> </blist> <blist> <bibtext> The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> The authors received no financial support for the research, authorship and/or publication of this article.</bibtext> </blist> <blist> <bibtext> Chris Felton https://orcid.org/0000-0001-9214-9985 Brandon M. Stewart https://orcid.org/0000-0002-7657-3089</bibtext> </blist> <blist> <bibtext> Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.</bibtext> </blist> <blist> <bibtext> The supplemental material for this article is available online.</bibtext> </blist> <blist> <bibtext> For identifying quantile effects, see [14]. For identifying other average effects of interest, see [32], [33], [19], or [31], among many others. For a review of partial identification with IV, see [77]. We provide further discussion in Online Appendix F.4.</bibtext> </blist> <blist> <bibtext> We make frequent use of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> subscripts to improve clarity, although for many assumptions they are not strictly necessary. We emphasize that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>X</mi><mi>i</mi></msub></math> </ephtml> denotes the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> th draw of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> from a population rather than enumerating particular units within that population. <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mtext>E</mtext><mo stretchy="false">[</mo><msub><mrow><mi mathvariant="italic">X</mi></mrow><mi>i</mi></msub><mo stretchy="false">]</mo></math> </ephtml> denotes the expectation of the random variable <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>X</mi><mi>i</mi></msub></math> </ephtml> , which can be thought of as the average across infinitely many samples of the random variable <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> from that population.</bibtext> </blist> <blist> <bibtext> This scenario is ruled out by the unconfoundedness assumption ("Unconfounded Instrument"), so some authors state the more general assumption that the instrument must be associated with the treatment, which covers both causal instruments and proxy instruments.</bibtext> </blist> <blist> <bibtext> This assumption goes by many names, including "ignorability" and "exogeneity." Exogeneity in particular sometimes refers to both the combination of unconfoundedness and the exclusion restriction (e.g., [28], 316 and [81], 89). While these two assumptions can be collapsed mathematically, we find it is easier to reason about them separately.</bibtext> </blist> <blist> <bibtext> More precisely, unconfoundededness requires all <emph>back-door paths</emph> are blocked ([64]). If plausible common causes are measured, we can block these paths by conditioning on them ([79]).</bibtext> </blist> <blist> <bibtext> In a linear, constant effects model, instrument–treatment confounding will not bias effect estimates. But when effects are heterogeneous and some effects are correlated with one another, it will produce bias. We include a short simulation illustrating this point on https://github.com/cmfelton/iv_checklist.</bibtext> </blist> <blist> <bibtext> While researchers may disagree on the correct level of geographic aggregation with which to measure "neighborhood" in any given analysis (e.g., [36]), our point is that if post-Katrina release causes within-parish moves that in turn affect the outcome, the exclusion restriction will be violated.</bibtext> </blist> <blist> <bibtext> See [55] for a formal discussion of conditions required for the bias to be upward in magnitude. To build intuition for why coarsely measured treatments often bias effect estimates upward in magnitude, consider how this measurement error affects the first-stage and intent-to-treat (ITT) estimates (explained in "Estimation"). The ITT estimate will remain unchanged, but the first-stage regression will underestimate the proportion of compliers, inflating the treatment effect estimate.</bibtext> </blist> <blist> <bibtext> Under monotonicity, the relevance assumption can be weakened to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo form="prefix" movablelimits="true">Pr</mo><mo stretchy="false">[</mo><msub><mi>D</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>Z</mi><mi>i</mi></msub><mo>=</mo>1<mo stretchy="false">)</mo><mo>=</mo><msub><mi>D</mi><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mi>Z</mi><mi>i</mi></msub><mo>=</mo>0<mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo><</mo>1</math> </ephtml> . If defiers are present, this weaker assumption would be insufficient for identification because there could potentially be an average first-stage effect of 0 with the first-stage effects for defiers perfectly cancelling out the first-stage effects for compliers.</bibtext> </blist> <blist> <bibtext> This is easiest to think about in the case with a binary instrument, a binary treatment, and no covariates. Changes beyond that alter the estimand (see Online Appendix F). We also note that when treatment effects are constant, we can identify the total average treatment effect without monotonicity. Constant-effect assumptions, however, are often implausible. See [19] and [31] for more thorough discussions of homogeneity assumptions for IV analysis.</bibtext> </blist> <blist> <bibtext> See [22], [32], [34], and [76] for critiques of the LATE as a causal contrast of interest. See Angrist and Imbens (1999) and [38] for defenses.</bibtext> </blist> <blist> <bibtext> [17] only examine effects for households with married couples, which could block some of these exclusion restriction violations, but also risks inducing confounding (see "Why Blocking Exclusion Restriction Violations Can Induce Unconfoundedness Violations").</bibtext> </blist> <blist> <bibtext> Although we were unable to replicate Conley and Glauber's ([17]) results exactly, our estimates are close to theirs. For white, second-born boys, we have a first-stage estimate of 0.077, compared with their estimate of 0.08. For the estimated LATE, we have <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>−</mo></math> </ephtml> 0.057, compared with their estimate of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>−</mo></math> </ephtml> 0.046. Our estimate of the standard error on the LATE is 0.020, compared with their estimate of 0.022.</bibtext> </blist> <blist> <bibtext> As written, this claim is true only of "over-identified" 2SLS—that is, models containing more instruments than treatments. The reason is that the single-instrument, single-treatment (just-identified) 2SLS estimator has no (finite) expectation because the sampling distribution has heavy tails ([61]; [6]). With no expectation, the bias of the estimator cannot be defined. We can, however, make a similar claim about just-identified 2SLS by shifting focus from the expectation of the sampling distribution to its median. Under homoskedasticity, the just-identified 2SLS sampling distribution will be centered on the OLS probability limit when the population first stage is zero ([7]). Limited information maximum likelihood (LIML) estimation is sometimes recommended as an unbiased alternative to 2SLS. LIML, however, requires multiple valid instruments ([6]). Furthermore, in the presence of effect heterogeneity, the LIML estimand can lie outside the convex hull of LATEs ([47]).</bibtext> </blist> <blist> <bibtext> [74] provide a set of critical values for different levels of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math> </ephtml> , the worst-case bias of 2SLS relative to the bias of OLS. Because the expectation of the just-identified 2SLS estimator is infinite, they provide relative-bias-based critical values only for over-identified 2SLS. See [7] for a critique of basing critical values on the worst-case relative bias.[74] also provide critical values indicating when the undercoverage of confidence intervals will be small (but not zero), including the just-identified setting.[50] show that truly valid <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>t</mi></math> </ephtml> -ratio inference for IV—i.e., zero undercoverage—requires an <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>F</mi><mo stretchy="false">^</mo></mrow><mo>></mo>104.7</math> </ephtml> . Valid inference with Anderson–Rubin, <emph>tF</emph>, and <emph>VtF</emph> confidence sets, which we discuss below, do not require meeting this threshold.[62] show that a robust <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>F</mi><mo stretchy="false">^</mo></mrow></math> </ephtml> of 23.11 in the just-identified setting corresponds to 10% worst-case relative bias, which is the threshold that justifies the conventional rule of thumb in [74]. Importantly, however, they rely on the [60] approximation of 2SLS bias and assess this bias relative to a slightly different benchmark than [74].</bibtext> </blist> <blist> <bibtext> [74] include a set of critical values for bounding the size (the maximum probability of committing a type-I error) of a 5% significance test. With a single treatment, 30 instruments, and a size of 0.10 (which is already larger than the nominal size of 0.05), we would need <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>F</mi><mo stretchy="false">^</mo></mrow><mo>></mo>86.17</math> </ephtml> .[30] did not report how many instruments they used, so it is unclear what critical value applies.</bibtext> </blist> <blist> <bibtext> Recently, [7] argue that in models with a single instrument and single treatment, conventional confidence intervals will have fairly accurate coverage. Crucially, however, they focus on the i.i.d. setting. In contrast, sociologists tend to use panel data in IV settings, where <emph>VtF</emph>, AR, or bootstrapped confidence sets may be more reliable.</bibtext> </blist> <blist> <bibtext> More exactly, the IV estimator's variance will be high if the first-stage effect is small relative to the conditional population variance of the outcome among compliers given the instrument, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mtext>Var</mtext><mo stretchy="false">[</mo><mrow><msub><mi mathvariant="italic">Y</mi><mi mathvariant="italic">i</mi></msub><mo>∣</mo><msub><mi mathvariant="italic">C</mi><mi mathvariant="italic">i</mi></msub></mrow><mo>=</mo>1<mo>,</mo><msub><mi mathvariant="italic">Z</mi><mi mathvariant="italic">i</mi></msub><mo mathvariant="italic" stretchy="false">]</mo></math> </ephtml> , where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>C</mi><mi>i</mi></msub></math> </ephtml> is an indicator for whether unit <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> is a complier ([9]).</bibtext> </blist> <blist> <bibtext> This line of reasoning assumes that the LATE is seldom larger in magnitude than the ATE. In cases where we expect always- and never-takers to have smaller treatment effects than compliers, however, the ATE will be smaller.</bibtext> </blist> <blist> <bibtext> A number of papers only report 2SLS models and others use alternative IV estimators for binary outcomes, such as Two-Stage Residual Inclusion (2SRI) models. Without publicly available replication code, we are unable to compare OLS and 2SLS estimates for these papers.</bibtext> </blist> </ref> <aug> <p>By Chris Felton and Brandon M. Stewart</p> <p>Reported by Author; Author</p> <p></p> <p>Chris Felton is a Postdoctoral Fellow in the Graduate School of Education at Harvard University. He earned his PhD in Sociology and Social Policy from Princeton University in 2023. He studies causal inference and mental health.</p> <p>Brandon M. Stewart is Associate Professor of Sociology and affiliate of the Office of Population Research at Princeton University. He is the author (with Justin Grimmer and Margaret Roberts) of Text as Data: A New Framework for Machine Learning and the Social Sciences (Princeton University Press, 2022).</p> </aug> <nolink nlid="nl1" bibid="bib44" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib49" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib10" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib59" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib72" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib68" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib83" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib46" firstref="ref15"></nolink> <nolink nlid="nl9" bibid="bib41" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib38" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib70" firstref="ref23"></nolink> <nolink nlid="nl12" bibid="bib37" firstref="ref24"></nolink> <nolink nlid="nl13" bibid="bib71" firstref="ref28"></nolink> <nolink nlid="nl14" bibid="bib11" firstref="ref30"></nolink> <nolink nlid="nl15" bibid="bib12" firstref="ref31"></nolink> <nolink nlid="nl16" bibid="bib57" firstref="ref37"></nolink> <nolink nlid="nl17" bibid="bib27" firstref="ref39"></nolink> <nolink nlid="nl18" bibid="bib55" firstref="ref40"></nolink> <nolink nlid="nl19" bibid="bib58" firstref="ref42"></nolink> <nolink nlid="nl20" bibid="bib48" firstref="ref43"></nolink> <nolink nlid="nl21" bibid="bib69" firstref="ref46"></nolink> <nolink nlid="nl22" bibid="bib24" firstref="ref49"></nolink> <nolink nlid="nl23" bibid="bib13" firstref="ref52"></nolink> <nolink nlid="nl24" bibid="bib14" firstref="ref54"></nolink> <nolink nlid="nl25" bibid="bib15" firstref="ref56"></nolink> <nolink nlid="nl26" bibid="bib16" firstref="ref57"></nolink> <nolink nlid="nl27" bibid="bib52" firstref="ref58"></nolink> <nolink nlid="nl28" bibid="bib17" firstref="ref59"></nolink> <nolink nlid="nl29" bibid="bib30" firstref="ref60"></nolink> <nolink nlid="nl30" bibid="bib21" firstref="ref61"></nolink> <nolink nlid="nl31" bibid="bib75" firstref="ref62"></nolink> <nolink nlid="nl32" bibid="bib78" firstref="ref65"></nolink> <nolink nlid="nl33" bibid="bib40" firstref="ref66"></nolink> <nolink nlid="nl34" bibid="bib35" firstref="ref69"></nolink> <nolink nlid="nl35" bibid="bib82" firstref="ref70"></nolink> <nolink nlid="nl36" bibid="bib39" firstref="ref71"></nolink> <nolink nlid="nl37" bibid="bib25" firstref="ref72"></nolink> <nolink nlid="nl38" bibid="bib66" firstref="ref73"></nolink> <nolink nlid="nl39" bibid="bib81" firstref="ref75"></nolink> <nolink nlid="nl40" bibid="bib65" firstref="ref77"></nolink> <nolink nlid="nl41" bibid="bib53" firstref="ref81"></nolink> <nolink nlid="nl42" bibid="bib54" firstref="ref82"></nolink> <nolink nlid="nl43" bibid="bib20" firstref="ref83"></nolink> <nolink nlid="nl44" bibid="bib18" firstref="ref84"></nolink> <nolink nlid="nl45" bibid="bib29" firstref="ref85"></nolink> <nolink nlid="nl46" bibid="bib43" firstref="ref87"></nolink> <nolink nlid="nl47" bibid="bib80" firstref="ref92"></nolink> <nolink nlid="nl48" bibid="bib19" firstref="ref93"></nolink> <nolink nlid="nl49" bibid="bib74" firstref="ref103"></nolink> <nolink nlid="nl50" bibid="bib62" firstref="ref104"></nolink> <nolink nlid="nl51" bibid="bib22" firstref="ref112"></nolink> <nolink nlid="nl52" bibid="bib51" firstref="ref114"></nolink> <nolink nlid="nl53" bibid="bib23" firstref="ref121"></nolink> <nolink nlid="nl54" bibid="bib26" firstref="ref123"></nolink> <nolink nlid="nl55" bibid="bib495" firstref="ref135"></nolink> <nolink nlid="nl56" bibid="bib42" firstref="ref142"></nolink> <nolink nlid="nl57" bibid="bib67" firstref="ref147"></nolink> <nolink nlid="nl58" bibid="bib56" firstref="ref150"></nolink> <nolink nlid="nl59" bibid="bib45" firstref="ref151"></nolink> <nolink nlid="nl60" bibid="bib73" firstref="ref152"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1495936
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Chris+Felton%22">Chris Felton</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9214-9985">0000-0001-9214-9985</externalLink>)<br /><searchLink fieldCode="AR" term="%22Brandon+M%2E+Stewart%22">Brandon M. Stewart</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7657-3089">0000-0002-7657-3089</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2026 55(1):3-50.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 48
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Social+Science+Research%22">Social Science Research</searchLink><br /><searchLink fieldCode="DE" term="%22Sociology%22">Sociology</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Inference%22">Statistical Inference</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Bias%22">Statistical Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241241235900
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Instrumental variables (IV) analysis is a powerful, but fragile, tool for drawing causal inferences from observational data. Sociologists increasingly turn to this strategy in settings where unmeasured confounding between the treatment and outcome is likely. This paper reviews the assumptions required for IV and the consequences of violating them, focusing on sociological applications. We highlight three methodological problems IV faces: (i) identification bias, an asymptotic bias from assumption violations; (ii) estimation bias, a finite-sample bias that persists even when assumptions hold; and (iii) type-M error, the exaggeration of effect size given statistical significance. In each case, we emphasize how weak instruments exacerbate these problems and make results sensitive to minor violations of assumptions. We survey IV papers from top sociology journals, finding that assumptions often go unstated and robust uncertainty measures are rarely used. We provide a practical checklist to show how IV, despite its fragility, can still be useful when handled with care.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1495936
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1495936
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241241235900
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 48
        StartPage: 3
    Subjects:
      – SubjectFull: Social Science Research
        Type: general
      – SubjectFull: Sociology
        Type: general
      – SubjectFull: Statistical Inference
        Type: general
      – SubjectFull: Statistical Bias
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Research Methodology
        Type: general
    Titles:
      – TitleFull: Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Chris Felton
      – PersonEntity:
          Name:
            NameFull: Brandon M. Stewart
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 55
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1