View in EDS HTML Full Text PDF Full Text

Data Imbalances in Coincidence Analysis: A Simulation Study

Saved in:
Bibliographic Details
Title: Data Imbalances in Coincidence Analysis: A Simulation Study
Language: English
Authors: Martyna Daria Swiatczak (ORCID 0000-0002-7537-1813), Michael Baumgartner (ORCID 0000-0003-1536-2816)
Source: Sociological Methods & Research. 2025 54(2):739-771.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 33
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Causal Models, Comparative Analysis, Data Analysis, Statistical Distributions, Statistical Data
DOI: 10.1177/00491241241227039
ISSN: 0049-1241
1552-8294
Abstract: In this paper, we investigate the conditions under which data imbalances, a common data characteristic that occurs when factor values are unevenly distributed, are problematic for the performance of Coincidence Analysis (CNA). We further examine how such imbalances relate to fragmentation and noise in data. We show that even extreme data imbalances, when not combined with fragmentation or noise, do not negatively affect CNA's performance. However, an extended series of simulation experiments on fuzzy-set data reveals that, when mixed with fragmentation or noise, data imbalances may substantially impair CNA's performance. Furthermore, we find that the performance impairment is higher when endogenous factors are imbalanced than when exogenous factors are concerned. Our results allow us to quantify these impacts and demarcate degrees at which data imbalances should be considered as problematic. Thus, applied researchers can use our demarcation guidelines to enhance the validity of their studies.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1473620
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFQmkpNz4m9LMhI7-J-hs9-AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDArZBM1YqjS-R4aAngIBEICBmzz469n1JkERNAapOyBVPBVd1MdSeGEveK2NNszvUVocFQMrmlVJqVNAg6BkKXJXUoL8TalS72b9A5BVjgjmPXjtmjaVTRwhzq93arkJGKRrlTWska3ZLxEPaHHXTNuXE5kGtXfZvoqJbqaH3XeKpnF63u1sg1Ywyzl3jlwF4I_TKWjU5KjsCvwGVQCM9xzFG_L9BGA2GQWgQTBt
Text:
  Availability: 1
  Value: <anid>AN0184233960;som01may.25;2025Apr07.05:58;v2.2.500</anid> <title id="AN0184233960-1">Data Imbalances in Coincidence Analysis: A Simulation Study </title> <p>In this paper, we investigate the conditions under which data imbalances, a common data characteristic that occurs when factor values are unevenly distributed, are problematic for the performance of Coincidence Analysis (CNA). We further examine how such imbalances relate to fragmentation and noise in data. We show that even extreme data imbalances, when not combined with fragmentation or noise, do not negatively affect CNA's performance. However, an extended series of simulation experiments on fuzzy-set data reveals that, when mixed with fragmentation or noise, data imbalances may substantially impair CNA's performance. Furthermore, we find that the performance impairment is higher when endogenous factors are imbalanced than when exogenous factors are concerned. Our results allow us to quantify these impacts and demarcate degrees at which data imbalances should be considered as problematic. Thus, applied researchers can use our demarcation guidelines to enhance the validity of their studies.</p> <p>Keywords: configurational causal modeling; configurational comparative methods; Coincidence Analysis; distributional imbalances; skewness; membership ratio; method benchmarking</p> <hd id="AN0184233960-2">Introduction</hd> <p>Coincidence Analysis (CNA; [<reflink idref="bib6" id="ref1">6</reflink>]) is a novel method of causal learning that belongs to the family of configurational comparative methods (CCMs; [<reflink idref="bib21" id="ref2">21</reflink>]). Unlike most methods of data analysis, CNA can handle complex causal structures involving conjunctivity (when multiple factors interact to produce an outcome) and disjunctivity (when alternative pathways produce the same outcome independently of one another), which do not necessarily exhibit pairwise dependencies between a cause and its effect. CNA accomplishes this by fitting complex Boolean functions as a whole to the data, and it is the only method of its kind capable of detecting links between multiple outcomes (sequentiality), which are characteristic for causal chains.</p> <p>As such, CNA has been increasingly applied in a wide range of fields, including political science (e.g., [<reflink idref="bib14" id="ref3">14</reflink>]), environmental studies (e.g., [<reflink idref="bib9" id="ref4">9</reflink>]), public health (e.g., [<reflink idref="bib33" id="ref5">33</reflink>]), medical informatics (e.g., [<reflink idref="bib32" id="ref6">32</reflink>]), sociology (e.g., [<reflink idref="bib10" id="ref7">10</reflink>]), and organizational behavior (e.g., [<reflink idref="bib25" id="ref8">25</reflink>]). In parallel, methodological research has substantially improved the quality of CNA's data analysis approach (e.g., [<reflink idref="bib18" id="ref9">18</reflink>]). However, data distribution requirements have not yet been investigated for CNA. This is all the more striking as, for example, within the framework of statistical methods, assessing data distributions is a crucial pre-analytical step for selecting appropriate methods and determining the expected accuracy of analyses. Accordingly, various data distribution characteristics have been widely shown to create issues for these methods (e.g., [<reflink idref="bib31" id="ref10">31</reflink>]; [<reflink idref="bib34" id="ref11">34</reflink>]).</p> <p>This study examines how the performance of CNA is affected by a common data distribution characteristic: data imbalances (also referred to as <emph>skewness</emph>), which occur if factors have value distributions that partition the cases in the data into sets of notably unequal sizes. Such imbalances are often encountered in CCM applications. For instance, the vast majority of countries are classified as not wealthy based on the world bank GDP classification system ([<reflink idref="bib13" id="ref12">13</reflink>]), decisions against the termination of pregnancy after a prenatal diagnosis of Down syndrome are very rare ([<reflink idref="bib7" id="ref13">7</reflink>]), educational poverty is a marginal occurrence in developed countries ([<reflink idref="bib12" id="ref14">12</reflink>]), and employees prevailingly consider themselves competent ([<reflink idref="bib25" id="ref15">25</reflink>]).</p> <p>Skewness has received some attention in the methodological literature on Qualitative Comparative Analysis (QCA; [<reflink idref="bib20" id="ref16">20</reflink>]), another method from the family of CCMs (e.g., [<reflink idref="bib17" id="ref17">17</reflink>]; [<reflink idref="bib22" id="ref18">22</reflink>]). However, on the one hand, those discussions have so far focused on particular data examples, which do not yield quantitative performance assessments or generalizable conclusions. On the other hand, findings on QCA cannot be transferred to CNA because of substantive algorithmic differences between the two methods ([<reflink idref="bib26" id="ref19">26</reflink>]). As a consequence, thus far, applied CNA researchers lack a means to determine whether their data are imbalanced to such an extent that countermeasures should be taken or reliable results can be expected.</p> <p>The aim of this study is to remedy this situation by systematically investigating to what extent data imbalances affect the quality of CNA's output. First, we demonstrate that, contrary to previous discussions on skewness, no general claims can be made about the effect of data imbalances in isolation of other aspects of data quality. More precisely, data imbalances do not affect CNA's performance if the data are completely free from noise and fragmentation. By contrast, it is far from clear how imbalances interact with noise and fragmentation. Do they affect CNA's performance solely by exacerbating noise and fragmentation or do they have their own impact on performance that is independent of other data deficiencies? Second, to answer these questions, we present the results of a series of simulation experiments benchmarking CNA's performance under varying degrees of distributional imbalances while controlling for other data deficiencies. Our experiments are designed as inverse search trials, meaning that we randomly draw data-generating causal structures from which we simulate data with varying imbalances and different combinations of other data deficiencies, consecutively analyze these data with CNA, and measure how frequently the original causal structures (or proper parts thereof) are contained in CNA's output.</p> <p>Overall, we find that increasing imbalances while keeping noise and fragmentation constant results in impaired performance, which our results allow to quantify. In other words, imbalances not only exacerbate other data deficiencies but also have a negative impact of their own. This impact is higher for imbalances in endogenous factors than in exogenous ones. Our study identifies degrees at which distributional imbalances should be considered problematic and proposes approaches to address them.</p> <hd id="AN0184233960-3">Data Imbalances in CNA</hd> <p></p> <hd id="AN0184233960-4">CNA Preliminaries</hd> <p>To infer causal structures featuring conjunctivity, disjunctivity, and sequentiality from data, CNA draws on the so-called <emph>(M) INUS theory of causation</emph> ([<reflink idref="bib5" id="ref20">5</reflink>]; [<reflink idref="bib16" id="ref21">16</reflink>]),[<reflink idref="bib6" id="ref22">6</reflink>] which is specifically designed for the analysis of such structures as it defines causation via complex Boolean dependencies. Factors are the basic modeling devices of the MINUS theory and of CNA. They are analogous to variables, meaning they are functions from (measured) properties into a range of values (typically integers). CNA can process data comprising crisp- and fuzzy-set or multi-value factors ([<reflink idref="bib6" id="ref23">6</reflink>]). For reasons of space, the ensuing discussion will, however, focus on crisp- and fuzzy-set factors only.</p> <p>Values of a crisp- and fuzzy-set factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> can be interpreted as membership scores in the set of cases exhibiting the property represented by <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> . That is, a case of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi><mo>=</mo><mn>1</mn></math> </ephtml> is a full member of that set, a case of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="normal">X</mi></mrow><mo>=</mo><mn>0</mn></math> </ephtml> is a full non-member, and a case of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="normal">X</mi></mrow><mo>=</mo><msub><mi>χ</mi><mi>i</mi></msub></math> </ephtml> , where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0</mn><mo><</mo><msub><mi>χ</mi><mi>i</mi></msub><mo><</mo><mn>1</mn></math> </ephtml> , is a member to degree <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>χ</mi><mi>i</mi></msub></math> </ephtml> . A case is considered a member if its membership score <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>χ</mi><mi>i</mi></msub></math> </ephtml> is above the 0.5-anchor, that is, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>χ</mi><mi>i</mi></msub><mo>></mo><mn>0.5</mn></math> </ephtml> , and it is a non-member if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>χ</mi><mi>i</mi></msub><mo>≤</mo><mn>0.5</mn></math> </ephtml> . In the process of calibration, the meanings of full membership, full non-membership, and cross-over at the 0.5-anchor are defined for each set and then used to transform raw data into crisp or fuzzy membership scores (see e.g., [<reflink idref="bib29" id="ref24">29</reflink>]; [<reflink idref="bib17" id="ref25">17</reflink>] on calibration methods and algorithms).</p> <p>As the explicit "Factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>=</mo></math> </ephtml> value" notation yields convoluted syntactic expressions, we will use the following shorthand notation, which is conventional in Boolean algebra: " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> " signifies membership in the set of cases exhibiting the property represented by <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> and " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi></math> </ephtml> " signifies non-membership in that set. Italicization thus carries meaning: " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> " designates the factor and " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> " membership in the set of cases with values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> above <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> . Moreover, we write " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>*</mo><mi>Y</mi></math> </ephtml> " for the Boolean operation of <emph>conjunction</emph> " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> ", " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>+</mo><mi>Y</mi></math> </ephtml> " for the <emph>disjunction</emph> " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> ", " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo stretchy="false">→</mo><mi>Y</mi></math> </ephtml> " for the <emph>implication</emph> "if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> then <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> ", and " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo stretchy="false">↔</mo><mi>Y</mi></math> </ephtml> " for the <emph>equivalence</emph> " <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> if, and only if, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> ". For crisp-set factors, the Boolean operations are given a rendering in classical logic (e.g., [<reflink idref="bib15" id="ref26">15</reflink>]), and for fuzzy-set factors, these operations are rendered in fuzzy logic (e.g., [<reflink idref="bib6" id="ref27">6</reflink>]).[<reflink idref="bib7" id="ref28">7</reflink>] The implication operator is used to define the notions of <emph>sufficiency</emph> and <emph>necessity</emph>, which are the two Boolean dependencies exploited by the MINUS theory and CNA: <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is sufficient for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> if, and only if, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mspace width=".1em" /><mo stretchy="false">→</mo><mspace width=".1em" /><mi>Y</mi></math> </ephtml> ; and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is necessary for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> if, and only if, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi><mspace width=".1em" /><mo stretchy="false">→</mo><mspace width=".1em" /><mi>X</mi></math> </ephtml> .</p> <p>To reflect causation, sufficiency and necessity relations need to be rigorously freed of redundancies, which is accomplished if sufficient and necessary conditions are <emph>minimal</emph>, meaning they do not have proper parts that are, respectively, sufficient and necessary on their own ([<reflink idref="bib5" id="ref29">5</reflink>]). In sum, using techniques from Boolean algebra, set theory, and fuzzy logic, CNA infers minimally necessary disjunctions of minimally sufficient conditions of scrutinized outcomes (in disjunctive normal form), so-called <emph>MINUS-formulas</emph>, from data. The following is an example: <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>b</mi><mspace width=".1em" /><mo>+</mo><mspace width=".1em" /><mi>c</mi><mo>*</mo><mi>D</mi><mspace width="0.25em" /><mo stretchy="false">↔</mo><mspace width="0.25em" /><mi>E</mi></math> </ephtml></p> <p>Graph</p> <p>When causally interpreted, (<reflink idref="bib1" id="ref30">1</reflink>) entails that each of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>c</mi></math> </ephtml> , and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>D</mi></math> </ephtml> is a cause of outcome <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi></math> </ephtml> and that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math> </ephtml> conjunctively cause <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi></math> </ephtml> on one path while <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>c</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>D</mi></math> </ephtml> operate on another path.</p> <p>In view of its embedding in the MINUS theory, CNA—unlike most other methods—does not infer its output from associations (e.g., effect sizes) in the data as a whole; rather, it exploits difference-making evidence on the level of individual factor configurations instantiated by cases in the data. For example, if two configurations <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>j</mi></msub></math> </ephtml> coincide in all measured factors except for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Y</mi></math> </ephtml> , such that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> features <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>j</mi></msub></math> </ephtml> features <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>y</mi></math> </ephtml> , this is evidence—assuming the homogeneity of the unmeasured causal background (for details, see [<reflink idref="bib6" id="ref31">6</reflink>])—that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> makes a difference to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> in the causal background of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>j</mi></msub></math> </ephtml> .[<reflink idref="bib8" id="ref32">8</reflink>] It follows that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> must be part of some conjunction causally relevant for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> . Correspondingly, a pair of configurations as <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>j</mi></msub></math> </ephtml> is called a <emph>difference-making pair</emph> for the causal relevance of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> ([<reflink idref="bib5" id="ref33">5</reflink>]).</p> <p>Table 1. Subtables (a) and (b) feature ideal data on structure (<reflink idref="bib2" id="ref34">2</reflink>).</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="center" colspan="6">(a)</th><th align="center" colspan="6">(b)</th></tr><tr><th align="left" /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">Y</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">n</mi></math></p></th><th align="left" /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">Y</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">n</mi></math></p></th></tr></thead><tbody><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>1</mn></msub></math></p></td><td>1</td><td>1</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>1</mn></msub></math></p></td><td>0.52</td><td>0.66</td><td>0.82</td><td>0.82</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>2</mn></msub></math></p></td><td>0</td><td>1</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>2</mn></msub></math></p></td><td>0.08</td><td>0.64</td><td>0.88</td><td>0.88</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>3</mn></msub></math></p></td><td>1</td><td>0</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>3</mn></msub></math></p></td><td>0.88</td><td>0.04</td><td>0.84</td><td>0.88</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>4</mn></msub></math></p></td><td>1</td><td>1</td><td>0</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>4</mn></msub></math></p></td><td>0.98</td><td>0.60</td><td>0.44</td><td>0.98</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>5</mn></msub></math></p></td><td>0</td><td>0</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>5</mn></msub></math></p></td><td>0.02</td><td>0.10</td><td>0.72</td><td>0.72</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>6</mn></msub></math></p></td><td>0</td><td>1</td><td>0</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>6</mn></msub></math></p></td><td>0.48</td><td>0.82</td><td>0.10</td><td>0.82</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>7</mn></msub></math></p></td><td>1</td><td>0</td><td>0</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>7</mn></msub></math></p></td><td>0.80</td><td>0.10</td><td>0.48</td><td>0.80</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>8</mn></msub></math></p></td><td>0</td><td>0</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>8</mn></msub></math></p></td><td>0.28</td><td>0.28</td><td>0.14</td><td>0.28</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr></tbody></table> </ephtml> </p> <p>1 The first (line-separated) column in both tables labels the configurations, the last column indicates the frequency of a corresponding configuration. In both tables, the membership ratios are as follows: <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.875</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>A</mi><mo>/</mo><mi>B</mi><mo>/</mo><mi>C</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.5</mn></math> </ephtml> .</p> <hd id="AN0184233960-5">The Problem of Imbalanced Data</hd> <p>As difference-making pairs are the main drivers of CNA's inference to causation, the value distributions of factors above and below the 0.5-anchor may affect CNA's performance. Notably in extreme scenarios, where all values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Y</mi></math> </ephtml> are above or below the 0.5-anchor, the data do not contain any difference-making pairs whatsoever, for the simple reason that all cases are uniformly in or out of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> . Without differences in set memberships, no difference-making pairs and, thus, no difference-making evidence. In order for data to contain reliably exploitable difference-making evidence, the value distributions of analyzed factors must be balanced so that an appropriate number of cases fall above and below the 0.5-anchor. Of course, the exact meaning of "appropriate" requires specification—which is the very topic of this paper.</p> <p>Based on [<reflink idref="bib17" id="ref35">17</reflink>], we define the <emph>membership ratio</emph> ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi></math> </ephtml> ) in a crisp or fuzzy set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> to be the ratio of cases in data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> with X <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mspace width="0.25em" /><mo>></mo><mn>0.5</mn></math> </ephtml> to all cases in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> .[<reflink idref="bib9" id="ref36">9</reflink>] For example, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.8</mn></math> </ephtml> means that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> takes a value <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>></mo><mspace width=".1em" /><mn>0.5</mn></math> </ephtml> in 80% of the cases in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> and a value <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo>≤</mo><mspace width=".1em" /><mn>0.5</mn></math> </ephtml> in 20% of the cases. Whenever the value distributions of a factor do not partition the cases in the data into sets of roughly equal size, we speak of an <emph>imbalanced (or skewed) factor distribution</emph>. Imbalanced distributions come in degrees and are the higher the farther away membership ratios are from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> . However, data imbalances are the norm in applied research and most of them are unproblematic because they do not impair CNA's performance. For <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> to contain an appropriate amount of difference-making evidence, it suffices that membership ratios lie in some interval centered around 0.5. A factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> only counts as problematically imbalanced if the membership ratio in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is outside of this moderate interval. Problematic imbalances are those imbalances that our subsequent investigation shows to significantly weaken CNA's performance, on average.</p> <p>For QCA, [<reflink idref="bib17" id="ref37">17</reflink>] propose that "as a rule of thumb" (p. 48) ratios outside of the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.8</mn><mo>,</mo><mn>0.2</mn><mo stretchy="false">]</mo></math> </ephtml> are problematic. But they do not provide an argument for why this, rather than, say, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.9</mn><mo>,</mo><mn>0.1</mn><mo stretchy="false">]</mo></math> </ephtml> is the relevant interval and they stress themselves that these are not fixed thresholds. Moreover, as there are many algorithmic differences between CNA and QCA ([<reflink idref="bib26" id="ref38">26</reflink>]), the question as to the interval outside of which membership ratios should be considered problematic is entirely open for CNA at this point. Note that the CCM literature typically discusses data imbalances under the label of <emph>skewness</emph> (e.g., [<reflink idref="bib22" id="ref39">22</reflink>]; [<reflink idref="bib17" id="ref40">17</reflink>]; [<reflink idref="bib30" id="ref41">30</reflink>]), which must not be confused with skewness in statistics, where it is a measure for the asymmetry of the distribution of a variable around its mean (e.g., [<reflink idref="bib28" id="ref42">28</reflink>]). What matters for difference-making evidence in CCMs, however, are neither distributional symmetries nor mean factor values, but only the ratios of cases above and below the 0.5-anchor.[<reflink idref="bib10" id="ref43">10</reflink>] For that reason, we prefer to speak of data imbalances or imbalanced distributions. Even so, we acknowledge that the term skewness has an established usage in CCMs and, occasionally, also speak of skewness or skewed distributions.[<reflink idref="bib11" id="ref44">11</reflink>]</p> <hd id="AN0184233960-6">The Case of Ideal Data</hd> <p>Before turning to the problem of demarcating the interval of problematic membership ratios, the special case of ideal data requires separate treatment. The reason is that, in ideal data, membership ratios can be extreme without any negative consequences for the performance of CNA. To see this, we first have to specify when data are ideal in configurational causal modeling. This is best accomplished by example. Thus, assume that the behavior of the factors in the set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>1</mn></msub><mo>=</mo><mo fence="false" stretchy="false">{</mo><mtext>A</mtext><mo>,</mo><mtext>B</mtext><mo>,</mo><mtext>C</mtext><mo>,</mo><mtext>Y</mtext><mo fence="false" stretchy="false">}</mo></math> </ephtml> is regulated by the causal structure corresponding to this simple MINUS-formula, which we will refer to as the <emph>ground truth</emph>: <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mspace width=".1em" /><mo>+</mo><mspace width=".1em" /><mi>B</mi><mspace width=".1em" /><mo>+</mo><mspace width=".1em" /><mi>C</mi><mspace width="0.25em" /><mo stretchy="false">↔</mo><mspace width="0.25em" /><mi>Y</mi></math> </ephtml></p> <p>Graph</p> <p>(<reflink idref="bib2" id="ref45">2</reflink>) entails that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>B</mi></math> </ephtml> , and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>C</mi></math> </ephtml> are three alternative causes of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> . If we take the factors in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>1</mn></msub></math> </ephtml> to be crisp-set and hold additional causes of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> not contained in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>1</mn></msub></math> </ephtml> constant, it follows that the factors in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>1</mn></msub></math> </ephtml> can be combined in exactly the eight configurations listed in Table 1a. To generalize for the fuzzy-set case, Table 1b contains fuzzy-set data corresponding to the crisp-set configurations in Table 1a. As the factors in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>1</mn></msub></math> </ephtml> take a value above the 0.5-anchor in one of these tables exactly if they take such a value in the other one, both tables feature the same configurations.</p> <p>There are <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mn>2</mn><mn>3</mn></msup><mo>=</mo><mn>8</mn></math> </ephtml> logically possible ways of combining values above and below the 0.5-anchor of the 3 exogenous factors in (<reflink idref="bib2" id="ref46">2</reflink>), all of which are contained in Tables 1a and 1b. Moreover, these tables do not contain any configurations incompatible with (<reflink idref="bib2" id="ref47">2</reflink>), that is, (<reflink idref="bib2" id="ref48">2</reflink>) is true in all configurations of both tables.[<reflink idref="bib12" id="ref49">12</reflink>] It follows that Tables 1a and 1b feature neither <emph>fragmentation</emph> nor <emph>noise</emph>.[<reflink idref="bib13" id="ref50">13</reflink>] Fragmentation of a data set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> generated by a ground truth <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> over a factor set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mi>i</mi></msub></math> </ephtml> is defined as the ratio of configurations of the factors in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mi>i</mi></msub></math> </ephtml> compatible with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> that are missing from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> . By contrast, data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> feature noise when some configurations in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> are incompatible with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> , which obtains if the left-hand and right-hand sides of the ' <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">↔</mo></math> </ephtml> ' in the MINUS-formula corresponding to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>r</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> , are non-identical (e.g., due to measurement error or confounding). The higher the mean differences between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>r</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> , the higher <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> 's noise level. Accordingly, noise can be measured in terms of the mean absolute difference between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>r</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><mo stretchy="false">)</mo></math> </ephtml> in the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>n</mi></math> </ephtml> cases of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>δ</mi></math> </ephtml> : <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mfrac><mrow><msubsup><mo movablelimits="false">∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></msubsup><mo fence="false" stretchy="false">|</mo><mspace width=".1em" /><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><msub><mo stretchy="false">)</mo><mi>i</mi></msub><mo>−</mo><mi>r</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><mi mathvariant="normal">Δ</mi><msub><mo stretchy="false">)</mo><mi>i</mi></msub><mspace width=".1em" /><mo fence="false" stretchy="false">|</mo></mrow><mi>n</mi></mfrac></mrow></math> </ephtml></p> <p>Graph</p> <p>Data with zero fragmentation and zero noise are <emph>ideal data</emph>. Thus, Tables 1a and 1b contain ideal data on ground truth (<reflink idref="bib2" id="ref51">2</reflink>) with each configuration realized by exactly one case. While the distributions of the exogenous factors are balanced in both tables, Y takes a value above <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> in 7 of 8 cases, yielding <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.875</mn></math> </ephtml> . That is, even though Tables 1a and 1b contain ideal data, they feature an endogenous factor that is imbalanced to a degree that counts as problematic for QCA subject to Oana et al.'s ([<reflink idref="bib17" id="ref52">17</reflink>]) rule of thumb. Clearly though, this extreme imbalance is all but problematic. Rather, it is induced by the form of the ground truth (<reflink idref="bib2" id="ref53">2</reflink>) itself. If case frequencies are kept constant, that is, if we ensure that all configurations are realized by an equal amount of cases, ideal data on structure (<reflink idref="bib2" id="ref54">2</reflink>) will always have very high membership ratios in <emph>Y</emph>. Despite the extreme imbalance of Y, CNA easily infers the MINUS-formula (<reflink idref="bib2" id="ref55">2</reflink>) from Tables 1a and 1b. Factor Y is imbalanced because, subject to the ground truth (<reflink idref="bib2" id="ref56">2</reflink>), there are three independent paths to produce <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> , each of which is activated by simply instantiating one cause. That means the overwhelming majority of all logically possible configurations of the exogenous factors produce <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> , which, correspondingly, occurs frequently in ideal data with constant case frequencies.</p> <p>Plainly, not only high but also low membership ratios can be induced by the structure of the ground truth. Assume that, instead of (<reflink idref="bib2" id="ref57">2</reflink>), the following is the ground truth: <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>B</mi><mo>*</mo><mi>C</mi><mspace width="0.25em" /><mo stretchy="false">↔</mo><mspace width="0.25em" /><mi>Y</mi></math> </ephtml></p> <p>Graph</p> <p>If the three causes of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> are not disjunctively concatenated as in (<reflink idref="bib2" id="ref58">2</reflink>) but conjunctively as in (<reflink idref="bib4" id="ref59">4</reflink>), three factors must jointly take values above 0.5 for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> to occur, which only happens in one of eight configurations in the ideal data on (<reflink idref="bib4" id="ref60">4</reflink>) in Tables 2a and 2b, where the membership ratio in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> is <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.125</mn></math> </ephtml> , while <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>B</mi><mo stretchy="false">)</mo></math> </ephtml> , and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mo stretchy="false">)</mo></math> </ephtml> are again perfectly balanced at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> . Just as in case of Table 1, CNA straightforwardly infers (<reflink idref="bib4" id="ref61">4</reflink>) from Tables 2a and 2b.</p> <p>Table 2. Subtable (a) features ideal crisp-set data on structure (<reflink idref="bib4" id="ref62">4</reflink>), subtable (b) ideal fuzzy-set data on structure (<reflink idref="bib4" id="ref63">4</reflink>), and subtable (c) comprises ideal crisp-set data, with unequal case frequencies, on structure (<reflink idref="bib4" id="ref64">4</reflink>).</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="left" /></colgroup><thead><tr><th align="center" colspan="6">(a)</th><th align="center" colspan="6">(b)</th><th align="center" colspan="6">(c)</th></tr><tr><th align="left" /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">Y</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">n</mi></math></p></th><th align="left" /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">Y</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">n</mi></math></p></th><th align="left" /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">B</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">C</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">Y</mi></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">n</mi></math></p></th></tr></thead><tbody><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>1</mn></msub></math></p></td><td>1</td><td>1</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>1</mn></msub></math></p></td><td>0.60</td><td>0.84</td><td>0.64</td><td>0.60</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>1</mn></msub></math></p></td><td>1</td><td>1</td><td>1</td><td>1</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">33</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>2</mn></msub></math></p></td><td>0</td><td>1</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>2</mn></msub></math></p></td><td>0.20</td><td>0.66</td><td>0.56</td><td>0.20</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>2</mn></msub></math></p></td><td>0</td><td>1</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>3</mn></msub></math></p></td><td>1</td><td>0</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>3</mn></msub></math></p></td><td>0.90</td><td>0.10</td><td>0.86</td><td>0.10</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>3</mn></msub></math></p></td><td>1</td><td>0</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>4</mn></msub></math></p></td><td>1</td><td>1</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>4</mn></msub></math></p></td><td>0.86</td><td>0.96</td><td>0.04</td><td>0.04</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>4</mn></msub></math></p></td><td>1</td><td>1</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>5</mn></msub></math></p></td><td>0</td><td>0</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>5</mn></msub></math></p></td><td>0.16</td><td>0.36</td><td>0.88</td><td>0.16</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>5</mn></msub></math></p></td><td>0</td><td>0</td><td>1</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>6</mn></msub></math></p></td><td>0</td><td>1</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>6</mn></msub></math></p></td><td>0.14</td><td>0.58</td><td>0.38</td><td>0.14</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>6</mn></msub></math></p></td><td>0</td><td>1</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>7</mn></msub></math></p></td><td>1</td><td>0</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>7</mn></msub></math></p></td><td>0.68</td><td>0.14</td><td>0.04</td><td>0.04</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>7</mn></msub></math></p></td><td>1</td><td>0</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr><tr><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>8</mn></msub></math></p></td><td>0</td><td>0</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>8</mn></msub></math></p></td><td>0.24</td><td>0.24</td><td>0.00</td><td>0.00</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub xmlns=""><mi>σ</mi><mn>8</mn></msub></math></p></td><td>0</td><td>0</td><td>0</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">1</mn></math></p></td></tr></tbody></table> </ephtml> </p> <p>2 The first (line-separated) column in all subtables labels the configurations, the last column indicates the frequency of a corresponding configuration. The data in (a)-(b) have membership ratios of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.125</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>A</mi><mo>/</mo><mi>B</mi><mo>/</mo><mi>C</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.5</mn></math> </ephtml> , the data in (c) have <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.825</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>A</mi><mo>/</mo><mi>B</mi><mo>/</mo><mi>C</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> .</p> <p>In general, membership ratios in outcomes in ideal data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>δ</mi><mrow><mi>i</mi><mi>d</mi></mrow></msup></math> </ephtml> with constant case frequencies depend on the structural properties of the ground truth <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> in the following manner: Given a fixed number of conjuncts in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> , the more disjuncts <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> has, the higher the membership ratio in the outcome in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>δ</mi><mrow><mi>i</mi><mi>d</mi></mrow></msup></math> </ephtml> , as the outcome can be produced more easily (via more paths), that is, more frequently. Conversely, given a fixed number of disjuncts in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> , the more conjuncts <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">Δ</mi></math> </ephtml> has, the lower the membership ratio in the outcome in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>δ</mi><mrow><mi>i</mi><mi>d</mi></mrow></msup></math> </ephtml> , as more conditions must be satisfied to produce the outcome, which is more difficult to accomplish and, correspondingly, occurs less frequently. We refer to imbalances that are induced by the properties of the ground truth as <emph>structure-induced</emph>. Note that, in ideal data with constant case frequencies, exogenous factors are always perfectly balanced, even when endogenous factors are affected by structure-induced imbalances.</p> <p>However, ideal data are not required to have constant case frequencies. Some configurations may be realized by more cases than others in ideal data. To illustrate, consider Table 2c, which, like Table 2a, contains ideal crisp-set data on structure (<reflink idref="bib4" id="ref65">4</reflink>). But while all configurations in Tables 2a and 2b are realized by exactly one case, in Table 2c, configuration <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mn>1</mn></msub></math> </ephtml> is realized much more frequently than all others (i.e., by 33 cases). This mismatch in case frequencies is not due to structural properties of (<reflink idref="bib4" id="ref66">4</reflink>), rather, cases realizing configuration <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mn>1</mn></msub></math> </ephtml> just happen to be more frequent than cases realizing the other configurations. The frequency mismatch in Table 2c yields that all membership ratios are high: <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.825</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>A</mi><mo>/</mo><mi>B</mi><mo>/</mo><mi>C</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> . Plainly, if configuration <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mn>8</mn></msub></math> </ephtml> instead of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mn>1</mn></msub></math> </ephtml> was realized more frequently, the result would not be high but low membership ratios. We refer to imbalances that are not induced by the properties of the ground truth as <emph>frequency-induced</emph>. As the example in Table 2c demonstrates, both exogenous and endogenous factors can be affected by frequency-induced imbalances in ideal data. But importantly, what we showed for CNA's analysis of Tables 1a, 1b, 2a, and 2b also holds for Table 2c: The MINUS-formula inferred by CNA corresponds to the ground truth.</p> <p>Overall, extreme imbalances, whether structure- or frequency-induced, do not impair CNA's performance provided that the data are free of fragmentation and noise, that is, ideal. As indicated in the last section, difference-making pairs of configurations constitute the main inferential lever of CNA. Ideal data contain exactly those difference-making pairs that are characteristic for the underlying ground truth. How often factors take values above the 0.5-anchor or how frequently configurations are realized by cases is irrelevant for the difference-making evidence contained in ideal data and, correspondingly, for CNA's analysis of such data. If the data contain all and only the difference-making pairs compatible with a ground truth, the latter will always be recovered by CNA. A demonstration of this is provided in an R-script contained in the paper's supplemental online materials.</p> <p>The same cannot be expected for fragmented or noisy data. CNA has to fit its models to non-ideal data by lowering the thresholds on its fit measures of <emph>consistency</emph> and <emph>coverage</emph> ([<reflink idref="bib6" id="ref67">6</reflink>]), and these measures are sensitive to distributional imbalances generated by case frequencies. Frequency-induced imbalances may push consistency and coverage scores up or down in non-ideal data, thereby distort the signal in the data, and make it difficult to distinguish signal from noise. In the following simulation experiments we determine how data imbalances affect CNA's performance when analyzing non-ideal data.</p> <hd id="AN0184233960-7">Simulation Experiments</hd> <p>We run a series of simulation experiments benchmarking the performance of CNA with noisy or fragmented data featuring varying degrees of distributional imbalances. The experiments are designed as inverse search trials. That is, we first randomly draw ground truths, second, simulate data from these ground truths featuring systematically varied membership ratios in all possible combinations with noise or fragmentation—which we hold constant in most of the trials—, third, analyze the data with CNA, and fourth, measure how frequently the ground truths (or proper parts thereof) are contained in CNA's output. We use the implementation of CNA in the R-libraries <bold>cna</bold> and <bold>frscore</bold> ([<reflink idref="bib1" id="ref68">1</reflink>]; [<reflink idref="bib19" id="ref69">19</reflink>]). The code of the test series is available in the paper's supplemental online materials.</p> <hd id="AN0184233960-8">Test Setup and Data Simulation</hd> <p>The series consists of four experiments, which differ in the investigated data characteristics. The data analyzed in all experiments are simulated from a stock of 1,000 ground truths <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mn>1</mn></msub></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mn>1000</mn></msub></math> </ephtml> , randomly drawn from the factor set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>2</mn></msub><mo>=</mo><mo fence="false" stretchy="false">{</mo><mtext>A, B, C, D, E, F</mtext><mo fence="false" stretchy="false">}</mo></math> </ephtml> . As the execution time of the CNA algorithm increases, on average, with the complexity of the models to be built and as we process a total of over 100,000 data sets in the whole series, we have to restrict the maximal complexity of the ground truths <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . Hence, our <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> have one outcome only and a maximum of three alternative paths (i.e., disjuncts), with a maximum of three causes on each path (i.e., conjuncts), producing the outcome. While many real-life CNA applications actually target causal structures within that complexity range, it must also be emphasized that this restriction has consequences for our experiments. Ground truths drawn within that complexity range tend to have endogenous factors with slightly structure-induced imbalances. More precisely, the average membership ratio in the outcome in ground truths satisfying our complexity restriction is about 0.4. How this affects our findings will be discussed in the results section. Finally, to test how frequently data imbalances induce CNA to erroneously include causally irrelevant factor values (i.e., non-causes) in its models, we ensure that, in all <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , there is at least one element of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>2</mn></msub></math> </ephtml> missing, values of which, thus, are causally irrelevant.</p> <p>The first step of the data simulation process then is the same in all four experiments: For every <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , we generate ideal fuzzy-set data on the factors in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow><mn>2</mn></msub></math> </ephtml> , with a sample size of 50 cases each, yielding 1,000 ideal data sets <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mn>1</mn><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mn>1000</mn><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> . For every <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> it holds that the left- and right-hand sides of the MINUS-formula corresponding to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> have identical membership scores in each row of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> and that all configurations compatible with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> are represented by at least one case in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> . In the second and third step, we add noise or fragmentation to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> (see Table 3). In experiment 1, we add fragmentation but no noise, in experiment 2, we add noise but no fragmentation, and in experiments 3 and 4 we add both fragmentation and noise. Contrary to experiments 1, 2, and 3, where fragmentation is kept constant at the expense of varying sample sizes, we keep sample sizes constant in experiment 4 and allow fragmentation to vary.</p> <p>Table 3. Overview of investigated data characteristics in experiments 1-4.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="left">Experiment 1</th><th align="left">Experiment 2</th><th align="left">Experiment 3</th><th align="left">Experiment 4</th></tr></thead><tbody><tr><td>Resulting mean <italic>noise</italic><italic>ratio</italic> randomly introduced from <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false" xmlns="">[</mo><mn xmlns="">0</mn><mo xmlns="">,</mo><mn xmlns="">0.3</mn><mo stretchy="false" xmlns="">]</mo></math></p>9</td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.15</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.15</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.15</mn></math></p></td></tr><tr><td>Resulting mean <italic>fragmentation ratio</italic> randomly introduced from <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false" xmlns="">[</mo><mn xmlns="">0.2</mn><mo xmlns="">,</mo><mn xmlns="">0.5</mn><mo stretchy="false" xmlns="">]</mo></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.35</mn></math></p></td><td>0</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.35</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">0.21</mn><mo xmlns="">−</mo><mn xmlns="">0.41</mn></math></p></td></tr><tr><td>Resulting mean <italic>sample size</italic></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">23</mn><mo xmlns="">−</mo><mn xmlns="">123</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">53</mn><mo xmlns="">−</mo><mn xmlns="">282</mn></math></p></td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn xmlns="">36</mn><mo xmlns="">−</mo><mn xmlns="">182</mn></math></p></td><td>57</td></tr><tr><td>Manipulated <italic>membership ratios (MR)</italic></td><td colspan="4"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">M</mi><mi xmlns="">R</mi><mo stretchy="false" xmlns="">(</mo><mi xmlns="">O</mi><mi xmlns="">U</mi><mi xmlns="">T</mi><mo stretchy="false" xmlns="">)</mo></math></p>, <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">M</mi><mi xmlns="">R</mi><mo stretchy="false" xmlns="">(</mo><mi xmlns="">C</mi><mi xmlns="">A</mi><mi xmlns="">U</mi><mo stretchy="false" xmlns="">)</mo></math></p>, <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">M</mi><mi xmlns="">R</mi><mo stretchy="false" xmlns="">(</mo><mi xmlns="">n</mi><mi xmlns="">o</mi><mi xmlns="">n</mi><mi xmlns="">C</mi><mi xmlns="">A</mi><mi xmlns="">U</mi><mo stretchy="false" xmlns="">)</mo></math></p> varied to each value in <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo fence="false" stretchy="false" xmlns="">⟨</mo><mn xmlns="">0.1</mn><mo xmlns="">,</mo><mn xmlns="">0.2</mn><mo xmlns="">,</mo><mn xmlns="">0.3</mn><mo xmlns="">,</mo><mn xmlns="">0.4</mn><mo xmlns="">,</mo><mn xmlns="">0.5</mn><mo xmlns="">,</mo><mn xmlns="">0.6</mn><mo xmlns="">,</mo><mn xmlns="">0.7</mn><mo xmlns="">,</mo><mn xmlns="">0.8</mn><mo xmlns="">,</mo><mn xmlns="">0.9</mn><mo fence="false" stretchy="false" xmlns="">⟩</mo></math></p></td></tr></tbody></table> </ephtml> </p> <p>3 The first two rows indicate if noise or fragmentation was added. In experiments 1-3, fragmentation is kept constant at the expense of varying sample sizes. In experiment 4, sample size is kept constant at the expense of varying fragmentation ratios.</p> <p>Whenever noise or fragmentation are introduced, this is done at random. To randomly introduce noise into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> , we first draw a number <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>γ</mi></math> </ephtml> from the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0</mn><mo>,</mo><mn>0.3</mn><mo stretchy="false">]</mo></math> </ephtml> . Second, we draw a sequence <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>ϵ</mi></math> </ephtml> of normally distributed random errors from the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mo>−</mo><mn>1</mn><mo>,</mo><mn>1</mn><mo stretchy="false">]</mo></math> </ephtml> with a length equal to the number of rows of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> such that, over all rows of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> , the mean absolute difference between the scores of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub><mo stretchy="false">)</mo></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mi>h</mi><mi>s</mi><mo stretchy="false">(</mo><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mi>ϵ</mi></math> </ephtml> is equal to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>γ</mi></math> </ephtml> . Third, we replace the outcome value in every row <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math> </ephtml> of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> by the sum of that outcome value and the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>j</mi><mrow><mi>t</mi><mi>h</mi></mrow></msup></math> </ephtml> element of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>ϵ</mi></math> </ephtml> . The resulting data have a noise ratio equal to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>γ</mi></math> </ephtml> , meaning anywhere between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.3</mn></math> </ephtml> . To randomly fragment a data set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> , we draw a ratio from the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.5</mn><mo>,</mo><mn>0.8</mn><mo stretchy="false">]</mo></math> </ephtml> and sample that ratio of configurations from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> (without replacement). The resulting data have a fragmentation ratio anywhere between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.2</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> . The upshot of introducing noise or fragmentation into each <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mrow><mi>i</mi><mi>d</mi></mrow></msubsup></math> </ephtml> in accordance with the requirements of the different experiments are <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>4</mn><mo>×</mo><mn>1</mn><mo>,</mo><mn>000</mn></math> </ephtml><emph>non-ideal</emph> base data sets of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> , where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math> </ephtml> refers to the experiment and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> numbers the data set. For example, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mn>543</mn><mn>2</mn></msubsup></math> </ephtml> designates the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mn>543</mn><mrow><mi>r</mi><mi>d</mi></mrow></msup></math> </ephtml> base data for experiment 2.</p> <p>In these base data sets, we then, in the fourth step, systematically manipulate case frequencies in order to modify selected membership ratios such that these ratios are transformed to each value in the following <emph>variation sequence</emph>: <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mo fence="false" stretchy="false">⟨</mo><mn>0.1</mn><mo>,</mo><mn>0.2</mn><mo>,</mo><mn>0.3</mn><mo>,</mo><mn>0.4</mn><mo>,</mo><mn>0.5</mn><mo>,</mo><mn>0.6</mn><mo>,</mo><mn>0.7</mn><mo>,</mo><mn>0.8</mn><mo>,</mo><mn>0.9</mn><mo fence="false" stretchy="false">⟩</mo><mo>.</mo><mrow><msup><mrow /><mn>1</mn></msup></mrow><mrow><msup><mrow /><mn>0</mn></msup></mrow></math> </ephtml></p> <p>Graph [<reflink idref="bib14" id="ref70">14</reflink>]</p> <p>This is done in three different legs of each experiment. In the first leg, case frequencies in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> are manipulated such that the membership ratio in the <emph>outcome (OUT)</emph> is transformed to each value of the variation sequence. In the second leg, a <emph>cause (CAU)</emph> is randomly selected in each <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> and frequencies in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> manipulated such that the membership ratio in that cause assumes all values in the variation sequence. In the third leg, a <emph>non-cause (nonCAU)</emph> is selected in each <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> and its membership ratio correspondingly transformed. That is, in every leg of an experiment, 9 (length of the variation sequence) frequency-manipulated data sets are built from every base set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> . As there are three legs in each of the four experiments, we end up with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>9</mn><mo>×</mo><mn>3</mn><mo>×</mo><mn>4</mn><mo>×</mo><mn>1</mn><mo>,</mo><mn>000</mn><mo>=</mo><mn>108</mn><mo>,</mo><mn>000</mn></math> </ephtml><emph>test data sets</emph> for the whole series. We will refer to them by <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> , where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math> </ephtml> indicates the experiment, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>r</mi></math> </ephtml> the targeted membership ratio, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>t</mi></math> </ephtml> the leg of the experiment, and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math> </ephtml> numbers the data. For example, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mn>3</mn><mo>/</mo><mn>34</mn></mrow><mrow><mn>2</mn><mo>/</mo><mn>0.4</mn></mrow></msubsup></math> </ephtml> designates the 34 <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow /><mrow><mi>t</mi><mi>h</mi></mrow></msup></math> </ephtml> data set in the 3 <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow /><mrow><mi>r</mi><mi>d</mi></mrow></msup></math> </ephtml> leg of experiment 2, in which the membership ratio in a non-cause is transformed to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.4</mn></math> </ephtml> .</p> <p>As these data transformations are done by changing case frequencies in the base data, all resulting membership ratios are frequency-induced. In experiments 1–3, case frequencies are modified by suitably selecting cases from the base <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> in such a way that the fragmentation and noise ratios of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> are retained (as closely as possible) in the transformed data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> . Depending on what the initial membership ratio is in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> , this selection process may cause <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> to have a much larger sample size than <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> . Also, the sample sizes of the test data vary greatly within each leg of an experiment. On average, test sets at the lower and upper ends of the variation sequence have much larger sample sizes than test sets targeting membership ratios around <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> .</p> <p>As sample sizes may influence the performance of CNA, experiment 4 varies membership ratios in such a way that, apart from noise ratios, also sample sizes are held constant across all data transformations. Selecting cases from the base set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> in such a way that noise and sample size stay the same can only be accomplished at the expense of varying fragmentation. That is, the test data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> may have much higher fragmentation than the corresponding base <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mi>i</mi><mi>k</mi></msubsup></math> </ephtml> and fragmentation also varies within each leg of an experiment. Test sets targeting membership ratios around <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> tend to have lower fragmentation than sets aiming for extreme ratios. Table 3 summarizes the settings for the four experiments.</p> <hd id="AN0184233960-9">Data Analysis and Benchmark Criteria</hd> <p>The 108,000 test data sets are analyzed by CNA using the robustness analysis protocol developed by [<reflink idref="bib18" id="ref71">18</reflink>]. That means that each <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> is not only analyzed at one designated tuning setting of consistency and coverage thresholds but re-analyzed at all settings in a whole sequence of consistency and coverage thresholds. For our analysis we choose the sequence <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo fence="false" stretchy="false">⟨</mo><mn>0.65</mn><mo>,</mo><mn>0.70</mn><mo>,</mo><mn>0.75</mn><mo>,</mo><mo>...</mo><mo>,</mo><mn>1</mn><mo fence="false" stretchy="false">⟩</mo></math> </ephtml> . All MINUS-formulas CNA recovers in that re-analysis series are collected and their robustness and overall model fit measured and scored. For every <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> , we then return the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>95</mn></math> </ephtml><sups><emph>th</emph></sups> percentile of top-performing MINUS-formulas as CNA's output set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">S</mi></mrow></math> </ephtml> for that data.</p> <p>The elements of such a set, which contains between 1 and 6 models in our test series, are indistinguishable on the basis of the evidence contained in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> by current model selection standards used in CNA. Accordingly, if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">S</mi></mrow></math> </ephtml> comprises more than one MINUS-formula, CNA cannot determine which of those formulas truthfully represents the ground truth <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> ; all that can be said is that at least one of them is true of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . It follows that a set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">S</mi></mrow></math> </ephtml> featuring, say, three MINUS-formulas <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>1</mn></msub></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>2</mn></msub></math> </ephtml> , and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>3</mn></msub></math> </ephtml> is to be causally interpreted disjunctively: <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>1</mn></msub><mspace width=".1em" /></math> </ephtml> OR <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mspace width=".1em" /><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>2</mn></msub><mspace width=".1em" /></math> </ephtml> OR <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mspace width=".1em" /><msub><mrow><mi mathvariant="bold">m</mi></mrow><mn>3</mn></msub></math> </ephtml> is true of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> .[<reflink idref="bib16" id="ref72">16</reflink>] If CNA returns multiple models in a real-life discovery context, an analyst has to rely on data-external sources of information as theoretical background knowledge to select among the candidates.[<reflink idref="bib17" id="ref73">17</reflink>] As such a data-external background is not available for simulated data, we take the set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">S</mi></mrow></math> </ephtml> inferred from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> to be CNA's final output for these data. All in all, analyzing the data of our entire test series yields 108,000 output sets <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> .</p> <p>We assess the quality of all those output sets based on three complimentary benchmark criteria, which Table 4 summarizes. The first is a qualitative <emph>correctness</emph> criterion, which has been repeatedly used in CCM benchmarking before (e.g., [<reflink idref="bib6" id="ref74">6</reflink>]; [<reflink idref="bib3" id="ref75">3</reflink>]; [<reflink idref="bib4" id="ref76">4</reflink>]).</p> <p>Table 4. Overview of used benchmark criteria with example formulas that (do not) pass respective criteria assuming that (<reflink idref="bib1" id="ref77">1</reflink>), i.e. <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>b</mi><mo>+</mo><mi>c</mi><mo>*</mo><mi>D</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> , is the ground truth.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left">Criterion</th><th align="center">Substance</th><th align="center">Output</th><th align="center">Examples based on (1)</th></tr></thead><tbody><tr><td><italic>Correctness</italic></td><td>non-empty output without false positives</td><td>(not) passed</td><td>passed: <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">*</mo><mi xmlns="">b</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>; <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">+</mo><mi xmlns="">D</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>not passed: <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">*</mo><mi xmlns="">B</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>; <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">+</mo><mi xmlns="">b</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p></td></tr><tr><td><italic>Completness</italic></td><td>degree of ground truth captured</td><td>passed to degree</td><td><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">*</mo><mi xmlns="">b</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>; <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">+</mo><mi xmlns="">D</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>both score <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mn>2</mn><mo>/</mo><mn>4</mn></mrow><mo xmlns="">=</mo><mn xmlns="">0.5</mn></math></p></td></tr><tr><td><italic>Error-freeness</italic></td><td>no false positives or empty output</td><td>(not) passed</td><td>passed: <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">*</mo><mi xmlns="">b</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p>; <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal" xmlns="">∅</mi></math></p>not passed: <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">A</mi><mo xmlns="">*</mo><mi xmlns="">B</mi><mo stretchy="false" xmlns="">↔</mo><mi xmlns="">E</mi></math></p></td></tr></tbody></table> </ephtml> </p> <p>According to that criterion, what CNA infers from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> counts as correct if, and only if, that inference is true of the underlying ground truth <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . As we have seen above, that is the case if, and only if, at least one MINUS-formula <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>j</mi></msub></math> </ephtml> in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> is true of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , which, in turn, holds if, and only if, all factor values contained in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>j</mi></msub></math> </ephtml> are in fact causes of the outcome of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> and all conjunctive and disjunctive groupings in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>j</mi></msub></math> </ephtml> are in agreement with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . In other words, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>j</mi></msub></math> </ephtml> is correct if, and only if, it entails no false positives.[<reflink idref="bib18" id="ref78">18</reflink>] For example, if formula (<reflink idref="bib1" id="ref79">1</reflink>) from our previous example, i.e. <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>b</mi><mo>+</mo><mi>c</mi><mo>*</mo><mi>D</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> , is the ground truth, models as <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>b</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>+</mo><mi>D</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> are correct because all factor values contained in these models are in fact causes of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi></math> </ephtml> and all conjunctive and disjunctive groupings are true of (<reflink idref="bib1" id="ref80">1</reflink>). By contrast, a model as <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>B</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> is incorrect because <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>B</mi></math> </ephtml> is not in fact a cause of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>E</mi></math> </ephtml> , or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>+</mo><mi>b</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> is incorrect because <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math> </ephtml> are conjunctively and not disjunctively grouped in (<reflink idref="bib1" id="ref81">1</reflink>). If CNA does not infer anything from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> and, thus, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> is empty—say, because consistency or coverage thresholds cannot be met— <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> does not pass the correctness benchmark.</p> <p>The second benchmark is a <emph>completeness</emph> criterion that quantifies the informativeness of correct MINUS-formulas. Making only true claims about <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , as is required to pass the correctness benchmark, can be easily accomplished by models that make only very few causal claims. Also, of two correct MINUS-formulas one can be more complex than the other and, hence, reveal <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> more completely. As more informative models are preferable, the completeness benchmark measures the degree to which the correct MINUS-formulas in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> exhaustively reveal <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . More specifically, completeness amounts to the ratio of the complexity of the most complex correct MINUS-formula in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> to the complexity of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , where complexity of a MINUS-formula <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>i</mi></msub></math> </ephtml> is understood as the number of factor values in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>i</mi></msub></math> </ephtml> . That is, contrary to correctness, which can only be either satisfied or not, the second benchmark can be passed by degree. For example, if (<reflink idref="bib1" id="ref82">1</reflink>) is the ground truth, models as <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>*</mo><mi>b</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi><mo>+</mo><mi>D</mi><mo stretchy="false">↔</mo><mi>E</mi></math> </ephtml> score <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>2</mn><mo>/</mo><mn>4</mn></mrow><mo>=</mo><mn>0.5</mn></math> </ephtml> on completeness, as they recover two of the four factor values contained in the ground truth. When <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> is either empty or does not contain a correct MINUS-formula, completeness is 0 by default.</p> <p>As a supplement, we measure a third auxiliary criterion, <emph>error-freeness</emph>, that is also qualitative and counts as passed if, and only if, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> as a whole is not false. Contrary to correctness and completeness, error-freeness is non-zero both if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> contains a MINUS-formula <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>i</mi></msub></math> </ephtml> that does not entail a false positive and if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> is empty. As an empty output set is uninformative and thus suboptimal, error-freeness is not a benchmark on a par with correctness and completeness, and our subsequent discussion will mainly focus on the latter two benchmarks. Nonetheless, error-freeness deserves some attention because an empty output is still preferable over a false one.</p> <hd id="AN0184233960-10">Results</hd> <p>The results of experiments 1 and 2 are plotted in the bar-charts in Figure 1, Figure 2 shows the results of experiments 3 and 4. Black bars represent correctness scores, dark gray bars completeness scores, and light gray bars depict error-freeness scores. The effects of varying membership ratios in an outcome, a cause, and a non-cause are presented in separate panels. The exact values of all scores can be found in the score tables in the paper's supplemental online materials. Apart from the benchmark scores, the plots also display fragmentation and noise ratios as well as sample sizes. All values are means over 1,000 CNA analyses of 1,000 test data <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mi>t</mi><mo>/</mo><mi>i</mi></mrow><mrow><mi>k</mi><mo>/</mo><mi>r</mi></mrow></msubsup></math> </ephtml> . For example, the correctness score of 0.99 depicted by the first (black) bar in the leftmost panel in the plot of experiment 1 in Figure 1 means that CNA found a correct MINUS-formula for 99% of the 1,000 test data of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mn>1</mn><mo>/</mo><mi>i</mi></mrow><mrow><mn>1</mn><mo>/</mo><mn>0.1</mn></mrow></msubsup></math> </ephtml> . The fragmentation score of 0.35 superimposed over that bar means that, on average, 35% of rows compatible with the corresponding ground truths are missing from the 1,000 <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>δ</mi><mrow><mn>1</mn><mo>/</mo><mi>i</mi></mrow><mrow><mn>1</mn><mo>/</mo><mn>0.1</mn></mrow></msubsup></math> </ephtml> .[<reflink idref="bib19" id="ref83">19</reflink>]</p> <p>Graph: Figure 1. Results of experiments 1 and 2, subdivided by their three legs. Membership ratios are plotted on the x-axis, benchmark scores (black and shades of gray), noise (red), and fragmentation ratios (orange) are on the left y-axis, sample sizes (blue) on the right y-axis. All values are means over 1,000 CNA runs.</p> <p>Graph: Figure 2. Results of experiments 3 and 4, subdivided by their three legs. Membership ratios are plotted on the x-axis, benchmark scores (black and shades of gray), noise (red), and fragmentation ratios (orange) are on the left y-axis, sample sizes (blue) on the right y-axis. All values are means over 1,000 CNA runs.</p> <p>First and foremost, our results demonstrate, that increasing distributional imbalances may be associated with decreasing performance even when fragmentation, noise, or sample size are kept constant. This, in turn, shows that data imbalances do not only exacerbate the negative effects of other data deficiencies but also have an independent negative effect. In what follows, we break this main finding down in more detail.</p> <hd id="AN0184233960-11">Fragmentation vs. Noise</hd> <p>Data imbalances have weaker effects when paired only with fragmentation than when paired with noise. In all three legs of experiment 1, where fragmented but noise-free data are processed, correctness scores remain almost maximal (between 0.98 and 1). Maximal completeness, however, cannot be achieved because of the fragmentation, which amounts to missing empirical information about ground truths <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . In legs 1 and 3, varying membership ratios only barely affect completeness scores (which remain between 0.85 and 0.90) beyond the impact of fragmentation. In leg 2, distributional imbalances drag completeness down noticeably. If <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo></math> </ephtml> is set to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.1</mn></math> </ephtml> or <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.2</mn></math> </ephtml> , the cause is so rare that it is no longer needed to cover the outcome and thus is not built into the models. By contrast, in the trials with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.8</mn><mo>/</mo><mn>0.9</mn></math> </ephtml> , the cause is so frequent that it covers the outcome even without other causes, which therefore become redundant and are not built into the models. In both scenarios, some causes are missing from CNA's models in addition to those that are missing due to fragmentation alone.</p> <p>In experiments 2–4, where data feature noise, both correctness and completeness scores drop significantly compared to experiment 1 because CNA cannot persistently avoid false positives in the presence of noise, which, after all, amounts to incorrect information about <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . Moreover, in trials when all models in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> make false claims about <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> , both correctness and completeness are 0. Completeness drops more than correctness because CNA is designed to keep false positives to a minimum, meaning that the method abstains from including a factor value in a model, if its causal relevance is not sufficiently corroborated by the data. The more cautiously a method operates, the less causal inferences it draws, the lower the chances that false positives are committed, yet the less completely <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> is recovered.</p> <p>Owing to its cautiousness, CNA frequently abstains from drawing any inferences when the outcome is very rare in experiments 2–4. The result are empty output sets <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> in up to 25% of the trials, which can be read off the large difference in correctness and error-freeness scores at the lower end of the variation sequence. Error-free trials that do not pass correctness are trials with empty outputs. In consequence, error-freeness scores remain above 0.75 in all experiments even when the outcome is very rare. By contrast, when <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> , there are no longer any empty outputs, to the effect that error-freeness and correctness coincide and drop well below 0.75. The reason is overfitting. Due to the noise, conjunctions of only one or two factor values are not consistently sufficient for the outcome, such that CNA builds rather complex minimally sufficient conditions, on average. And in order to cover a very frequent outcome with complex conditions, large disjunctions of many of these conditions are needed. In 35%–40% of the trials on noisy data at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> , CNA's output sets contain only overfitted models.</p> <hd id="AN0184233960-12">Performance Peaks</hd> <p>Another feature of the results obtained in the first leg of experiments 2–4 is that, across all variations of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo></math> </ephtml> , CNA scores highest on completeness at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.4</mn></math> </ephtml> and highest on correctness at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.6</mn></math> </ephtml> . Why are these performance peaks off-centered? To answer this question recall from the test setup that outcomes have an average structure-induced membership ratio of roughly <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.4</mn></math> </ephtml> in the ground truths <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mn>1</mn></msub></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mn>1000</mn></msub></math> </ephtml> , which, in turn, stems from the complexity restriction imposed on them. Deviations from structure-induced membership ratios in our experiments are due to biased case frequencies and hence tend not to be faithful to the structural properties of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . The more we increase that frequency bias as we move through the variation sequence, the higher the chances that CNA builds elements into its models that fail to have counterparts in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . The complex models contained in output sets <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> have the highest chances of being true of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> in trials at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.4</mn></math> </ephtml> because, in these trials, case frequencies are manipulated the least, on average, meaning that membership ratios are most faithful to the structural properties of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . The higher the chances that complex CNA models in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">S</mi></mrow><mi>i</mi></msub></math> </ephtml> are true, the higher CNA's completeness scores.</p> <p>But then, why are the correctness scores, even though they are good at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.4</mn></math> </ephtml> (i.e., between 0.85 and 0.88), not the highest in those trials as well? To understand that, recall that we analyze each data set at a whole range of threshold combinations, going down to consistency and coverage set to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.65</mn></math> </ephtml> . The lower these thresholds, the less accurately models are required to account for the outcome, the higher the chances that very sparse models pass the thresholds. And the sparser a MINUS-formula <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>i</mi></msub></math> </ephtml> , the less causal claims it makes, which, in return, increases the probability that <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>i</mi></msub></math> </ephtml> does not make any false claims and, thus, is true of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> . The sparsest possible MINUS-formulas are <emph>one-cause formulas</emph> of type <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>B</mi><mo stretchy="false">↔</mo><mi>A</mi></math> </ephtml> . In our test series, CNA's output sets contain the most one-cause formulas at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.6</mn></math> </ephtml> . The fact that the MINUS-formulas with the highest a priori probability of being true are the most frequent in the trials at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.6</mn></math> </ephtml> pushes CNA's correctness score even higher than it is at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.4</mn></math> </ephtml> (i.e., to scores between 0.98 and 0.99).</p> <hd id="AN0184233960-13">Outcome vs. Causes vs. Non-Causes</hd> <p>Finally, our results show that extreme membership ratios have varying performance impacts depending on whether they affect the outcome, a cause, or a non-cause. In all experiments, extreme membership ratios in non-causes have no significant effect on performance beyond fragmentation and noise. That means they do not induce CNA to erroneously include non-causes in models more frequently than they are included because of other data deficiencies. In contrast, extreme membership ratios in causes have sizeable effects on completeness scores in all experiments. The same mechanism as in experiment 1 accounts for this finding in all other experiments. That is, rare causes are often not included in models and frequent causes tend to render other causes redundant, which then are not included. In addition, extreme membership ratios in causes, when combined with noise (i.e., in experiments 2–4), induce a noticeable drop in correctness. Still, correctness scores do not fall below 0.79 and correctness drops are counterbalanced by error-freeness scores well above 0.8, meaning that CNA repeatedly issues no models at all.</p> <p>In general, when data feature noise (i.e., in experiments 2–4) the performance impact is much higher if the endogenous factors are imbalanced than if exogenous factors are concerned. The reason is that while every <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> has only one outcome, most <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> have several causes, yet only the frequency of one of these is manipulated in our experiments. Thus, whereas frequency distortion in one cause can be counterbalanced by correctly inferred causal claims on other causes, this is not possible with frequency distortion in the outcome which then tends to hinder the correct recovery of the whole <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> .</p> <hd id="AN0184233960-14">Discussion</hd> <p>We set out to answer the question whether data imbalances have their own impact on CNA's performance. In the case of ideal data, even extreme imbalances have no such impact. However, we have seen that in case of non-ideal data, which are common in real-life research settings, extreme membership ratios affect CNA's performance. It remains to be determined, first, which membership ratios should count as problematic for CNA, second, what counter-measures can be taken to resolve problematic distributional imbalances, and, third, what our study's limitations are.</p> <hd id="AN0184233960-15">Demarcating Problematic Membership Ratios</hd> <p>In light of our results, the answer to the first question depends on an array of conditions such as the quality of the data at hand, the type of factor(s) with extreme imbalances, whether the analysts are primarily interested in correct or complete models, and how willing they are to take a risk. Accordingly, there does not exist a general and objective demarcation between problematic and unproblematic membership ratios. In what follows, we determine whether a performance drop in a particular benchmark measure within a leg of an experiment is problematic by comparing it to the best benchmark score in that leg. In order for a drop to count as problematic we require the difference to the best performance to be higher than 20%. Readers with a different assessment of what counts as problematic have to correspondingly adjust our subsequent demarcations.</p> <p>If it can be plausibly assumed that the data are collected against a homogeneous background, such that noise is negligible and the only serious data deficiency is fragmentation, distributional imbalances tend not to affect performance beyond fragmentation. The only exception, as the second leg of experiment 1 shows, is that causes with extreme membership ratios noticeably reduce the completeness of CNA's models. At <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.5</mn></math> </ephtml> , CNA recovers 91% of the ground truths, on average, whereas this percentage drops to 76% at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn></math> </ephtml> and to 73% at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> , which amount to performance drops of 16.5% and 19.8%, respectively. Although the latter drop borders on the problem zone identified above, the fact that the overall completeness scores remain high throughout the second leg of experiment 1 lets us confidently conclude that these completeness drops, though sizeable, are not to be considered problematic.</p> <p>This changes when the data are noisy. In the second leg of experiment 2, the best completeness score is 0.55 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.5</mn></math> </ephtml> ; it drops by more than 20% at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>⩾</mo><mn>0.8</mn></math> </ephtml> , and similarly in the second legs of experiments 3 and 4. Hence, in the presence of noise, membership ratios of causes outside of the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.2</mn><mo>,</mo><mn>0.7</mn><mo stretchy="false">]</mo></math> </ephtml> drag down completeness to a problematic extent. It follows that somebody with an interest in learning as much as possible about the ground truth should consider countermeasures.</p> <p>The same does not hold if analysts are primarily interested in correct models, that is, in reliably finding parts of the data-generating structure. While <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> has no effect on correctness at all, correctness drops from a solid score of 0.9 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.5</mn><mo>/</mo><mn>0.6</mn></math> </ephtml> by about 12% to 0.79 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn></math> </ephtml> when the data are noisy. But at the same time, error-freeness remains between 0.83 and 0.85, meaning that the loss in correctness is to a substantive degree due to empty outputs. That is, there is a slightly increased risk of inferring something false at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>C</mi><mi>A</mi><mi>U</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn></math> </ephtml> , but that increase is not severe enough to call for countermeasures against distributional imbalances; if an error risk of 15% is considered too high, taking measures against the noise in the data would be more effective.</p> <p>The most problematic impact extreme membership ratios have on the correctness of CNA's output occurs when endogenous factors are imbalanced in noisy data. In experiments 2–4, correctness and error-freeness collapse at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> . While correctness and error-freeness are between 0.84 and 0.87 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.8</mn></math> </ephtml> in the experiments with noise, these scores drop to values between 0.6 and 0.65 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> . Compared to the best correctness scores at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.6</mn></math> </ephtml> , that is a performance drop of almost 40%, down to a level where, in 2 out of 5 CNA runs, all resulting models are fallacious. That is unquestionably an instance of a problematic distributional imbalance.</p> <p>The situation is not so clear in case of low membership ratios in outcomes. At <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn><mo>/</mo><mn>0.2</mn></math> </ephtml> , CNA only recovers a true model in 58%–67% of the trials, but these low correctness scores are largely due to the fact that CNA often abstains from inferring any models at all. When conditionalized on the trials that produce non-empty outputs, correctness does not fall below 0.75 at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn><mo>/</mo><mn>0.2</mn></math> </ephtml> in any of our experiments. If a fallacy risk of 25%—in contexts where up to 30% of the observations are distorted by noise—is acceptable to the analyst, even an extremely rare outcome hence does not call for immediate countermeasures. Instead, CNA could be run to see if an output is produced. If that is not the case, the low membership ratio in the outcome is the likely source of the problem, making it an instance of a problematic imbalance after all.</p> <p>Finally, at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.1</mn></math> </ephtml> the completeness of the output CNA infers from noisy data is reduced by about 25%, which is deep within the problem zone. Similarly, high outcome membership ratios are very consequential for the completeness of CNA's output in the presence of noise. At <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.7</mn></math> </ephtml> completeness drops between 25% and 30% compared to the optimal membership ratio; at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.8</mn></math> </ephtml> completeness is cut in half, and at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> two thirds of the information CNA recovers at the optimal membership ratio are lost, on average. Hence, if maximal informativeness is a research objective and the data cannot plausibly be assumed to be noise-free, membership ratios in the outcome outside of the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.2</mn><mo>,</mo><mn>0.6</mn><mo stretchy="false">]</mo></math> </ephtml> should be considered problematic.</p> <p>In sum, based on our convention that performance drops need to be larger than 20% to count as problematic, we propose the following demarcation lines for problematic data imbalances, assuming a typical real-life research with the following two characteristics: First, neither fragmentation nor noise can plausibly be excluded, and second, only outcomes and candidate causes can be distinguished, but the latter cannot be grouped into causes and non-causes. If the analyst is primarily interested in finding a correct model, only <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.9</mn></math> </ephtml> is problematic. Yet, if the research context requires learning as much as possible about the data-generating structure, membership ratios in outcomes outside of the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.2</mn><mo>,</mo><mn>0.6</mn><mo stretchy="false">]</mo></math> </ephtml> or membership ratios in candidate causes outside of the interval <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><mn>0.2</mn><mo>,</mo><mn>0.7</mn><mo stretchy="false">]</mo></math> </ephtml> are problematic.</p> <hd id="AN0184233960-16">Resolving Problematic Distributional Imbalances</hd> <p>When it comes to taking countermeasures, problematic distributional imbalances affecting the correctness of CNA's output must be distinguished from problematic imbalances affecting completeness. The former type requires countermeasures <emph>before</emph> CNA is applied to the data, whereas the latter type may be addressed <emph>after</emph> initial applications of CNA are found not to deliver satisfactory results. If an outcome is imbalanced at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>O</mi><mi>U</mi><mi>T</mi><mo stretchy="false">)</mo><mo>≥</mo><mn>0.9</mn></math> </ephtml> , CNA's output cannot be trusted in non-ideal discovery circumstances, making it imperative to take immediate action. By contrast, if the data are imbalanced in a way that does not create problems for correctness but only for completeness, say, an outcome has a membership ratio of 0.2 or 0.1, the analyst may well run an analysis and inspect the results before any action is taken. Our simulations indicate that the likelihood of not receiving any model are high. But if a non-empty output is produced under such circumstances, it may be given consideration. In fact, we have seen that such outputs are correct in 80% of the trials. Thus, if these outputs are informative enough for the research question at hand and analysts are ready to accept a fallacy risk of 20%, they can take such outputs seriously without addressing distributional imbalances. Yet, if it turns out that CNA does not return any models or that returned models are not informative enough, resolving distributional imbalances will be a promising path forward.</p> <p>There are three main approaches to resolve distributional imbalances: (<reflink idref="bib1" id="ref84">1</reflink>) adding or removing cases, (<reflink idref="bib2" id="ref85">2</reflink>) adjusting membership scores via recalibration, and (<reflink idref="bib3" id="ref86">3</reflink>) negating values of imbalanced factors. We take them in turn. Approach (<reflink idref="bib1" id="ref87">1</reflink>) consists in suitably changing the sample of cases in the data. To resolve a problematically high membership ratio in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> , cases can be added to the data in which the factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> takes values below 0.5 or cases can be removed in which <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> takes values above <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> . Analogously, if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo></math> </ephtml> is problematically low, cases with membership scores above <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> can be added or cases with membership scores below <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.5</mn></math> </ephtml> removed. Both adding and removing cases require to reassess the case selection decisions in the study design. For instance, complementing the data in a study analyzing employees from a particular industry by cases featuring employees from another (comparable) industry shifts the study's analytical focus to employees in the union of both industries. Or, cases can be removed from that study by shifting the level of analysis from all employees in an industry to a particular group of employees from that industry, say, employees without leadership positions. Plainly, such case selection adjustments should not only be assessed based on their capacity to resolve problematic distributional imbalances but, in the first place, they must be theoretically meaningful and in line with the research interests at hand. Avoiding problematic imbalances is only one constraint among many to be taken into account when selecting cases. Moreover, note that after cases have been added or removed in order to resolve the problematic imbalance of some factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> , the distributions of <emph>all</emph> factors must be reassessed, not just of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> . The reason is that changing the data basis might resolve the distributional problem for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> but create it for another factor.</p> <p>Moreover, other constraints must be kept in mind when adding and removing cases to and from the data. First, added cases should have homogenous causal backgrounds and they should be comparable to the cases already contained in the data. For example, if a study affected by problematic imbalances is concerned with Western democracies, adding cases from the set of Asian autocracies will, in all likelihood, induce homogeneity violations and, thereby, render the resulting models uninterpretable. Second, removing cases should, if possible, not increase fragmentation, that is, it should not reduce the number of configurations instantiated in the data and thus reduce the amount of difference-making evidence. In other words, cases should primarily be removed that instantiate configurations which have various other cases instantiating them in the data.</p> <p>By contrast, when membership ratios are modified by recalibration in the vein of approach (<reflink idref="bib2" id="ref88">2</reflink>), the imbalance of one factor <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> can be tackled without affecting the distribution of the other factors. To this end, the number of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> 's values above and below the 0.5-anchor is changed by moving the cross-over calibration threshold defining the 0.5-anchor. If <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is too frequent, that threshold is moved up such that less cases are calibrated to instantiate <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> , whereas if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is too rare, the cross-over threshold is moved down such that more cases instantiate <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> .</p> <p>Note that recalibration also requires changes to the study design, as shifting calibration thresholds changes the subject matter of the analysis; it moves the analytical interest from a problematically imbalanced factor to a different, non-problematic one. Such adjustments must, of course, also be theoretically justified and in line with the study's research goals. To illustrate, shifting the subject matter via recalibration can be an option when the imbalanced factor represents a concept for which a bias towards higher or lower values that is not induced by the causal structure under investigation can be observed. A concept as self-perceived competence (or self-efficacy) is an example for which many studies observe a general positivity bias resulting in high membership ratios (e.g., [<reflink idref="bib25" id="ref89">25</reflink>]; [<reflink idref="bib11" id="ref90">11</reflink>]; [<reflink idref="bib23" id="ref91">23</reflink>]). If the aim of the study is not to investigate the causal structure underlying overly positive competence assessments, the bias can be counterbalanced by changing the subject matter of the analysis from, say, "competent employees" to "highly competent employees". In addition, it is clear that such recalibrations should also follow general calibration guidelines (e.g., [<reflink idref="bib17" id="ref92">17</reflink>]). Unproblematic distributions are merely one constraint among many to be considered in the calibration process.</p> <p>Finally, while it can happen that approaches (<reflink idref="bib1" id="ref93">1</reflink>) and (<reflink idref="bib2" id="ref94">2</reflink>) are inapplicable because there may not be justifiable ways of changing the data basis or the calibration thresholds, approach (<reflink idref="bib3" id="ref95">3</reflink>) is always applicable. It amounts to simply replacing an <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> with a problematic membership ratio by its negation <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi></math> </ephtml> , that is, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>1</mn><mo>−</mo><mi>X</mi></math> </ephtml> . To illustrate, consider a scenario where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is an outcome with a membership ratio of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.7</mn></math> </ephtml> and an initial CNA analysis produces a model that is not informative enough. According to our findings, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> 's membership ratio counts as problematic in that scenario. If <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> is now replaced by <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi></math> </ephtml> , the previously problematic membership ratio becomes a ratio of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>M</mi><mi>R</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.3</mn></math> </ephtml> (i.e., <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>1</mn><mo>−</mo><mn>0.7</mn></math> </ephtml> ), which our results show not to count as problematic. In fact, we have found that an outcome membership ratio of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.3</mn></math> </ephtml> is almost optimal for completeness maximization in the presence of fragmentation and noise.</p> <p>Not only is approach (<reflink idref="bib3" id="ref96">3</reflink>) always applicable, it also does not require intricate considerations on changing the data basis or the calibration. On the downside, however, it is not always possible to resolve problematic imbalances by simple negation. In particular, when both <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>1</mn><mo>−</mo><mi>X</mi></math> </ephtml> fall into ranges of problematic membership ratios, which, according to our findings, holds if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">X</mi></math> </ephtml> is an exogenous factor, approach (<reflink idref="bib3" id="ref97">3</reflink>) does not solve the problem. Also, while the adjustments of the study design induced by approaches (<reflink idref="bib1" id="ref98">1</reflink>) and (<reflink idref="bib2" id="ref99">2</reflink>) may be small, negating outcomes or causes amounts to reversing the subject matter of the study entirely. It shifts the focus from investigating the presence of factors to their absence.</p> <hd id="AN0184233960-17">Limitations</hd> <p>Our study's limitations originate from design decisions we had to take when simulating data. For reasons of computational feasibility, we had to restrict the complexity of the assumed ground truths to structures with one outcome and a complexity range of one to three disjuncts, each consisting of one to three conjuncts. Ground truths were randomly drawn from that complexity range to the effect that resulting ground truth complexities are normally distributed in that range. As pointed out before, the outcomes in those ground truths have mean structure-induced membership ratios of about <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mn>0.4</mn></math> </ephtml> , which, ultimately, off-centered the demarcation lines we found for problematic imbalances.</p> <p>While we are confident that the (unknown) ground truths in a majority of actual CNA applications fall into our complexity range, causal structures analyzed by CNA may, of course, have higher complexities, which moreover may not be normally distributed. We suspect that the intervals demarcating problematic distributional imbalances will be more centered, on average, if more complex grounds truths are taken into account, but our study provides no basis for making such a determination. And it is an open question what membership ratio intervals should count as problematic if the complexities of real-life causal structures are not normally distributed. At the same time, as ground truth complexities do not influence the distributions of mutually independent exogenous factors, we have every reason to expect that our findings on causes and non-causes can be generalized to single-outcome ground truths with higher complexities—even if the latter are not normally distributed.</p> <p>The same does not hold for ground truths with multiple outcomes. Although its capacity to analyze data stemming from multi-outcome structures is one of CNA's distinctive qualities, we could not integrate that additional layer of complexity into this study. Investigating problematic membership ratios for such structures, hence, remains an important open question.</p> <p>Furthermore, again for reasons of computational feasibility, we could not systematically vary fragmentation and noise in a controlled manner, even though it is very likely that demarcation lines for problematic data imbalances change with varying levels of fragmentation and noise. Also, fragmentation and noise may be biased in real-life research contexts, to the effect that imbalances affect CNA's performance in ways unforeseen by our analysis. Finally, when manipulating distributions of exogenous factors we did so for only one exogenous factor in each trial. But, of course, in real-life settings multiple exogenous factors may be imbalanced to varying degrees. It is to be expected that extreme imbalances in more than one such factor interact and negatively impact on performance much beyond the impact we found in our experiments. But quantifying that impact has to await another occasion.</p> <p>Finally, the reader shall be reminded that our criterion for demarcating problematic from unproblematic performance drops is negotiable. A reader with a different view of what counts as a problematic drop in performance is invited to correspondingly adjust the membership ratio intervals that call for measures against imbalances.</p> <hd id="AN0184233960-18">Outlook</hd> <p>This is the first study investigating how data imbalances affect the performance of CNA, in particular, and the first study quantifying that impact for a CCM, in general. Even though CCMs, contrary to many other methods, do not infer causation from distributional properties of the data but from difference-making pairs contained therein, our results show that extreme imbalances can affect both the correctness and the completeness of CNA's output. The reason, in a nutshell, is that extreme distributional imbalances induce CNA to mistake noise for signal. That mechanism remains underinvestigated despite our study. Further analyses are needed to determine how varying membership ratios impact on performance under the discovery circumstances we had to bracket for reasons of computational feasibility. And we submit that similar studies aiming at quantifying the performance impact are needed for other CCMs, such as QCA.</p> <p>Another avenue for future research derives from the fact that even extreme distributional imbalances do not negatively affect CNA's performance when applied to ideal data. We envisage that the closer analyzed data are to ideal data, the lower the negative impact of such imbalances. It follows that a technique estimating the closeness of the data to ideality would likewise provide an estimate for the severity of the performance impact to be expected from problematic distributional imbalances. Such a technique is lacking at the moment.</p> <p>In sum, our study shows that the problems posed by data imbalances are to be taken seriously—more seriously than they currently are—by both methodologists and applied researchers. The former need to address the many remaining questions surrounding distributional imbalances and CNA (or CCMs, more generally). The latter should learn to take analytical decisions, from the study design to the interpretation of the results, with an eye on distributional imbalances.</p> <hd id="AN0184233960-19">Acknowledgments</hd> <p>We thank the audiences at the 9 <emph>th</emph> International QCA Expert Workshop in Zürich and the 1 <emph>st</emph> International Conference on Current Issues in Coincidence Analysis in Bergen, where earlier versions of this article were presented, for valuable exchanges. We also thank Judith Glaesser, Reiping Huang, and Sebastian Klein, as well as two anonymous reviewers for helpful comments and suggestions.</p> <ref id="AN0184233960-20"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref30" type="bt">1</bibl> <bibtext> The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref34" type="bt">2</bibl> <bibtext> We gratefully acknowledge the support by the Toppforsk program of the University of Bergen, co-financed by the Trond Mohn Foundation under grant number 811886.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref75" type="bt">3</bibl> <bibtext> Martyna D. Swiatczak https://orcid.org/0000-0002-7537-1813 Michael Baumgartner https://orcid.org/0000-0003-1536-2816</bibtext> </blist> <blist> <bibl id="bib4" idref="ref59" type="bt">4</bibl> <bibtext> The datasets generated during this study are available or can be replicated via the replication materials in the OSF repository ([27]).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref20" type="bt">5</bibl> <bibtext> Supplemental and replication materials for this article are available in the OSF repository ([27]).</bibtext> </blist> <blist> <bibl id="bib6" idref="ref1" type="bt">6</bibl> <bibtext> "INUS" originally is an acronym referring to <emph>I</emph>nsufficient but <emph>N</emph>on-redundant parts of <emph>U</emph>nnecessary but <emph>S</emph>ufficient conditions ([16], p.62). Today, it is often used as a mere name for a theoretical framework. In contrast, "MINUS" explicitly refers to the corresponding causal theory located in the INUS tradition that assigns causation to only those sufficiency and necessity relations that are rigorously freed of redundancies, i.e. that are <emph>minimal</emph>, where <emph>M</emph> stands for <emph>M</emph>inimally.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref13" type="bt">7</bibl> <bibtext> In short, the fuzzy logic rendering relevant for CNA is as follows: A negation <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="normal">¬</mi><mi>X</mi></math> </ephtml> amounts to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML">1<mo>−</mo><mi>X</mi></math> </ephtml> , a conjunction <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>*</mo><mi>Y</mi></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo form="prefix" movablelimits="true">min</mo><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>Y</mi><mo stretchy="false">)</mo></math> </ephtml> , a disjunction <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>+</mo><mi>Y</mi></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo form="prefix" movablelimits="true">max</mo><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>Y</mi><mo stretchy="false">)</mo></math> </ephtml> , an implication <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo stretchy="false">→</mo><mi>Y</mi></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>≤</mo><mi>Y</mi></math> </ephtml> , and an equivalence <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo stretchy="false">↔</mo><mi>Y</mi></math> </ephtml> to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>X</mi><mo>=</mo><mi>Y</mi></math> </ephtml> .</bibtext> </blist> <blist> <bibl id="bib8" idref="ref32" type="bt">8</bibl> <bibtext> As an example consider the configurations <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi>7</msub></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi>8</msub></math> </ephtml> in Table 1a. Everything is constant in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi>7</msub></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi>8</msub></math> </ephtml> except for factors A and Y. These configurations thus provide evidence for the relevance of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>A</mi></math> </ephtml> for <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> .</bibtext> </blist> <blist> <bibl id="bib9" idref="ref4" type="bt">9</bibl> <bibtext> For multi-value factors, which are beyond the scope of this study, the notion of membership ratio has to be re-defined to reflect the distribution of cases across all admissible values.</bibtext> </blist> <blist> <bibtext> Contrary to many regression methods ([28]) or Bayesian network methods ([24]), CCMs do not rely on distributional normality or symmetry assumptions.</bibtext> </blist> <blist> <bibtext> This follows terminological conventions in machine learning, where classification categories that are not equally represented in the data are interchangeably referred to as imbalanced and skewed (e.g., [8]).</bibtext> </blist> <blist> <bibtext> (2) is true in a configuration <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> if, and only if, membership in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>Y</mi></math> </ephtml> is equal to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo form="prefix" movablelimits="true">max</mo><mo stretchy="false">(</mo><mtext>A</mtext><mo>,</mo><mtext>B</mtext><mo>,</mo><mtext>C</mtext><mo stretchy="false">)</mo></math> </ephtml> in <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>σ</mi><mi>i</mi></msub></math> </ephtml> , which is the fuzzy logic rendering of disjunction (see Endnote 2).</bibtext> </blist> <blist> <bibtext> The CNA notions of fragmentation and noise are related, but not identical, to the QCA notions of limited diversity and inconsistency. For more see [4].</bibtext> </blist> <blist> <bibtext> While fragmentation ratios can be strictly retained, noise ratios may vary slightly (i.e., by 1% at most) due to the fuzzy-set nature of the data.</bibtext> </blist> <blist> <bibtext> Membership ratios cannot be set to 0 and 1 because, as we have seen in the previous section, all difference-making evidence would be gone under these conditions, inducing CNA to return nothing.</bibtext> </blist> <blist> <bibtext> Note that disjunctively interpreting multiple models is not the same as disjunctivity. Disjunctivity is given when one model <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow>1</msub></math> </ephtml> entails that more than one causal path produces an outcome. By contrast, an output set <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">S</mi></mrow></math> </ephtml> containing multiple models entails that at least one of these models is true but it is indeterminate which one.</bibtext> </blist> <blist> <bibtext> On a par with Bayesian network methods, but different from typical regression methods, CCMs automatically build all equally data-fitting models (for more on CCM model ambiguities, see, e.g., [2]).</bibtext> </blist> <blist> <bibtext> These conditions are satisfied if <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">m</mi></mrow><mi>j</mi></msub></math> </ephtml> is a submodel of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi mathvariant="normal">Δ</mi><mi>i</mi></msub></math> </ephtml> (e.g., [6]). Furthermore, note that we do not <emph>quantify</emph> correctness because there currently does not exist a satisfactory quantitative correctness measure for MINUS-formulas. It is an intricate problem to quantify the seriousness of errors or the proximity to the ground truth.</bibtext> </blist> <blist> <bibtext> We do not depict the distribution of the benchmark scores using, say, whisker plots because whiskers would be barely visible. We deliberately chose 1,000 ground truths for each trial because the means of the resulting scores calculated from different samples of that size were found to stabilize with very small standard errors of the means (e.g., between 0.0005 and 0.018). Thus, we can have high confidence that trials on a sample of 1,000 ground truths drawn from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow>2</msub></math> </ephtml> with the complexity constraints described in the previous section are representative of the population of all ground truths that can be built from <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow><mi mathvariant="bold">F</mi></mrow>2</msub></math> </ephtml> in that manner.</bibtext> </blist> <blist> <bibtext> [4]</bibtext> </blist> </ref> <ref id="AN0184233960-21"> <title> References </title> <blist> <bibtext> Ambühl Mathias, Baumgartner Michael. 2023. cna: Causal modeling with Coincidence Analysis. R Package Version 3.5.0. https://cran.r-project.org/package=cna.</bibtext> </blist> <blist> <bibtext> Baumgartner Michael, Thiem Alrik. 2017. " Model Ambiguities in Configurational Comparative Research." Sociological Methods & Research. 46 (4): 954–987.</bibtext> </blist> <blist> <bibtext> Baumgartner Michael, Thiem Alrik. 2020. " Often Trusted But Never (Properly) Tested: Evaluating Qualitative Comparative Analysis." Sociological Methods & Research. 49: 279–311.</bibtext> </blist> <blist> <bibtext> Baumgartner Michael, Falk Christoph. 2023. " Configurational Causal Modeling and Logic Regression." Multivariate Behavioral Research. 58 (2): 292–310.</bibtext> </blist> <blist> <bibtext> Baumgartner Michael, Falk Christoph. 2023. " Boolean Difference-Making: A Modern Regularity Theory of Causation." The British Journal for the Philosophy of Science. 74 (1): 171–197.</bibtext> </blist> <blist> <bibtext> Baumgartner Michael, Ambühl Mathias. 2020. " Causal Modeling with Multi-Value and Fuzzy-Set Coincidence Analysis." Political Science Research and Methods. 8: 526–542.</bibtext> </blist> <blist> <bibtext> Britt David W., Risinger Samantha T., Miller Virginia, Mans Mary K., Krivchenia Eric L., Evans Mark I. 2000. " Determinants of Parental Decisions After the Prenatal Diagnosis of Down Syndrome: Bringing in Context." American Journal of Medical Genetics. 93 (5): 410–416.</bibtext> </blist> <blist> <bibtext> Chawla Nitesh V. 2010. "Data Mining for Imbalanced Datasets: An Overview." Pp. 875-886 in Data Mining and Knowledge Discovery Handbook, edited by Oded Maimon and Lior Rokach. New York: Springer.</bibtext> </blist> <blist> <bibtext> Edianto Achmed, Trencher Gregory, Matsubae Kazuyo. 2022. " Why Do Some Countries Receive More International Financing for Coal-fired Power Plants Than Renewables? Influencing Factors in 23 Countries." Energy for Sustainable Development. 66: 177–188.</bibtext> </blist> <blist> <bibtext> Epple Ruedi, Schief Sebastian. 2016. " Fighting (For) Gender Equality: The Roles of Social Movements and Power Resources." Interface. 8 (2): 394–432.</bibtext> </blist> <blist> <bibtext> Gabriel Allison S., Campbell Joanna Tochman, Djurdjevic Emilija, Johnson Russell E., Rosen Christopher C. 2018. " Fuzzy Profiles: Comparing and Contrasting Latent Profile Analysis and Fuzzy Set Qualitative Comparative Analysis for Person-Centered Research." Organizational Research Methods. 21 (4): 877–904.</bibtext> </blist> <blist> <bibtext> Glaesser Judith. 2022. " Relative Educational Poverty: Conceptual and Empirical Issues." Quality & Quantity. 56: 2803–2820.</bibtext> </blist> <blist> <bibtext> Goertz Gary. 2020. Social Science Concepts and Measurement: New and Completely Revised Edition. Princeton: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Haesebrouck Tim. 2023. " The Populist Radical Right and Military Intervention: A Coincidence Analysis of Military Deployment Votes." International Interactions. 49 (3): 345–371.</bibtext> </blist> <blist> <bibtext> Lemmon E. J. 1965. Beginning Logic. London: Chapman & Hall.</bibtext> </blist> <blist> <bibtext> Mackie John L. 1974. The Cement of the Universe. A Study of Causation. Oxford: Clarendon Press.</bibtext> </blist> <blist> <bibtext> Oana Ioana-Elena, Schneider Carsten Q., Thomann Eva. 2021. Qualitative Comparative Analysis (QCA) Using R: A Beginner's Guide. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Parkkinen Veli-Pekka, Baumgartner Michael. 2023. " Robustness and Model Selection in Configurational Causal Modeling." Sociological Methods & Research. 52 (1): 176–208.</bibtext> </blist> <blist> <bibtext> Parkkinen Veli-Pekka, Baumgartner Michael, Ambühl Mathias. 2021. frscore: Functions for Calculating Fit-Robustness of CNA-Solutions. R Package Version 0.1.1. https://cran.r-project.org/package=frscore.</bibtext> </blist> <blist> <bibtext> Ragin Charles C.2009. Redesigning Social Inquiry: Fuzzy Sets and Beyond. Chicago: University of Chicago Press.</bibtext> </blist> <blist> <bibtext> Rihoux Benoît, Ragin, Charles C. ed. 2009. Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. London: Sage Publications.</bibtext> </blist> <blist> <bibtext> Schneider Carsten Q., Wagemann Claudius. 2012. Set-Theoretic Methods For the Social Sciences: A Guide to Qualitative Comparative Analysis. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Schnell Tatjana, Höge Thomas, Pollet Edith. 2013. " Predicting Meaning in Work: Theory, Data, Implications." Journal of Positive Psychology. 8 (6): 543–554.</bibtext> </blist> <blist> <bibtext> Spirtes Peter, Glymour Clark, Scheines Richard. 2000. Causation, Prediction, and Search. 2 ed. Cambridge: MIT Press.</bibtext> </blist> <blist> <bibtext> Swiatczak Martyna D. 2021. " Towards a Neo-configurational Theory of Intrinsic Motivation." Motivation and Emotion. 45 (6): 769–789.</bibtext> </blist> <blist> <bibtext> Swiatczak Martyna D. 2022. " Different Algorithms, Different Models." Quality & Quantity. 56: 1913–1937.</bibtext> </blist> <blist> <bibtext> Swiatczak Martyna D., Baumgartner Michael. 2024. " Replication Material for 'Data Imbalances in Coincidence Analysis: A Simulation Study.'" OSF. osf.io/9nc68/</bibtext> </blist> <blist> <bibtext> Tabachnick Barbara G., Fidell Linda S. 2019. Using Multivariate Statistics. 7 ed. Boston: Pearson.</bibtext> </blist> <blist> <bibtext> Thiem Alrik, Duşa Adrian. 2013. Qualitative Comparative Analysis with R: A User's Guide. New York, NY: Springer.</bibtext> </blist> <blist> <bibtext> Thomann Eva, Maggetti Martino. 2020. " Designing Research With Qualitative Comparative Analysis (QCA): Approaches, Challenges, and Tools." Sociological Methods & Research. 49 (2): 356–386.</bibtext> </blist> <blist> <bibtext> von Hippel Paul T. 2013. " Should a Normal Imputation Model be Modified to Impute Skewed Variables? " Sociological Methods & Research. 42 (1): 105–138.</bibtext> </blist> <blist> <bibtext> Womack Dana M., Miech Edward J., Fox Nicholas J., Silvey Linus C., Somerville Anna M., Eldredge Deborah H., Steege Linsey M. 2022. " Coincidence Analysis: A Novel Approach to Modeling Nurses' Workplace Experience." Applied Clinical Informatics. 13 (4): 794–802.</bibtext> </blist> <blist> <bibtext> Yakovchenko Vera, Miech Edward J., Chinman Matthew J., Chartier Maggie, Gonzalez Rachel, Kirchner JoAnn E., Morgan Timothy R., Park Angela, Powell Byron J., Proctor Enola K., Ross David, Waltz Thomas J., Rogal Shari S. 2020. " Strategy Configurations Directly Linked to Higher Hepatitis C Virus Treatment Starts: An Applied Use of Configurational Comparative Methods." Medical Care. 58 (5): e31–e38.</bibtext> </blist> <blist> <bibtext> Yuan Ke-Hai, Bentler Peter M., Zhang Wei. 2005. " The Effect of Skewness and Kurtosis on Mean and Covariance Structure Analysis: The Univariate Case and Its Multivariate Implication." Sociological Methods & Research. 34 (2): 240–258.</bibtext> </blist> </ref> <aug> <p>By Martyna Daria Swiatczak and Michael Baumgartner</p> <p>Reported by Author; Author</p> <p></p> <p>Martyna Daria Swiatczak is a postdoctoral research fellow at the Department of Philosophy of the University of Bergen. She applies the configurational method of Coincidence Analysis (CNA) in her field of expertise, organizational behavior, and methodologically develops CNA. Her methodological research focuses on topics related to the algorithmic foundations of configurational comparative methods and to the systematic understanding of data preparation processes for such methods.</p> <p>Michael Baumgartner is a full professor at the Department of Philosophy of the University of Bergen, with a specialization in the philosophy of science and logic. He developed the configurational method Coincidence Analysis (CNA) and has numerous publications on causation, causal reasoning, and data analysis with different methods. Moreover, he has worked on mechanistic constitution, cognition, interventionism, determinism, and logical formalization; and he is a co-developer of the CNA software libraries for the R environment for statistical computing.</p> </aug> <nolink nlid="nl1" bibid="bib21" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib14" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib33" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib32" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib10" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib25" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib18" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib31" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib34" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib13" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib20" firstref="ref16"></nolink> <nolink nlid="nl13" bibid="bib17" firstref="ref17"></nolink> <nolink nlid="nl14" bibid="bib22" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib26" firstref="ref19"></nolink> <nolink nlid="nl16" bibid="bib16" firstref="ref21"></nolink> <nolink nlid="nl17" bibid="bib29" firstref="ref24"></nolink> <nolink nlid="nl18" bibid="bib15" firstref="ref26"></nolink> <nolink nlid="nl19" bibid="bib30" firstref="ref41"></nolink> <nolink nlid="nl20" bibid="bib28" firstref="ref42"></nolink> <nolink nlid="nl21" bibid="bib11" firstref="ref44"></nolink> <nolink nlid="nl22" bibid="bib19" firstref="ref69"></nolink> <nolink nlid="nl23" bibid="bib23" firstref="ref91"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1473620
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Data Imbalances in Coincidence Analysis: A Simulation Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Martyna+Daria+Swiatczak%22">Martyna Daria Swiatczak</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7537-1813">0000-0002-7537-1813</externalLink>)<br /><searchLink fieldCode="AR" term="%22Michael+Baumgartner%22">Michael Baumgartner</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1536-2816">0000-0003-1536-2816</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2025 54(2):739-771.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 33
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Causal+Models%22">Causal Models</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Analysis%22">Comparative Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Distributions%22">Statistical Distributions</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Data%22">Statistical Data</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241241227039
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In this paper, we investigate the conditions under which data imbalances, a common data characteristic that occurs when factor values are unevenly distributed, are problematic for the performance of Coincidence Analysis (CNA). We further examine how such imbalances relate to fragmentation and noise in data. We show that even extreme data imbalances, when not combined with fragmentation or noise, do not negatively affect CNA's performance. However, an extended series of simulation experiments on fuzzy-set data reveals that, when mixed with fragmentation or noise, data imbalances may substantially impair CNA's performance. Furthermore, we find that the performance impairment is higher when endogenous factors are imbalanced than when exogenous factors are concerned. Our results allow us to quantify these impacts and demarcate degrees at which data imbalances should be considered as problematic. Thus, applied researchers can use our demarcation guidelines to enhance the validity of their studies.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1473620
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1473620
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241241227039
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 33
        StartPage: 739
    Subjects:
      – SubjectFull: Causal Models
        Type: general
      – SubjectFull: Comparative Analysis
        Type: general
      – SubjectFull: Data Analysis
        Type: general
      – SubjectFull: Statistical Distributions
        Type: general
      – SubjectFull: Statistical Data
        Type: general
    Titles:
      – TitleFull: Data Imbalances in Coincidence Analysis: A Simulation Study
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Martyna Daria Swiatczak
      – PersonEntity:
          Name:
            NameFull: Michael Baumgartner
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 54
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1