Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork
Saved in:
| Title: | Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork |
|---|---|
| Language: | English |
| Authors: | Verónica Pérez Bentancur (ORCID |
| Source: | Sociological Methods & Research. 2024 53(2):729-759. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 31 |
| Publication Date: | 2024 |
| Intended Audience: | Researchers |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Research Design, Social Science Research, Public Opinion, Punishment, Foreign Countries, Experiments, Mixed Methods Research, Field Studies, Data Collection, Pretesting |
| Geographic Terms: | Uruguay |
| DOI: | 10.1177/00491241221082595 |
| ISSN: | 0049-1241 1552-8294 |
| Abstract: | Experimental designs in the social sciences have received increasing attention due to their power to produce causal inferences. Nevertheless, experimental research faces limitations, including limited external validity and unrealistic treatments. We propose combining qualitative fieldwork and experimental design iteratively--moving back-and-forth between elements of a research design--to overcome these limitations. To properly evaluate the strength of experiments researchers need information about the context, data, and previous knowledge used to design the treatment. To support our argument, we analyze 338 pre-analysis plans submitted to the Evidence in Governance and Politics repository in 2019 and the design of a study on public opinion support for punitive policing practices in Montevideo, Uruguay. The paper provides insights about using qualitative fieldwork to enhance the external validity, transparency and replicability of experimental research, and a practical guide for researchers who want to incorporate iteration to their research designs. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Accession Number: | EJ1423057 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwH70Eqao4AZsN8XHpk_EYALAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDIpSlv75RYpR92bb9gIBEICBm-T6zWxnJjxb8hz63VsoD7a8Qs32LDWmjeDgbvS4hwwM7BQKy-BWoNjvHRFBOdNgDWN7W10AddbaFPidoSP6q6CdB7ReH0lw3q94AEst53ISKZlXtlAwIKVuokEU-1ryZPmkXPJIm9Szuo9xPYinYhbdG1F6-_E-2jarcc2AsoXR3B1y-gDA5wcWnpRmyBTLqGP9g8qlD1_K916w Text: Availability: 1 Value: <anid>AN0176861674;som01may.24;2024Apr30.03:38;v2.2.500</anid> <title id="AN0176861674-1">Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork , </title> <p>Experimental designs in the social sciences have received increasing attention due to their power to produce causal inferences. Nevertheless, experimental research faces limitations, including limited external validity and unrealistic treatments. We propose combining qualitative fieldwork and experimental design iteratively—moving back-and-forth between elements of a research design—to overcome these limitations. To properly evaluate the strength of experiments researchers need information about the context, data, and previous knowledge used to design the treatment. To support our argument, we analyze 338 pre-analysis plans submitted to the Evidence in Governance and Politics repository in 2019 and the design of a study on public opinion support for punitive policing practices in Montevideo, Uruguay. The paper provides insights about using qualitative fieldwork to enhance the external validity, transparency and replicability of experimental research, and a practical guide for researchers who want to incorporate iteration to their research designs.</p> <p>Keywords: Iteration; research design; experiments; qualitative fieldwork; mixed-methods</p> <hd id="AN0176861674-2">Introduction</hd> <p>Social science researchers frequently discuss how to improve causal inference—why and how something occurs—to better understand the inner workings of social life. This concern has motivated efforts to improve research methodology. As part of these efforts, experimental designs have become increasingly popular in the social sciences ([<reflink idref="bib15" id="ref1">15</reflink>]). In general, experiments are research strategies that can be used for descriptive and causal exploration ([<reflink idref="bib5" id="ref2">5</reflink>]). For example, list experiments can be used as tools to document stages of a process to fill in evidentiary gaps ([<reflink idref="bib28" id="ref3">28</reflink>]), and randomized controlled trials (RCTs) can be used to infer causality ([<reflink idref="bib17" id="ref4">17</reflink>]; [<reflink idref="bib24" id="ref5">24</reflink>]; [<reflink idref="bib5" id="ref6">5</reflink>]). Nevertheless, critics argue that randomization has limits, and that experiments typically lack generalizability and are often unrealistic. A nascent body of literature highlights the advantages of using qualitative tools to refine experiments in the design phase as a tool to overcome these limitations ([<reflink idref="bib1" id="ref7">1</reflink>]; [<reflink idref="bib2" id="ref8">2</reflink>]; [<reflink idref="bib16" id="ref9">16</reflink>], [<reflink idref="bib17" id="ref10">17</reflink>]; [<reflink idref="bib19" id="ref11">19</reflink>]; [<reflink idref="bib54" id="ref12">54</reflink>], [<reflink idref="bib55" id="ref13">55</reflink>]; [<reflink idref="bib59" id="ref14">59</reflink>]).</p> <p>We build on this body of work and make three contributions: 1) we clarify what is entailed in building realistic treatments and specify how qualitative tools can contribute to this goal; 2) we describe how qualitative methods can be used to improve measurement during the experimental design phase; 3) we propose iteration as a specific procedure to improve measurement by operationalizing untested assumptions in experiments ([<reflink idref="bib54" id="ref15">54</reflink>], [<reflink idref="bib55" id="ref16">55</reflink>]). Iteration—the process of alternating between different elements of the research design ([<reflink idref="bib21" id="ref17">21</reflink>]; [<reflink idref="bib38" id="ref18">38</reflink>])—is a means to combine qualitative fieldwork and experiments. Using our own research as an example, we provide concrete steps on how to mix these two methods applying iteration in the design phase.</p> <p>Iteration is an important tool for at least three reasons: First, it improves causal identification. Iteration between theory and fieldwork produces a more refined understanding of the context and more realistic interventions—those that more accurately depict real-life situations ([<reflink idref="bib55" id="ref19">55</reflink>]). Second, when made explicit, iteration allows the research design to be more transparent. By outlining the iterative steps, researchers facilitate replication. Third, transparency about iteration aids generalization: if readers are told how a research design was informed by or tailored to features of the study context, then they are in a better position to understand which aspects of the findings are more or less likely to travel to other contexts. Gaining a deeper understanding of the context can clarify the scope of the theory being tested, and thus provide insights regarding the extent to which findings can be generalized.</p> <p>Our approach is different from using qualitative fieldwork in the analytical phase of the research process, typically employed to discover causal mechanisms ([<reflink idref="bib18" id="ref20">18</reflink>]; [<reflink idref="bib41" id="ref21">41</reflink>]). Emphasizing iteration can help researchers structure their projects before carrying out the analysis. Iteration is also different from pre-testing the experimental design. While pre-testing focuses on piloting the mechanics of a design on a smaller sample prior to fielding a study, it may or may not allow researchers to discover whether the elements of the design reflect the context.</p> <p>The paper is organized as follows: We first review scholarly work combining experiments and qualitative evidence and we describe the process of iteration with qualitative fieldwork in experimental research designs. We show that the authors of experimental studies that mix methods, rarely make the steps they follow explicit. To illustrate this point, we analyze 338 pre-analysis plans submitted to the Evidence in Governance and Politics (EGAP) repository in 2019. Then, we illustrate our proposed iterative process by describing a study of public opinion support for punitive policing practices in Montevideo, Uruguay, that combined a survey experiment and qualitative fieldwork. Finally, we summarize our contributions regarding the value of iterating between qualitative fieldwork and experiments, we discuss some limitations of this approach, and we include a practical guide for researchers who wish to incorporate iteration in their research designs.</p> <hd id="AN0176861674-3">How to Iteratively mix Methods in the Context of an Experimental Design</hd> <p>Social science research seeks to provide accurate causal inferences to advance understanding of social, economic and political phenomena. In recent years, this concern has manifested in a growing interest in improving several aspects of the research process, including, among other issues, improving research designs ([<reflink idref="bib8" id="ref22">8</reflink>]; [<reflink idref="bib25" id="ref23">25</reflink>]; [<reflink idref="bib27" id="ref24">27</reflink>]; [<reflink idref="bib39" id="ref25">39</reflink>]; [<reflink idref="bib42" id="ref26">42</reflink>]; [<reflink idref="bib54" id="ref27">54</reflink>]) increasing transparency in qualitative research sources ([<reflink idref="bib43" id="ref28">43</reflink>]) and in quantitative and qualitative designs through pre-registration processes ([<reflink idref="bib4" id="ref29">4</reflink>]; [<reflink idref="bib20" id="ref30">20</reflink>]).</p> <p>Social science researchers in the quantitative tradition have increasingly used experimental designs, including field experiments, lab experiments, natural experiments, and others. Though initially marginal, experimental designs are widely used, owing to their particular strength at producing causal inferences ([<reflink idref="bib24" id="ref31">24</reflink>]; [<reflink idref="bib35" id="ref32">35</reflink>]; [<reflink idref="bib30" id="ref33">30</reflink>]). Experiments derive their potency from two features: the combination of a relatively large number of cases with random assignment to treatment and control groups. Random assignment to treatment is meant to eliminate systematic group differences at the outset of the study such that any observed difference between treatment and control groups in the outcome variable can be attributed to the treatment of interest. The main advantage of experiments for producing strong causal inferences thus lies in their ability to eliminate the effects of confounding variables present in most observational designs in both quantitative and qualitative research ([<reflink idref="bib17" id="ref34">17</reflink>]; [<reflink idref="bib24" id="ref35">24</reflink>]). Experiments can also support causal inference by demonstrating the plausibility of a certain causal path by filling in evidentiary gaps in cases where mechanisms are hard to observe directly ([<reflink idref="bib28" id="ref36">28</reflink>]: 16–18). Nevertheless, scholars have pointed out that whether experiments are the best tool for causal inference depends on the questions we are asking, as well as prior knowledge ([<reflink idref="bib14" id="ref37">14</reflink>]: 2).</p> <p>In this paper we focus on two limits of experimental research: unrealistic treatments and lack of external validity ([<reflink idref="bib6" id="ref38">6</reflink>]; [<reflink idref="bib18" id="ref39">18</reflink>]; [<reflink idref="bib54" id="ref40">54</reflink>]). Realistic interventions are such that participants could be expected to reproduce real-life behaviors. Constructing real-life scenarios requires considering the context to identify interventions that are adequate reflections of the context ([<reflink idref="bib54" id="ref41">54</reflink>]: 166–167; [<reflink idref="bib55" id="ref42">55</reflink>]: 371). Realistic interventions are central to external validity. Replicating an experimental design requires adapting it to a new context ([<reflink idref="bib14" id="ref43">14</reflink>]; [<reflink idref="bib55" id="ref44">55</reflink>]). We argue that to overcome these limitations researchers can use a mixed-methods approach that explicitly incorporates qualitative elements through an iterative process in the design phase. As a result, researchers improve the theory and refine the experimental intervention, which aids the inferential power of the argument. An added benefit of making the iterative process explicit is increased transparency of the design process, which aids replicability as well as generalizability.</p> <p>A mixed-methods approach involves combining strategies for data collection and analysis from different methodological schools, usually from quantitative and qualitative approaches ([<reflink idref="bib27" id="ref45">27</reflink>]; [<reflink idref="bib42" id="ref46">42</reflink>]; [<reflink idref="bib54" id="ref47">54</reflink>]). Combinations of methods from different epistemological traditions are multiple: for example, surveys can be combined with interviews, focus groups, or process tracing; regression analyses of a large number of cases can be combined with an in-depth study of one or two cases; formal theory can be combined with case studies, among other possibilities ([<reflink idref="bib13" id="ref48">13</reflink>]; [<reflink idref="bib19" id="ref49">19</reflink>]; [<reflink idref="bib42" id="ref50">42</reflink>]; [<reflink idref="bib41" id="ref51">41</reflink>]; [<reflink idref="bib52" id="ref52">52</reflink>]; [<reflink idref="bib54" id="ref53">54</reflink>]). Mixed methods can be used to probe the untested assumptions of other methods, such as assumptions about measurement validity or about the nature of data ([<reflink idref="bib54" id="ref54">54</reflink>]: 48–55).</p> <p>A central distinction is between mixed-methods and the process through which methods are mixed. "Mixing methods" refers to the combination of quantitative and qualitative methods.[<reflink idref="bib7" id="ref55">7</reflink>] Different methods can be combined through different processes: nested analysis ([<reflink idref="bib42" id="ref56">42</reflink>]), parallel ([<reflink idref="bib32" id="ref57">32</reflink>]), embedded ([<reflink idref="bib34" id="ref58">34</reflink>]), integrative ([<reflink idref="bib54" id="ref59">54</reflink>]), iterative ([<reflink idref="bib38" id="ref60">38</reflink>]). While nested, parallel, embedded and integrative techniques are generally employed in the <emph>analytical phase</emph> of the research process, we argue that iteration is particularly useful in the <emph>design phase</emph>. Iteration is one possible way to combine methods. It refers to the alternation between different elements of the design (for example, using in-depth interviews to refine a survey measurement instrument and then validating it with additional interviews), thus making the researcher more conscious of each method's contribution to inference.</p> <p>Related, another relevant distinction is between integration and iteration. Integration refers to the combination of two or more methods to support a unified causal inference, which corresponds to the analytical phase of the research design ([<reflink idref="bib54" id="ref61">54</reflink>]). Iteration, we contend, does not necessarily involve an analytical component, it focuses on strengthening the design prior to implementation and analysis ([<reflink idref="bib38" id="ref62">38</reflink>]).</p> <p>Iteration is generally understood as the movement back-and-forth in the research process. There are different ways to conceptualize iteration: for example, between methods of reasoning (induction and deduction), or between different elements of the design itself (theory and fieldwork) ([<reflink idref="bib38" id="ref63">38</reflink>]; [<reflink idref="bib62" id="ref64">62</reflink>]). Iteration has a dual objective: the refinement of theory, and the refinement of the research design itself ([<reflink idref="bib16" id="ref65">16</reflink>]; [<reflink idref="bib21" id="ref66">21</reflink>]; [<reflink idref="bib38" id="ref67">38</reflink>]; [<reflink idref="bib62" id="ref68">62</reflink>]). Through iteration researchers can revise and update different parts of the design: the research question, concepts, hypotheses, cases and instruments for data collection, such as interview protocols and survey questionnaires ([<reflink idref="bib38" id="ref69">38</reflink>]: 23). Iteration is also a mechanism to improve measurement because it allows to probe the untested assumptions of different methods, rectifying when necessary ([<reflink idref="bib54" id="ref70">54</reflink>]). For example, an experiment designed only following an extensive review of the literature likely makes several assumptions about measurement and face validity.[<reflink idref="bib8" id="ref71">8</reflink>]</p> <p>In some cases, the distinction between iteration in the design and analytical phases may not be so clear cut. For example, in designs that implement process-tracing or comparative historical analysis, researchers may iterate with insights from the analytical phase to refine their hypotheses and theory ([<reflink idref="bib62" id="ref72">62</reflink>]). However, when designs include components that are "one shot" such as the implementation of a survey, or an experiment, the distinction between iteration in the analytical and design phases is more clearly defined (since the experiment cannot be redone once it is fielded)[<reflink idref="bib9" id="ref73">9</reflink>]. This is also possible in qualitative designs, for example, those using process-tracing, where iterating with interviews can lead to theory refinement and modifications in interview questionnaires that allow for gathering better pieces of evidence ([<reflink idref="bib29" id="ref74">29</reflink>]).</p> <p>Despite being a common research practice, iteration is often unacknowledged, as it is viewed as pre-scientific ([<reflink idref="bib2" id="ref75">2</reflink>]; [<reflink idref="bib62" id="ref76">62</reflink>]). We argue that when iteration is made explicit and public it makes the research process more transparent, as it requires one to make all the steps in the design known. This paper's contribution lies in specifying how to iterate between methods—in particular, the combination of qualitative fieldwork and experimentation—to improve a research design prior to testing with the goal of strengthening causal inference. In doing so, we bring to bear mixed methods research containing experiments, research that mixes methods in the design phase without making it explicit, and methodological research that explicitly discusses iteration.</p> <p>Although there is a growing body of research that mixes experiments with qualitative fieldwork (for some examples, see [<reflink idref="bib1" id="ref77">1</reflink>]; [<reflink idref="bib11" id="ref78">11</reflink>]; [<reflink idref="bib16" id="ref79">16</reflink>], [<reflink idref="bib18" id="ref80">18</reflink>]; [<reflink idref="bib19" id="ref81">19</reflink>]; [<reflink idref="bib41" id="ref82">41</reflink>]; [<reflink idref="bib51" id="ref83">51</reflink>]; [<reflink idref="bib59" id="ref84">59</reflink>]), the process of mixing methods is not always acknowledged. Within those researchers who mix methods, a common practice is to incorporate qualitative methods during the analytical phase, to illustrate causal mechanisms. For example, [<reflink idref="bib41" id="ref85">41</reflink>] deployed a survey that had an experimental component as well as open ended questions. Using the information from the open-ended questions to contextualize the experimental results, the author concluded that combining experiments and fieldwork produces a more complete picture of the causal effects resulting from the experiment ([<reflink idref="bib41" id="ref86">41</reflink>]: 61). Other scholars might use qualitative fieldwork to strengthen the design but without making it explicit ([<reflink idref="bib2" id="ref87">2</reflink>]), and methodological research highlighting the importance of iteration in the research process places less emphasis on its use to improve experimental designs prior to testing ([<reflink idref="bib38" id="ref88">38</reflink>]; [<reflink idref="bib62" id="ref89">62</reflink>]).</p> <p>We build on previous approaches highlighting the value of putting fieldwork at the service of experiments to improve the latter's ability to yield valid and powerful causal inferences ([<reflink idref="bib16" id="ref90">16</reflink>]; [<reflink idref="bib38" id="ref91">38</reflink>]; [<reflink idref="bib54" id="ref92">54</reflink>]). Specifically, we identify and stylize an iterative sequence for refining a research design that can be applied more generally. In addition, we contend that making iteration explicit forces researchers to specify every step of the design, thus contributing to transparency and increasing potential generalizability.</p> <p>Using iteration in mixed-methods designs helps researchers make stronger causal inferences than they would have been able to make otherwise. Combining methods through iterative steps allows researchers to leverage the strengths of both quantitative and qualitative approaches in the design phase. However, iteration is not an indefinite process. In a design containing an experiment, iteration ends when the experiment is fielded. To make decisions about the exact duration of the iterative phase, scholars should consider substantive and pragmatic criteria. Substantively, iteration ends when qualitative fieldwork does not reveal new insights that would significantly improve the experimental design. This is akin to the notion of "saturation" in qualitative research: when conducting interviews or participant observation, the researcher does not gain additional confirmatory evidence, or new evidence that challenges their argument ([<reflink idref="bib53" id="ref93">53</reflink>]). Pragmatically, when to end iteration is conditioned by the amount of resources available to the researcher (time, and economic resources) ([<reflink idref="bib56" id="ref94">56</reflink>]). Other pragmatic considerations relate to safety and security conditions on the ground ([<reflink idref="bib10" id="ref95">10</reflink>]).</p> <hd id="AN0176861674-4">The Benefits of Iteration</hd> <p>In this section we describe a few examples of research designs that illustrate some of the benefits of iteration. It is worth noting that not all mixed-methods research using experiments requires iteration. For example, a common way of mixing methods to generate designs are natural experiments. In natural experiments, qualitative methods are a core element used to produce the design; their key function is to provide evidence of the pseudo-random nature of treatment assignment. Archival research, review of documents, interviews, and other qualitative tools provide in-depth knowledge of cases that show random assignment of treatments in the real world. Qualitative methods are necessary to show the mechanisms whereby treatment and outcome are connected ([<reflink idref="bib17" id="ref96">17</reflink>]; [<reflink idref="bib22" id="ref97">22</reflink>]; [<reflink idref="bib40" id="ref98">40</reflink>]; [<reflink idref="bib54" id="ref99">54</reflink>]).</p> <p>One example that illustrates the importance of iteration in experimental contexts is [<reflink idref="bib11" id="ref100">11</reflink>]. The authors conducted a survey experiment to assess whether women candidates suffer gender bias in Malawi. In parallel with the survey, they gathered biographical information about the candidates and conducted focus groups with them. Initially, they aimed to use the qualitative data to contextualize the experimental results. However, their main hypothesis—that in conservative contexts citizens prefer male candidates—was not supported by the experimental results, making it difficult to align the survey findings with the qualitative evidence. In reviewing the latter, they realized that the candidate profiles they used in the experiment were not entirely realistic. A relevant finding from the focus groups is that women candidates in Malawi face defamation campaigns, which impacts citizens' relative evaluations of female and male candidate profiles. The authors conclude that this contextual information was not captured in the experiment, and that this can explain their null finding. They also conclude that they would have benefitted from conducting the focus groups before the experiment, to improve the design of the latter by including a "'rumor mongering condition' to test whether and how voter biases are activated in the electoral process" ([<reflink idref="bib11" id="ref101">11</reflink>]: 622). This is an example of fieldwork conducted in parallel with the implementation of the experiment and used mainly for analytical purposes. Had the authors implemented iteration between fieldwork and the experiment in the design phase, their experiment would have been stronger and their findings might have differed. Similarly, [<reflink idref="bib51" id="ref102">51</reflink>] conduct extensive ethnography in parallel with a survey-based RCT. They use the ethnographic fieldwork analytically to unpack the reasons for the failure of the RCT and to illustrate small treatment effects that would not have been detected through statistical analyses.</p> <p>An example that uses ethnographic methods for purposes of strengthening the design of a conjoint experiment is [<reflink idref="bib1" id="ref103">1</reflink>], who conducted research on slum leaders in India. To create realistic vignettes, they use ethnographic fieldwork to define accurate experimental attributes ([<reflink idref="bib1" id="ref104">1</reflink>]: 476–477). The use of ethnography results in a compelling research design with realistic treatments. However, the authors do not specify the concrete steps they followed to improve the treatment. From the paper it is unclear whether they designed the experiment first, and then iterated with fieldwork to adjust the original design, or they sequentially conducted fieldwork, and designed the experiment.</p> <p>Explicit examples of the benefits of iteration are Thachil (2017, [59]), [<reflink idref="bib19" id="ref105">19</reflink>], and [<reflink idref="bib37" id="ref106">37</reflink>], who conducted experiments to understand identity politics in India, ethnic cleavages in Mali, and the construction of local leadership in India, respectively. Thachil ([<reflink idref="bib58" id="ref107">58</reflink>], [<reflink idref="bib59" id="ref108">59</reflink>]) uses ethnographic fieldwork with the dual purpose of defining his sample and to improve a vignette experiment. The original experimental intervention was adjusted using information from fieldwork to better reflect the context. In turn, [<reflink idref="bib19" id="ref109">19</reflink>] constructed a preliminary matrix of common Malian last names—as proxies for ethnicity—based on interviews, to assign subjects to the different treatment conditions. They then validated and refined the original matrix with a second round of interviews with qualified informants in the field. This process allowed them to add more last names to the matrix. Prior to conducting the experiment, they tested the revised matrix with 169 people, which allowed them to further refine it ([<reflink idref="bib16" id="ref110">16</reflink>]: 21–22). Using a participatory econometrics approach, [<reflink idref="bib37" id="ref111">37</reflink>]: 231–232) constructed survey instruments based on focus groups discussions, in-depth interviews, and participant observation, which they then pre-tested and adjusted through additional interviews.</p> <p>The examples above illustrate how research designs containing an experiment can benefit from the use of iteration between qualitative data collection and the design of the experiment prior to implementation. The examples also highlight how the absence of iteration can lead to poor causal identification strategies. In the following section we present a more systematic analysis by evaluating a set of 338 pre-analysis plans.</p> <hd id="AN0176861674-5">How Common is Iteration in Experimental Designs?</hd> <p>The process of combining methods and iterating between different parts of the design in experimental contexts is rarely discussed. Thus, researchers have little guidance on how to iterate or about why making the process explicit is valuable. Although experimental designs rely on fieldwork, interviews, and case knowledge, this fact is rarely made explicit or public. Yet the strength of an experiment can only be properly evaluated if researchers know what kind of data and prior knowledge went into its design.</p> <p>To illustrate this point, we analyze 338 pre-analysis plans (PAPs) pre-registered in the Evidence in Governance and Politics (EGAP) repository in 2019 (out of a total of 457 pre-registered designs) for which documentation was available.[<reflink idref="bib10" id="ref112">10</reflink>] EGAP is a cross-disciplinary network of researchers and practitioners focused on experimental designs.[<reflink idref="bib11" id="ref113">11</reflink>] EGAP members' research designs encompass a variety of topics, such as inequality and poverty, governance and institutions, conflict and violence, among others, across disciplines such as sociology, political science, economics, public health. These designs are not limited to program evaluations, they also include academic research. PAPs can be pre-registered before conducting the research, or prior to looking at the data, after pre-testing. EGAP's repository is one of the most flexible in terms of what can be pre-registered ([<reflink idref="bib7" id="ref114">7</reflink>]). Unless they are gated, these documents are available to the public.</p> <p>In their simplest form PAPs contain the basic structure of the research design: hypotheses, the structure of the experiment, the population to which it will be applied and the analytical strategy. The main goal of pre-registration is to prevent analytical biases, such as ad hoc analyses ex post, or choosing which results to report ([<reflink idref="bib4" id="ref115">4</reflink>]; [<reflink idref="bib36" id="ref116">36</reflink>]). Another goal of pre-registration is related to increasing research transparency[<reflink idref="bib12" id="ref117">12</reflink>]. Transparency in every step of the research design makes replication possible ([<reflink idref="bib43" id="ref118">43</reflink>]; [<reflink idref="bib48" id="ref119">48</reflink>]). An added value of pre-registration, much less discussed or acknowledged, relates to making the process of design itself public. Many studies are not published due to a wide variety of biases in the publication process, and a strong reluctance to publish null results ([<reflink idref="bib36" id="ref120">36</reflink>]). In this context, pre-registration is key because it allows other researchers to learn from the knowledge that is generated in the process of designing research ([<reflink idref="bib49" id="ref121">49</reflink>]).</p> <p>We chose PAPs for pragmatic and substantive reasons: pragmatically, PAPs are centralized in one repository, with the same guidelines, containing a diversity of projects across disciplines. More importantly, there is a powerful substantive reason to focus on PAPs: they contain explicit decisions related to the design phase of the research process, which is our focus. Therefore, if the different steps in the construction of the design are specified, and if there is iteration between those steps, we should expect to find evidence in PAPs. An alternative analytical strategy could have been to select articles published in academic journals. However, conducting this selection process with journals poses challenges in terms of the substantive requirements, variation across disciplines and journal access. In addition, had we chosen to analyze published articles we would not have been able to consider studies that have not been published, either because they have null findings or because of other biases inherent in the publication process ([<reflink idref="bib7" id="ref122">7</reflink>]:341; [<reflink idref="bib36" id="ref123">36</reflink>]:240; [<reflink idref="bib44" id="ref124">44</reflink>]:356). Besides, the process of writing the PAP is an opportunity to think carefully about the design, and potentially, iterate; other considerations (such as word limits) may lead researchers to avoid providing too much detail in a manuscript.[<reflink idref="bib13" id="ref125">13</reflink>],[<reflink idref="bib14" id="ref126">14</reflink>]</p> <p>To conduct our analysis, we downloaded all available PAPs submitted in 2019, including amendments and updates, and processed them using text-analysis tools. Our goal was to identify how many of these PAPs contained qualitative tools as part of the design. To do so, we searched the documents for keywords such as <emph>fieldwork</emph>, <emph>archive</emph>, <emph>ethnography</emph>, <emph>interview</emph>, <emph>triangulation</emph>, <emph>oral</emph>, <emph>history</emph>, <emph>qualitative</emph>, <emph>participant observation</emph>, and <emph>process tracing</emph>. We took the absence of such words from the design description as evidence that the project did not include a qualitative fieldwork component. The results of this search are summarized in Figure 1.</p> <p>Graph: Figure 1. Text analysis using Pre-analysis plans in EGAP, 2019.</p> <p>Figure 1 shows that of the 338 documents analyzed, about a third (<reflink idref="bib108" id="ref127">108</reflink>) contain one or more of the relevant keywords, whereas two-thirds (<reflink idref="bib230" id="ref128">230</reflink>) do not. We then carefully read the subset of 108 documents that contain keywords to identify each study's purpose in combining methods. We classified the different purposes as either <emph>analytical</emph>, <emph>refine design</emph>, or <emph>other</emph>. "Analytical" refers to the combination of methods to identify mechanisms and build theory. "Refine design" refers to the combination of methods to improve some aspect of the research design, such as to improve the treatment, to adjust the treatment and control groups, to add information about the context, or to adjust hypotheses. The category "other" identifies those cases where terms such as "interview" refer to the administration of survey or field experiments but do not refer to interviews in a qualitative sense, as well as to documents that were misclassified; for example, the word "field" only appeared in the bibliography. The results of that additional step are presented in Figure 2.</p> <p>Graph: Figure 2. Analysis of PAPs containing mixed-methods keywords.</p> <p>According to Figure 2, only 14 designs combine methods for the purpose of improving the design itself, which represents only 4% of the PAPs pre-registered in EGAP in 2019. The vast majority of the designs that employ mixed methods combine interviews with experiments; other tools such as participant observation or ethnographic methods are far less common, and iteration between elements of the design is rarely discussed explicitly.</p> <p>In a third and final step, we re-read the PAPs under "refine design" to identify how different methods were combined. For example, [<reflink idref="bib31" id="ref129">31</reflink>][<reflink idref="bib15" id="ref130">15</reflink>] study of excessive alcohol consumption and violence in bars in Bogota combines a survey experiment with fieldwork in four municipalities. Insights from fieldwork were used to identify where the survey would be deployed, based on security concerns. Even though one can infer that the geographical area for survey implementation was adjusted based on qualitative fieldwork, the authors do not provide explicit details of this process. In another example, [<reflink idref="bib3" id="ref131">3</reflink>][<reflink idref="bib16" id="ref132">16</reflink>] combine interviews and focus groups with a field experiment to study the effect of mining companies' social investment on citizens' perceptions of local governments in two mining communities in Burkina Faso. The PAP explicitly states that fieldwork is used to inform the identification of relevant experimental outcomes and to test and improve an audio-podcast. However, the design does not specify how these steps were taken, or the modifications that were included. These examples show that researchers mix methods with the purpose of refining their research designs, and in some cases they do so iteratively. The designs include creative combinations of methods, unfortunately, the PAPs do not include enough details to allow other researchers to learn from these strategies. One important caveat in our analysis is that these descriptive statistics might be biased. As we argued before, it is possible that authors leave out discussions of qualitative methods because they see them as secondary, or pre-scientific. Yet, when PAPs include such details it allows for a better understanding of the whole research process, and it gives other researchers more elements to replicate the design.</p> <p>Refining the research design is essential for good causal inference. A good research design allows for good data collection and analysis; this is critical for experiments. While some observational designs can be adjusted ex post, this is not possible for experiments; for example, once a field experiment is implemented, researchers cannot redo it ([<reflink idref="bib25" id="ref133">25</reflink>]). Experimental research design can benefit from iteration with qualitative elements. Iterating allows the researcher to adjust hypotheses, as well as the treatment and measurement tools, prior to implementation. If researchers do not combine methods, they run the risk of employing unrealistic treatments. By creating realistic treatments, researchers strengthen the design's internal validity: higher correspondence between the measurement tool and the phenomenon of interest. Furthermore, when researchers combine methodologies and make the steps involved in the combination explicit, they make the design more transparent, which allows others to replicate it.</p> <p>In the next section, we provide an example of a mixed-methods case study in which we refine an experimental design, as well as the subsequent analysis, in the context of a study of public opinion support for punitive policing practices in the city of Montevideo, Uruguay. We pay particular attention to iteration in the combination of qualitative fieldwork and experiments. We then discuss the contributions of the iterative process.</p> <hd id="AN0176861674-6">Iterating Between Experiments and Fieldwork: the Case of Punitive Security Policy in Uruguay</hd> <p>We illustrate our proposed iterative methodology with a study of public opinion support for punitive policing practices in the city of Montevideo, Uruguay ([<reflink idref="bib60" id="ref134">60</reflink>]). The study combines a survey experiment with intensive fieldwork, including direct observation and interviews. The steps we describe apply more generally to other mixed-methods designs, with or without experiments.</p> <p>Recently, Latin American countries have experienced a "security crisis," associated with a rise in crime and violence. Citizens' concern with security has pushed governments to adopt increasingly punitive responses that enjoy widespread support. Punitive policies—policing practices characterized by the escalation of the use of force, or threat thereof, and the selective application of the law—have yielded inconclusive results with respect to crime reduction ([<reflink idref="bib33" id="ref135">33</reflink>]). Furthermore, in many cases, these policies have led to increased levels of police violence ([<reflink idref="bib61" id="ref136">61</reflink>]).</p> <p>These seemingly divergent logics—the spread of punitive policing without positive results in terms of crime reduction—motivate the main research question in this study: why does public opinion support punitive policing? We argue that public opinion support for punitive policing is associated not with expected effectiveness of the practice, but with implicit beliefs and preferences about whether the target of punitive policing deserves such treatment. We further argue that individuals evaluate deservingness based on a target's socio-demographic attributes.</p> <p>To test this argument, we conducted an online survey experiment consisting of a factorial design with attributes embedded in a vignette. We administered the survey in Montevideo, the capital city of Uruguay, in 2020. Despite the country's relatively low level of violence, public opinion in Uruguay supports the implementation of punitive policing practices. Even though the study of support for punitive strategies tends to focus on countries where violence is higher, such as Mexico ([<reflink idref="bib23" id="ref137">23</reflink>]), recent work has identified similar policing strategies in low violence contexts ([<reflink idref="bib26" id="ref138">26</reflink>]). Prior to administering the survey, we iterated multiple times between theory, fieldwork, and the experimental design, which allowed us to more clearly specify the attributes of deservingness in a low violence context such as Uruguay.</p> <p>The decision to implement iteration systematically followed a deductive—inductive process. Initially, we followed the mixed-methods literature stating that it is important to verify that the components of an experiment resonate with lived experiences ([<reflink idref="bib55" id="ref139">55</reflink>]). Following this logic, we planned to gather insights directly from interviewees to probe two sets of assumptions: 1) assumptions in the theoretical literature about the attributes of deservingness; 2) assumptions in the experimental literature about the effects of experimental manipulations. However, we came to the decision to iterate multiple times inductively, as a result of the first phase of fieldwork, where we realized that we needed to adjust the design, and then verify whether it needed further adjustments.</p> <p>The following paragraphs describe the iterative process we employed in our theoretical and empirical design, emphasizing the use of qualitative fieldwork to refine the design: 1) preliminary theory and hypotheses and preliminary data collection strategies; 2) qualitative data collection (interviews, participant observation) to refine theory; 3) use of qualitative insights to refine the experimental component; 4) pre-test of quantitative component and subsequent refinement based on the results of the pre-test. We also employed qualitative data for analytical purposes (integration of qualitative and quantitative components of the design), but the iterative process takes place during the design phase (steps 1 through 4 in Figure 3), prior to testing. Figure 3 presents a summary of the entire process (design and analysis).</p> <p>Graph: Figure 3. Iteration in the refinement of testing tools.</p> <hd id="AN0176861674-7">Preliminary Theory and Hypotheses and Preliminary Data Collection Strategies (Step 1, no Iter...</hd> <p>Research designs usually start with a review of the literature and articulate a proto-theory including a set of hypotheses. Measurement instruments are selected and specified based on the variables defined in the theory.</p> <p>In our project, we first developed a theory based on insights from public policy research, which suggests that citizens support certain welfare policies based on whether they believe beneficiaries deserve the benefits ([<reflink idref="bib60" id="ref140">60</reflink>]). Research on welfare shows that individuals base their support for social welfare benefits on their perceptions of who deserves to receive them. For example, if beneficiaries are perceived as "lazy," individual support for welfare policies decreases. Deservingness is a cognitive shortcut; it operates by reinforcing pre-conceived notions based on incomplete information ([<reflink idref="bib50" id="ref141">50</reflink>]).</p> <p>[<reflink idref="bib60" id="ref142">60</reflink>] argue that much like welfare policy, support for punitive policing policies is based on perceptions of deservingness. Individuals have preconceived ideas about who is likely to be a criminal. Citizens see certain groups as criminal and as less deserving of guarantees of due process. Because deservingness is an abstract, complex concept, individuals use sociodemographic cues (as attributes of deservingness) to determine who does not deserve due process. For example, due to racial stereotypes, a black individual will be perceived as dangerous, likely to be a criminal, and thus, as less deserving of due process. These attributes are contextually constructed; therefore, they may vary across contexts.</p> <p>To identify which attributes of deservingness are relevant in a given context, we initially developed a forced-choice conjoint experiment. Conjoint experiments are popular in political science and they are frequently used to disentangle multidimensional preferences. They also minimize social desirability bias in survey responses. In regular surveys, respondents may be reluctant to state how they truly feel about individual features and police actions. Embedding each characteristic in a larger set of features and assigning them randomly to subjects may help reduce social desirability bias. Forced-choice conjoint experiments are usually embedded in a survey and consist of presenting respondents with a list of individual attributes and asking them to make a hypothetical choice ([<reflink idref="bib30" id="ref143">30</reflink>]). As explained in more detail below, after pre-testing our original design, we ended up presenting respondents with a vignette describing a hypothetical interaction between police and an individual in our final design. We also included an image of the purported individual, randomizing socio-demographic and appearance features. We then asked respondents to evaluate police behavior.</p> <hd id="AN0176861674-8">Qualitative Data Collection (Step 2)</hd> <p>Strategies for data collection can have multiple purposes in relation to the research design. In many cases, researchers collect data to test a theory. Qualitative data can also be used to build a theory, provide additional contextual information, or to refine hypotheses ([<reflink idref="bib62" id="ref144">62</reflink>]).</p> <p>We developed our argument in several iterations. After specifying our theory deductively, we used information obtained during fieldwork and we went back to refine the theory; this constitutes a first iteration in our design. We conducted our fieldwork over a period of three months in a pre-selected set of neighborhoods of Montevideo. We conducted interviews with about 40 individuals: government officials, police officers, community leaders, representatives of local governments, and ordinary neighbors. The purpose of the interviews was to obtain evidence about how the deservingness mechanism operates.</p> <p>As a result of the interviews, we were better able to understand respondents' perceptions of police behavior as well as their ideas about who deserves to be treated differently by the police. We also attended community meetings and reviewed documentation concerning security policy implementation. In addition, we reviewed press articles from four different newspapers to better grasp the context of crime, insecurity and police responses in Montevideo. These different sources were useful for triangulating and verifying information. We used insights from the interviews to refine our hypotheses and adjust the theory, and to refine the experimental design, as described below.</p> <hd id="AN0176861674-9">Refinement of Theory and Experimental Component (Step 3, Iteration)</hd> <p>In our original design, our intuition was that deservingness could be tied to attributes such as nationality, race, or location. However, because the concrete manifestation of some of these cleavages was not clear, and some of them are not particularly salient in the Uruguayan context (neither race nor xenophobic cleavages are particularly strong), we needed to use fieldwork systematically to refine our theory by identifying the relevant attributes of deservingness. Because deservingness is abstract and can be understood differently in different contexts, we needed to ensure that we were gathering evidence about it, and not some other related phenomenon. In many of our interviews, neighborhood residents stated that young, dark-skinned men who happened to be wearing baseball caps or hoodies, were a symbol of insecurity. For example:</p> <p>"I say to [my adolescent son] 'If you came up to me, and you weren't my son, I'd think you were going to rob me.' He says, 'But I don't dress bad!' 'No, but you wear that hat, those jeans, that jacket, and when you dress like that you look just like the rest of them.'"[<reflink idref="bib17" id="ref145">17</reflink>]</p> <p>Based on these insights, we adjusted the theory to incorporate social class as a relevant attribute of deservingness. We take attire to serve as a proxy for social class ([<reflink idref="bib61" id="ref146">61</reflink>]). This insight also led us to incorporate "attire" as a marker of deservingness in the set of descriptive features used in the experiment, which we would not have done had we not conducted interviews and informal conversations with neighborhood residents.</p> <p>Qualitative data in mixed-methods designs can also be used to refine the quantitative tools. For treatments to have an effect they should first resonate with individuals, they should be realistic ([<reflink idref="bib55" id="ref147">55</reflink>]). To improve our experiment, we incorporated observations from fieldwork by, 1) identifying relevant attributes to test in the experimental design; and, 2) incorporating descriptions of police actions and interactions with individuals to create more realistic scenarios.</p> <p>Here, we iterated again. In the original design, we selected a set of seven socio-demographic features—based on our review of the literature—as attributes associated with deservingness: sex, age, nationality, education, race, religion, and whether the individual had a criminal record. We also included actions by the hypothetical individual and police, as well as the neighborhood in which the encounter took place (see the appendix for a full description of the design). Once in the field, we used interviews and observations to validate the design. To ensure we had identified relevant attributes and situations, we asked interviewees to describe the kinds of interactions they had with the police (their own, or ones they knew about or had witnessed), and to provide information of the context in which the interactions occurred and of those involved. As a result, we learned about relevant manifestations of deservingness and common police behaviors. We then revised the characteristics listed in the experiment based on those descriptions, for example:</p> <p>"they are quite violent, they stopped a bus, got on (...) they wanted to arrest an adolescent who was on the bus with his uncle because they did not like the way he looked at them, they ended up throwing punches, the whole bus witnessed it"[<reflink idref="bib18" id="ref148">18</reflink>]</p> <p>"they would stop you on your way to work, take your backpack and throw all its contents on the floor, they were very aggressive"[<reflink idref="bib19" id="ref149">19</reflink>]</p> <p>We adjusted the original design based on the information from interviews. By contextualizing our design, we ensured that the instrument is an adequate measure of our concepts of interest ([<reflink idref="bib57" id="ref150">57</reflink>]). We employed an experiment, but mixed methods can be used to combine indicators such as scales, for example, or to identify new variables to include in regression analysis.</p> <hd id="AN0176861674-10">Pre-Tests (Step 4, Iteration)</hd> <p>Any research design containing a questionnaire benefits from pre-testing to check the mechanics and to ensure that the questions capture the phenomena of interest. Before administering the survey, we conducted a pre-test: we shared the survey with a small group to test its mechanics, and to evaluate our experimental design. We gathered data from about 35 respondents (undergraduate students and faculty members) and we asked them to take the survey and comment on the format and on anything they found unusual.</p> <p>The typical presentation of a forced-choice conjoint experiment includes two profiles that are being compared on a list of attributes. In our case, we presented respondents with two hypothetical individual profiles. Each profile comprised a series of socio-demographic attributes that varied at random, as well as a series of actions the police and a hypothetical individual were engaged in. After reviewing the comments, we concluded that this format made it hard for respondents to understand what they were expected to do, as several respondents mentioned they found the question hard to follow and a bit confusing. For example, two respondents stated that it was hard for them to compare the profiles, another one mentioned that she did not quite understand the sequence of actions, and yet another one suggested that a photo would summarize the physical characteristics more easily.</p> <p>Based on these results, we went back to the original design, and switched it: instead of presenting respondents with a list of attributes, we presented a vignette with an image. By substituting an image for the list of attributes, we reduced some of the task complexity because an individual's attributes were summarized in a photo. This change reduced the number of dimensions that respondents needed to compare and made it more engaging and easier to understand. Instead of asking respondents to compare between profiles, we presented the vignette four times, varying the attributes. We present a comparison between the two designs in the appendix.</p> <p>To obtain the photos, we hired actors. Once we adjusted the design incorporating the images, we conducted a manipulation check as part of the pre-test. Scholarly work on experimental research emphasizes the importance of conducting "manipulation checks," to ensure that the experimental treatment is working as expected ([<reflink idref="bib47" id="ref151">47</reflink>]). The manipulation check consisted of asking respondents in the pre-test to rate the images based on a series of characteristics (race, dangerousness, social class, attractiveness) to ensure that our treatments of interest were actually having an effect, and to rule out confounding factors. We showed respondents 15 images of 5 actors. Based on the manipulation check, we selected the specific images we presented. We finally pre-tested the whole survey again with the new experimental design.</p> <hd id="AN0176861674-11">Quantitative Data Collection and Analysis (Step 5 and 6, Analytical)</hd> <p>The last step is the implementation of the quantitative component and subsequent data analysis. In this stage, qualitative insights can be used to amplify the quantitative results. We administered our survey online between January 29 and March 17, 2020, targeting individuals at least 18 years old who resided in Montevideo. We collected about 2,900 responses. In the analytical phase of the research process, rather than iterating, we combined methodological approaches through the process of integration.</p> <p>We integrated the analysis of the experimental results with insights from our interviews. The experiment produced two important findings. First, whether someone is seen as deserving more violent treatment from the police depends on the individual's social class and their actions. Second, regardless of the identity and behavior of the individual in the vignette, respondents justify stop-and-frisk as a reasonable police action. Because the experiment does not tell us why individuals see this police practice as reasonable, we drew insights from our interviews to better contextualize this finding.</p> <p>During our conversations with police officers and neighborhood residents, we learned that stop-and-search is a very common practice. People are used to seeing the police conduct this activity, which suggests that this finding might be the result of a normalized routine practice. Residents of neighborhoods in Montevideo also state that police stop individuals based on the way they look—i.e. "profiling"—a practice they deem necessary to combat crime.</p> <p>This study could have been conducted by constructing the experiment based only on prior research, without incorporating fieldwork. However, successive iterations between fieldwork and theory, and between fieldwork and design and analysis, resulted in a much richer research design and analysis, and stronger causal inference. Fieldwork revealed the relevance of attributes we had not considered, and the irrelevance of others we previously thought were important. Though we have described our research process as a linear sequence of steps, the iterative process is far from linear, as captured in Figure 3.</p> <hd id="AN0176861674-12">Extending Iteration Beyond Experimental Designs</hd> <p>In our study of attitudes towards punitive policing, we improved the operationalization and measurement of our variables through iteration. Insights from fieldwork allowed us to identify and measure attributes of the concept of deservingness. Using interviews, we identified the most relevant attributes and adapted the experiment accordingly. Furthermore, because the notion of deservingness varies by context (in some cases it may be more closely linked to race while in other cases it may be linked more to social class, or to other attributes), using only deduction to select which attributes to study may lead to erroneous measurement and interpretation. The refinement of testing tools lies somewhere in between theory building and testing, and it involves more than pre-testing of instruments for data collection.</p> <p>Making iteration explicit in an experimental context allows for increased transparency in research design because researchers can show all the steps involved in the construction of the design. This process facilitates replication of research in different contexts, as it makes clear to researchers which parts of the design should be adapted.</p> <p>Although researchers value the benefits of iteration, the way in which it occurs is rarely made explicit, which obscures its benefits. Iteration also calls attention to research design as a non-linear process: it requires alternating back-and-forth between different elements of the design, where researchers update prior information ([<reflink idref="bib16" id="ref152">16</reflink>]; [<reflink idref="bib38" id="ref153">38</reflink>]; [<reflink idref="bib62" id="ref154">62</reflink>]). When an experiment includes insights from fieldwork, the researcher ensures that experimental results are not an artifact of the research design, thus lending added credibility to the findings.</p> <p>To make iteration explicit, researchers should document each step in the process and pre-register the entire process in a PAP. Figure 4 demonstrates a series of practical steps researchers can take to ensure proper iteration and documentation when combining fieldwork and experimental designs. The key issue is to ask whether theory, hypotheses and/or design need to be updated in light of fieldwork. If any of these elements need to be updated, another documentation phase is required, the previous PAP needs to be revised. In revising the PAP, researchers should explicitly state what evidence leads to the proposed change. If none of these elements are updated, the research process continues through a pre-test. In light of the pre-test, a similar question is asked: does the experimental design need to be updated? The researcher should repeat the same steps, documenting as needed, until quantitative data collection can begin. Documentation and updating of PAPs is fundamental to ensure transparency and replicability of the research process as a whole.</p> <p>Graph: Figure 4. Steps to implement iterative designs with fieldwork and experiments.</p> <p>Our approach is applicable to designs with other types of quantitative elements. Iteration works to strengthen theory and data collection tools and can be applied to designs containing interviews or survey questionnaires which are then used in, for example, regression analyses. The choice of qualitative technique to inform quantitative methods is at the service of the research question. In our example, the aim was to understand the attributes of deservingness to improve measurement, thus we needed to speak to individuals. We conducted one- on- one and group interviews to refine our theory. However, we could have taken an inductive starting point and used ethnographic methods instead. [<reflink idref="bib59" id="ref155">59</reflink>] and [<reflink idref="bib19" id="ref156">19</reflink>] used qualitative methods to build their respective samples. Thachil uses ethnography to identify the relevant socio-demographic attributes of his sample. The specific method may vary based on what tools the researcher needs to answer her research question and to probe the assumptions behind other methods ([<reflink idref="bib54" id="ref157">54</reflink>]). Yet, these choices may also be determined by pragmatic considerations: time, budget and access. Researchers should also factor in these elements when deciding how to iterate.</p> <hd id="AN0176861674-13">Conclusions</hd> <p>This paper highlights the value of iteration between different elements of the research design to improve research combining qualitative elements and experiments, prior to testing. We seek to fill a gap in the literature, namely, qualitative research that highlights the value of iteration does not address experimental work, and, in general, experimental work does not combine methods explicitly. According to our analysis of EGAP's pre-analysis plans, most experiments do not make the process of combining methods explicit. However, researchers would benefit from making the process more transparent.</p> <p>We make three contributions: First, we highlight the role of iteration in improving causal inference in mixed-methods research designs using experiments. Related, the project demonstrates improvements to concept operationalization and measurement gained by paying attention to the context. Second, we showcase the importance of making explicit the incorporation of qualitative elements in experimental designs. By outlining the steps involved in iteration, we also contribute to discussions of transparency in research design. Third, the paper offers insights to better understand the conditions under which experimental work is generalizable. More generally, the steps we propose can apply to other designs that combine qualitative and quantitative methods.</p> <p>Iteration helps a researcher to improve causal inference. For example, researchers usually begin the research process by proposing a theory, a few hypotheses, and a research design to test the argument. The researcher can choose to take a linear path between theory and testing, or to iterate. If she chooses the latter, she can, for example, use interviews to better understand the context of the experimental intervention. A better understanding of the context allows her to refine her hypotheses and create more realistic interventions, thus strengthening internal validity. Iteration is also relevant in non-experimental designs, where researchers start with deduction and refine their hypotheses in light of their interaction with data ([<reflink idref="bib62" id="ref158">62</reflink>]).</p> <p>Field research remains underutilized as a tool to improve causal inference and other aspects of the research design. This may result from a poor understanding of its advantages: aside from strengthening causal inference, iterating with fieldwork produces more realistic treatments because it increases contextual knowledge. In addition, being explicit about iteration makes research more transparent and more easily replicated. By documenting the steps involved in iteration and making the documentation public in a PAP allows for other researchers to replicate the design and to learn from other researchers' process regardless of publication status.</p> <p>One of our project's key contributions concerns the ability to generalize in experimental work. As mentioned before, one of the limitations of experimental designs is the context dependency of treatment effects. However, experimental researchers are concerned about assessing the extent to which experimental designs can "travel." Along these lines, [<reflink idref="bib6" id="ref159">6</reflink>] identify a series of shortcomings in current approaches for dealing with generalizability in experimental contexts. They propose a design-driven approach, using multi-arm bandit algorithms, to evaluate the generalizability of treatment effects. We suggest that iteration with fieldwork can also help ameliorate one of the most important limitations the authors highlight, namely how treatments are assigned to contexts.</p> <p>[<reflink idref="bib6" id="ref160">6</reflink>] 421) explain that researchers usually use "intuition" to choose the most effective intervention for a given context. This is problematic because, aside from eliminating other alternative treatments, it implies that pre-conceived notions about what might work are the basis for the design. Our work, however, shows that conducting fieldwork allows researchers to design treatments that take the context into consideration, thus minimizing the incorporation of researchers' pre-conceived notions in the research design. As Robert Bates has noted, conducting fieldwork allows researchers to change preconceived notions they might have and to adapt their theories to reality ([<reflink idref="bib46" id="ref161">46</reflink>]:25). A good understanding of contextual features helps clarify the scope of the theory being tested, and thus, where else it might apply.</p> <p>Even though iterating between fieldwork and experiments has many advantages, there are some practical limitations: costs, knowing when to stop, and the possibility of inducing confirmation bias. Fieldwork is time-consuming and costly, it entails coordinating meetings, interviews, traveling to remote or difficult-to-access locations, and may even jeopardize researchers' safety.[<reflink idref="bib20" id="ref162">20</reflink>] However, even under these circumstances, iteration is still possible. Other qualitative tools can be incorporated to understand the context and improve designs. For example, researchers can use information from newspapers or secondary literature about cases to contextualize their experimental interventions. Another risk that researchers should be aware of is the tendency to continue to refine the design indefinitely. A good place to stop is when field observations no longer provide new evidence to refine the design. For example, when information from interviews or observation begins to be repeated across different respondents ([<reflink idref="bib53" id="ref163">53</reflink>]). Last, researchers may wonder about whether incorporating realism might introduce demand effects—where respondents may guess what the expected response should be and act accordingly ([<reflink idref="bib45" id="ref164">45</reflink>]). We believe that concerns about priming participants are mitigated when researchers employ qualitative methods to enhance experimental realism. No method is completely exempt from confirmatory bias, but research suggests that the risk of generating demand effects in survey experiments is generally low ([<reflink idref="bib45" id="ref165">45</reflink>]). In addition, presenting real-life situations based on qualitative evidence instead of manufactured ones mitigates priming biases by accurately representing the context ([<reflink idref="bib9" id="ref166">9</reflink>]).</p> <p>Iteration in mixed-methods designs can happen in multiple ways, and it has slowly become more widely used. Researchers concerned with improving causal inference and producing transparent, replicable designs, would benefit from explicitly including iteration as part of their research process.</p> <hd id="AN0176861674-14">Supplemental Material</hd> <p>Graph: Supplemental material, sj-docx-1-smr-10.1177_00491241221082595 for Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork[<reflink idref="bib5" id="ref167">5</reflink>],[<reflink idref="bib6" id="ref168">6</reflink>] by Verónica Pérez Bentancur and Lucía Tiscornia in Sociological Methods &amp; Research</p> <ref id="AN0176861674-15"> <title> References </title> <blist> <bibl id="bib1" idref="ref7" type="bt">1</bibl> <bibtext> Auerbach Adam, Thachil Tariq. 2020. " Cultivating Clients: Reputation, Responsiveness, and Ethnic Indifference in India's Slums." American Journal of Political Science. 64(3):471‐87.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref8" type="bt">2</bibl> <bibtext> Beach Derek, Littvay Levente. 2020. "A More Conscious Experimental Design: How More Explicit Use of Process Tracing Can Improve Experiments." 116th APSA Annual Meeting, September 30 – October 3.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref131" type="bt">3</bibl> <bibtext> Bezzola, Selina, Fritz Brugger, Isabel Günther, and Dawit Sebhatu. 2021. " Do Social Investments by Mining Companies Harm Citizen-State Relations? Experimental Evidence from Burkina Faso." The Journal of Development Studies: 1‐19. https://doi.org/10.1080/00220388.2021.1983166.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref29" type="bt">4</bibl> <bibtext> Blair Graeme, Cooper Jasper, Coppock Alexander, Humphreys Macartan. 2019. " Declaring and Diagnosing Research Designs." American Political Science Review. 113(3):838‐59. https://doi.org/10.1017/S0003055419000194.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref2" type="bt">5</bibl> <bibtext> Blair Graeme, Coppock Alexander, Humphreys Macartan. Forthcoming. Research Design: Declaration, Diagnosis, Redesign. New Jersey: Princeton University Press.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref38" type="bt">6</bibl> <bibtext> Blair Graeme, McClendon Gwyneth. 2021. " Conducting Experiments in Multiple Contexts." Pp. 411‐28 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref55" type="bt">7</bibl> <bibtext> Boudreau Cheryl. 2021. " Transparency in Experimental Research." Pp. 339‐53 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref22" type="bt">8</bibl> <bibtext> Brady Henry, Collier David. 2010. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Maryland: Rowman &amp; Littlefield Publishers.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref73" type="bt">9</bibl> <bibtext> Carpenter Charli, Montgomery Alexander H., Nylen Alexandria. 2021. " Breaking Bad? How Survey Experiments Prime Americans for War Crimes." Perspectives on Politics. 19(3):912‐24. https://doi.org/10.1017/S1537592720002376.</bibtext> </blist> <blist> <bibtext> Chappuis Fairlie, Krause Jana. 2019. " Research Dilemmas in Dangerous Places." Pp. 112‐25 in Secrecy and Methods in Security Research: A Guide to Qualitative Fieldwork, edited by De Goede Marieke, Bosma Esmé, Pallister-Wilkins Polly. New York: Routledge.</bibtext> </blist> <blist> <bibtext> Clayton Amanda, Robinson Amanda Lea, Johnson Martha C., Muriaas Ragnhild. 2020. " (How) Do Voters Discriminate Against Women Candidates? Experimental and Qualitative Evidence From Malawi." Comparative Political Studies. 53(3–4):601‐30. https://doi.org/10.1177/0010414019858960.</bibtext> </blist> <blist> <bibtext> Cooperman Alicia Dailey. 2019. " Trading Favors: Local Politics and Development in Brazil." New York: Columbia University. https://academiccommons.columbia.edu/doi/10.7916/d8-azrz-6g03.</bibtext> </blist> <blist> <bibtext> Cyr Jennifer. 2019. " An Integrative Approach to Measurement: Focus Groups as a Survey Pretest." Quality &amp; Quantity. 53:897‐913. https://doi.org/10.1007/s11135-018-0795-5</bibtext> </blist> <blist> <bibtext> Deaton Angus, Cartwright Nancy. 2018. "Understanding and Misunderstanding Randomized Controlled Trials." Social Science &amp; Medicine, Randomized Controlled Trials and Evidence-based Policy: A Multidisciplinary Dialogue210(August):2‐21. https://doi.org/10.1016/j.socscimed.2017.12.005.</bibtext> </blist> <blist> <bibtext> Druckman James N., Green Donald P. 2021. " A New Era of Experimental Political Science." Pp. 1‐15 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Dunning Thad. 2008. " Natural and Field Experiments: The Role of Qualitative Methods." Qualitative &amp; Multi-Method Research. Fall:17‐23.</bibtext> </blist> <blist> <bibtext> Dunning Thad. 2012. Natural Experiments in the Social Sciences: A Design-Based Approach. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Dunning Thad. 2015. " Improving Process Tracing: The Case of Multi-Method Research." Pp. 211‐36 in Process Tracing. Fom Metaphor to Analytic Tool, edited by Bennett Andrew, Checkel Jeffrey T. Nueva York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Dunning Thad, Harrison Lauren. 2010. " Cross-Cutting Cleavages and Ethnic Voting: An Experimental Study of Cousinage in Mali." The American Political Science Review. 104(1):21‐39. https://doi.org/10.2307/27798538.</bibtext> </blist> <blist> <bibtext> Elman Colin, Gerring John, Mahoney James. 2020. The Production of Knowledge: Enhancing Progress in Social Science. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Fairfield Tasha, Charman Andrew E. 2017. " Explicit Bayesian Analysis for Process Tracing: Guidelines, Opportunities, and Caveats." Political Analysis. 25(3):363‐80. https://doi.org/10.1017/pan.2017.14.</bibtext> </blist> <blist> <bibtext> Ferwerda Jeremy, Miller Nicholas L. 2014. " Political Devolution and Resistance to Foreign Rule: A Natural Experiment." American Political Science Review. 108(3):642‐60. https://doi.org/10.1017/S0003055414000240.</bibtext> </blist> <blist> <bibtext> García-Ponce Omar, Young Lauren, Zeitzoff Thomas. 2019. "Anger and Support for Punitive Justicein Mexico's DrugWar." APSA Annual Meeting, Washington DC.</bibtext> </blist> <blist> <bibtext> Gerber Alan S., Green Donald P. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W. W. Norton.</bibtext> </blist> <blist> <bibtext> Gerring John. 2011. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Gingerich Daniel, Scartascini Carlos. 2018. " A Heavy Hand or a Helping Hand? Information Provision and Citizen Preferences for Anti-Crime Policies." Inter-American Development Bank.</bibtext> </blist> <blist> <bibtext> Goertz Gary. 2017. Multimethod Research, Causal Mechanisms, and Case Studies: An Integrated Approach. New Jersey: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Gonzalez-Ocantos Ezequiel, LaPorte Jody. 2019. " Process Tracing and the Problem of Missing Data." Sociological Methods &amp; Research, 50(3):1407–35. https://doi.org/10.1177/0049124119826153.</bibtext> </blist> <blist> <bibtext> Gonzalez-Ocantos Ezequiel, Masullo Juan. 2020. " Aligning Interviewing with Process Tracing." working paper.</bibtext> </blist> <blist> <bibtext> Hainmueller Jens, Hopkins Daniel J., Yamamoto Teppei. 2014. " Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments." Political Analysis. 22(1):1‐30. https://doi.org/10.1093/pan/mpt024.</bibtext> </blist> <blist> <bibtext> Ham, Andrés, Darío Maldonado, Michael Weintraub, Andrés Felipe Camacho, and Daniela Gualtero. 2019. " Alcohol and Violence: A Field Experiment with Bartenders in Bogotá, Colombia." Universidad de los Andes/Escuela de Gobierno Alberto Lleras Camargo. Working paper 70, https://egob.uniandes.edu.co/images/books/DT/DT-70.pdf.</bibtext> </blist> <blist> <bibtext> Harbers Imke, Ingram Matthew C. 2020. " Mixed-Methods Designs." Pp. 1117–32 in The SAGE Handbook of Research Methods in Political Science and International Relations, edited by Curini Luigi, Franzese Robert. United Kingdom: Sage.</bibtext> </blist> <blist> <bibtext> Holland Alisha C. 2013. " Right on Crime?: Conservative Party Politics and Mano Dura Policies in El Salvador." Latin American Research Review. 48(1):44‐67. https://doi.org/10.1353/lar.2013.0009.</bibtext> </blist> <blist> <bibtext> Hollstein Betina. 2014. " Mixed Methods Social Networks Research: An Introduction." Pp. 3–34 in Mixed Methods Social Networks Research. Design and Applications, edited by Hollstein Betina, Domínguez Silvia. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Jackson Michelle, Cox David R. 2013. " The Principles of Experimental Design and Their Application in Sociology." Annual Review of Sociology. 39:27‐49.</bibtext> </blist> <blist> <bibtext> Jacobs Alan M.2020. " Pre-Registration and Results-Free Review in Obsevational and Qualitative Research." Pp. 221–64 in The Production of Knowledge. Enhancing Progress in Social Science, edited by Elman Colin, Gerring John, Mahoney James. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Jha Saumitra, Rao Vijayendra, Woolcock Michael. 2007. " Governance in the Gullies: Democratic Responsiveness and Leadership in Delhi's Slums." World Development, Experiences of Combining Qualitative and Quantitative Approaches in Poverty Analysis. 35(2):230‐46. https://doi.org/10.1016/j.worlddev.2005.10.018.</bibtext> </blist> <blist> <bibtext> Kapiszewski Diana, MacLean Lauren M., Read Benjamin L. 2015. Field Research in Political Science. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> King Gary, Keohane Robert O., Verba Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. New Jersey: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Kocher Matthew A., Monteiro Nuno P. 2016. " Lines of Demarcation: Causation, Design-Based Inference, and Historical Research." Perspectives on Politics. 14(4):952‐75. https://doi.org/10.1017/S1537592716002863.</bibtext> </blist> <blist> <bibtext> Levy Paluck Elizabeth. 2010. " The Promising Integration of Qualitative Methods and Field Experiments." The ANNALS of the American Academy of Political and Social Science. 628(1):59‐71. https://doi.org/10.1177/0002716209351510.</bibtext> </blist> <blist> <bibtext> Lieberman Evan S. 2005. " Nested Analysis as a Mixed-Method Strategy for Comparative Research." American Political Science Review. 99(3):435‐52. https://doi.org/10.1017/S0003055405051762.</bibtext> </blist> <blist> <bibtext> Lupia Arthur, Elman Colin. 2014. " Openness in Political Science: Data Access and Research Transparency: Introduction." PS: Political Science &amp; Politics. 47(1):19‐42. https://doi.org/10.1017/S1049096513001716.</bibtext> </blist> <blist> <bibtext> Malhotra Neil. 2021. " Threats to the Scientific Credibility of Experiments: Publications Bias and P-Haking." Pp. 354‐68 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Mummolo Jonathan, Peterson Erik. 2019. " Demand Effects in Survey Experiments: An Empirical Assessment." American Political Science Review. 113(2):517‐29. https://doi.org/10.1017/S0003055418000837.</bibtext> </blist> <blist> <bibtext> Munck Gerardo L., Snyder Richard, eds. 2007. Passion, Craft, and Method in Comparative Politics. Baltimore: JHU Press.</bibtext> </blist> <blist> <bibtext> Mutz Diana. 2021. " Improving Experimental Treatments in Political Science." Pp. 219‐38 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Pérez Bentancur Verónica, Rodríguez Rafael Piñeiro, Rosenblatt Fernando. 2018. " Unexplored Advantages of DART for Qualitative Research." Qualitative &amp; Multi-Method Research. 16(2):31‐5. https://doi.org/10.5281/zenodo.3524354.</bibtext> </blist> <blist> <bibtext> Pérez Bentancur Verónica, Tiscornia Lucía. Forthcoming. " Transparency and Replicability in Mixed-Methods Designs Using Experiments." Qualitative and Multi-Method Research.</bibtext> </blist> <blist> <bibtext> Petersen Michael Bang. 2012. " Social Welfare as Small-Scale Help: Evolutionary Psychology and the Deservingness Heuristic." American Journal of Political Science. 56(1):1‐16. https://doi.org/10.1111/j.1540-5907.2011.00545.x.</bibtext> </blist> <blist> <bibtext> Rao Vijayendra, Ananthpur Kripa, Malik Kabir. 2017. " The Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India." World Development. 99(November):481‐97. https://doi.org/10.1016/j.worlddev.2017.05.037.</bibtext> </blist> <blist> <bibtext> Redlich Revkin Mara. 2020. " What Explains Taxation by Resource-Rich Rebels? Evidence from the Islamic State in Syria." Journal of Politics 82(2):757–64. https://doi.org/10.2139/ssrn.3023317.</bibtext> </blist> <blist> <bibtext> Saunders Benjamin, Sim Julius, Kingstone Tom, Baker Shula, Waterfield Jackie, Bartlam Bernadette, Burroughs Heather, Jinks Clare. 2018. " Saturation in Qualitative Research: Exploring its Conceptualization and Operationalization." Quality &amp; Quantity. 52(4):1893‐907.</bibtext> </blist> <blist> <bibtext> Seawright Jason. 2016. Multi-Method Social Science: Combining Qualitative and Quantitative Tools. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Seawright Jason. 2021. " What Can Multi-Method Research Add to Experiments? " Pp. 369‐84 in Advances in Experimental Political Science, edited by Druckman James N., Green Donald P. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Seawright Jason, Gerring John. 2008. " Case Selection Techniques in Case Study Research: A Menu of Qualitative and Quantitative Options." Political Research Quarterly. 61(2):294‐308. https://doi.org/10.1177/1065912907313077.</bibtext> </blist> <blist> <bibtext> Steiner Peter M., Atzmüller Christiane, Su Dan. 2016. " Designing Valid and Reliable Vignette Experiments for Survey Research: A Case Study on the Fair Gender Income Gap." Journal of Methods and Measurement in the Social Sciences. 7(2):52‐94. https://doi.org/10.2458/v7i2.20321.</bibtext> </blist> <blist> <bibtext> Thachil Tariq. 2017. " Do Rural Migrants Divide Ethnically in the City? Evidence from an Ethnographic Experiment in India." American Journal of Political Science. 61(4):908‐26. https://doi.org/10.1111/ajps.12315.</bibtext> </blist> <blist> <bibtext> Thachil Tariq. 2018. " Improving Surveys Through Ethnography: Insights from India's Urban Periphery." Studies in Comparative International Development. 53(3):281‐99.</bibtext> </blist> <blist> <bibtext> Tiscornia Lucía, Bentancur Verónica Pérez, Albarracín Juan, MacColman Leslie. 2021. "The Social Basis of Punitive Policing: Public Opinion and Perceptions of Deservingness." In Annual Meeting of the International Studies Association (ISA). Virtual Meeting.</bibtext> </blist> <blist> <bibtext> Wolf Sonja. 2017. Mano Dura: The Politics of Gang Control in El Salvador. Austin: University of Texas Press.</bibtext> </blist> <blist> <bibtext> Yom Sean. 2015. " From Methodology to Practice: Inductive Iteration in Comparative Research." Comparative Political Studies. 48(5):616‐44. https://doi.org/10.1177/0010414014554685.</bibtext> </blist> </ref> <ref id="AN0176861674-16"> <title> Footnotes </title> <blist> <bibtext> The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Agencia Nacional de Investigación e Innovación, Uruguay (grant number FSSC_1_2018_1_147720).</bibtext> </blist> <blist> <bibtext> Verónica Pérez Bentancur https://orcid.org/0000-0002-9855-3097 Lucía Tiscornia https://orcid.org/0000-0001-7871-1708</bibtext> </blist> <blist> <bibtext> Supplemental material for this article is available online.</bibtext> </blist> <blist> <bibtext> Authors in alphabetical order, equal contribution assumed.</bibtext> </blist> <blist> <bibtext> The authors acknowledge funding from Agencia Nacional de Investigación e Innovación (ANII, Uruguay), project number FSSC_1_2018_1_147720 (with Juan Albarracin and Leslie MacColman).</bibtext> </blist> <blist> <bibtext> Qualitative methodology encompasses a wide variety of tools with varying degrees of involvement and difficulty (in depth ethnography, interviews, review of press, focus groups, among others). Ultimately, the method selected will depend not only on the problem to be addressed, but also on considerations about time and resources.</bibtext> </blist> <blist> <bibtext> We are grateful to an anonymous reviewer for this insight.</bibtext> </blist> <blist> <bibtext> In process-tracing designs, some interviews are also "one shot". However, because interviews are sequenced, the researcher does not conduct all interviews at once and can make changes to the interview protocol. In addition, different sources of information can inform the same piece of evidence ([29]).</bibtext> </blist> <blist> <bibtext> For 119 of the registered designs, documentation is either gated or there is no additional documentation. available.</bibtext> </blist> <blist> <bibtext> See here: <ulink href="http://egap.org">http://egap.org</ulink></bibtext> </blist> <blist> <bibtext> This is explicitly stated in EGAP's statement of principles.</bibtext> </blist> <blist> <bibtext> In the appendix we provide results of two sets of additional analyses: 1) a search of manuscripts based on all the PAPs that stated they used mixed-methods, plus a random sample of 10 others, for a total of 33; 2) the 2020 issue of the American Political Science Review (APSR), one of the most important journals in political science. The results of this analysis are consistent with our argument.</bibtext> </blist> <blist> <bibtext> For an exception where a research design provides further detail than the PAP, see [12] "Trading Favors: Local Politics and Development in Brazil." In the appendix, she provides extensive detail about her iterative process between fieldwork and design for the refinement of theory and hypotheses in an observational study.</bibtext> </blist> <blist> <bibtext> Registered design available here: https://osf.io/nxh7v</bibtext> </blist> <blist> <bibtext> Registered design available here: https://osf.io/8v5tw</bibtext> </blist> <blist> <bibtext> Interview, neighborhood resident, June 27, 2019.</bibtext> </blist> <blist> <bibtext> Interview, government official, August 1, 2019.</bibtext> </blist> <blist> <bibtext> Interview, neighborhood resident, June 13, 2019.</bibtext> </blist> <blist> <bibtext> This challenge may be more acute for women researchers, who may face a higher risk of experiencing violence than men.</bibtext> </blist> </ref> <aug> <p>By Verónica Pérez Bentancur and Lucía Tiscornia</p> <p>Reported by Author; Author</p> <p></p> <p>Verónica Pérez Bentancur, PhD in Political Science (Universidad Torcuato Di Tella, 2019); research interests: Latin American politics, political parties, gender and politics; published in Comparative Political Studies, Politics and Gender. Book: "How Party Activism Survives: Uruguay's Frente Amplio," (CUP, 2020), winner of the Leon Epstein Outstanding Book Award from APSA.</p> <p>Lucía Tiscornia, PhD in Political Science (University of Notre Dame, 2019); research interests: political and criminal violence, security sector reform, conflict processes and transitional justice; published in Journal of Peace Research.</p> </aug> <nolink nlid="nl1" bibid="bib15" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib28" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib17" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib24" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref9"></nolink> <nolink nlid="nl6" bibid="bib19" firstref="ref11"></nolink> <nolink nlid="nl7" bibid="bib54" firstref="ref12"></nolink> <nolink nlid="nl8" bibid="bib55" firstref="ref13"></nolink> <nolink nlid="nl9" bibid="bib59" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib21" firstref="ref17"></nolink> <nolink nlid="nl11" bibid="bib38" firstref="ref18"></nolink> <nolink nlid="nl12" bibid="bib18" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib41" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib25" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib27" firstref="ref24"></nolink> <nolink nlid="nl16" bibid="bib39" firstref="ref25"></nolink> <nolink nlid="nl17" bibid="bib42" firstref="ref26"></nolink> <nolink nlid="nl18" bibid="bib43" firstref="ref28"></nolink> <nolink nlid="nl19" bibid="bib20" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib35" firstref="ref32"></nolink> <nolink nlid="nl21" bibid="bib30" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib14" firstref="ref37"></nolink> <nolink nlid="nl23" bibid="bib13" firstref="ref48"></nolink> <nolink nlid="nl24" bibid="bib52" firstref="ref52"></nolink> <nolink nlid="nl25" bibid="bib32" firstref="ref57"></nolink> <nolink nlid="nl26" bibid="bib34" firstref="ref58"></nolink> <nolink nlid="nl27" bibid="bib62" firstref="ref64"></nolink> <nolink nlid="nl28" bibid="bib29" firstref="ref74"></nolink> <nolink nlid="nl29" bibid="bib11" firstref="ref78"></nolink> <nolink nlid="nl30" bibid="bib51" firstref="ref83"></nolink> <nolink nlid="nl31" bibid="bib53" firstref="ref93"></nolink> <nolink nlid="nl32" bibid="bib56" firstref="ref94"></nolink> <nolink nlid="nl33" bibid="bib10" firstref="ref95"></nolink> <nolink nlid="nl34" bibid="bib22" firstref="ref97"></nolink> <nolink nlid="nl35" bibid="bib40" firstref="ref98"></nolink> <nolink nlid="nl36" bibid="bib37" firstref="ref106"></nolink> <nolink nlid="nl37" bibid="bib58" firstref="ref107"></nolink> <nolink nlid="nl38" bibid="bib36" firstref="ref116"></nolink> <nolink nlid="nl39" bibid="bib12" firstref="ref117"></nolink> <nolink nlid="nl40" bibid="bib48" firstref="ref119"></nolink> <nolink nlid="nl41" bibid="bib49" firstref="ref121"></nolink> <nolink nlid="nl42" bibid="bib44" firstref="ref124"></nolink> <nolink nlid="nl43" bibid="bib108" firstref="ref127"></nolink> <nolink nlid="nl44" bibid="bib230" firstref="ref128"></nolink> <nolink nlid="nl45" bibid="bib31" firstref="ref129"></nolink> <nolink nlid="nl46" bibid="bib60" firstref="ref134"></nolink> <nolink nlid="nl47" bibid="bib33" firstref="ref135"></nolink> <nolink nlid="nl48" bibid="bib61" firstref="ref136"></nolink> <nolink nlid="nl49" bibid="bib23" firstref="ref137"></nolink> <nolink nlid="nl50" bibid="bib26" firstref="ref138"></nolink> <nolink nlid="nl51" bibid="bib50" firstref="ref141"></nolink> <nolink nlid="nl52" bibid="bib57" firstref="ref150"></nolink> <nolink nlid="nl53" bibid="bib47" firstref="ref151"></nolink> <nolink nlid="nl54" bibid="bib46" firstref="ref161"></nolink> <nolink nlid="nl55" bibid="bib45" firstref="ref164"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1423057 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Verónica+Pérez+Bentancur%22">Verónica Pérez Bentancur</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-9855-3097">0000-0002-9855-3097</externalLink>)<br /><searchLink fieldCode="AR" term="%22Lucía+Tiscornia%22">Lucía Tiscornia</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7871-1708">0000-0001-7871-1708</externalLink>) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2024 53(2):729-759. – Name: Avail Label: Availability Group: Avail Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 31 – Name: DatePubCY Label: Publication Date Group: Date Data: 2024 – Name: Audience Label: Intended Audience Group: Audnce Data: Researchers – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Science+Research%22">Social Science Research</searchLink><br /><searchLink fieldCode="DE" term="%22Public+Opinion%22">Public Opinion</searchLink><br /><searchLink fieldCode="DE" term="%22Punishment%22">Punishment</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Experiments%22">Experiments</searchLink><br /><searchLink fieldCode="DE" term="%22Mixed+Methods+Research%22">Mixed Methods Research</searchLink><br /><searchLink fieldCode="DE" term="%22Field+Studies%22">Field Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Pretesting%22">Pretesting</searchLink> – Name: Subject Label: Geographic Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Uruguay%22">Uruguay</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1177/00491241221082595 – Name: ISSN Label: ISSN Group: ISSN Data: 0049-1241<br />1552-8294 – Name: Abstract Label: Abstract Group: Ab Data: Experimental designs in the social sciences have received increasing attention due to their power to produce causal inferences. Nevertheless, experimental research faces limitations, including limited external validity and unrealistic treatments. We propose combining qualitative fieldwork and experimental design iteratively--moving back-and-forth between elements of a research design--to overcome these limitations. To properly evaluate the strength of experiments researchers need information about the context, data, and previous knowledge used to design the treatment. To support our argument, we analyze 338 pre-analysis plans submitted to the Evidence in Governance and Politics repository in 2019 and the design of a study on public opinion support for punitive policing practices in Montevideo, Uruguay. The paper provides insights about using qualitative fieldwork to enhance the external validity, transparency and replicability of experimental research, and a practical guide for researchers who want to incorporate iteration to their research designs. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2024 – Name: AN Label: Accession Number Group: ID Data: EJ1423057 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1423057 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1177/00491241221082595 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 31 StartPage: 729 Subjects: – SubjectFull: Research Design Type: general – SubjectFull: Social Science Research Type: general – SubjectFull: Public Opinion Type: general – SubjectFull: Punishment Type: general – SubjectFull: Foreign Countries Type: general – SubjectFull: Experiments Type: general – SubjectFull: Mixed Methods Research Type: general – SubjectFull: Field Studies Type: general – SubjectFull: Data Collection Type: general – SubjectFull: Pretesting Type: general – SubjectFull: Uruguay Type: general Titles: – TitleFull: Iteration in Mixed-Methods Research Designs Combining Experiments and Fieldwork Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Verónica Pérez Bentancur – PersonEntity: Name: NameFull: Lucía Tiscornia IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 0049-1241 – Type: issn-electronic Value: 1552-8294 Numbering: – Type: volume Value: 53 – Type: issue Value: 2 Titles: – TitleFull: Sociological Methods & Research Type: main |
| ResultId | 1 |
