View in EDS HTML Full Text PDF Full Text

The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys

Saved in:
Bibliographic Details
Title: The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys
Language: English
Authors: Patricia Hadler (ORCID 0000-0003-4901-8618)
Source: Sociological Methods & Research. 2025 54(1):106-139.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 34
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Online Surveys, Discovery Processes, Test Items, Data Collection, Cost Effectiveness, Questionnaires, Response Rates (Questionnaires), Cognitive Processes, Foreign Countries, Regional Characteristics, Age, Sex, Sampling, Research Methodology
Geographic Terms: Germany, East Germany
DOI: 10.1177/00491241231176846
ISSN: 0049-1241
1552-8294
Abstract: Probes are follow-ups to survey questions used to gain insights on respondents' understanding of and responses to these questions. They are usually administered as open-ended questions, primarily in the context of questionnaire pretesting. Due to the decreased cost of data collection for open-ended questions in web surveys, researchers have argued for embedding more open-ended probes in large-scale web surveys. However, there are concerns that this may cause reactivity and impact survey data. The study presents a randomized experiment in which identical survey questions were run with and without open-ended probes. Embedding open-ended probes resulted in higher levels of survey break off, as well as increased backtracking and answer changes to previous questions. In most cases, there was no impact of open-ended probes on the cognitive processing of and response to survey questions. Implications for embedding open-ended probes into web surveys are discussed.
Abstractor: As Provided
Notes: https://doi.org/10.6084/m9.figshare.22499557.v1
Entry Date: 2025
Accession Number: EJ1458200
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEBixiObE5AGqswsIZ1NftRAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLP-LPYEqDkjGnYuEwIBEICBmzqcq2IJKg_3QWnIpLPDbostcTkKLnEsuytEUHHPpFO9WUAAXn67ZO42dDTsTqmAMe3hzDlfxFkAwVVVIs0DMO60hA_CL_5mhOT-V15DH2-VDJHaXqhYMTZB8AGuihcaOZddnOEymSs-WebqAy7izYphlVbj2lHJwWt6bcJTV4VlLNNef-FNEDVyGOd-hwtrvee9IEbkNgYQriOf
Text:
  Availability: 1
  Value: <anid>AN0182244233;som01feb.25;2025Jan17.04:27;v2.2.500</anid> <title id="AN0182244233-1">The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys </title> <p>Probes are follow-ups to survey questions used to gain insights on respondents' understanding of and responses to these questions. They are usually administered as open-ended questions, primarily in the context of questionnaire pretesting. Due to the decreased cost of data collection for open-ended questions in web surveys, researchers have argued for embedding more open-ended probes in large-scale web surveys. However, there are concerns that this may cause reactivity and impact survey data. The study presents a randomized experiment in which identical survey questions were run with and without open-ended probes. Embedding open-ended probes resulted in higher levels of survey break off, as well as increased backtracking and answer changes to previous questions. In most cases, there was no impact of open-ended probes on the cognitive processing of and response to survey questions. Implications for embedding open-ended probes into web surveys are discussed.</p> <p>Keywords: web probing; open-ended questions; reactivity; response quality; cognitive pretest</p> <hd id="AN0182244233-2">Introduction</hd> <p>In recent years, open-ended questions have experienced a renaissance ([<reflink idref="bib42" id="ref1">42</reflink>]), particularly in the context of web surveys ([<reflink idref="bib55" id="ref2">55</reflink>]), as the cost and effort of data collection ([<reflink idref="bib22" id="ref3">22</reflink>]; [<reflink idref="bib70" id="ref4">70</reflink>]) and coding of responses ([<reflink idref="bib51" id="ref5">51</reflink>]) are much decreased due to technological development. Open-ended narrative questions are considered the "classic open-ended question [...], in which respondents are invited to articulate their response using their own words" ([<reflink idref="bib15" id="ref6">15</reflink>]). Probes are a specific type of open-ended narrative question that directly relate to a forgoing closed survey question ([<reflink idref="bib9" id="ref7">9</reflink>]; [<reflink idref="bib52" id="ref8">52</reflink>]) and can be used to assess the validity and even cross-cultural comparability of survey questions ([<reflink idref="bib56" id="ref9">56</reflink>]). They are frequently used at the stage of question development and cognitive pretesting for the purpose of question evaluation, for instance, to examine whether a term in the survey question is understood in the way intended by the researcher or to gain insights on why respondents chose a response option ([<reflink idref="bib17" id="ref10">17</reflink>]; [<reflink idref="bib61" id="ref11">61</reflink>]). The implementation of open-ended probes in large-scale surveys is less common, though researchers have repeatedly argued for this to clarify reasons for a response ([<reflink idref="bib52" id="ref12">52</reflink>]), gain insights on reasons for lack of measurement invariance ([<reflink idref="bib38" id="ref13">38</reflink>]), or even to encourage more truthful answers ([<reflink idref="bib14" id="ref14">14</reflink>]; [<reflink idref="bib85" id="ref15">85</reflink>]).</p> <p>Next to these described benefits, there are concerns that open-ended probes impact surrounding survey questions. Recent studies that embedded open-ended probes in web surveys indicated an increase in survey break offs and item nonresponse ([<reflink idref="bib53" id="ref16">53</reflink>]) as well as slight shifts in response behavior ([<reflink idref="bib14" id="ref17">14</reflink>]; [<reflink idref="bib18" id="ref18">18</reflink>]). If embedding open-ended probes affects the response behavior to web survey questions, the comparability of survey questions asked with and without open-ended probes may be compromised. This would affect settings such as the one proposed by [<reflink idref="bib52" id="ref19">52</reflink>], in which probes are asked to a random subsample within a survey, or longitudinal analysis of panel data, if probes are implemented in some, but not all waves. Moreover, the validity of insights gained from web probing responses depends on respondents understanding and answering the survey questions in the same way regardless of whether or not they receive probing questions.</p> <p>The present article sets out to examine the effects of open-ended probes on web survey questions. The background section begins with an overview of previous studies that examined the effect of open-ended probes on closed survey questions and points to current research gaps. Next, the differences between open-ended and closed questions in terms of burden, cognitive processing and response behavior are summarized, and the notion of measurement reactivity in surveys is introduced. From this, the research questions and hypotheses are derived. A between-subject experiment is reported which assessed the effects of embedding open-ended probes on the processing of and response to closed web survey questions. The experiment examined survey break off, backtracking, and answer changes to previous survey questions, as well as response times, nonresponse, and response behavior to successive survey questions. Finally, the benefits and potential adverse effects of embedding open-ended probes into web surveys are discussed.</p> <hd id="AN0182244233-3">Background</hd> <p></p> <hd id="AN0182244233-4">Previous Research on the Impact of Open-Ended Probes on Survey Response</hd> <p>In the realm of web surveys, three studies have examined the impact of open-ended probes on closed survey questions. [<reflink idref="bib53" id="ref20">53</reflink>] examined the effect of embedding an open-ended probe on survey break off and item nonresponse to a closed opinion question. He found that a probe displayed on the same survey page as the question it pertained to increased survey break off by 0.6 and item nonresponse by more than 25 percentage points. When using a paging design—that is displaying the probe on a separate survey page—there was a stronger impact on survey break off of 1.4 percentage points, but no effect on item nonresponse as compared to inserting no probe. It must be noted that the probe in this experiment more resembled an open-ended text field at the end of a closed survey question than a typical open-ended narrative probe and was worded in a strongly nonmandatory manner ("If you like, you can add some bullet points to your response.").</p> <p>[<reflink idref="bib14" id="ref21">14</reflink>] reported the results of two experiments that inserted open-ended probes into a ten-item scale on attitudes towards immigrants in a probability panel in the Netherlands. In the first experiment, respondents were presented with one mandatory open-ended probe after each item using a paging design. In the second experiment, respondents were presented with an optional open-ended probe on the same screen as the respective closed survey item. In both experiments, there was a small but significant difference in the overall means of the item battery between the experimental condition with probes and the control group, with respondents reporting lower levels of prejudice in the condition with probes. In the experiment using the paging design, this effect occurred as of the second-shown survey item.</p> <p>Finally, a study by [<reflink idref="bib18" id="ref22">18</reflink>] compared responses to survey questions depending on probe placement. Respondents answered nine closed items on perceptions of neighborhood walkability, such as the presence of sidewalks, trails, or paths. On one condition, they received four open-ended probes on the survey page directly after the item battery. In the other condition, respondents were presented with the probes retrospectively at the end of the survey, that is, with several unrelated survey questions in between. Results showed a small but significant effect of probe placement on the mean walkability score, with respondents who received the probes directly after the survey questions reporting slightly enhanced perceptions of walkability. It must be noted that the study was not a strictly randomized experiment, as the condition that included probes directly after the survey items was fielded three weeks before the condition with retrospective probes, and the sample was not representative of the U.S. population in terms of demographics.</p> <p>In sum, previous studies lend support for the notion that open-ended probes impact whether a respondent continues the survey, answers survey questions, and how they answer them. However, the studies also raise many questions. Regarding survey break off and item nonresponse, the effect sizes found by [<reflink idref="bib53" id="ref23">53</reflink>] merit further examination. The study only examined one survey question, and the probe was rather atypical. Effects may vary across both probe and question types. Regarding response behavior to the survey questions, [<reflink idref="bib14" id="ref24">14</reflink>] found that using a paging design influenced response behavior to subsequent items (i.e., as of the second-shown item), whereas [<reflink idref="bib18" id="ref25">18</reflink>] found an effect on their item battery despite presenting their probes on a separate page after the survey questions. A possible explanation for this effect could be that respondents in Fowler and Willis' study backtracked to the previous survey page and changed their answers; however, the study provides no details on this. Most importantly, the reasons for the shift in the overall means found in both studies remain unclear. [<reflink idref="bib14" id="ref26">14</reflink>] had assumed that open-ended text fields in which respondents could justify their responses would reduce the threat of sensitive questions and lead to an increase in socially undesirable answers; however, the shift in response behavior indicated the opposite effect. Moreover, in both studies, the effects on the overall means were rather small. Possibly, effects on response behavior can be better examined using other measures, such as indicators of response quality or response styles. However, there is currently no framework to predict such effects.</p> <hd id="AN0182244233-5">Cognitive Processing of Open-Ended and Closed Questions</hd> <p>The following section draws on literature on open-ended probes in the context of cognitive interviewing and web probing, as well as open-ended narrative questions in general. Other types of open-ended (such as numeric) questions or probes with closed response options, also known as targeted embedded probes ([<reflink idref="bib48" id="ref27">48</reflink>], [<reflink idref="bib74" id="ref28">74</reflink>]), are not considered.</p> <p>The process of survey response optimally consists of several cognitive steps ([<reflink idref="bib91" id="ref29">91</reflink>]). Respondents must interpret the pragmatic meaning of a survey question. They embark on an information retrieval process, which is truncated when respondents have gathered enough information to form a judgment of sufficient certainty. The relevance of the accessible information is assessed, and an internal judgment is formed. This is then adjusted to the response format of the survey question.</p> <p>For closed questions, the available response options may contribute to construing the meaning of a question ([<reflink idref="bib83" id="ref30">83</reflink>]) and impact the perceived relevance of the retrieved information. For open-ended questions, neither question interpretation nor the assessment of which accessible information is relevant to form a judgment is guided—and potentially limited—by predefined response options. However, open response formats also bear the risk that respondents deem aspects irrelevant if they consider them self-evident, or that information retrieval is truncated before relevant information is retrieved ([<reflink idref="bib56" id="ref31">56</reflink>]).</p> <p>The differences in these processes impact whether and how respondents answer open-ended and closed questions. In general, the respondent tasks associated with open-ended questions are considered more demanding and burdensome ([<reflink idref="bib32" id="ref32">32</reflink>]; [<reflink idref="bib90" id="ref33">90</reflink>]). In line with this, a higher number of open-ended questions in a survey is associated with an increased likelihood of survey break off ([<reflink idref="bib21" id="ref34">21</reflink>]), and inserting multiple open-ended questions on one page has a particularly strong effect ([<reflink idref="bib44" id="ref35">44</reflink>]). The study by [<reflink idref="bib53" id="ref36">53</reflink>] confirmed the negative impact on survey break off for open-ended probes, in particular when the probe is presented on a separate survey page. Moreover, open-ended questions result in higher levels of item nonresponse than corresponding closed questions ([<reflink idref="bib45" id="ref37">45</reflink>]; [<reflink idref="bib98" id="ref38">98</reflink>]), particularly among lower educated respondents ([<reflink idref="bib4" id="ref39">4</reflink>]; [<reflink idref="bib40" id="ref40">40</reflink>]; [<reflink idref="bib49" id="ref41">49</reflink>]; [<reflink idref="bib50" id="ref42">50</reflink>]; [<reflink idref="bib99" id="ref43">99</reflink>]). These findings have been confirmed for open-ended as compared to closed probes ([<reflink idref="bib41" id="ref44">41</reflink>]).</p> <p>Differences can also be found in response distributions between open-ended and closed survey questions ([<reflink idref="bib45" id="ref45">45</reflink>]) and probes ([<reflink idref="bib41" id="ref46">41</reflink>]). Responses not included in a closed format are unlikely to be named by respondents, even when an open-ended "other" field is included. On the other hand, any given opinion, theme, or topic is less likely to be volunteered in an open response format than when it must simply be "recognized" in a closed question ([<reflink idref="bib9" id="ref47">9</reflink>]).</p> <p>Open-ended probes are more directed than other open-ended questions as they directly pertain to the preceding closed survey question ([<reflink idref="bib17" id="ref48">17</reflink>]; [<reflink idref="bib41" id="ref49">41</reflink>]; [<reflink idref="bib84" id="ref50">84</reflink>]). In the context of web probing, three types of probes are mainly employed. <emph>Comprehension probes</emph> ask about the understanding of a term used in the survey question. In <emph>category selection probes</emph> respondents are requested to explain why they chose their response option. <emph>Specific probes</emph> encourage respondents to provide additional information on a particular detail of the item ([<reflink idref="bib8" id="ref51">8</reflink>], [<reflink idref="bib9" id="ref52">9</reflink>]; [<reflink idref="bib58" id="ref53">58</reflink>]). These probe types ask respondents to focus on different aspects of their survey responses. Due to the differences in the response process between open-ended and closed questions, probes may lead respondents to consider additional or different aspects of a question than they did while answering it, or respondents may come to a different evaluation as to which retrieved information should be relevant to their judgment. Moreover, simply the process of repeated thinking about survey questions may impact how a respondent answers it. This could lead to an interaction of probing and survey questions. The following section describes the impact that questions within a survey can have on each other, known as measurement reactivity, and applies it to probing questions.</p> <hd id="AN0182244233-6">Measurement Reactivity in Surveys</hd> <p>The notion that examining a phenomenon can alter the phenomenon itself is discussed in many areas of research, from physics to behavioral psychology. In survey research, the notion of measurement reactivity was examined in a series of experiments using personality measures. [<reflink idref="bib44" id="ref54">44</reflink>] argued that thinking about questions has consequences for question construal and that increased reflection on a topic makes a certain interpretation more salient, leading to a polarization of judgment. They postulated that later items within a measure (or items in a repeated measurement) show more extreme, but also more reliable and consistent responses. To examine this, the order of multi-item measures was randomized ([<reflink idref="bib43" id="ref55">43</reflink>]), with later items showing higher reliability and more extreme answers. Importantly, there was generally no visible effect on the mean value of these items. The studies demonstrated that increased reflection about survey questions influences both cognitive processing and response to survey items and that these effects must not (necessarily) be visible by a simple comparison of means.</p> <p>Whether respondents' verbalized reflection on survey questions causes reactivity has been subject to debate since the dawn of cognitive testing. The early standard of verbal protocols required respondents to think aloud while answering a survey question ([<reflink idref="bib24" id="ref56">24</reflink>], [<reflink idref="bib25" id="ref57">25</reflink>]). This was criticized by researchers as potentially increasing the effort required to create a response ([<reflink idref="bib57" id="ref58">57</reflink>], [<reflink idref="bib58" id="ref59">58</reflink>]), especially after an experimental study demonstrated that think-aloud protocols impact task accuracy and response times for some tasks ([<reflink idref="bib72" id="ref60">72</reflink>]). The debate gave rise to the use of probing questions, which are administered after the respondent has completed the survey question. [<reflink idref="bib6" id="ref61">6</reflink>] argued that using probes in cognitive interviews may be less likely to cause reactivity than employing the think-aloud technique. However, other researchers argued that probes may likewise lead to invalid or reactive reports ([<reflink idref="bib12" id="ref62">12</reflink>]) by interfering with the natural flow of the survey interview ([<reflink idref="bib6" id="ref63">6</reflink>]), and a recent meta-analysis supported the notion that directive probing can impact task accuracy ([<reflink idref="bib19" id="ref64">19</reflink>]). A further study showed some indication of increased respondent motivation through verbal probing, but remained inconclusive ([<reflink idref="bib88" id="ref65">88</reflink>]).</p> <hd id="AN0182244233-7">Research Questions and Hypotheses</hd> <p>This study aims to enhance our understanding of the impact of open-ended probes on survey responses. I differentiate between effects on the survey in terms of survey completion, effects on the questions being probed, and effects on subsequent questions.</p> <p></p> <ulist> <item> <bold> RQ1: </bold> Does embedding open-ended probes into web surveys impact survey break off?</item> <p></p> <item> <bold> RQ2: </bold> Does embedding open-ended probes into web surveys impact the survey questions the probes pertain to?</item> <p></p> <item> <bold> RQ3: </bold> Does embedding open-ended probes into web surveys impact subsequent survey questions?</item> </ulist> <p>Based on the findings from previous studies and literature on open-ended questions and measurement reactivity, I put forward several hypotheses. The strongest possible adverse effect of an open-ended question occurs if a respondent chooses to discontinue the survey. The sum of past research indicates that adding open-ended probes to a web survey results in higher levels of survey break off ([<reflink idref="bib21" id="ref66">21</reflink>]; [<reflink idref="bib53" id="ref67">53</reflink>]; [<reflink idref="bib44" id="ref68">44</reflink>]).</p> <p></p> <ulist> <item> <bold> H1: </bold> Embedding open-ended probes into web surveys increases survey break off.</item> </ulist> <p>Embedding open-ended probes may impact the survey questions they relate to, either if probes are presented alongside the survey question on the same page (as in some of the experimental conditions in [<reflink idref="bib14" id="ref69">14</reflink>]; [<reflink idref="bib53" id="ref70">53</reflink>]), or if respondents have the possibility to return to previous questions in a paging design (which would explain the effects found by [<reflink idref="bib18" id="ref71">18</reflink>]). A probe may cause respondents to reconsider their interpretation of a survey question, access other information, or include other information in their judgment. Previous research has indicated that reverse question order effects may arise when respondents have the possibility to return to previous questions ([<reflink idref="bib88" id="ref72">88</reflink>]), meaning that subsequent questions can influence responses to previous ones ([<reflink idref="bib8" id="ref73">8</reflink>]; [<reflink idref="bib53" id="ref74">53</reflink>]; [<reflink idref="bib81" id="ref75">81</reflink>]). Therefore, I hypothesize that embedding open-ended probes leads to an increase in backtracking and changing one's answer to previous survey questions:</p> <p></p> <ulist> <item> <bold> H2a: </bold> Embedding open-ended probes into web surveys increases backtracking to previous survey questions.</item> <p></p> <item> <bold> H2b: </bold> Embedding open-ended probes into web surveys increases answer changes to previous survey questions.</item> </ulist> <p>Next to the effects on the survey questions they relate to, open-ended probes may impact how respondents process and answer subsequent questions. The following hypotheses rest on the assumption that probes cause respondents to reflect on their previous survey responses, and that respondents process survey questions more deeply when they are expecting these questions to be followed by probes. [<reflink idref="bib43" id="ref76">43</reflink>] demonstrated that increased thinking about questions leads to judgment polarization and more consistent responses.</p> <p>Regarding cognitive effort, response times are considered "the most important means for investigating hypotheses about mental processing" ([<reflink idref="bib62" id="ref77">62</reflink>]). Findings from think-aloud and verbal probing ([<reflink idref="bib19" id="ref78">19</reflink>]; [<reflink idref="bib72" id="ref79">72</reflink>]) indicate that response times increase in interviewer-administered settings for some questions. Therefore, I hypothesize that response times to closed survey questions increase when open-ended probes are embedded:</p> <p></p> <ulist> <item> <bold> H3: </bold> Embedding open-ended probes into web surveys increases response times to subsequent survey questions.</item> </ulist> <p>Unfortunately, previous studies did not report on nonresponse ([<reflink idref="bib14" id="ref80">14</reflink>]; [<reflink idref="bib18" id="ref81">18</reflink>]) or only examined one question ([<reflink idref="bib53" id="ref82">53</reflink>]). However, if embedding open-ended probes causes respondents to reflect on survey questions more deeply and this leads to judgment polarization, it can be assumed that nonresponse decreases for subsequent items:</p> <p></p> <ulist> <item> <bold> H4: </bold> Embedding open-ended probes into web surveys decreases nonresponse for subsequent survey questions.</item> </ulist> <p>Previous research has found effects of embedding open-ended probes on the mean sum score of multi-item measures ([<reflink idref="bib14" id="ref83">14</reflink>]; [<reflink idref="bib18" id="ref84">18</reflink>]), but the effect size was small and the direction could not be predicted. [<reflink idref="bib44" id="ref85">44</reflink>] argued that increased thinking about answers impacts response behavior but that this must not necessarily be visible in the form of a mean shift. Rather, increased thinking about questions leads to judgment polarization and more consistent responses. Therefore, I hypothesize that embedding open-ended probes does not (consistently) impact means. Instead, differences become visible in the form of increased extreme responding and nondifferentiation:</p> <p></p> <ulist> <item> <bold> H5a: </bold> Embedding open-ended probes into web surveys does not impact mean scores for subsequent survey questions.</item> <p></p> <item> <bold> H5b: </bold> Embedding open-ended probes into web surveys increases extreme responding for subsequent survey questions.</item> <p></p> <item> <bold> H5c: </bold> Embedding open-ended probes into web surveys increases non-differentiation for subsequent survey questions.</item> </ulist> <hd id="AN0182244233-8">Method</hd> <p>An experiment was designed with the aim of comparing closed survey questions that were either accompanied by open-ended probes or not. Respondents received six survey pages with closed attitude questions. A between-subject design was used, in which respondents were randomly assigned to experimental condition (A) which embedded open-ended probes between the survey pages with closed questions, or condition (B) which contained only the closed questions.</p> <hd id="AN0182244233-9">Survey and Probing Questions</hd> <p>The closed survey questions presented a mix of single- and multi-item measures using common constructs in social science research, such as political attitudes, personality, and well-being. Measures that have been accompanied by open-ended probes in other studies were chosen when possible. Multi-item measures were presented in a grid format on one survey page. The exact wording of the closed survey questions and the open-ended probes from condition (A) can be found in Online Appendix Table A1.</p> <p>The first closed survey question was a single-item measure of <emph>left–right orientation</emph> (Q1), which is considered a "central element of political science research" ([<reflink idref="bib99" id="ref86">99</reflink>]:28). Left–right orientation is implemented in several general population surveys, such as the German General Social Survey ([<reflink idref="bib24" id="ref87">24</reflink>]) or the German Longitudinal Election Study ([<reflink idref="bib25" id="ref88">25</reflink>]). The question has repeatedly been complemented by open-ended probes to gain additional insights into the meaning of left and right ([<reflink idref="bib5" id="ref89">5</reflink>]; [<reflink idref="bib29" id="ref90">29</reflink>], [<reflink idref="bib20" id="ref91">20</reflink>]; [<reflink idref="bib50" id="ref92">50</reflink>]; [<reflink idref="bib99" id="ref93">99</reflink>]). It was succeeded by another political construct, the two-item short scale on <emph>political cynicism</emph> (Q2) ([<reflink idref="bib2" id="ref94">2</reflink>]), which captures respondents' general trust in politicians' honesty. It was developed for the Austrian National Election Study ([<reflink idref="bib31" id="ref95">31</reflink>]) and has been implemented by other researchers since ([<reflink idref="bib67" id="ref96">67</reflink>]). The third question was the six-item short scale for the <emph>Gamma factor of social desirability</emph> (Q3) ([<reflink idref="bib29" id="ref97">29</reflink>]). Social desirability responding is the tendency to give overly positive self-descriptions ([<reflink idref="bib65" id="ref98">65</reflink>]). The construct's Gamma factor is implemented into questionnaires to control whether self-reports may be biased by social desirability responding ([<reflink idref="bib43" id="ref99">43</reflink>]). Survey research has indicated reactivity in personality measures ([<reflink idref="bib43" id="ref100">43</reflink>]; [<reflink idref="bib44" id="ref101">44</reflink>]). <emph>General life satisfaction</emph> (Q4) and <emph>relationship satisfaction</emph> (Q5) ([<reflink idref="bib7" id="ref102">7</reflink>]; [<reflink idref="bib82" id="ref103">82</reflink>]) were implemented as further single-item measures. They have been followed by open-ended probes in the past to analyze determinants of self-reported satisfaction ([<reflink idref="bib23" id="ref104">23</reflink>]). Furthermore, past research has repeatedly demonstrated that the communicative context of these questions impacts how the items are answered and how strongly they relate to one another ([<reflink idref="bib79" id="ref105">79</reflink>]; [<reflink idref="bib54" id="ref106">54</reflink>]). The measure of <emph>intergenerational social support</emph> (Q6) consists of six items on family support ([<reflink idref="bib23" id="ref107">23</reflink>]; [<reflink idref="bib35" id="ref108">35</reflink>]). All questions included the nonsubstantive response option "I don't want to answer" except the personality measure (Q3).</p> <p>In experimental condition (A), each closed question was followed by one open-ended probe. For the multi-item inventories (Q2, Q3, and Q6), respondents were randomly presented with a comprehension or category selection probe. The single-item measures on life and relationship satisfaction (Q4 and Q5) were each followed by a specific probe. The question on left–right orientation (Q1) was followed by two probes on the understanding of the terms "left" and "right" as in previous studies ([<reflink idref="bib99" id="ref109">99</reflink>]). To keep the survey setting identical across conditions and to be able to attribute survey break offs to the probing situation, the probes were not announced on the welcome page of the survey. Instead, the first probe was introduced with the words "We would like to receive more information on the previous question." Probes were presented using a paging design, with the survey question being repeated on the page with the probe. In addition, the selected survey response was repeated for category selection and specific probes.</p> <hd id="AN0182244233-10">Web Survey</hd> <p>An online survey was carried out with a nonprobability sample between November 20 and December 2, 2020, with the panel provider Respondi AG. The sample included quotas to depict the German online population in terms of gender (male, female),[<reflink idref="bib6" id="ref110">6</reflink>] age (18–29, 30–39, 40–49, 50–59, and 60 or more years), education (low, medium, and high) and region (former East and West Germany). Respondents were randomly assigned to experimental condition (A) or (B). There were no significant differences regarding demographics or devices used between experimental groups (see Online Appendix Table A2 for the sample composition). The web survey included several experiments, which were randomized independently of each other. The reported study was placed towards the beginning of the survey after the screening and quota questions and three short scales. No open-ended questions were implemented before the experiment.</p> <p>The Universal Client-Side Paradata script by [<reflink idref="bib40" id="ref111">40</reflink>] was implemented to ensure a more exact measure of response latency ([<reflink idref="bib62" id="ref112">62</reflink>]) and collect questionnaire navigation data ([<reflink idref="bib15" id="ref113">15</reflink>]). The script records response behavior sequentially so that the resulting string variables enable coding backtracking to previous survey pages and answer changes to items. In accordance with both legal and ethical research standards ([<reflink idref="bib1" id="ref114">1</reflink>]; [<reflink idref="bib47" id="ref115">47</reflink>]), respondents were informed about the collection and use of client-side paradata on the welcome page of the survey.</p> <p>Of the 9,731 panelists invited to participate in the survey, 2,441 started the survey and of those, 241 broke off before completing it (before, during, or after the reported experiment), resulting in 2,200 completed questionnaires. This leads to a participation rate of 22.6 percent ([<reflink idref="bib3" id="ref116">3</reflink>]) and a break off rate of 9.9 percent ([<reflink idref="bib10" id="ref117">10</reflink>]). Respondents received €1.50 for survey completion. About a quarter (27.1%; <emph>n</emph> = 597) of respondents filled out the survey using a mobile device. Average survey completion was 21.0 min (median: 17.3; <emph>n</emph> = 1,096) for respondents in condition (A) and 15.9 min (median: 12.4; <emph>n</emph> = 1,046) for respondents in condition (B).</p> <hd id="AN0182244233-11">Probe Response Quality and Content</hd> <p>Prior to examining the survey responses, probe response quality and content were analyzed. Low probe response quality may point to poorly designed probing questions. Probe response content was examined to gain insights into the respondents' cognitive process of survey response and the quality of the survey questions. To ascertain probe response quality, the share of nonsubstantive responses was determined. Responses were coded as non-substantive when respondents left the probe empty, entered random characters, typed a "don't know" answer or an explicit refusal, or other nonintelligible or noncodable content ([<reflink idref="bib8" id="ref118">8</reflink>]). Between 12.2 percent and 21.6 percent of respondents in condition (A) gave nonsubstantive responses to the probing questions (see Table 1), which coincides with previous web probing studies ([<reflink idref="bib10" id="ref119">10</reflink>]; [<reflink idref="bib57" id="ref120">57</reflink>]). To examine probe response content, the substantive probe responses were subjected to cognitive coding ([<reflink idref="bib59" id="ref121">59</reflink>]) to determine whether the probe responses hinted at issues in respondents' cognitive process of survey response. This approach is also known as an error perspective as it gives insights into possible reasons behind measurement errors ([<reflink idref="bib57" id="ref122">57</reflink>]). Errors or problems may occur at the stages of question comprehension, information retrieval, judgment, or response formatting ([<reflink idref="bib60" id="ref123">60</reflink>]). For all but one question, error codes were detected for under 5 percent of respondents (see Table 1). Reported problems included misinterpreting words central to the question (i.e., the terms "left" or "right" in Q1) or mismatches between survey and probe responses (i.e., a low score of political cynicism in Q2, but probe responses indicating very low trust in politicians). Question Q5 on relationship satisfaction showed an unusually high share of error coding, with 25.5 percent of responses pointing to difficulties with judgment and/or response formatting. Almost all of these respondents reported that they were missing an answer to indicate that they were currently not in a relationship. A complete list of reported problems is available from the author on request.</p> <p>Table 1. Probe Response Quality and Content (Basis: Condition A, N = 1,096).</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="char" char="(" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" rowspan="2" /><th align="center" rowspan="2">Nonsubstantive response</th><th align="center" rowspan="2">Error detected</th><th align="center" colspan="2">Error related to...</th></tr><tr><th align="center">Question comprehension/Information recall</th><th align="center">Judgment/Response formatting</th></tr></thead><tbody><tr><td>Q1<xref ref-type="table-fn" rid="tfn1">a</xref></td><td>21.6% (237)</td><td>4.7% (51)</td><td>3.7% (41)</td><td>1.0% (11)</td></tr><tr><td>Q2</td><td>13.0% (143)</td><td>4.7% (51)</td><td>2.1% (23)</td><td>2.6% (29)</td></tr><tr><td>Q3</td><td>13.8% (151)</td><td>1.9% (21)</td><td>0.5% (6)</td><td>1.4% (15)</td></tr><tr><td>Q4</td><td>12.2% (134)</td><td>0.9% (10)</td><td>0.2% (2)</td><td>0.7% (8)</td></tr><tr><td>Q5</td><td>15.5% (170)</td><td>26.3% (288)</td><td>0.7% (8)</td><td>25.5% (280)</td></tr><tr><td>Q6</td><td>19.3% (212)</td><td>3.4% (37)</td><td>2.2% (24)</td><td>1.2% (13)</td></tr></tbody></table> </ephtml> </p> <p>1 For Q1, responses were coded as nonsubstantive when both probes contained nonsubstantive content; an error was coded when at least one probe response contained the error code.</p> <hd id="AN0182244233-12">Dependent Measures and Data Analysis</hd> <p>All dependent measures were compared across the two experimental conditions (1 = condition (A) open-ended probes; 0 = condition (B) without probes).</p> <hd id="AN0182244233-13">Survey break off, backtracking, and answer changes</hd> <p>Break offs that occurred during the reported experiment, that is on the pages containing the closed survey questions (Q1–Q6) or the open-ended probes (P1a/b-P6) were included in the analysis. Backtracking was recorded for respondents when the client-side paradata string recorded more than one page visit. Multiple page visits were coded into a binary variable for each respondent on page level (1 = backtracking to survey page; 0 = no backtracking to survey page) and aggregated across all six closed survey questions (1 = backtracking to at least one survey page; 0 = no backtracking to any survey pages). Likewise, the prevalence of answer changes after backtracking was coded on page level ([<reflink idref="bib27" id="ref124">27</reflink>], [<reflink idref="bib38" id="ref125">38</reflink>]) (1 = answer change after backtracking; 0 = no answer change/no backtracking) and aggregated across all six closed survey questions (1 = answer change after backtracking on at least one survey page; 0 = no answer change after backtracking for any closed survey questions).</p> <p>Chi-square tests of independence were used to examine whether the share of survey break offs, backtracking, and answer changes after backtracking differed between conditions. For backtracking and subsequent answer changes, analyses were additionally carried out on page level, with Bonferroni adjusted alpha levels for multiple comparisons.</p> <hd id="AN0182244233-14">Response times</hd> <p>Client-side response times were captured for each survey page. Response time data is positively skewed and subject to outliers, which makes decisions about outlier definition, handling, and potential transformation of response times prior to analysis of utmost importance ([<reflink idref="bib33" id="ref126">33</reflink>]). A variety of response time outlier definitions exist ([<reflink idref="bib37" id="ref127">37</reflink>]). Detecting outliers based on the mean and standard deviation remains a common procedure ([<reflink idref="bib61" id="ref128">61</reflink>]), but has been criticized, as the mean value is in turn influenced by outliers. Researchers have increasingly recommended using median-based outlier definitions ([<reflink idref="bib28" id="ref129">28</reflink>]). Employing 2.5 times the median absolute deviation is considered the most robust outlier threshold ([<reflink idref="bib52" id="ref130">52</reflink>]), and applied in the present study. This method led to between 9 percent and 12 percent of response times being identified as outliers. Response time outliers were omitted from response time analysis, as were all instances in which a survey page was visited more than once (backtracking).</p> <p>To test the third hypothesis, a multivariate analysis of covariance (MANCOVA) was applied to the adjusted response times as of the second survey question (Q2–Q6), with the experimental condition as the main predictor. The model included age, education, and device used (PC/laptop or mobile) as covariates, as it was not possible to include baseline reading speed, which otherwise accounts for much of the variance within response times ([<reflink idref="bib16" id="ref131">16</reflink>]; [<reflink idref="bib36" id="ref132">36</reflink>]; [<reflink idref="bib62" id="ref133">62</reflink>]). Because outlier exclusion as described above leads to a sizeable decrease in available cases for a MANCOVA, separate analyses of covariances (ANCOVAs) were run for each survey page (thus, only excluding outliers page-wise).</p> <p>The robustness of response time analyses should be tested by applying different outlier detection methods ([<reflink idref="bib46" id="ref134">46</reflink>]). The alternative approach used excluded observations beyond the upper and lower one percentile from analysis ([<reflink idref="bib62" id="ref135">62</reflink>]). A MANCOVA and separate ANCOVAs were run using the same predictors. All analyses led to the same results.</p> <hd id="AN0182244233-15">Response behavior</hd> <p></p> <hd id="AN0182244233-16">Survey response</hd> <p>T-tests were used to test for differences between conditions. For the single-item measures (Q4 and Q5), the scale values coincide with the items' raw values. For the multi-item measures, scoring was carried out according to the instruments' documentation. For the two-item measure on political cynicism (Q2), the second item was recoded so that the scale direction was the same across items, with higher values indicating higher cynicism. The score is the sum of both items divided by the number of items ([<reflink idref="bib2" id="ref136">2</reflink>]). For the social desirability responding measure (Q3), the two factors exaggerating positive qualities (PQ+) and minimizing negative qualities (NQ−) were calculated separately using the sum score, again dividing by the number of items ([<reflink idref="bib29" id="ref137">29</reflink>]). For the intergenerational social support inventory (Q6), the fourth item was recoded so that the scale direction remained the same across items, and principal components factor analysis was conducted on the six items with oblique rotation (Direct Oblimin). The Kaiser-Meyer-Olkin measure of sampling adequacy was 0.65, and Bartlett's test of sphericity reached statistical significance (χ<sups>2</sups>(<reflink idref="bib15" id="ref138">15</reflink>) = 1,947.06, <emph>p</emph> <.001). The analysis identified two factors with an Eigenvalue greater than 1, explaining 58.84 percent of the variance. All items loaded above.60. The two factors could be attributed to support provided by older to younger generations (F1) and support provided by younger to older generations (F2).</p> <hd id="AN0182244233-17">Item nonresponse</hd> <p>Item nonresponse can occur in the form of (<reflink idref="bib1" id="ref139">1</reflink>) <emph>skipping</emph> an item entirely, that is, when respondents produce missing data by leaving an item blank, or (<reflink idref="bib2" id="ref140">2</reflink>) by choosing a <emph>nonsubstantive</emph> response option, such as "I don't want to answer" or "I don't know" ([<reflink idref="bib13" id="ref141">13</reflink>]). Skipping items is the more precise indicator of satisficing and low response quality, as choosing a nonsubstantive response option at the least requires reading and choosing the respective response option ([<reflink idref="bib79" id="ref142">79</reflink>]). Because these types of nonresponse may occur for different reasons, they may be impacted differently by embedding open-ended probes and are analyzed separately.</p> <p>Chi-square tests of independence were used to examine whether the frequency of skipping items or choosing nonsubstantive response options differed between conditions. Analyses were additionally carried out on page level (for Q2–Q6), with Bonferroni-adjusted alpha levels for multiple comparisons. For multi-item inventories (Q2, Q3, and Q6), nonresponse was aggregated across all items. The item battery on social desirability responding (Q3) did not to include a non-substantive response option and is not included in this analysis.</p> <hd id="AN0182244233-18">Nondifferentiation.</hd> <p>Nondifferentiation, also referred to as straightlining, was examined for the two multi-item batteries Q3 and Q6. A dichotomous and a metric measure of straightlining were calculated. The dichotomous measure indicated whether a respondent chose the identical response option for all items within one battery. The second measure "mean root of pairs" was chosen to "capture variations in the choice of answers in a battery" ([<reflink idref="bib30" id="ref143">30</reflink>]). It is calculated by producing a temporary index by computing the mean of the root of the absolute differences between all pairs of items in a battery and then rescaling the temporary index to range from 0, indicating least straightlining, to 1, indicating most straightlining ([<reflink idref="bib11" id="ref144">11</reflink>]). To examine whether nondifferentiation differed between conditions, chi-square tests of independence were used for the absolute and <emph>T</emph>-tests for the metric measure of nondifferentiation.</p> <hd id="AN0182244233-19">Extreme responding</hd> <p>Extreme responding was defined as choosing either endpoint of a response scale (apart from non-substantive response options). Chi-square tests of independence were carried out on item level with Bonferroni-adjusted alpha levels for multiple comparisons.</p> <p>All analyses were carried out using IBM SPSS Statistics Version 24.0.</p> <hd id="AN0182244233-20">Results</hd> <p></p> <hd id="AN0182244233-21">Effect on Survey Break Off</hd> <p>The first hypothesis postulated that embedding open-ended probes into a web survey increases survey break off. Seventy-nine respondents dropped out of the survey during the reported experiment. Most of these break offs (82.3%; <emph>n</emph> = 65) occurred in condition (A) with open ended probes. This difference was significant (<emph>n</emph> = 2,279; χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref145">1</reflink>)</subs> = 32.153; <emph>p</emph> <.001); therefore hypothesis 1 can be confirmed.</p> <p>More than half of the break offs in condition (A) (<emph>n</emph> = 44) occurred during the first two survey questions (Q1 and Q2) or their respective probes (P1a, P1b, and P2). Respondents who broke off the survey during the reported experiment were significantly more likely to have lower education (low/medium: <emph>n</emph> = 59 vs. high: <emph>n</emph> = 20; χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref146">1</reflink>)</subs> = 7.567; <emph>p</emph> =.006) and be women (women: <emph>n</emph> = 55 vs. men: <emph>n</emph> = 24; χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref147">1</reflink>)</subs> = 12.914; <emph>p</emph> <.001), consistent with previous research ([<reflink idref="bib47" id="ref148">47</reflink>]) that respondents who break off surveys systematically differ from those who complete them.</p> <hd id="AN0182244233-22">Effect on Preceding Survey Questions</hd> <p>The second hypothesis stated that embedding open-ended probes into a web survey increases backtracking (H2a) and subsequent answer changes (H2b) to previous survey questions. In total, 6.1% (<emph>n</emph> = 134) of respondents backtracked to a previous survey page with closed questions. Of these, three quarters (76.1%; <emph>n</emph> = 102) were assigned to condition (A) with open-ended probes. A chi-square test of independence confirmed the significant difference between conditions (χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref149">1</reflink>)</subs> = 39.483; <emph>p</emph> <.001). Backtracking to the first-shown question (Q1) occurred most often (<emph>n</emph> = 46). Analysis of question level showed the same tendency for all six questions (see Table 2). Differences between conditions were significant for Q1, Q3, and Q4 based on Bonferroni-adjusted alpha levels of.0083 per test (.05/6).</p> <p>Table 2. Backtracking to Previous Survey Pages and Answer Changes After Backtracking on the Question Level.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="center" /><th align="center">Condition (A): Open-ended probes</th><th align="center">Condition (B): Without probes</th><th align="center">Chi-square test of independence</th></tr><tr><th align="left" /><th align="left">Valid <italic>n</italic></th><th align="center">% (<italic>n</italic>)</th><th align="center">% (<italic>n</italic>)</th><th align="center">χ<sup>2</sup><sub>(df)</sub>; <italic>p</italic></th></tr></thead><tbody><tr><td colspan="5"><italic>Backtracking to previous survey question</italic></td></tr><tr><td>Q1</td><td>2,197</td><td>3.3% (36)</td><td>0.9% (10)</td><td>χ2<sub>(1)</sub> = 15.180; <italic>p</italic> <.001</td></tr><tr><td>Q2</td><td>2,196</td><td>1.4% (15)</td><td>0.6% (7)</td><td>χ2<sub>(1)</sub> = 2.998; <italic>p</italic> =.083</td></tr><tr><td>Q3</td><td>2,196</td><td>2.0% (22)</td><td>0.5% (5)</td><td>χ2<sub>(1)</sub> = 10.963; <italic>p</italic> =.001</td></tr><tr><td>Q4</td><td>2,196</td><td>1.3% (14)</td><td>0.1% (1)</td><td>χ2<sub>(1)</sub> = 11.440; <italic>p</italic> =.001</td></tr><tr><td>Q5</td><td>2,196</td><td>1.3% (14)</td><td>0.5% (6)</td><td>χ2<sub>(1)</sub> = 3.289; <italic>p</italic> =.070</td></tr><tr><td>Q6</td><td>2,196</td><td>1.4% (15)</td><td>0.6% (7)</td><td>χ2<sub>(1)</sub> = 2.998; <italic>p</italic> =.083</td></tr><tr><td colspan="5"><italic>Answer change after backtracking</italic></td></tr><tr><td>Q1</td><td>2,197</td><td>0.7% (8)</td><td>0.4% (4)</td><td>χ2<sub>(1)</sub> = 1.370; <italic>p</italic> =.242</td></tr><tr><td>Q2</td><td>2,196</td><td>0.5% (5)</td><td>0.4% (4)</td><td>χ2<sub>(1)</sub> = 0.119; <italic>p</italic> =.730</td></tr><tr><td>Q3</td><td>2,196</td><td>0.5% (5)</td><td>0.1% (1)</td><td>χ2<sub>(1)</sub> = 2.703; <italic>p</italic> =.100</td></tr><tr><td>Q4</td><td>2,196</td><td>0.4% (4)</td><td>0.1% (1)</td><td>χ2<sub>(1)</sub> = 1.826; <italic>p</italic> =.177</td></tr><tr><td>Q5</td><td>2,196</td><td>0.8% (9)</td><td>0.1% (1)</td><td>χ2<sub>(1)</sub> = 6.488; <italic>p</italic> =.011</td></tr><tr><td>Q6</td><td>2,196</td><td>0.5% (5)</td><td>0.2% (2)</td><td>χ2<sub>(1)</sub> = 1.312; <italic>p</italic> =.252</td></tr></tbody></table> </ephtml> </p> <p>Answer changes after backtracking were carried out by 2.0% of respondents (<emph>n</emph> = 45). Of these, 71.1% (<emph>n</emph> = 32) were respondents in condition (A) with open-ended probes. A Chi-square test of independence confirmed that the difference between conditions was significant (χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref150">1</reflink>)</subs> = 8.332; <emph>p</emph> =.004), though there were no significant differences in question level when using the Bonferroni adjusted alpha levels (see Table 2). It should be noted that across both conditions, a similar share of respondents who returned to a previous survey page decided to change their answers. Thus, while embedding open-ended probes increases backtracking and subsequently answer changes, it does not increase the likelihood of backtrackers to change their survey response. In summary, hypothesis 2 can be confirmed.</p> <hd id="AN0182244233-23">Effects on Subsequent Survey Questions</hd> <p></p> <hd id="AN0182244233-24">Response times</hd> <p>The third hypothesis stated that embedding open-ended probes into a web survey increases cognitive effort and thus response time invested in subsequent questions. To examine this, the response times to Q2–Q6 between conditions were examined using MANCOVA. After outlier exclusion, 1,509 cases remained for analysis (condition A: <emph>n</emph> = 737; condition B: <emph>n</emph> = 772). There was a significant main effect of condition on overall response time (<emph>Wilks-Lambda</emph> = 0.98; <emph>F</emph><subs>(<reflink idref="bib5" id="ref151">5</reflink>,1500)</subs> = 6.01; <emph>p</emph> <.001; <emph>η</emph><sups>2</sups> =.02). Age and device used were significant covariates, whereas education was not.[<reflink idref="bib7" id="ref152">7</reflink>]Table 3 depicts the mean response times and separate ANCOVAs run for each survey question. Results showed that response times differed significantly for only one of the examined questions: respondents who were presented with open-ended probes spent significantly longer on average (7.4 s) to respond to the question on relationship satisfaction (Q5) than respondents in condition (B) (6.7 s). Thus, based on the result of the MANCOVA, hypothesis 3 can be confirmed; however, the effect only seems to apply in specific situations.</p> <p>Table 3. Analysis of Covariances (ANCOVAs) of Response Times.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="center">Q2</th><th align="center">Q3</th><th align="center">Q4</th><th align="center">Q5</th><th align="center">Q6</th></tr><tr><th align="left"><italic>n</italic></th><th align="center">1,999</th><th align="center">1,977</th><th align="center">1,954</th><th align="center">1,971</th><th align="center">2,012</th></tr></thead><tbody><tr><td colspan="6"><italic>Condition (A)</italic></td></tr><tr><td>Mean response times (std dev)</td><td>16.7 (7.5)</td><td>40.9 (17.9)</td><td>8.0 (3.4)</td><td>7.4 (3.3)</td><td>45.8 (21.5)</td></tr><tr><td colspan="6"><italic>Condition (B)</italic></td></tr><tr><td>Mean response times (std dev)</td><td>16.5 (7.5)</td><td>40.5 (18.1)</td><td>8.1 (3.7)</td><td>6.7 (3.2)</td><td>46.9 (23.0)</td></tr><tr><td><italic>F</italic></td><td>1.12</td><td>1.73</td><td>0.15</td><td>30.68</td><td>0.45</td></tr><tr><td><italic>p</italic></td><td>0.29</td><td>0.19</td><td>0.70</td><td>0.00</td><td>0.50</td></tr><tr><td>Partial η2</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.00</td></tr></tbody></table> </ephtml> </p> <hd id="AN0182244233-25">Nonresponse</hd> <p>The fourth hypothesis postulated that nonresponse is lower in the condition with open-ended probes. Table 4 shows the occurrence of skipping items and choosing non-substantive response options on question level. Across Q2-Q6, 8% (<emph>n</emph> = 175) of respondents skipped at least one item. A chi-square test of independence showed no significant difference between conditions (χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref153">1</reflink>)</subs> = 0.251; <emph>p</emph> =.616) across the five questions. Question level tests using Bonferroni adjusted alpha levels of.01 (.05/5) confirmed this, though based on the global alpha level, more respondents left the question on relationship satisfaction (Q5) unanswered in condition (B) without probes.</p> <p>Table 4. Nonresponse.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="center">Condition (A): Open-ended probes</th><th align="center">Condition (B): Without probes</th><th align="center">Chi-square test of independence</th></tr><tr><th align="left">Question</th><th align="center">% (<italic>n</italic>)</th><th align="center">% (<italic>n</italic>)</th><th align="center">χ<sup>2</sup><sub>(df)</sub>; <italic>p</italic></th></tr></thead><tbody><tr><td colspan="4"><italic>Item skipping</italic></td></tr><tr><td>Q2</td><td>0.7% (8)</td><td>1.3% (14)</td><td>χ2<sub>(1)</sub> = 1.61; <italic>p</italic> =.205</td></tr><tr><td>Q3</td><td>4.3% (47)</td><td>3.9% (43)</td><td>χ2<sub>(1)</sub> = 0.22; <italic>p</italic> =.641</td></tr><tr><td>Q4</td><td>0.1% (1)</td><td>0.1% (1)</td><td>χ2<sub>(1)</sub> = 0.00; <italic>p</italic> =.996</td></tr><tr><td>Q5</td><td>0.4% (4)</td><td>1.3% (14)</td><td>χ2<sub>(1)</sub> = 5.53; <italic>p</italic> =.019</td></tr><tr><td>Q6</td><td>2.6% (28)</td><td>2.5% (28)</td><td>χ2<sub>(1)</sub> = 0.00; <italic>p</italic> =.978</td></tr><tr><td colspan="4"><italic>Nonsubstantive response</italic></td></tr><tr><td>Q2</td><td>4.7% (52)</td><td>5.2% (57)</td><td>χ2<sub>(1)</sub> = 0.21; <italic>p</italic> =.651</td></tr><tr><td>Q4</td><td>2.1% (23)</td><td>0.8% (9)</td><td>χ2<sub>(1)</sub> = 6.32; <italic>p</italic> =.012</td></tr><tr><td>Q5</td><td>16.0% (175)</td><td>13.3% (147)</td><td>χ2<sub>(1)</sub> = 3.10; <italic>p</italic> =.078</td></tr><tr><td>Q6</td><td>6.8% (75)</td><td>6.3% (69)</td><td>χ2<sub>(1)</sub> = 0.32; <italic>p</italic> =.574</td></tr></tbody></table> </ephtml> </p> <p>Respondents had the option to choose a nonsubstantive answer ("I don't want to answer") for all questions except Q3. In total, 22.7% (<emph>n</emph> = 499) of respondents chose a nonsubstantive response to at least one item. A chi-square test of independence showed a significant difference between conditions (χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref154">1</reflink>)</subs> = 3.905; <emph>p</emph> =.048). However, contrary to the hypothesis, more respondents in condition (A) with open-ended probes chose a nonsubstantive response (24.5%; <emph>n</emph> = 268) than in condition (B) (20.9%; <emph>n</emph> = 231). This difference remained significant on the question level for the question on general life satisfaction (Q4) based on the Bonferroni adjusted alpha level of.0125 (.05/4). Thus, hypothesis 4 could not be confirmed.</p> <hd id="AN0182244233-26">Mean scores</hd> <p>In line with hypothesis 5a, there were no significant differences between means for any of the single-item or multi-item measures (see Table 5).</p> <p>Table 5. Mean Scores.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="char" char="(" /><col align="char" char="(" /><col align="center" /></colgroup><thead><tr><th align="left">Question</th><th align="left">Range</th><th align="center">Condition (A): Open-ended probesmean (std)</th><th align="center">Condition (B): Without probesmean (std)</th><th align="center"><italic>T</italic><sub>(df)</sub>; <italic>p</italic></th></tr></thead><tbody><tr><td>Q1</td><td>1 (left) to 10 (right)</td><td>5.30 (1.78)</td><td>5.26 (1.86)</td><td>T<sub>(1918)</sub> =.524; <italic>p</italic> =.600</td></tr><tr><td>Q2</td><td>1 (low) to 5 (high)</td><td>3.63 (.86)</td><td>3.66 (.88)</td><td>T<sub>(2067)</sub> = −.631; <italic>p</italic> =.528</td></tr><tr><td>Q3: PQ+</td><td>1 (low) to 5 (high)</td><td>3.58 (.70)</td><td>3.53 (.77)</td><td>T<sub>(2126)</sub> = 1.617; <italic>p</italic> =.106</td></tr><tr><td>Q3: NQ−</td><td>1 (low) to 5 (high)</td><td>2.04 (.82)</td><td>2.03 (.81)</td><td>T<sub>(2154)</sub> =.376; <italic>p</italic> =.707</td></tr><tr><td>Q4</td><td>1 to 11 (extremely satisfied)</td><td>7.19 (2.19)</td><td>7.03 (2.35)</td><td>T<sub>(2163)</sub> = 1.580; <italic>p</italic> =.114</td></tr><tr><td>Q5</td><td>1 to 11 (extremely satisfied)</td><td>7.90 (3.19)</td><td>7.69 (2.91)</td><td>T<sub>(1830)</sub> = 1.464; <italic>p</italic> =.143</td></tr><tr><td>Q6: F1</td><td>−2.79 (disagree) to 3.15 (agree)</td><td>−.02 (1.01)</td><td>.02 (.99)</td><td>T<sub>(1997)</sub> = −.830; <italic>p</italic> =.406</td></tr><tr><td>Q6: F2</td><td>−2.55 (disagree) to 2.87 (agree)</td><td>−.01 (1.00)</td><td>.01 (1.00)</td><td>T<sub>(1997)</sub> = −.501; <italic>p</italic> =.616</td></tr></tbody></table> </ephtml> </p> <hd id="AN0182244233-27">Extreme responding</hd> <p>According to hypothesis 5b, more extreme responding should occur in condition (A) with open-ended probes. Chi-square tests were conducted for each of the eight probed items of Q2–Q6, using Bonferroni adjusted alpha levels of.00625 (.05/8). Table 6 shows the share of extreme responding across conditions on item level. Extreme responding was more likely for the single-item measure of relationship satisfaction (Q5). In line with the hypothesis, significantly more respondents reported that they were extremely satisfied or unsatisfied with their current relationship in condition (A) that included open-ended probes (37.4%, <emph>n</emph> = 343) than when no probes were embedded (25.7%; <emph>n</emph> = 242) (χ<sups>2</sups><subs>(<reflink idref="bib1" id="ref155">1</reflink>)</subs> = 29.73; <emph>p</emph> <.001). There were no further significant differences between conditions. Thus, similar to the findings regarding response times, hypothesis 5b can be confirmed for one question only, that being the question on relationship satisfaction.</p> <p>Table 6. Extreme Responding (Probed Items Only).</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="center" /><th align="center">Condition (A): Open-ended probes</th><th align="center">Condition (B): Without probes</th><th align="center">Chi-square test of independence</th></tr><tr><th align="left">Question</th><th align="left">Valid <italic>n</italic></th><th align="center">% (<italic>n</italic>)</th><th align="center">% (<italic>n</italic>)</th><th align="center">χ<sup>2</sup><sub>(df)</sub>; <italic>p</italic></th></tr></thead><tbody><tr><td>Q2 (item 1)</td><td>2,113</td><td>24.4% (259)</td><td>27.5% (290)</td><td>χ2<sub>(1)</sub> = 2.65; <italic>p</italic> =.104</td></tr><tr><td>Q2 (item 2)</td><td>2,093</td><td>20.6% (216)</td><td>22.0% (230)</td><td>χ2<sub>(1)</sub> =.61; <italic>p</italic> =.435</td></tr><tr><td>Q3 (item 1)</td><td>2,186</td><td>31.9% (347)</td><td>31.4% (345)</td><td>χ2<sub>(1)</sub> =.04; <italic>p</italic> =.835</td></tr><tr><td>Q3 (item 5)</td><td>2,185</td><td>54.1% (589)</td><td>53.0% (581)</td><td>χ2<sub>(1)</sub> =.30; <italic>p</italic> =.582</td></tr><tr><td>Q4</td><td>2,166</td><td>5.7% (61)</td><td>6.5% (71)</td><td>χ2<sub>(1)</sub> =.61; <italic>p</italic> =.437</td></tr><tr><td>Q5</td><td>1,860</td><td>37.4% (343)</td><td>25.7% (242)</td><td>χ2<sub>(1)</sub> = 29.73; <italic>p</italic> =.000</td></tr><tr><td>Q6 (item 2)</td><td>2,136</td><td>22.6% (240)</td><td>24.5% (263)</td><td>χ2<sub>(1)</sub> = 1.06; <italic>p</italic> =.304</td></tr><tr><td>Q6 (item 4)</td><td>2,097</td><td>22.4% (234)</td><td>23.6% (249)</td><td>χ2<sub>(1)</sub> =.42; <italic>p</italic> =.518</td></tr></tbody></table> </ephtml> </p> <hd id="AN0182244233-28">Nondifferentiation</hd> <p>Hypothesis 5c assumed higher levels of nondifferentiation among respondents in condition (A) with open-ended probes. This was examined for the two multi-item batteries Q3 and Q6. However, neither the absolute nor the metric measure using the mean root of pairs showed significant differences between conditions (see Table 7). Therefore, an impact on nondifferentiation cannot be confirmed.</p> <p>Table 7. Nondifferentiation.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" /><th align="center">Condition (A): Open-ended probes</th><th align="center">Condition (B): Without probes</th><th align="center">Significance level</th></tr></thead><tbody><tr><td>Absolute nondifferentiation</td><td>% (<italic>n</italic>)</td><td>% (<italic>n</italic>)</td><td /></tr><tr><td>Q3</td><td>3.3% (36)</td><td>4.1% (45)</td><td>χ2<sub>(1)</sub> =.971; <italic>p</italic> =.324</td></tr><tr><td>Q6</td><td>10.6% (116)</td><td>9.3% (103)</td><td>χ2<sub>(1)</sub> =.965; <italic>p</italic> =.326</td></tr><tr><td>Mean root of pairs</td><td>Mean (std)</td><td>Mean (std)</td><td /></tr><tr><td>Q3</td><td>.335 (.193)</td><td>.338 (.203)</td><td>T<sub>(2198)</sub> = −.435; <italic>p</italic> =.664</td></tr><tr><td>Q6</td><td>.483 (.242)</td><td>.473 (.234)</td><td>T<sub>(2198)</sub> = 1.046; <italic>p</italic> =.295</td></tr></tbody></table> </ephtml> </p> <hd id="AN0182244233-29">Discussion and Conclusion</hd> <p>The purpose of the presented research was to determine whether and in which ways embedding open-ended probes into web surveys impacts the process of responding to closed survey questions. In doing so, it took a different perspective than many current studies in the area of open-ended questions, which examine contextual effects on the response quality to open-ended questions and probes.[<reflink idref="bib8" id="ref156">8</reflink>] The study differentiated between the effects of open-ended probes on survey completion, on the survey questions the probes pertain to, and on subsequent survey questions. To this end, a randomized web survey experiment was carried out, in which closed survey questions were fielded with and without open-ended probes using a paging design. Inserting open-ended probes increased survey break off and impacted the survey questions the probes pertained to in the form of increased backtracking and answer changes. Effects on subsequent questions occurred in single cases (see Table 8 for an overview of the hypotheses and results).</p> <p>Table 8. Overview of Hypotheses and Results.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left" colspan="2">Hypotheses:</th><th align="left">Result</th></tr></thead><tbody><tr><td colspan="2">Embedding open-ended probes into web surveys...</td><td /></tr><tr><td colspan="2"><italic>Impact on survey break off</italic></td><td /></tr><tr><td>H1</td><td>... increases survey break off</td><td>Confirmed</td></tr><tr><td colspan="2"><italic>Impact on preceding survey questions</italic></td><td /></tr><tr><td>H2a</td><td>... increases backtracking</td><td>Confirmed</td></tr><tr><td>H2b</td><td>... increases answer changes</td><td>Confirmed</td></tr><tr><td colspan="2"><italic>Impact on subsequent survey questions</italic></td><td /></tr><tr><td>H3</td><td>... increases response times</td><td>Confirmed for one question only</td></tr><tr><td>H4</td><td>... decreases nonresponse in terms of item skipping and choosing nonsubstantive response options</td><td>Not confirmed</td></tr><tr><td>H5a</td><td>... does not impact mean scores</td><td>Confirmed</td></tr><tr><td>H5b</td><td>... increases extreme responding</td><td>Confirmed for one question only</td></tr><tr><td>H5c</td><td>... increases nondifferentiation</td><td>Not confirmed</td></tr></tbody></table> </ephtml> </p> <p>The majority of break offs occurred in the condition with open-ended probes, particularly after the first- and second-shown probe. The open-ended probes were not announced at the beginning of the survey, which may have contributed to the increase in break offs for the first-shown probes (rather than at the announcement that probes will be asked). Importantly, respondents with lower education and women were more likely to break off the survey. Although embedding open-ended probes did not lead to an unusually high level of survey break off, survey researchers should consider this potential nonresponse bias when implementing open-ended probes in web surveys.</p> <p>Embedding open-ended probes significantly increased backtracking and answer changes to previous survey questions. In the present study, 9 percent of respondents who received open-ended probes returned to a previous question, while only 3 percent of respondents did this in the condition without probes. Across both conditions, about one of three respondents who backtracked changed their response to the preceding survey question. Like survey break off, backtracking and answer changes to previous questions occurred most often in response to the first open-ended probes.</p> <p>Asking open-ended probes did not impact subsequent questions for the most part. There were no significant effects on mean scores, item skipping or non-differentiation for any of the examined questions. For four of the five examined questions, there were no significant effects on response time, choosing a nonsubstantive response option or extreme responding. This indicates that, in most cases, the cognitive processing of survey questions and subsequent web survey data are not impacted by inserting open-ended probes. However, there are notable exceptions.</p> <p>Respondents were significantly more likely to choose a nonsubstantive response option to the single-item measure of life satisfaction (Q4) in the condition with open-ended probes. Moreover, respondents took significantly longer to answer and were more likely to give an extreme response to the question on relationship satisfaction (Q5) in the condition with open-ended probes.</p> <p>There are several possible explanations for these findings. First, the responses to the open-ended probes indicated that the question on relationship satisfaction was flawed because it lacked a response category to indicate that one was currently not in a relationship. By the time respondents in condition (A) reached the survey question on relationship satisfaction, they were certainly expecting an open-ended probe to follow. Possibly, respondents lacking a suitable response option dealt with this irritation differently when they were expecting to be able to explain their response in an open-ended text field than when they were not expecting this. The majority of respondents who indicated that they were not in a relationship in the probe either chose the available non-substantive response option ("I do not want to answer this question") or an extreme response option (i.e., "very unsatisfied"), while only one respondent in condition (A) chose to skip the question.[<reflink idref="bib9" id="ref157">9</reflink>]</p> <p>However, alternative explanations should be considered. Perhaps, the effects of embedding open-ended probes on the response to subsequent closed survey questions are more likely to occur when there is a close connection to the preceding survey and probing questions. In the present study, the question on relationship satisfaction was directly preceded by the closely related construct on life satisfaction ([<reflink idref="bib82" id="ref158">82</reflink>]).</p> <p>Probing techniques and probe design vary strongly, as do the survey questions they pertain to, and generalizing the results of any given study to all settings is not possible. In the present study, the effect of open-ended probing on the response time and behavior of only one of the examined questions highlights that future research should establish which question, probe, and respondent characteristics determine when open-ended probes impact surrounding survey questions.</p> <p>Thus, the present study has several limitations which point the way to future research. For instance, it could be that certain probe types, such as category selection probing, increase the likelihood of respondents backtracking and changing their answers more than other probes that do not prompt respondents to reconsider their survey response. Different spacing designs should be examined to understand whether asking probes after (almost) each survey question leads to other effects than spacing probes throughout the survey or only inserting one (random) probe. Future research should include other question types, such as behavior and factual questions. Finally, future studies should employ further measures of data quality, such as test–retest reliability ([<reflink idref="bib44" id="ref159">44</reflink>]).</p> <p>In summary, embedding open-ended probes can increase survey break off, backtracking, and answer changes to previous questions, though fortunately, none of these outcomes occurred very often in the present study. Survey researchers may omit a back button to prevent effects on previous questions in practice. Of course, effects on response behavior to subsequent survey questions cannot be prevented technically; however, for this study such effects were seldom, and there is no reason to assume a worrisome impact on data collection. More than ever, [<reflink idref="bib52" id="ref160">52</reflink>] suggestion to ask single open-ended probes to a subsample of a survey seems a timely and pragmatic compromise in order to control for rare effects of open-ended probes on response behavior while gaining insights into respondents' thought processes and thereby validating survey responses.</p> <hd id="AN0182244233-30">Supplemental Material</hd> <p>Graph: Supplemental material, sj-docx-1-smr-10.1177_00491241231176846 for The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys by Patricia Hadler in Sociological Methods & Research</p> <ref id="AN0182244233-31"> <title> References </title> <blist> <bibl id="bib1" idref="ref114" type="bt">1</bibl> <bibtext> ADM, ASI, BVM and DGOF. 2021. "Richtlinie für Online-Befragungen." (https://<ulink href="http://www.adm-ev.de/wp-content/uploads/2021/03/RL-Online-2021-neu.pdf">www.adm-ev.de/wp-content/uploads/2021/03/RL-Online-2021-neu.pdf</ulink>).</bibtext> </blist> <blist> <bibl id="bib2" idref="ref94" type="bt">2</bibl> <bibtext> Aichholzer Julian, Kritzinger Sylvia. 2016. "Kurzskala politischer Zynismus (KPZ)." Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS) (https://zis.gesis.org/skala/Aichholzer-Kritzinger-Kurzskala-politischer-Zynismus-(KPZ)).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref116" type="bt">3</bibl> <bibtext> American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. AAPOR. (https://aapor.org/wp-content/uploads/2022/11/Standard-Definitions20169theditionfinal.pdf)</bibtext> </blist> <blist> <bibl id="bib4" idref="ref39" type="bt">4</bibl> <bibtext> Andrews Mark. 2005. Who Is Being Heard? Response Bias in Open-Ended Responses in a Large Government Employee Survey. Florida: Miami Beach (<ulink href="http://www.asasrms.org/Proceedings/y2005/files/JSM2005-000924.pdf">http://www.asasrms.org/Proceedings/y2005/files/JSM2005-000924.pdf</ulink>).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref89" type="bt">5</bibl> <bibtext> Bauer Paul C., Barbera Pablo, Ackermann Kathrin, Venetz Aaron. 2017. " Is the Left-Right Scale a Valid Measure of Ideology? Individual-Level Variation in Associations With "Left" and "Right" and Left-Right Self-Placement." Political Behavior. 39:553‐83. doi:10.1007/s11109-016-9368-2.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref61" type="bt">6</bibl> <bibtext> Beatty Paul C. 2004 "The Dynamics of Cognitive Interviewing." pp. 45‐66 in Methods for Testing and Evaluating Survey Questionnaires, edited by Presser S., Rothgeb J. M., Couper M. P., Lessler J. T., Martin E., Martin J., Singer E., Groves R. M., Kalton G., Rao J. N. K., Schwarz N., Skinner C. Hoboken, NJ, USA: John Wiley & Sons, Inc.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref102" type="bt">7</bibl> <bibtext> Beatty Paul C., Willis Gordon B. 2007. " Research Synthesis. The Practice of Cognitive Interviewing." Public Opinion Quarterly. 71(2):287‐311. doi:10.1093/poq/nfm006.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref51" type="bt">8</bibl> <bibtext> Behr Dorothée, Braun Michael, Kaczmirek Lars, Bandilla Wolfgang. 2012a. " Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys." Field Methods. 25(2):124‐41. doi:10.1177/1525822X12462525.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref7" type="bt">9</bibl> <bibtext> Behr Dorothée, Kaczmirek Lars, Bandilla Wolfgang, Braun Michael. 2012b. " Asking Probing Questions in Web Surveys: Which Factors Have an Impact on the Quality of Responses?" Social Science Computer Review. 30(4):487‐98. doi:10.1177/0894439311435305.</bibtext> </blist> <blist> <bibtext> Behr Dorothée, Meitinger Katharina, Braun Michael, Kaczmirek Lars. 2017. Web Probing – Implementing Probing Techniques From Cognitive Interviewing in Web Surveys With the Goal to Assess the Validity of Survey Questions. Mannheim: GESIS – Leibniz-Institute for the Social Sciences.</bibtext> </blist> <blist> <bibtext> Beierlein Constanze, Kovaleva Anastassyia, László Z., Kemper Christoph J., Rammstedt Beatrice. 2014. Kurzskala zur Erfassung der Allgemeinen Lebenszufriedenheit (L-1). doi:10.6102/zis229.</bibtext> </blist> <blist> <bibtext> Bishop George F., Hippler Hans-Jürgen, Schwarz Norbert, Strack Fritz. 1988. "A Comparison of Response Effects in Self-Administered and Telephone Surveys." pp. 321‐40 in Telephone Survey Methodology, edited by Groves R. M., Biemer P., Lyberg L., Massey J. T., Nicholls W. L., Waksberg J. New York: Wiley.</bibtext> </blist> <blist> <bibtext> Bradburn Norman M.1983. "Response Effects." pp. 289‐328 in Handbook of Survey Research, edited by Rossi P., Wright J., Anderson A. New York: Academic Press.</bibtext> </blist> <blist> <bibtext> Callegaro Mario, DiSogra Charles. 2008. " Computing Response Metrics for Online Panels." Public Opinion Quarterly. 72(5):1008‐32. doi:10.1093/poq/nfn065.</bibtext> </blist> <blist> <bibtext> Callegaro Mario, Manfreda Katja Lozar, Vehovar Vasja. 2015. Web Survey Methodology. Los Angeles: Sage Publications.</bibtext> </blist> <blist> <bibtext> Chang Linchiat, Krosnick Jon A. 2009. " National Surveys via Rdd Telephone Interviewing Versus the Internet. Comparing Sample Representativeness and Response Quality." Public Opinion Quarterly. 73(4):641‐78. doi:10.1093/poq/nfp075.</bibtext> </blist> <blist> <bibtext> Collins Debbie, ed. 2015. Cognitive Interviewing Practice. London: Sage Publications.</bibtext> </blist> <blist> <bibtext> Conrad Frederick G., Blair Johnny, Tracy Elena. 1999. Verbal Reports Are Data! A Theoretical Approach to Cognitive Interviews. Washington, DC (https://<ulink href="http://www.bls.gov/osmr/research-papers/1999/st990240.htm">www.bls.gov/osmr/research-papers/1999/st990240.htm</ulink>).</bibtext> </blist> <blist> <bibtext> Cornesse Carina, Blom Annelies G. 2020. " Response Quality in Nonprobability and Probability-Based Online Panels." Sociological Methods & Research. 21(1):online first. doi:10.1177/0049124120914940.</bibtext> </blist> <blist> <bibtext> Couper Mick P.2013. " Research Note: Reducing the Threat of Sensitive Questions in Online Surveys? " Survey Methods: Insights from the Field: 1‐9. doi:10.13094/SMIF-2013-00008.</bibtext> </blist> <blist> <bibtext> Couper Mick P., Kennedy Courtney, Conrad Frederick G., Tourangeau Roger. 2011. " Designing Input Fields for Non-Narrative Open-Ended Responses in Web Surveys." Journal of Official Statistics. 27(1):65‐85.</bibtext> </blist> <blist> <bibtext> Couper Mick P., Kreuter Frauke. 2013. " Using Paradata to Explore Item Level Response Times in Surveys." Journal of the Royal Statistical Society: Series A (Statistics in Society). 176(1):271‐86. doi:10.1111/j.1467-985X.2012.01041.x.</bibtext> </blist> <blist> <bibtext> Edwards Lisa M., Lopez Shane J. 2006. " Perceived Family Support, Acculturation, and Life Satisfaction in Mexican American Youth: a Mixed-Methods Exploration." Journal of Counseling Psychology. 53(3):279‐87.</bibtext> </blist> <blist> <bibtext> Ericsson K. Anders, Simon Herbert A. 1980. " Verbal Reports as Data." Psychological Review. 87(3):215‐51. doi:10.1037/0033-295X.87.3.215.</bibtext> </blist> <blist> <bibtext> Ericsson K. Anders, Simon Herbert A. 1993. Protocol Analysis: Verbal Reports as Data, 2nd ed. Cambridge, Massachusetts: MIT Press.</bibtext> </blist> <blist> <bibtext> Foddy William.1998. " An Empirical Evaluation of In-Depth Probes Used to Pretest Survey Questions." Sociological Methods & Research. 27(1):103‐33.</bibtext> </blist> <blist> <bibtext> Fowler Stephanie L., Willis Gordon B. 2020. "The Practice of Cognitive Interviewing Through Web Probing." pp. 451‐69 in Advances in Questionnaire Design, Development, Evaluation and Testing, edited by Beatty P., Collins D., Kaye L., Padilla J. L., Willis G. B., Wilmot A. Hoboken, New Jersey: Wiley.</bibtext> </blist> <blist> <bibtext> Fox Mark C., Ericsson K. Anders, Best Ryan. 2011. " Do Procedures for Verbal Reporting of Thinking Have to be Reactive? A Meta-Analysis and Recommendations for Best Reporting Methods." Psychological Bulletin. 137(2):316‐44. doi:10.1037/a0021663.</bibtext> </blist> <blist> <bibtext> Fuchs Dieter, Klingemann Hans-Dieter. 1989. "Das Links-Rechts-Schema als politischer Code: ein interkultureller Vergleich auf inhaltsanalytischer Grundlage." pp. 484‐98 in Kultur und Gesellschaft. Verhandlungen des 24. Deutschen Soziologentags, des 11. Österreichischen Soziologentags und des 8. Kongresses der Schweizerischen Gesellschaft für Soziologie in Zürich 1988, edited by Haller M., Hoffmann-Nowotny H.-J., Zapf W. Frankfurt/Main: Campus-Verlag.</bibtext> </blist> <blist> <bibtext> Fuchs Dieter, Klingemann Hans-Dieter. 1990. "The Left-Right Schema." pp. 203‐34 in De Gruyter Studies on North America, vol. 5, Continuities in Political Action. A Longitudinal Study of Political Orientations in Three Western Democracies, edited by Jennings M. K., van Deth J. W. Berlin, Boston: DE GRUYTER.</bibtext> </blist> <blist> <bibtext> Galesic Mirta.2006. " Dropouts on the Web: Effects of Interest and Burden Experienced During an Online Survey." Journal of Official Statistics. 22(2):313‐28.</bibtext> </blist> <blist> <bibtext> Gavras Konstantin, Höhne Jan K. 2020. " Evaluating Political Parties: Criterion Validity of Open Questions With Requests for Text and Voice Answers." International Journal of Social Research Methodology. 37(3):1‐7. doi:10.1080/13645579.2020.1860279.</bibtext> </blist> <blist> <bibtext> Gerlitz Jean-Yves.2014. Intergenerationale familiale Verpflichtung. Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS) (https://zis.gesis.org/skala/Gerlitz-Intergenerationale-familiale-Verpflichtung). doi: 10.6102/zis212.</bibtext> </blist> <blist> <bibtext> GESIS. 2020. Allgemeine Bevölkerungsumfrage der Sozialwissenschaften ALLBUS – Kumulation 1980-2018: ZA5274 Datenfile Version 1.0.0. Cologne: GESIS Data Archive.</bibtext> </blist> <blist> <bibtext> GLES. 2019. Pre-election Cross Section (GLES 2017). Cologne: GESIS Data Archive.</bibtext> </blist> <blist> <bibtext> Hadler Patricia.2021. " Question Order Effects in Cross-Cultural Web Probing – Pretesting Behavior and Attitude Questions." Social Science Computer Review 39(6):1292-1312. doi:10.1177/0894439321992779.</bibtext> </blist> <blist> <bibtext> Heerwegh Dirk.2003. " Explaining Response Latencies and Changing Answers Using Client-Side Paradata From a Web Survey." Social Science Computer Review. 21(3):360‐73. doi:10.1177/0894439303253985.</bibtext> </blist> <blist> <bibtext> Heerwegh Dirk.2011. "Internet Survey Paradata." pp. 325‐48 in Social and Behavioral Research and the Internet. Advances in Applied Methods and Research Strategies, edited by Das M., Ester P., Kaczmirek L. Florence: Routledge.</bibtext> </blist> <blist> <bibtext> Höhne Jan K., Schlosser Stephan. 2018. " Investigating the Adequacy of Response Time Outlier Definitions in Computer-Based Web Surveys Using Paradata SurveyFocus." Social Science Computer Review. 36(3):369‐78. doi:10.1177/0894439317710450.</bibtext> </blist> <blist> <bibtext> Kaczmirek Lars, Neubarth Wolfgang. 2007. "Nicht-reaktive Datenerhebung: Teilnahmeverhalten bei Befragungen mit Paradaten evaluieren." pp. 293‐311 in Online-Forschung 2007: Grundlagen und Fallstudien, edited by DGOF (Deutsche Gesellschaft für Online-Forschung) e.V. Köln: Herbert von Halem Verlag.</bibtext> </blist> <blist> <bibtext> Kemper Christoph J., Beierlein Constanze, Bensch Doreen, Kovaleva Anastassiya, Rammstedt Beatrice. 2014. Eine Kurzskala zur Erfassung des Gamma-Faktors sozial erwünschten Antwortverhaltens: Die Kurzskala Soziale Erwünschtheit-Gamma (KSE-G). 2012|25. Köln: GESIS (https://<ulink href="http://www.gesis.org/fileadmin/kurzskalen/working%5fpapers/KSE%5fG%5fWorkingpaper.pdf">www.gesis.org/fileadmin/kurzskalen/working%5fpapers/KSE%5fG%5fWorkingpaper.pdf</ulink>).</bibtext> </blist> <blist> <bibtext> Kim Yujin, Dykema Jennifer, Stevenson John, Black Penny, Moberg D. Paul. 2019. " Straightlining. Overview of Measurement, Comparison of Indicators, and Effects in Mail–Web Mixed-Mode Surveys." Social Science Computer Review. 37(2):214‐33. doi:10.1177/0894439317752406.</bibtext> </blist> <blist> <bibtext> Knowles Eric S.1988. " Item Context Effects on Personality Scales. Measuring Changes the Measure." Journal of Personality and Social Psychology. 55(2):312‐20. doi:10.1037/0022-3514.55.2.312.</bibtext> </blist> <blist> <bibtext> Knowles Eric S., Coker Michelle C., Cook Deborah A., Diercks Steven R., Irwin Mary E., Lundeen Edward J., Neville John W., Sibicky Mark E. 1992. "Order Effects Within Personality Measures." pp. 221‐36 in Context Effects in Social and Psychological Research, edited by Schwarz N., Sudman S. New York: Springer.</bibtext> </blist> <blist> <bibtext> Kritzinger Sylvia, Johann David, Aichholzer Julian, Glinitzer Konstantin, Glantschnigg Christian, Thomas Kathrin, Wagner Markus, Zeglovits Eva. 2014. AUTNES Rolling-Cross-Section Panel Study 2013: ZA5857 Data File Version 2.0.0. Cologne: GESIS Data Archive.</bibtext> </blist> <blist> <bibtext> Krosnick Jon A.1999. " Survey Research." Annual Review of Psychology. 50:537‐67.</bibtext> </blist> <blist> <bibtext> Kunz Tanja, Beuthner Christoph, Hadler Patricia, Roßmann Joss, Schaurer Ines. 2020. Informing About Web Paradata Collection and Use. Mannheim, Germany: GESIS – Leibniz Institute for the Social Sciences.</bibtext> </blist> <blist> <bibtext> Kunz Tanja, Hadler Patricia. 2020. Web Paradata in Survey Research. Mannheim, Germany: GESIS – Leibniz Institute for the Social Sciences.</bibtext> </blist> <blist> <bibtext> Kunz Tanja, Quoß Franziska, Gummer Tobias. 2020. " Using Placeholder Text in Narrative Open-Ended Questions in Web Surveys." Journal of Survey Statistics and Methodology(Online First):1‐21. doi:10.1093/jssam/smaa039.</bibtext> </blist> <blist> <bibtext> Legewie J., Gerlitz Jean-Yves, Mühleck K., Scheller P., Schrenker M. 2007. Dokumentation des International Social Justice Projekt 2006 für Deutschland. Vol. 118. Berlin: Humboldt Universität zu Berlin.</bibtext> </blist> <blist> <bibtext> Lenzner Timo, Kaczmirek Lars, Lenzner Alwine. 2010. " Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment." Applied Cognitive Psychology. 24(7):1003‐20. doi:10.1002/acp.1602.</bibtext> </blist> <blist> <bibtext> Leys Christophe, Ley Christophe, Klein Olivier, Bernard Philippe, Licata Laurent. 2013. " Detecting Outliers: Do Not Use Standard Deviation Around the Mean, Use Absolute Deviation Around the Median." Journal of Experimental Social Psychology. 49(4):764‐66. doi:10.1016/j.jesp.2013.03.013.</bibtext> </blist> <blist> <bibtext> Luebker Malte.2021. " How Much is a Box? The Hidden Cost of Adding an Open-Ended Probe to an Online Survey." methods, data, analyses. 15(1):7‐42. doi:10.12758/mda.2020.09.</bibtext> </blist> <blist> <bibtext> Matjašič Miha, Vehovar Vasja, Manfreda Katja Lozar. 2018. " Web Survey Paradata on Response Time Outliers: a Systematic Literature Review." Metodološki Zvezki. 15(1):23‐41.</bibtext> </blist> <blist> <bibtext> Meitinger Katharina.2017. " Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a Complementary Tool." Public Opinion Quarterly. 81(2):447‐72. doi:10.1093/poq/nfx009.</bibtext> </blist> <blist> <bibtext> Meitinger Katharina.2018. " What Does the General National Pride Item Measure? Insights From Web Probing." International Journal of Comparative Sociology. 59(5-6):428‐50. doi:10.1177/0020715218805793.</bibtext> </blist> <blist> <bibtext> Meitinger Katharina, Behr Dorothée. 2016. " Comparing Cognitive Interviewing and Online Probing: Do They Find Similar Results?" Field Methods. 28(4):363‐80. doi:10.1177/1525822X15625866.</bibtext> </blist> <blist> <bibtext> Meitinger Katharina, Braun Michael, Behr Dorothée. 2018. " Sequence Matters in Web Probing: The Impact of the Order of Probes on Response Quality, Motivation of Respondents, and Answer Content." Survey Research Methods. 12(2):103‐20. doi:10.18148/srm/2018.v12i2.7219.</bibtext> </blist> <blist> <bibtext> Meitinger Katharina, Kunz Tanja. 2022. " Visual Design and Cognition in List-Style Open-Ended Questions in Web Probing." Sociological Methods & Research. 28(1): Online first. doi:10.1177/00491241221077241.</bibtext> </blist> <blist> <bibtext> Miller Angie L., Lambert Amber D. 2014. " Open-Ended Survey Questions: Item Nonresponse Nightmare or Qualitative Data Dream? " Survey Practice. 7(5):1‐11. doi:10.29115/SP-2014-0024.</bibtext> </blist> <blist> <bibtext> Miller Kristen, Willson Stephanie, Chepp Valerie, Padilla José-Luis, eds. 2014. Cognitive Interviewing Methodology. Hoboken, NJ, USA: John Wiley & Sons, Inc.</bibtext> </blist> <blist> <bibtext> Neuert Cornelia E., Meitinger Katharina, Behr Dorothée. 2021. " Open-Ended Versus Closed Probes. Assessing Different Formats of Web Probing." Sociological Methods & Research. 35(2):1‐35. doi:10.1177/00491241211031271.</bibtext> </blist> <blist> <bibtext> Neuert Cornelia E., Meitinger Katharina, Behr Dorothée, Schonlau Matthias. 2021. " The Use of Open-Ended Questions in Surveys." methods, data, analyses (Special Issue). 15(1):3-6.</bibtext> </blist> <blist> <bibtext> Nießen Désirée, Partsch Melanie V., Kemper Christoph J., Rammstedt Beatrice. 2019. " An English-Language Adaptation of the Social Desirability–Gamma Short Scale (KSE-G)." Measurement Instruments for the Social Sciences. 1(2):1‐10. doi:10.1186/s42409-018-0005-1.</bibtext> </blist> <blist> <bibtext> Paulhus Delroy L.2002. "Socially Desirable Responding. The Evolution of a Construct." pp. 49‐69 in The Role of Constructs in Psychological and Educational Measurement, edited by Braun H. I., Jackson D. N., Wiley D. E. Mahwah, NJ: Erlbaum.</bibtext> </blist> <blist> <bibtext> Peytchev Andy.2009. " Survey Breakoff." Public Opinion Quarterly. 73(1):74‐97. doi:10.1093/poq/nfp014.</bibtext> </blist> <blist> <bibtext> Prochazka Fabian.2020. Vertrauen in Journalismus unter Online-Bedingungen: Zum Einfluss von Personenmerkmalen, Qualitätswahrnehmungen und Nachrichtennutzung. Wiesbaden: Springer VS.</bibtext> </blist> <blist> <bibtext> Reja Ursa, Manfreda Katja Lozar, Hlebec Valentina, Vehovar Vasja. 2003. " Open-Ended vs. Close-Ended Questions in Web Questionnaires." Metodološki Zvezki. 19:159‐77.</bibtext> </blist> <blist> <bibtext> Revilla Melanie, Couper Mick P. 2018. " Comparing Grids With Vertical and Horizontal Item-by-Item Formats for PCs and Smartphones." Social Science Computer Review. 36(3):349‐68. doi:10.1177/0894439317715626.</bibtext> </blist> <blist> <bibtext> Revilla Melanie, Couper Mick P. 2019. " Improving the Use of Voice Recording in a Smartphone Survey." Social Science Computer Review. 39:(6):1159-78. doi:10.1177/0894439319888708.</bibtext> </blist> <blist> <bibtext> Roßmann Joss, Blumenstiel Jan E., Steinbrecher Markus. 2015. " Why Do Respondents Break Off Web Surveys and Does it Matter? Results From Four Follow-Up Surveys." International Journal of Public Opinion Research. 27(2):289‐302. doi:10.1093/ijpor/edv030.</bibtext> </blist> <blist> <bibtext> Russo J. Edward, Johnson Eric J., Stephens Debra L. 1989. " The Validity of Verbal Protocols." Memory & Cognition. 17(6):759‐69. doi:10.3758/BF03202637.</bibtext> </blist> <blist> <bibtext> Scanlon Paul J.2019. " The Effects of Embedding Closed-Ended Cognitive Probes in a Web Survey on Survey Response." Field Methods. 31(4):328‐43. doi:10.1177/1525822X19871546.</bibtext> </blist> <blist> <bibtext> Scanlon Paul J.2020. "Using Targeted Embedded Probes to Quantify Cognitive Interviewing Findings." pp. 427‐49 in Advances in Questionnaire Design, Development, Evaluation and Testing, edited by Beatty P., Collins D., Kaye L., Padilla J. L., Willis G. B., Wilmot A. Hoboken, New Jersey: Wiley.</bibtext> </blist> <blist> <bibtext> Schmidt Katharina, Gummer Tobias, Roßmann Joss. 2020. " Effects of Respondent and Survey Characteristics on the Response Quality to an Open-Ended Attitude Question in Web Surveys." methods, data, analyses. 14(1):3‐34. doi:10.12758/mda.2019.05.</bibtext> </blist> <blist> <bibtext> Scholz Evi, Zuell Cornelia. 2012. " Item non-Response in Open-Ended Questions: Who Does not Answer on the Meaning of Left and Right? " Social Science Research. 41(6):1415‐28. doi:10.1016/j.ssresearch.2012.07.006.</bibtext> </blist> <blist> <bibtext> Schonlau Matthias, Couper Mick P. 2016. " Semi-Automated Categorization of Open-Ended Questions." Survey Research Methods. 10(2):143‐52. doi:10.18148/srm/2016.v10i2.6213.</bibtext> </blist> <blist> <bibtext> Schuman Howard.1966. " The Random Probe: a Technique for Evaluating the Validity of Closed Questions." American Sociological Review 32(2):218‐22.</bibtext> </blist> <blist> <bibtext> Schuman Howard, Presser Stanley. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York: Academic Press.</bibtext> </blist> <blist> <bibtext> Schwarz Norbert, Hippler Hans-J. 1995. " Subsequent Questions May Influence Answers to Preceding Questions in Mail Surveys." Public Opinion Quarterly. 59(1):93‐7. doi:10.1086/269460.</bibtext> </blist> <blist> <bibtext> Schwarz Norbert, Strack Fritz, Hippler Hans-J., Bishop George. 1991. " The Impact of Administration Mode on Response Effects in Survey Measurement." Applied Cognitive Psychology. 5(3):193‐212. doi:10.1002/acp.2350050304.</bibtext> </blist> <blist> <bibtext> Schwarz Norbert, Strack Fritz, Mai Hans-Peter. 1991. " Assimilation and Contrast Effects in Part-Whole Question Sequences: a Conversational Logic Analysis." Public Opinion Quarterly. 55(1):3‐23. https://nbn-resolving.org/urn:nbn:de:0168-ssoar-67184</bibtext> </blist> <blist> <bibtext> Schwarz Norbert, Strack Fritz, Müller Gesine, Chassein Brigitte. 1988. " The Range of Response Alternatives May Determine the Meaning of the Question. Further Evidence on Informative Functions of Response Alternatives." Social Cognition. 6(2):107‐17. doi:10.1521/soco.1988.6.2.107.</bibtext> </blist> <blist> <bibtext> Silber Henning, Zuell Cornelia, Kuehnel Steffen-M. 2020. " What Can We Learn From Open Questions in Surveys? A Case Study on Non-Voting Reported in the 2013 German Longitudinal Election Study." Methodology (European Journal of Research Methods for the Behavioral and Social Sciences). 16(1):41‐58. doi:10.5964/meth.2801.</bibtext> </blist> <blist> <bibtext> Singer Eleanor, Couper Mick P. 2017. " Some Methodological Uses of Responses to Open Questions and Other Verbatim Comments in Quantitative Surveys." methods, data, analyses. 11(2):115‐34. doi:10.12758/mda.2017.01.</bibtext> </blist> <blist> <bibtext> Smith Tom W.1982. Conditional Order Effects. GSS Technical Report no. 13. Chicago: NORC (https://gss.norc.org/Documents/reports/methodological-reports/MR020.pdf).</bibtext> </blist> <blist> <bibtext> Smyth Jolene D., Dillman Don A., Christian Leah M., Mcbride Mallory. 2009. " Open-Ended Questions in Web Surveys: Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality? " Public Opinion Quarterly. 73(2):325‐37. doi:10.1093/poq/nfp029.</bibtext> </blist> <blist> <bibtext> Sudman Seymour, Bradburn Norman M., Schwarz Norbert. 1996. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco, Calif.: Jossey-Bass.</bibtext> </blist> <blist> <bibtext> Tourangeau Roger, Conrad Frederick G., Couper Mick P., Ye Cong. 2014. " The Effects of Providing Examples in Survey Questions." Public Opinion Quarterly. 78(1):100‐25. doi:10.1093/poq/nft083.</bibtext> </blist> <blist> <bibtext> Tourangeau Roger, Rasinski Kenneth A. 1988. " Cognitive Processes Underlying Context Effects in Attitude Measurement." Psychological Bulletin. 103:299‐314.</bibtext> </blist> <blist> <bibtext> Tourangeau Roger, Rips Lance J., Rasinski Kenneth A. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Willis Gordon B.1994. " Cognitive Interviewing and Questionnaire Design: a Training Manual." Cognitive Methods Staff Working Paper Series. 7:1‐56.</bibtext> </blist> <blist> <bibtext> Willis Gordon B.2005. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks: Sage.</bibtext> </blist> <blist> <bibtext> Willis Gordon B.2015. Analysis of the Cognitive Interview in Questionnaire Design. Oxford: Oxford University Press.</bibtext> </blist> <blist> <bibtext> Willis Gordon B., Schechter Susan, Whitaker Karen. 1999. "A Comparison of Cognitive Interviewing, Expert Review, and Behavior Coding: What Do They Tell Us?" (<ulink href="http://www.asasrms.org/Proceedings/papers/1999%5f006.pdf">http://www.asasrms.org/Proceedings/papers/1999%5f006.pdf</ulink>).</bibtext> </blist> <blist> <bibtext> Yan Ting, Olson Kristen. 2013. "Analyzing Paradata to Investigate Measurement Error." Pp. 73‐95 in Improving Surveys with Paradata. Analytic Uses of Process Information, edited by Kreuter F. Hoboken, New Jersey: John Wiley & Sons, Inc.</bibtext> </blist> <blist> <bibtext> Yan Ting, Tourangeau Roger. 2008. " Fast Times and Easy Questions. The Effects of Age, Experience and Question Complexity on Web Survey Response Times." Applied Cognitive Psychology. 22(1):51‐68. doi:10.1002/acp.1331.</bibtext> </blist> <blist> <bibtext> Zuell Cornelia, Menold Natalja, Körber Sabine. 2015. " The Influence of the Answer Box Size on Item Nonresponse to Open-Ended Questions in a Web Survey." Social Science Computer Review. 33(1):115‐22. doi:10.1177/0894439314528091.</bibtext> </blist> <blist> <bibtext> Zuell Cornelia, Scholz Evi. 2015. " Who is Willing to Answer Open-Ended Questions on the Meaning of Left and Right? " Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique. 127(1):26‐42. doi:10.1177/0759106315582199.</bibtext> </blist> </ref> <ref id="AN0182244233-32"> <title> Footnotes </title> <blist> <bibtext> The quantitative data set of this study and analysis file is available under the following link: Hadler, Patricia (2023): Hadler 2023 SMR_Effect of openended probes_Analysis.sps. figshare. Dataset.https://doi.org/10.6084/m9.figshare.22499557.v1. The answers to the open-ended questions are not publicly available due to them containing information that could compromise participant privacy.</bibtext> </blist> <blist> <bibtext> The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> The author(s) received no financial support for the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibtext> Patricia Hadler https://orcid.org/0000-0003-4901-8618</bibtext> </blist> <blist> <bibtext> Supplemental material for this article is available online.</bibtext> </blist> <blist> <bibtext> Respondents could also choose the nonbinary category "divers"; this was, however, not subjected to quotas.</bibtext> </blist> <blist> <bibtext> MANCOVA results with respect to the covariates (1) age: <emph>Wilks-Lambda</emph> = 0.87; <emph>F</emph><subs>(5,1500)</subs> = 46.59; <emph>p</emph> <.001; <emph>η</emph> 2 =.13; (2) device used: <emph>Wilks-Lambda</emph> = 0.92; <emph>F</emph><subs>(5,1500)</subs> = 25.76; <emph>p</emph> <.001; <emph>η</emph> 2 =.08; (3) education: <emph>Wilks-Lambda</emph> = 0.996; <emph>F</emph><subs>(5,1500)</subs> = 1.18; <emph>p</emph> =.319 n.s.</bibtext> </blist> <blist> <bibtext> A series of studies have examined the impact of respondent and survey characteristics ([49]) on response quality to open-ended questions, including the size of the answer box ([39]; [55]; [98]), the use of placeholder text ([34]), the order of the closed survey questions ([26]) and the sequence of the open-ended probes ([58]).</bibtext> </blist> <blist> <bibtext> Unfortunately, as the relationship status of respondents in condition B without probes was not determined in the course of the study, it cannot be determined whether respondents in condition B who were not in a relationship at the time of the survey systematically chose other survey response options.</bibtext> </blist> </ref> <aug> <p>By Patricia Hadler</p> <p>Reported by Author</p> <p></p> <p>Patricia Hadler studied communication sciences, psychology, and French literature at Johannes Gutenberg-Universität Mainz (Magister Artium 2009). She is a PhD candidate in psychology at the University of Mannheim and a researcher at the GESIS pretest lab. Her research focus is cognitive pretesting methodology.</p> </aug> <nolink nlid="nl1" bibid="bib42" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib55" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib22" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib70" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib51" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib15" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib52" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib56" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib17" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib61" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib38" firstref="ref13"></nolink> <nolink nlid="nl12" bibid="bib14" firstref="ref14"></nolink> <nolink nlid="nl13" bibid="bib85" firstref="ref15"></nolink> <nolink nlid="nl14" bibid="bib53" firstref="ref16"></nolink> <nolink nlid="nl15" bibid="bib18" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib48" firstref="ref27"></nolink> <nolink nlid="nl17" bibid="bib74" firstref="ref28"></nolink> <nolink nlid="nl18" bibid="bib91" firstref="ref29"></nolink> <nolink nlid="nl19" bibid="bib83" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib32" firstref="ref32"></nolink> <nolink nlid="nl21" bibid="bib90" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib21" firstref="ref34"></nolink> <nolink nlid="nl23" bibid="bib44" firstref="ref35"></nolink> <nolink nlid="nl24" bibid="bib45" firstref="ref37"></nolink> <nolink nlid="nl25" bibid="bib98" firstref="ref38"></nolink> <nolink nlid="nl26" bibid="bib40" firstref="ref40"></nolink> <nolink nlid="nl27" bibid="bib49" firstref="ref41"></nolink> <nolink nlid="nl28" bibid="bib50" firstref="ref42"></nolink> <nolink nlid="nl29" bibid="bib99" firstref="ref43"></nolink> <nolink nlid="nl30" bibid="bib41" firstref="ref44"></nolink> <nolink nlid="nl31" bibid="bib84" firstref="ref50"></nolink> <nolink nlid="nl32" bibid="bib58" firstref="ref53"></nolink> <nolink nlid="nl33" bibid="bib43" firstref="ref55"></nolink> <nolink nlid="nl34" bibid="bib24" firstref="ref56"></nolink> <nolink nlid="nl35" bibid="bib25" firstref="ref57"></nolink> <nolink nlid="nl36" bibid="bib57" firstref="ref58"></nolink> <nolink nlid="nl37" bibid="bib72" firstref="ref60"></nolink> <nolink nlid="nl38" bibid="bib12" firstref="ref62"></nolink> <nolink nlid="nl39" bibid="bib19" firstref="ref64"></nolink> <nolink nlid="nl40" bibid="bib88" firstref="ref65"></nolink> <nolink nlid="nl41" bibid="bib81" firstref="ref75"></nolink> <nolink nlid="nl42" bibid="bib62" firstref="ref77"></nolink> <nolink nlid="nl43" bibid="bib29" firstref="ref90"></nolink> <nolink nlid="nl44" bibid="bib20" firstref="ref91"></nolink> <nolink nlid="nl45" bibid="bib31" firstref="ref95"></nolink> <nolink nlid="nl46" bibid="bib67" firstref="ref96"></nolink> <nolink nlid="nl47" bibid="bib65" firstref="ref98"></nolink> <nolink nlid="nl48" bibid="bib82" firstref="ref103"></nolink> <nolink nlid="nl49" bibid="bib23" firstref="ref104"></nolink> <nolink nlid="nl50" bibid="bib79" firstref="ref105"></nolink> <nolink nlid="nl51" bibid="bib54" firstref="ref106"></nolink> <nolink nlid="nl52" bibid="bib35" firstref="ref108"></nolink> <nolink nlid="nl53" bibid="bib47" firstref="ref115"></nolink> <nolink nlid="nl54" bibid="bib10" firstref="ref117"></nolink> <nolink nlid="nl55" bibid="bib59" firstref="ref121"></nolink> <nolink nlid="nl56" bibid="bib60" firstref="ref123"></nolink> <nolink nlid="nl57" bibid="bib27" firstref="ref124"></nolink> <nolink nlid="nl58" bibid="bib33" firstref="ref126"></nolink> <nolink nlid="nl59" bibid="bib37" firstref="ref127"></nolink> <nolink nlid="nl60" bibid="bib28" firstref="ref129"></nolink> <nolink nlid="nl61" bibid="bib16" firstref="ref131"></nolink> <nolink nlid="nl62" bibid="bib36" firstref="ref132"></nolink> <nolink nlid="nl63" bibid="bib46" firstref="ref134"></nolink> <nolink nlid="nl64" bibid="bib13" firstref="ref141"></nolink> <nolink nlid="nl65" bibid="bib30" firstref="ref143"></nolink> <nolink nlid="nl66" bibid="bib11" firstref="ref144"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1458200
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Patricia+Hadler%22">Patricia Hadler</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4901-8618">0000-0003-4901-8618</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2025 54(1):106-139.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 34
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Online+Surveys%22">Online Surveys</searchLink><br /><searchLink fieldCode="DE" term="%22Discovery+Processes%22">Discovery Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Cost+Effectiveness%22">Cost Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Questionnaires%22">Questionnaires</searchLink><br /><searchLink fieldCode="DE" term="%22Response+Rates+%28Questionnaires%29%22">Response Rates (Questionnaires)</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Processes%22">Cognitive Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Regional+Characteristics%22">Regional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Age%22">Age</searchLink><br /><searchLink fieldCode="DE" term="%22Sex%22">Sex</searchLink><br /><searchLink fieldCode="DE" term="%22Sampling%22">Sampling</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Germany%22">Germany</searchLink><br /><searchLink fieldCode="DE" term="%22East+Germany%22">East Germany</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241231176846
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Probes are follow-ups to survey questions used to gain insights on respondents' understanding of and responses to these questions. They are usually administered as open-ended questions, primarily in the context of questionnaire pretesting. Due to the decreased cost of data collection for open-ended questions in web surveys, researchers have argued for embedding more open-ended probes in large-scale web surveys. However, there are concerns that this may cause reactivity and impact survey data. The study presents a randomized experiment in which identical survey questions were run with and without open-ended probes. Embedding open-ended probes resulted in higher levels of survey break off, as well as increased backtracking and answer changes to previous questions. In most cases, there was no impact of open-ended probes on the cognitive processing of and response to survey questions. Implications for embedding open-ended probes into web surveys are discussed.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://doi.org/10.6084/m9.figshare.22499557.v1
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1458200
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1458200
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241231176846
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 34
        StartPage: 106
    Subjects:
      – SubjectFull: Online Surveys
        Type: general
      – SubjectFull: Discovery Processes
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: Cost Effectiveness
        Type: general
      – SubjectFull: Questionnaires
        Type: general
      – SubjectFull: Response Rates (Questionnaires)
        Type: general
      – SubjectFull: Cognitive Processes
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Regional Characteristics
        Type: general
      – SubjectFull: Age
        Type: general
      – SubjectFull: Sex
        Type: general
      – SubjectFull: Sampling
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Germany
        Type: general
      – SubjectFull: East Germany
        Type: general
    Titles:
      – TitleFull: The Effects of Open-Ended Probes on Closed Survey Questions in Web Surveys
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Patricia Hadler
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 54
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1