View in EDS HTML Full Text PDF Full Text

Updating 'The Future of Coding': Qualitative Coding with Generative Large Language Models

Saved in:
Bibliographic Details
Title: Updating 'The Future of Coding': Qualitative Coding with Generative Large Language Models
Language: English
Authors: Nga Than (ORCID 0000-0002-6845-6253), Leanne Fan, Tina Law (ORCID 0000-0001-7631-6763), Laura K. Nelson (ORCID 0000-0001-8948-300X), Leslie McCall (ORCID 0000-0002-7700-3969)
Source: Sociological Methods & Research. 2025 54(3):849-888.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 40
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Artificial Intelligence, Coding, Qualitative Research, Cues, Open Source Technology, Accuracy, Natural Language Processing, Social Science Research
DOI: 10.1177/00491241251339188
ISSN: 0049-1241
1552-8294
Abstract: Over the past decade, social scientists have adapted computational methods for qualitative text analysis, with the hope that they can match the accuracy and reliability of hand coding. The emergence of GPT and open-source generative large language models (LLMs) has transformed this process by shifting from programming to engaging with models using natural language, potentially mimicking the in-depth, inductive, and/or iterative process of qualitative analysis. We test the ability of generative LLMs to replicate and augment traditional qualitative coding, experimenting with multiple prompt structures across four closed- and open-source generative LLMs and proposing a workflow for conducting qualitative coding with generative LLMs. We find that LLMs can perform nearly as well as prior supervised machine learning models in accurately matching hand-coding output. Moreover, using generative LLMs as a natural language interlocutor closely replicates traditional qualitative methods, indicating their potential to transform the qualitative research process, despite ongoing challenges.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1475799
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGv4CNRAV4bn3y44pevNs-xAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDFiM2008VvPrMEs-hwIBEICBmwWUhqngRdn9aeRCeZ664apTlErZg2d54vKdEOiAOQFhTLdJIvqWE9j56-OAa6AYXY-ubI0ixIgqwtDp5G7AILJa_RChvgCYIQsI6sQc48Of_UpjzhUX61q3DuMdYWgA7_QiTVpRC4pntfOfFDVCcIP8DXig1QhAe8NHlQplmyXNXzlwqB55avnRF2Ro2dC9_mF8imHNX7QiUMYw
Text:
  Availability: 1
  Value: <anid>AN0186128775;som01aug.25;2025Jun26.00:54;v2.2.500</anid> <title id="AN0186128775-1">Updating "The Future of Coding": Qualitative Coding with Generative Large Language Models </title> <p>Over the past decade, social scientists have adapted computational methods for qualitative text analysis, with the hope that they can match the accuracy and reliability of hand coding. The emergence of GPT and open-source generative large language models (LLMs) has transformed this process by shifting from programming to engaging with models using natural language, potentially mimicking the in-depth, inductive, and/or iterative process of qualitative analysis. We test the ability of generative LLMs to replicate and augment traditional qualitative coding, experimenting with multiple prompt structures across four closed- and open-source generative LLMs and proposing a workflow for conducting qualitative coding with generative LLMs. We find that LLMs can perform nearly as well as prior supervised machine learning models in accurately matching hand-coding output. Moreover, using generative LLMs as a natural language interlocutor closely replicates traditional qualitative methods, indicating their potential to transform the qualitative research process, despite ongoing challenges.</p> <p>Keywords: generative large language models; qualitative coding; epistemology; qualitative methods; computational methods; computational text analysis</p> <hd id="AN0186128775-2">Introduction</hd> <p>The exploration of automated methods for discovering, refining, and annotating concepts and other social artifacts in textual data has been a major topic of research in the social sciences over the past decade. Computational social scientists have focused on refining new techniques from computer and information sciences to enable greater accuracy in identifying the often subtle and complex subject matter present in texts of interest to social scientists (e.g., [<reflink idref="bib23" id="ref1">23</reflink>]). The goal for many is twofold: to reach a level of accuracy and reliability commensurate with hand coding, facilitating the incorporation of larger volumes of data, and to enable the discovery of new patterns and topics not readily discoverable via qualitative reading alone ([<reflink idref="bib19" id="ref2">19</reflink>]). The introduction of each new application of a different computational technique generally requires advanced quantitative and programming skills ([<reflink idref="bib28" id="ref3">28</reflink>]). That cycle was arguably broken with the emergence in 2022 of a comparatively user-friendly product, ChatGPT by OpenAI.</p> <p>The decades-long shift from statistical natural language processing to the deep learning paradigm, represented in part by the GPT models (the models underlying ChatGPT), has fundamentally transformed computational text analysis in (at least) three respects. First, there has been a shift from methods that involve the manual engineering of text into numerical representations, such as document-term matrices, to deep learning techniques that learn, via neural network modeling, distributed representations of words and their context ([<reflink idref="bib5" id="ref4">5</reflink>]; [<reflink idref="bib38" id="ref5">38</reflink>]). Second, transfer learning and large pre-trained models have led to greater accuracy for many tasks while greatly reducing data and compute requirements ([<reflink idref="bib15" id="ref6">15</reflink>]). Third, and more recently, there has been a shift from an interface that uses programming languages exclusively to an interface that also uses natural language inputs and outputs ([<reflink idref="bib25" id="ref7">25</reflink>]). As we argue here, the deep learning and transfer learning paradigms, when combined with natural language interfaces, have the potential to transform the qualitative coding process, when validated and used with care.</p> <p>Indeed, the growing interest in testing the ability for generative Large Language Models (LLMs) like GPT to augment or substantially replace hand coding is a testament to their projected transformative power (see, e.g., [<reflink idref="bib3" id="ref8">3</reflink>]; [<reflink idref="bib7" id="ref9">7</reflink>]; [<reflink idref="bib13" id="ref10">13</reflink>]; [<reflink idref="bib15" id="ref11">15</reflink>]; [<reflink idref="bib50" id="ref12">50</reflink>]). However, the results on accuracy have thus far been mixed, and, in part because of the natural language interface, there are currently no widely agreed-upon guidelines on how best to use LLMs for qualitative coding practices that involve labeling, classification, annotation, or discovery tasks (but see [<reflink idref="bib11" id="ref13">11</reflink>]; [<reflink idref="bib21" id="ref14">21</reflink>]; [<reflink idref="bib46" id="ref15">46</reflink>]; [<reflink idref="bib49" id="ref16">49</reflink>]). Moreover, much of the recent work examining the capabilities of LLMs has sought to compare the accuracy of LLM classifications to, for example, crowd workers or expert coders across a range of tasks (e.g., [<reflink idref="bib2" id="ref17">2</reflink>]; [<reflink idref="bib11" id="ref18">11</reflink>]; [<reflink idref="bib15" id="ref19">15</reflink>]; [<reflink idref="bib50" id="ref20">50</reflink>]; [<reflink idref="bib59" id="ref21">59</reflink>]). Yet <emph>generative</emph> LLMs do more than produce text classifications; they take natural language as input and output natural language. This processual shift, we argue, could mark a turning point for the qualitative text analysis process in the social sciences (see also [<reflink idref="bib25" id="ref22">25</reflink>]).</p> <p>In this article, we examine whether instruction-tuned generative LLMs—LLMs fine-tuned to better understand and follow user instructions and then used to generate text—can replicate and reliably augment the traditional iterative qualitative process of alternating between deductive and inductive analysis when annotating complex social science textual data ([<reflink idref="bib14" id="ref23">14</reflink>]; [<reflink idref="bib15" id="ref24">15</reflink>]; [<reflink idref="bib25" id="ref25">25</reflink>]; [<reflink idref="bib41" id="ref26">41</reflink>]). Our test case is derived from an original, in-depth, fully qualitative project in which graduate and undergraduate students hand coded a corpus of over 1,200 articles appearing in U.S.-based newsweeklies from 1980 to 2012. We refer to this process as text classification, in accordance with the terminology used in machine learning, but our data and classification tasks are distinct in two interconnected ways. First, the documents are longer than the data LLMs have been tested on thus far for classification and related content-analysis tasks in the social sciences.[<reflink idref="bib6" id="ref27">6</reflink>] Second, the classification task requires a holistic and contextual reading of entire documents; that is, the concept to be identified cannot usually be gleaned from a single sentence or paragraph. In addition to enabling us to benchmark our results against those of human coders, this dataset allows us to propose a more general workflow that converts qualitative coding processes into a new approach that emphasizes strategic interaction with machines while retaining key qualitative coding tools, such as a detailed coding guide and deep human reading at key points throughout the coding process (see Figure 1, discussed further below).</p> <p>Graph: Figure 1. Researcher-LLM-Researcher workflow.</p> <p>Concretely, we examine whether instructing LLMs to interpret and categorize text based on holistic qualitative criteria detailed in extensive coding guides can generate not only accurate outputs but also insights into the coding process itself comparable to those gained from the iterative process of training a group of research assistants to apply codes by hand. For instance, rather than hand coding a large number of documents for a supervised machine learning task, research assistants could instead participate in the development of instructions for LLMs and conduct more limited but essential hand coding for the purposes of concept refinement and output validation ([<reflink idref="bib8" id="ref28">8</reflink>]; [<reflink idref="bib15" id="ref29">15</reflink>]; [<reflink idref="bib25" id="ref30">25</reflink>]; also, see again Figure 1). Alternatively, scholars without research assistants, including graduate students themselves, can engage in this same process on their own, including scholars who are embarking on a new text analysis project, or those wishing to modify an existing qualitative coding project or to extend it to a new and larger corpus. Given the format of our data, the processes we explore here are likely best suited for the content analysis of long-form documentary evidence (e.g., legacy news, organizational documents, oral histories), which constitutes a central line of investigation in the social sciences ([<reflink idref="bib17" id="ref31">17</reflink>]; [<reflink idref="bib20" id="ref32">20</reflink>]; [<reflink idref="bib22" id="ref33">22</reflink>]; [<reflink idref="bib35" id="ref34">35</reflink>]; [<reflink idref="bib42" id="ref35">42</reflink>]; [<reflink idref="bib44" id="ref36">44</reflink>]).</p> <p>With this overarching goal in mind, we contribute to the emerging literature on generative LLMs in two more specific ways. First, drawing from the existing literature, we experimented with multiple methods of interacting with, or "prompting," LLMs. We explore how to convert a coding guide into a prompt or sequence of prompts, and, in particular, whether and how to derive a definition of a multidimensional concept from the coding guide to use as part of a natural language prompt (or sequence of prompts). For instance, we deductively derived definitions of the focal concept from the hand-coding guide ourselves (i.e., manually) and then in a separate test deployed LLMs to generate these definitions from the guide, thus automating parts of the prompting process itself ([<reflink idref="bib29" id="ref37">29</reflink>]; [<reflink idref="bib56" id="ref38">56</reflink>]; [<reflink idref="bib57" id="ref39">57</reflink>]). Like others, we also tested whether to provide examples of pre-labeled documents within the prompt (i.e., zero- and few-shot prompts, see [<reflink idref="bib11" id="ref40">11</reflink>]; [<reflink idref="bib50" id="ref41">50</reflink>]; [<reflink idref="bib59" id="ref42">59</reflink>]), though our work highlights the challenges of these formats when analyzing long documents specifically. Given the almost infinite variation in prompts that is possible with generative LLMs, due in large part to the vagaries of the natural language interface itself, our tests across a wide range of prompting strategies offer initial guidance for some of the more challenging tasks in automated textual analysis in the social sciences.</p> <p>Second, across all the main tests in which we systematically vary the prompting strategy, we compare the performance of both closed-source and open-source LLMs. We focused in particular on open-source LLMs that are readily available on the platform Ollama, which allows others to immediately replicate our process without the need for specialized hardware or accounts with proprietary platforms.[<reflink idref="bib7" id="ref43">7</reflink>] There is also greater transparency about the training data for open-source models, enabling users to determine whether their own data are part of the training set, which could lead to increased accuracy. We compare the results from (pre-quantized) open-source LLMs to GPT4, as GPT4 continues to be the industry standard, though, like others, we maintain, and illustrate why, it should not generally be used for social science research because of its hidden, proprietary architecture ([<reflink idref="bib46" id="ref44">46</reflink>]; [<reflink idref="bib51" id="ref45">51</reflink>]).[<reflink idref="bib8" id="ref46">8</reflink>]</p> <p>In the remainder of the paper, we begin in Section "Reconciling Qualitative and Computational Approaches to Textual Analysis" by further describing our overarching goal, which is to use the traditional qualitative coding process of developing a detailed codebook and training research assistants as a template for instructing and interacting with LLMs. In the following section, we turn to a discussion of LLMs themselves and the potentially important distinction between closed- and open-source models, which we test in the empirical section of the paper. We then describe our data and specific classification task, and how it relates to other kinds of textual analysis in the social sciences, in Section "Data and Classification Task," followed in Section "Prompting Strategies" by a description of how we convert that classification task into prompting strategies. Sections "Results" and "Discussion" provide the results and concluding discussion. Although experimentation with LLMs will be ongoing for some time to come, in part because the terrain of LLMs is itself changing rapidly, our tests suggest that LLMs have the potential to allow researchers to deploy (invaluable yet resource-intensive) qualitative hand coding more selectively and strategically in their projects ([<reflink idref="bib37" id="ref47">37</reflink>]).</p> <hd id="AN0186128775-3">Reconciling Qualitative and Computational Approaches to Textual Analysis</hd> <p>Traditional qualitative coding in the social sciences typically involves researchers iteratively—through both deductive and inductive approaches—developing comprehensive codebooks to define and measure complex themes in the text, followed by extensive training of research assistants to apply these guidelines accurately while refining the codebook and measures during this process ([<reflink idref="bib12" id="ref48">12</reflink>]; [<reflink idref="bib14" id="ref49">14</reflink>]; [<reflink idref="bib17" id="ref50">17</reflink>]; [<reflink idref="bib24" id="ref51">24</reflink>]; [<reflink idref="bib41" id="ref52">41</reflink>]). This process ensures that elaborate and sometimes fuzzy themes are defined clearly enough that multiple research assistants independently code them in the same way, and it preserves nuanced human understanding across analyses. In some cases, including the case we explore here, the coding of themes cannot happen at the word, phrase, sentence, or even paragraph level, but requires a holistic and contextualized reading of an entire document (as illustrated further below). For all these reasons, a humanistic approach will certainly continue to play an essential role in the analysis of social texts even when automated methods are introduced to "augment" the qualitative coding process ([<reflink idref="bib23" id="ref53">23</reflink>]).</p> <p>Nonetheless, the process of qualitative coding presents several challenges, all well recognized by qualitative researchers. Hand coding is time consuming and thus inflexible: once a significant amount of text has been hand coded, it is difficult to change coding schemes if desired, although this rigidity is relaxed to a certain extent by qualitative data analysis software ([<reflink idref="bib14" id="ref54">14</reflink>]). Reliability across (and even within) coders is often difficult to achieve, necessitating complicated intercoder reliability tests and metrics (e.g., [<reflink idref="bib34" id="ref55">34</reflink>]). And, as datasets grow in size and complexity, qualitative researchers often do not feel hand coding is capable of discovering and capturing the full nuance and range of themes contained within. Indeed, quantitative researchers bemoan the inadequacy of conventional deductive quantitative methods for the same reason ([<reflink idref="bib9" id="ref56">9</reflink>]; [<reflink idref="bib15" id="ref57">15</reflink>]; [<reflink idref="bib23" id="ref58">23</reflink>]; [<reflink idref="bib55" id="ref59">55</reflink>]).</p> <p>Still, many qualitative researchers have already invested an enormous amount of time in the development of lengthy hand-coding guides, and still others will need to begin new qualitative coding projects with the development of a guide. This is because such researchers are often interested in subject matter—what we will refer to as concept identification—that cannot be easily traced in an accounting of specific words deductively produced by an expert in the field of analysis ([<reflink idref="bib8" id="ref60">8</reflink>]; [<reflink idref="bib15" id="ref61">15</reflink>]; [<reflink idref="bib31" id="ref62">31</reflink>]; [<reflink idref="bib43" id="ref63">43</reflink>]; [<reflink idref="bib52" id="ref64">52</reflink>]; [<reflink idref="bib54" id="ref65">54</reflink>]). This may be true for many reasons. Relevant words may be either polysemic or nearly absent in texts where alternative—more accessible, euphemistic, subtle, implicit, or intentionally obscured—ways of referencing the concept are used. Specific words may also be insufficient on their own to describe the multifaceted or multidimensional nature of many social science concepts. Under these circumstances, even computational researchers advocate for the use of highly trained human coders for the labeling of a subset of data for validation purposes. In comparison, training coders is relatively straightforward and less time-intensive for tasks involving entity recognition and stance detection in short pieces of text ([<reflink idref="bib15" id="ref66">15</reflink>]; [<reflink idref="bib50" id="ref67">50</reflink>]).</p> <p>A growing body of research in the computational analysis of textual data recognizes the challenges of using computational methods to code complex concepts ([<reflink idref="bib8" id="ref68">8</reflink>]; [<reflink idref="bib15" id="ref69">15</reflink>]; [<reflink idref="bib26" id="ref70">26</reflink>]; [<reflink idref="bib43" id="ref71">43</reflink>]; [<reflink idref="bib54" id="ref72">54</reflink>]), but a gap in the literature remains in how to translate the process of defining concepts in a coding guide into the process of instructing and interacting with generative LLMs. As noted above, potential similarities in these processes are enabled by the fact that the generative LLM interface takes natural language as input and produces natural language as output. Moreover, LLMs model recurring patterns in extended text sequences, leveraging (typically) dense representations of words to encode both common syntactic structures and subtle semantic variations, such as diverse styles, tones, and topics of discussion. We thus investigate the practical use of generative LLMs for iterative qualitative coding, testing the accuracy of LLM output as well as their ability to replicate the processes traditionally done by human researchers while maintaining a crucial role for human interpretation and intervention ([<reflink idref="bib23" id="ref73">23</reflink>]).</p> <hd id="AN0186128775-4">Variation among Generative LLMs</hd> <p>Thus far, we have stressed the fact that instruction-tuned generative LLMs produce human-like output by mimicking the complexities of natural language. Yet, scholars have strongly cautioned social scientists against some of the AI industry's "hype" ([<reflink idref="bib46" id="ref74">46</reflink>]). Closed-source or proprietary LLMs, such as ChatGPT, have garnered particularly harsh critique for a wide range of reasons ([<reflink idref="bib46" id="ref75">46</reflink>]; [<reflink idref="bib51" id="ref76">51</reflink>]). The lack of transparency in training data and algorithms prevents researchers from being able to fully account for what the models encode or what they can accurately do. Closed-source models also restrict customization and adaptability, which limits researchers' ability to tailor them to specific research questions or contexts. The reliance on closed-source models additionally raises issues of reproducibility, as researchers cannot verify results without access to the same (often expensive) tools and data, and even then, the same models can produce different results when the underlying model structure is updated without notice. These challenges pose ultimately insurmountable barriers to reproducible and ethical social science research ([<reflink idref="bib46" id="ref77">46</reflink>]; [<reflink idref="bib51" id="ref78">51</reflink>]).</p> <p>In response, corporations and the scientific community are embracing open-source models for their greater transparency and collaborative potential. The release of models such as OpenAI's GPT2 laid the groundwork for subsequent open-source initiatives. Organizations like Hugging Face, The Allen Institute for AI, and EleutherAI and platforms such as Ollama have made significant strides in democratizing access to powerful language models, enabling researchers to inspect, modify, and build upon existing architectures using local machines without specialized hardware. And Meta and Google have released their own open-source models—Llama ([<reflink idref="bib16" id="ref79">16</reflink>]) and Gemma ([<reflink idref="bib40" id="ref80">40</reflink>]), respectively—that are beginning to perform on par with closed-source alternatives for many tasks (as we show below, this is the case with our experiments). These open-source models provide researchers with the flexibility to tailor models to specific needs, ensuring greater accuracy in and relevance to their work. Additionally, open-source initiatives promote reproducibility and peer validation, as researchers can freely access and test models. This collaborative environment accelerates innovation but also invites the kind of broader scrutiny needed to mitigate well-documented biases in language models ([<reflink idref="bib4" id="ref81">4</reflink>]; [<reflink idref="bib46" id="ref82">46</reflink>]; [<reflink idref="bib51" id="ref83">51</reflink>]).</p> <p>In short, with the boom in the development of open-source LLMs and their intermediary platforms (e.g., Ollama), the social science community is in a position to rigorously consider how LLMs of various kinds—and the interactive, generative use of them—might change the landscape of computational, qualitative text analysis. Here we focus in particular on the implications of LLMs for coding subject matter in the social sciences that is challenging even for humans to identify systematically.</p> <hd id="AN0186128775-5">Data and Classification Task</hd> <p>Our current work extends an original, fully qualitative project conducted by one of our co-authors ([<reflink idref="bib39" id="ref84">39</reflink>]) and a replication study conducted by two of our co-authors ([<reflink idref="bib43" id="ref85">43</reflink>]). In the original project, more than a half dozen undergraduate and graduate research assistants coded a sample of 1,253 newsweekly articles appearing in <emph>Time</emph>, <emph>Newsweek</emph>, and <emph>U.S. News & World Report</emph> between 1980 and 2012 ([<reflink idref="bib39" id="ref86">39</reflink>]). The original task entailed the identification of a broad, multifaceted concept used widely in the social sciences—socio-economic inequality—and its manifestation through a new empirical reality unfolding over time—the rise in wage, earnings, income, and wealth inequality beginning in the late 1970s. This coding task is like many others in politics and political sociology where scholars qualitatively analyze media and other discursive data to better understand whether and how a new issue becomes known and politicized in the public sphere (e.g., [<reflink idref="bib22" id="ref87">22</reflink>], on welfare; [<reflink idref="bib17" id="ref88">17</reflink>], on abortion). Additionally, the multifaceted nature of the concepts to be identified, which can straddle multiple paragraphs of a long document, are characteristics shared by other kinds of long-form qualitative data, such as legacy news, interview/audio transcripts, oral histories, and organizational documents ([<reflink idref="bib14" id="ref89">14</reflink>]; [<reflink idref="bib36" id="ref90">36</reflink>]; [<reflink idref="bib41" id="ref91">41</reflink>]).</p> <p>The specific data and classification task in this project has three other characteristics that could be of more general interest to qualitative researchers: (<reflink idref="bib1" id="ref92">1</reflink>) the process of identifying articles that are relevant to a concept is time-intensive and unfolds over multiple stages of a long-term research project; (<reflink idref="bib2" id="ref93">2</reflink>) keyword searches are insufficient for selecting relevant articles; and (<reflink idref="bib3" id="ref94">3</reflink>) relevant articles discuss a concept either implicitly or explicitly.</p> <p>First, the challenges encountered in "simply" identifying, or classifying, articles in which the target subject matter appeared (about a quarter of the sample in our data) meant that we were unable in the original analysis to hand code other aspects of the articles, such as content related to the causes and consequences of rising economic inequality or to intersections with race, ethnic, and gender inequality. Yet these aspects of an article can critically impact how readers might potentially understand and respond to the issue of socio-economic inequality. In subsequent work, then, we can experiment with automated methods not only to corroborate the original hand-coding results but to extend the substantive research and coding process into new, adjacent territory, and into new corpora as well. In this respect, our extension of this work using LLMs is an example of the "agnostic" approach to textual analysis advocated by [<reflink idref="bib23" id="ref95">23</reflink>]: 26–28), in which "building, refining, and testing social science theories requires iteration and cumulation" across multiple methods and targets of analysis.</p> <p>Second, and related, the data and classification task adopted in this article provides an illustration of the steps needed to build a corpus appropriate for deeper content analysis ([<reflink idref="bib48" id="ref96">48</reflink>]). These steps are necessary for researchers who do not have access to a pre-defined corpus (i.e., a corpus where all documents are by definition relevant, such as an oral history archive or an organization's newsletters). For instance, in the first step of the original study, a comprehensive keyword-based search on topics potentially related to the rise in economic inequality in the United States was conducted. A random 10–15 percent sample of articles obtained from this search was then selected in each year from 1980 to 2012 to reduce the scale of hand coding. However, in the final step involving hand coders, only about a quarter of the sampled articles ended up in the crucial sub-corpus of articles hand coded as mentioning economic inequality. Because the original research sought to determine <emph>whether</emph> and <emph>when</emph> the media was covering inequality, this labor-intensive step was not solely in service of a third, more in-depth step focused on inductively exploring <emph>how</emph> the media was doing so in the relevant sub-corpus. Nonetheless, researchers without such an incentive may still want to be more discerning in the selection of their corpus—in other words, in the initial classification steps—than keyword searches allow, and we suggest an expedited way of doing so with generative LLMs.</p> <p>Third, the reason that only a minority of articles was originally classified as being about economic inequality was because we had developed—through first deductive and then subsequent inductive iterations through the data—a strict but elaborate <emph>definition</emph> of both the concept of economic inequality and its empirical manifestation in the current era of rising income, wealth, and earnings inequality. This definition was spelled out in a 14-page, single-spaced, hand-coding guide, as well as a set of appendices. (A one-page outline of the definition of inequality from the original study is provided in Appendix A in the online supplement.) Namely, the guide sought to isolate all articles that <emph>potentially</emph> conveyed the idea, either explicitly or implicitly, of inequalities stemming from a wide range of phenomenon. Executive compensation and other firm-level wage-setting policies, minimum wage regulations, attacks on collective bargaining, social insurance and safety net spending, and changes to the tax code, all could result in discussions of income inequality without the term inequality itself ever being used. Conversely, economic phenomenon that was <emph>not</emph> about <emph>income/wealth/earnings</emph> inequality, such as trends in unemployment, needed to be defined as negative cases in the guide as well.</p> <p>We provide an example of the challenges of this coding task using an article selected from the corpus and reproduced in Figure 2. This article was hand coded as "implicitly" mentioning inequality, which is a coding category described in Appendix A, section 4a in the online supplement. According to the original codebook, the classification of this article requires contextual knowledge that two economic groups in hierarchical relation to one another are being mentioned: yachts are for the rich and welfare is for the poor without the article using the words "rich" or "poor" or "low" or "high" income. Moreover, the reader must be able to identify, via the gently snarky tone in the article, the implicit connection between yachts, welfare, and inequality through a holistic reading of the entire document, as these references do not occur together in a single sentence or paragraph and are never explicitly compared in the article (leading to the "implicit" rather than "explicit" designation by the original hand coders). In short, the very nature of our task—and, we contend, many others like it—means that text cannot be analyzed at the level of words, sentences, or paragraphs as is common in most other classification research in the social sciences.[<reflink idref="bib9" id="ref97">9</reflink>]</p> <p>Graph: Figure 2. Example news article discusses inequality without using the word inequality.</p> <p>Table 1. Categories and Binary Schemas From Hand-Coding Guide and Analysis.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Categories and Binary Schemas</th><th align="left">Number</th></tr></thead><tbody><tr><td>Categories (shorthand in hand-coding guide) a:</td><td /></tr><tr><td> Relevant Articles</td><td /></tr><tr><td> Inequalityb (relinequality)</td><td align="center">1</td></tr><tr><td> Economic Conditions (releconomy)</td><td align="center">2</td></tr><tr><td> Changes in Wages and Income (relchanges)</td><td align="center">3</td></tr><tr><td> Irrelevant Articles (irrelevant)</td><td align="center">4</td></tr><tr><td>Binary Classification Schemas:</td><td /></tr><tr><td> Relevant versus Irrelevant</td><td align="center">1/0</td></tr><tr><td> Inequality versus Other Relevant</td><td align="center">1/0</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Notes:</emph><sups>a</sups> In the hand-coding guide from the original study ([<reflink idref="bib39" id="ref98">39</reflink>]), these categories are given shorthands which are referenced in some of our prompts because the guide itself is evaluated by the LLMs. <sups>b</sups> This category is broken down into two categories (explicit and implicit inequality) in the original work and in the replication but is simplified in most of the present work to be a single category (except for tests in Appendix G in the online supplement, Table G1, Panel B, Rows 1 and 2).</p> <p>Drawing from the prior research but adapting it to the generative LLM context, our full classification task is presented in Table 1. In terms of the categories of analysis, articles that were hand coded as mentioning <emph>economic inequality</emph>, both explicitly and implicitly, are referred to as category 1. The remaining articles fell into three mutually exclusive categories: a category that mentions <emph>employment conditions</emph> without mention of wages or income (category 2); a category that mentions <emph>wages and income</emph> for single groups of workers or occupations without mention of differences among groups of higher (e.g., rich) and lower (e.g., poor) economic status (category 3); and a final category that was deemed <emph>irrelevant</emph> in the original project because the discussion focused on other countries or on gender and/or racial inequality without any mention of income or class inequality (category 4). The first three categories are also together denoted as <emph>relevant</emph> articles.</p> <p>In terms of classification schemas, our prior replication of the original study using non-LLM computational methods examined multiple classification schemas ([<reflink idref="bib43" id="ref99">43</reflink>]), whereas we have simplified the classification schemas for our analyses given the variations that we introduce instead around the definition of the targeted concept of inequality and with respect to different kinds of LLMs. In the prior replication, we examined schemas with two categories (e.g., inequality versus non-inequality articles, or category 1 versus categories 2–4) or three categories (inequality versus other economic versus irrelevant articles, or category 1 versus categories 2–3 versus category 4). We focus on two different binary schemas in this article because our prompting workflow evolved over time, described further below, into a two-step sequence of first removing irrelevant articles (category 4 versus categories 1–3) and then isolating articles on inequality (category 1) from articles on other economic topics (categories 2–3) among the relevant articles ([<reflink idref="bib21" id="ref100">21</reflink>]).</p> <p>In assessing the performance of LLMs in classifying articles into these schemas, we again draw from our prior replication for guidance. In that work, we examined whether three categories of computational methods—dictionaries, supervised machine learning, and unsupervised machine learning—could produce results comparable to hand coding, We found that supervised machine learning methods performed both well and the best of the three tested methods. Our objective in terms of assessing accuracy in the present work, then, is to determine whether LLMs can match or exceed the performance of supervised machine learning methods on the same corpus of hand-coded articles but using natural language as the input and output.[<reflink idref="bib10" id="ref101">10</reflink>] Our efforts focused on using the coding guide (created in the original project) to develop prompts for generative LLMs that will enable them to correctly classify the corpus of articles into the binary classification schemes described in the preceding paragraph. We tested three open-source models that have previously performed well: Llama3:70b, Gemma2:27b, and Llama3.1:70b, comparing the results from the open-source models to the industry standard but proprietary GPT4 ([<reflink idref="bib47" id="ref102">47</reflink>]).</p> <hd id="AN0186128775-6">Prompting Strategies</hd> <p>We have mentioned the term "iterative" several times in the course of describing the qualitative hand-coding process of alternating between deductively and inductively derived instructions for identifying target concepts in documents. This is represented more formally by the first box of our proposed workflow in Figure 1. We now highlight two broad approaches that researchers can take in adapting this process to generative LLMs, as represented in the second box of the figure.</p> <p>The first involves interacting in real time with generative LLMs in the chatbot or context window with queries in natural language, where the goal is to develop and refine a set of coding instructions for the LLM to successfully execute with a set of documents. This approach will work especially well when a coding guide does not already exist, and [<reflink idref="bib25" id="ref103">25</reflink>] provide an excellent description of this approach. The second approach involves converting an already existing coding guide, or a new but well-developed coding guide, into a prompting strategy that is run non-interactively on the full corpus (or smaller sub-samples in a preliminary testing phase) through an API (Application Programming Interface) that is called using simple programming code. In practice, researchers often use both approaches, depending on how much interactive testing, along the lines of the first approach, is needed when first embarking on a project. As described in this section, we found the need for the first approach to be limited and thus moved quickly to the second approach.</p> <p>We began with some initial discussion and experimentation because of the wide range of parameters we and others have considered in this endeavor ([<reflink idref="bib46" id="ref104">46</reflink>]; [<reflink idref="bib53" id="ref105">53</reflink>]). One set of elementary considerations concerned the use of specific terms of art to describe inequality—simply, whether LLMs first needed a tailored definition of the specific terms used in a more elaborate definition of the ways that inequality could be covered in the media. Initial inquiries using the interactive chatbot window, along the lines of the first approach described above, convinced us that systematic testing and defining of terms of art was unnecessary because they were already recognizable by the pre-trained LLMs. Our tests then followed the second, non-interactive approach of prompting generative LLMs through an API. We describe our final prompting strategy in terms of the <emph>structure, length,</emph> and <emph>sequence</emph> of the prompts.</p> <p>There are many ways to <emph>structure</emph> a prompt to provide background knowledge for a generative LLM to complete a task. We experimented with three main features. First, we provided references drawn from our data to enable the LLM to better understand our specific task: sample articles and their codes in a binary schema (see again Table 1) were inserted into the prompt as examples to further train the model, which is called "few-shot'" learning in LLM speak ([<reflink idref="bib10" id="ref106">10</reflink>]; [<reflink idref="bib11" id="ref107">11</reflink>]; [<reflink idref="bib50" id="ref108">50</reflink>]). When no sample input of articles and codes is inserted into the prompt, this is called "zero-shot" learning ([<reflink idref="bib32" id="ref109">32</reflink>]). Few-shot and zero-shot learning are different ways of structuring a prompt, which can have implications for the length of a prompt but need not do so. In our analyses of long documents, we found that the length of the sample input in few-shot learning most likely affected the performance of the task negatively (as discussed further below).</p> <p>Second, we manipulated whether the prompt includes a definition (e.g., "Inequality is defined as ...") in addition to a command (e.g., "If an article is about inequality, respond yes."). Both types of instruction are included in the prompts we discuss below, though we focus especially on the construction of definitions. As a baseline, we also experimented with prompts without definitions of inequality, allowing the LLMs to rely on their own training data to determine what it means to cover the topic in the media. Finally, all prompts begin with a directive about the type of persona that LLMs should emulate (e.g., "You are a helpful assistant.") and in some cases also include a more extended description of the task to be completed by LLMs. These are called system prompts and meta-statements, respectively. Because the natural language interface permits the structure of a prompt to vary in nearly infinite and not currently well-understood ways, we cannot say with certainty that the structure we adopted is the most optimal, only that it produced the most consistently accurate results among the variations we tested.[<reflink idref="bib11" id="ref110">11</reflink>]</p> <p>The <emph>length</emph> of a prompt is a major consideration in work like ours because of the complexity of the subject matter, the length of the articles (mean length is 1,089 words and 1,389 tokens), and the limits on the length of prompts imposed by various versions of LLMs (i.e., the maximum number of tokens, a sub-word unit, is 8,182 for Llama3, Gemma2, and GPT 4.0, and 128,000 for Llama3.1). Although we hypothesize that a major advantage of LLMs is their ability to recognize complex thematic content in long documents, it is still possible that longer prompts perform less well than shorter prompts even when they are within the token limit. In other words, longer token limits may enable the input of longer definitions and documents but there is no guarantee that accuracy will increase as a result (at least at the current stage of LLM development).</p> <p>Finally, the <emph>sequencing</emph> aspect of prompting strategies can operate in a variety of ways—sequencing of code within a single prompt, which we refer to above as part of the <emph>structure</emph> of the prompt, or sequencing of multiple prompts. The latter is the type of sequencing that we are referring to and that is connected to the format of our data. As described above and in Table 1, the data are partitioned into four mutually exclusive categories, with some groups likely sharing more porous boundaries than others, even though all articles contain economic content of some kind. A comprehensively inductive sequencing strategy might have been to test all logical sequence permutations. Instead, we qualitatively decided to test, as the first target category, what we hypothesized to be the most distinct of the categories, which is the irrelevant category (category 4 versus categories 1–3). This proved to be successful, and thus we moved on to testing a second step of the sequence, which was to isolate inequality articles among the remaining relevant articles (i.e., category 1 versus categories 2 and 3).</p> <p>These tests of different prompt <emph>structures</emph>, <emph>lengths</emph>, and <emph>sequences</emph> were at first researcher-created and largely deductive: we began with the detailed hand-coding guide and a summary of the guide that appeared in the original and replication studies (reproduced in Appendix A in the online supplement) to inform our own design of the structure, length, and sequence of the prompts (see also [<reflink idref="bib53" id="ref111">53</reflink>]). Subsequently, we developed a core prompt structure within which we experimented with an automated, inductive version of the concept definition performed nearly entirely by LLMs ([<reflink idref="bib56" id="ref112">56</reflink>]). Our final tests took a stepped approach, beginning with no definition, then using definitions produced by researchers themselves, and finally using LLM-generated definitions, the latter two mimicking the use of hand-coding guides traditionally used for complex coding tasks. Crucially, this approach not only encourages researchers to be deeply embedded in their coding guides and definition and prompt development, something we consider integral to qualitative research, it also produces multiple estimates of the classification task that later can be compared to one another in a reliability and validation step (see the third box in Figure 1), as we demonstrate in the next section.</p> <hd id="AN0186128775-7">Results</hd> <p>We first present the final, core prompt structure that is replicated across tests, and then follow with three subsections of results from the experiments with no definition (the baseline), a researcher-generated definition, and an LLM-generated definition, respectively. In the final two subsections, we discuss the possibility of qualitatively interpreting the natural language output from the LLMs and of utilizing the output across prompts and models to calculate interprompt and intermodel agreement metrics for reliability and validation purposes.</p> <hd id="AN0186128775-8">Core Prompt Structure</hd> <p>Table 2 provides a description of the core prompt structure that we implement. First, we used the system prompt "You are a news classifier," aligning with our task and priming the LLM with the word <emph>news,</emph> a genre of text. Second, we included a meta-statement: "We categorize articles that are related to issues of income inequality, changes in income or wealth, general economic conditions." This statement incorporated the word "categorize" followed by key terms from the qualitative codebook, guiding the output toward our conception of the relevant classification categories. Third, we added a definition of the target category, depending on which step in the sequence of tests we were at (i.e., the first step of identifying all relevant articles in Table 2, Panel A, or the second step of identifying inequality articles among all relevant articles in Table 2, Panel B), prefaced with "Read this definition: ". Fourth, we included the actual article to be classified, prefaced with "Read this article: ". Finally, we issued our command, such as "Does the article reference American economic inequality? Respond with 'Yes'...." We found that this prompting strategy resulted in more consistent output with clear binary indicators followed by a justification than other prompt structures we tested, which are described in Appendix C in the online supplement.</p> <p>Table 2. Core Prompt Structure for Zero-Shot Prompts.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /></colgroup><thead><tr><th align="left">Zero-shot Prompts</th></tr></thead><tbody><tr><td /></tr><tr><td>Panel A: Relevant vs. irrelevant articles (categories 1-3 vs. category 4; <italic>N</italic> = 1,253)</td></tr><tr><td>"role": "system",</td></tr><tr><td>"content": "You are a news classifier."</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "We categorize articles that are related to issues of income inequality, changes in income or wealth, general economic conditions."</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Read this definition: " + irrelevant_definitiona,</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Read this article: " + text,</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Is the article relevant? Answer relevant or irrelevant, and why in 1 sentence." Panel B: Inequality vs. non-inequality among relevant articles (category 1 vs. categories 2-3, <italic>N</italic> = 786)</td></tr><tr><td>"role": "system",</td></tr><tr><td>"content": "You are a news classifier."</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "We categorize articles that are related to issues of income inequality, changes in income or wealth, general economic conditions."</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Read this definition: " + inequality_definition,</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Read this article: " + text,</td></tr><tr><td>"role": "user",</td></tr><tr><td>"content": "Does the article reference American economic inequality? Respond with 'Yes' if article meets any or all criteria referencing American economic inequality and 'No' if article meets none of the criteria, and explain why in 1 sentence."b</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Notes:</emph><sups>a</sups> As discussed in the text, we provide a definition of irrelevant rather than relevant articles. <sups>b</sups> This command is provided when researcher- and LLM-generated definitions are included, otherwise the command is shorter when no definition is provided.</p> <p>Looking at the zero-shot examples of code in Table 2, we see the sequence of two tests represented in Panels A and B, respectively. The <emph>irrelevant_definition</emph> and the <emph>inequality_definition</emph> variables take on the values of either no definition, researcher-generated definition, or LLM-generated definition. The variable <emph>text</emph> contains the title of each article followed by the entire text of the article, truncated as necessary if the prompt length goes beyond the maximum context length of each respective LLM. For our few-shot tests, not shown in Table 2, we appended two randomly selected sample articles as input followed by the output for each after the final command prompt. Note that we only performed the few-shot learning tests for the second step in the sequence of prompts shown in Panel B, because this step represented a more challenging task, and thus our total number of tasks is three (i.e., Panel A and Panel B in Table 2, and a few-shot version of Panel B). For all tests, we set a seed and set the temperature parameter to 0 to minimize output variation and to enhance reliability and reproducibility,[<reflink idref="bib12" id="ref113">12</reflink>] and we set the context window to the maximum allowed for each model. Table 3 shows the weighted average recall, precision, and F1 scores across the three tasks in the rows, varying the LLM and the type of definition we provided in the columns.</p> <p>Table 3. Accuracy Metrics for Three Classification Tests (rows) by LLM and Definition Input (columns).</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left" /><th align="center" colspan="4">No Definition</th><th align="center" colspan="4">Researcher Definition</th><th align="center" colspan="4">LLM-Generated Definition</th></tr><tr><th align="center" /><th align="center">Llama3(70b)</th><th align="center">Llama3.1(70b)</th><th align="center">Gemma2(27b)</th><th align="center">GPT4 </th><th align="center">Llama3(70b)</th><th align="center">Llama3.1(70b)</th><th align="center">Gemma2(27b)</th><th align="center">GPT4</th><th align="center">Llama3(70b)</th><th align="center">Llama3.1(70b)</th><th align="center">Gemma2(27b)</th><th align="center">GPT4 </th></tr></thead><tbody><tr><td>Relevant:Zero Shot(<italic>n</italic> = 1253)</td><td>P: 0.74R: 0.73F1: 0.74</td><td>P: 0.80R: 0.81F1: 0.80</td><td>P: 0.78R: 0.79F1: 0.79</td><td>P: 0.82R: 0.84F1: 0.83</td><td>P: 0.76R: 0.74F1: 0.73</td><td>P: 0.79R: 0.78F1: 0.78</td><td>P: 0.82R: 0.83F1: 0.82</td><td>P: 0.83R: 0.83<bold>F1: 0.83</bold></td><td>P: 0.82R: 0.82F1: 0.82</td><td>P: 0.79R: 0.82F1: 0.80</td><td>P: 0.79R: 0.83F1: 0.80</td><td>P: 0.81R: 0.83F1: 0.81</td></tr><tr><td>Inequality:Zero Shot(<italic>n</italic> = 786)</td><td>P: 0.71R: 0.72F1: 0.70</td><td>P: 0.72R: 0.72F1: 0.72</td><td>P: 0.73R: 0.73F1: 0.73</td><td>P: 0.75R: 0.74F1: 0.74</td><td>P: 0.74R: 0.75F1: 0.73</td><td>P: 0.72R: 0.72F1: 0.71</td><td>P: 0.74R: 0.74F1: 0.74</td><td>P: 0.77R: 0.77<bold>F1: 0.77</bold></td><td>P: 0.76R: 0.77F1: 0.75</td><td>P: 0.73R: 0.73F1: 0.73</td><td>P: 0.75R: 0.75F1: 0.75</td><td>P: 0.74R: 0.74F1: 0.73</td></tr><tr><td>Inequality:Few Shot(<italic>n</italic> = 786)</td><td>P: 0.72R: 0.73F1: 0.73</td><td>P: 0.73R: 0.75F1: 0.70</td><td>P: 0.61R: 0.77F1: 0.60</td><td>P: 0.75R: 0.75F1: 0.75</td><td>P: 0.69R: 0.72F1: 0.69</td><td>P: 0.64R: 0.77F1: 0.62</td><td>P: 0.49R: 0.94F1: 0.59</td><td>P: 0.75R: 0.75<bold>F1: 0.75</bold></td><td align="center">–</td><td align="center">–</td><td align="center">–</td><td align="center">–</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note:</emph> Average weighted precision (P), recall (R), and F1 metrics across all tests. F1 is the harmonic mean of recall and precision. All tests use the core prompt structure presented in Table 2.</p> <hd id="AN0186128775-9">Baseline: No Definition</hd> <p>In our first baseline test (<emph>N</emph> = 1,253), providing no definition, using the entire corpus of articles (categories 1–4) and identifying relevant versus irrelevant articles (i.e., categories 1–3 versus category 4), we achieved weighted F1 scores of 0.74 (Llama3), 0.80 (Llama3.1), 0.79 (Gemma2), and 0.83 (GPT4) (see Table 3, row 1, columns 1–4). With careful prompt structures but no definition of the concept we were using to classify documents, and no examples to learn from, these accuracy scores are on par with acceptable human-coding intercoder reliability scores ([<reflink idref="bib45" id="ref114">45</reflink>]), and with the acceptable F1 scores for traditional supervised machine learning in the prior replication study.[<reflink idref="bib13" id="ref115">13</reflink>]</p> <p>In our second baseline test (<emph>N</emph> = 786), providing no definition of inequality, using only relevant articles (i.e., excluding articles hand-coded in category 4), and identifying whether an article mentions economic inequality (i.e., category 1 vs. categories 2–3), we achieved weighted F1 scores of 0.70 (Llama3), 0.72 (Llama3.1), 0.73 (Gemma2), and 0.74 (GPT4) (see Table 3, row 2, columns 1–4). While lower than the first baseline tests, which is perhaps expected given the more nuanced task, these weighted F1 scores are still within acceptable accuracy scores for most complex hand-coding tasks. For these two baseline tests, GPT4 achieved the highest F1 scores, but only by 0.03 and 0.01 points, respectively (on a 0.0 to 1.0 scale).</p> <p>For our third baseline test (<emph>N</emph> = 786), the accuracy metrics across the four models varied. Providing two example articles (one about inequality and one not, each randomly selected) improved Llama3's performance slightly over the second baseline test, with a weighted F1 of 0.73, and a comparable performance with GPT4, with a weighted F1 of 0.75. By contrast, the accuracy was worse with Llama3.1 and Gemma2, with weighted F1s of 0.70 and 0.60, respectively. We think this might be due to the inclusion of three full-length articles—two examples and one to be classified. The prompt context window may have become too long for these LLMs to properly parse. Few-shot learning thus might work better for short-text (e.g., sentence- or paragraph-length) classification tasks (e.g., [<reflink idref="bib11" id="ref116">11</reflink>]; [<reflink idref="bib15" id="ref117">15</reflink>]; [<reflink idref="bib50" id="ref118">50</reflink>]) than for ones such as ours. Because these initial few-shot tests were not encouraging, we did not engage in additional systematic tests (e.g., varying the types or numbers of articles selected), which should be the subject of further research.</p> <hd id="AN0186128775-10">Researcher-Generated Definition</hd> <p>Seeking to match and improve upon these results (and following suggestions by [<reflink idref="bib53" id="ref119">53</reflink>]), we generated definitions of irrelevant and inequality articles for the two steps in the sequence of tests, respectively. This portion of the prompt was issued in the third section of the core prompt in Table 2 (i.e., "Read this definition:" followed by our definition). Our two definitions are reported in Appendix D in the online supplement (in the sections under researcher-generated definitions). We found a concise, one-paragraph excerpt from the original coding guide of the "irrelevant" category and used that as our researcher-generated definition (i.e., the <emph>irrelevant_definition</emph> variable in Table 2) for that step. The analogous definition of inequality (i.e., the <emph>inequality_definition</emph> variable in Table 2) is based on the one-page summary definition of media coverage of inequality published in the prior work (reproduced in Appendix A in the online supplement), which itself is a summary of the detailed 14-page hand-coding guide. Still, it is a long definition if incorporated verbatim, and therefore we sought to substantially reduce it.</p> <p>Specifically, the original summary is comprised of four sections or criteria defining "types of inequality," "causes and policy solutions associated with inequality," "social class groups," and "relational or comparative language connecting social class groups." These four sections in turn had subsections that defined, for instance, the "types of inequality" and "social class groups" that were to be identified in media coverage of inequality. Our initial tests reported in Appendix G in the online supplement consisted of variations on these four section headings and subsection descriptions to help clarify the intended content of the four main sections. For instance, we needed to convey whether all four criteria needed to be present or some combination of them. In the end, we settled on a definition with three bullet points, any one of which was sufficient to identify an article on inequality. This decision rule is reflected in the command statement (see bottom of Table 2, Panel B).</p> <p>The results from these tests are reported in Table 3, columns 5–8. The F1 scores across the three tasks (in the rows) and the four models (in the columns) were similar to the F1 scores from the baseline test, sometimes slightly lower, sometimes slightly higher. In other words, providing a definition of inequality did not consistently improve performance. The test for the first step in the sequence (relevant or irrelevant) had weighted F1 scores of 0.73 (Llama3), 0.78 (Llama3.1), 0.82 (Gemma2), and 0.83 (GPT4), compared to 0.74, 0.80, 0.79, and 0.83 for the baseline tests without definitions, respectively. For the second step in the sequence (mentioning inequality or not among relevant articles), the weighted F1 scores were 0.73 (Llama3), 0.71 (Llama3.1), 0.74 (Gemma2), and 0.77 (GPT4), compared to 0.70, 0.72, 0.73, and 0.74 for the baseline tests without definitions, respectively. For the few-shot test, the weighted F1 scores were 0.69 (Llama3), 0.62 (Llama3.1), 0.59 (Gemma2), and 0.75 (GPT4), compared to 0.73, 0.70, 0.60, and 0.75 for the baseline tests without definitions, respectively. For these tests, GPT4 again performed the best, but not more than 0.03 points better than the best-performing open-source model, except, notably, in the few-shot tests where it was far superior.</p> <p>We contemplated additional tests that would modify and expand the definition in different ways or use a larger number of articles as examples in the few-shot learning approach, but most LLMs' token limit precluded us from greatly increasing the length of the definition and number of input articles. Simultaneously, we tested an alternative approach to providing a definition—allowing the LLM to generate its own definition from the codebook.</p> <hd id="AN0186128775-11">LLM-Generated Definition</hd> <p>One of the purported capabilities of LLMs is to accurately summarize documents into easily digestible bullet points ([<reflink idref="bib58" id="ref120">58</reflink>]). We thus tested the possibility that LLMs could summarize nuanced codebooks into definitions to guide each LLM in its classification task, which also further automates the LLM-assisted coding process. This was accomplished in two separate, pre-processing prompts followed by a hand-editing step to merge the output from the two separate prompts. In the first pre-processing prompt, we gave each LLM the full coding guide and a command to create a longer definition (than our researcher-generated definition) with 7–10 bullet points of the irrelevant or inequality categories (depending on which step in the sequence we were testing). We did the same in the second pre-processing prompt, except that a detailed appendix to the main coding guide was submitted instead of the main guide itself (prompts and results not shown). The results from the two steps were merged and duplicates were removed, all by hand. Note that the LLM-generated definitions differ across the models, as each model generated a different definition. These definitions are reported in Appendix D in the online supplement (in the sections under LLM-generated definitions).</p> <p>The results from using the LLM-generated definition are reported in Table 3, columns 9–12. The accuracy metrics are similar to or marginally higher compared to the previous tests. For the first step in the sequence (identifying relevant vs. irrelevant articles), the weighted F1s were 0.82 (Llama3), 0.80 (Llama3.1), 0.80 (Gemma2), and 0.81 (GPT4). For the more nuanced second step (identifying inequality vs. non-inequality articles), the LLM-generated definition performed marginally better across all models except GPT4, compared to both the baseline (no definition) and the researcher-generated definition test, with weighted F1s of 0.75 (Llama3), 0.73 (Llama3.1), 0.75 (Gemma2), and 0.73 (GPT4).</p> <hd id="AN0186128775-12">Natural Language Output</hd> <p>The input in the prompts for these models is in natural language, but here we discuss the fact that the output is also in natural language, which presents both drawbacks and affordances. One drawback is that the model may not always provide the desired binary classification (e.g., relevant/irrelevant or yes/no), necessitating an additional step where the researcher must interpret the output to fit the desired classification (which is discussed further in Appendix C in the online supplement). However, the natural language output can also benefit researchers, particularly those in qualitative fields. Recall that our instructions for the models' output included a request for a one-sentence summary. Researchers can use this summary to assess which words may influence the LLM's classification decisions. This serves as a validity check on the output itself and provides additional qualitative insights into the corpus, the codebook, and the chosen categories. For the tests using the core prompt structure in Table 2, the model did reliably produce a yes/no or relevant/irrelevant output and an additional justification sentence(s).</p> <p>For example, in the Llama3 model reported in Table 3 (row 1, column 5), the output reliably produced a single-word classification for every document, and the subsequent sentence provided a substantive justification, for example:</p> <p>Irrelevant, because while the article mentions immigration reform and its potential impact on the economy, it does not explicitly discuss income or wealth inequality in the United States.</p> <p>Relevant, because although the article primarily discusses affirmative action and racial inequality, it also touches on broader issues of economic inequality, such as how government policies can create unfair advantages and concentrate wealth among a few individuals.</p> <p>The sentence-length justification often referenced aspects of the definition provided. For example, the definition provided explicitly mentioned that articles about racial inequality should be considered irrelevant, except where forms of class inequality are also mentioned. The sentence in the second example above gave this precise rationale, stating that although the article was primarily about racial inequality, it was also about the concentration of wealth. The justifications also typically detail which groups are mentioned (e.g., immigrants), which could provide further insight into how inequality is discussed over time, though we did not do that here. As is the case with training research assistants, this informational justification provides insight into the entire classification endeavor.</p> <hd id="AN0186128775-13">Interprompt and Intermodel Agreement Tests</hd> <p>Finally, assessing the reliability and validity of the classification task is a crucial component of both qualitative and quantitative coding procedures. Fortunately, the multiple estimates generated from our tests of different prompts and models enable us to conduct such an analysis (as shown in the third box of Figure 1). Recall that the F1 scores presented in Table 3 varied across prompting strategies and models, ranging from 0.59 to 0.83, but this variation was not consistent across prompts and models. For example, sometimes Gemma2 performed the best of the open-source models, sometimes the worst. Similarly, the LLM-generated definition for the second (inequality) test produced the best F1 score for Gemma2 compared to other prompts, but the worst for GPT4. To our knowledge, there is currently no way to know a priori which models and prompting strategies might be more or less accurate. In our case, we know the variation in accuracy because we were working with already hand-coded documents. Yet one goal of introducing LLMs into the qualitative coding pipeline is to reduce the time spent hand-coding documents or at least to redirect that time in a more strategic and efficient manner.</p> <p>We propose leveraging the multi-model analysis, and resulting variation across models, for reliability and validity testing of classification tasks with no pre-determined ground truth (see, e.g., [<reflink idref="bib58" id="ref121">58</reflink>]), as well as for benchmarking (as was done in [<reflink idref="bib15" id="ref122">15</reflink>] and [<reflink idref="bib50" id="ref123">50</reflink>]). To do so, we used standard "interrater reliability" measures to calculate two forms of prompt and model agreement from our output: <emph>interprompt agreement</emph>, defined as the degree of agreement within one LLM across prompts, and <emph>intermodel agreement</emph>, defined as the degree of agreement within one prompt across different LLMs. Full results are presented in Appendix E in the online supplement, but, in summary, interprompt and intermodel Fleiss' kappa ([<reflink idref="bib18" id="ref124">18</reflink>]) ranged from 0.62 to 0.88. Like the F1 metrics, these metrics range from low to high.</p> <p>Scholars without pre-hand-coded documents could use these metrics in three ways. First, similar to the use of interrater reliability, the metric itself can be used to assess stability across prompts and models—a test of the clarity of the classification task itself. Second, scholars could use a "majority voting" strategy to produce the ultimate classifications. For the latter, we assigned the classification agreed on by two or more prompts or models and recalculated the F1 scores across our tests (see Appendix E in the online supplement). Using this majority voting method did not achieve significantly higher F1 scores (the highest F1 score in these tests was 0.84 versus 0.83 in the original tests in Table 3), but the floor was much higher (the lowest F1 score was 0.75 versus 0.59 in our original tests). In other words, the majority voting method can screen out low-accuracy outliers.</p> <p>Third, identifying documents with full agreement among all prompts or models can help researchers strategically decide where to invest time in qualitative coding. Even when LLMs and prompts show high agreement, researchers should remain cautious, as such consistency may reflect shared model biases or training data rather than accuracy vis-à-vis human evaluation. Validation involving human judgment remains essential. Researchers could use these agreement metrics to, for example, identify documents where all prompts/models agreed on the classification, validate a selection of those by hand, and then intensively analyze the (hopefully smaller) set of documents with disagreements, also by hand. In our case, the three prompts or three models agreed on one classification for 74 to 91 percent of the documents (see Appendix E in the online supplement). This approach will not necessarily reduce the total amount of time spent interpreting and hand-coding documents, but it could allocate that time in more efficient and strategic ways.</p> <hd id="AN0186128775-14">Discussion</hd> <p>We have examined whether LLMs can classify documents into given categories on par with either human coders or previous supervised machine learning methods, but our main contribution has been to focus on the process of qualitative coding itself via LLMs using data from an original qualitative hand-coding project. Interacting with generative LLMs using natural language is a marked shift from previous computational text analysis methods that required translating text and instructions into machine-readable code. Some code is also necessary for the present work (e.g., see Table 2), but the shift to generative LLMs means that qualitative researchers can, more or less, replicate traditional methods involving the construction of comprehensive codebooks and the iterative testing of the reliability of those codebooks (see Figure 1). We systematically varied LLM prompts derived from such a codebook and found that LLMs can achieve classifications comparable to hand coding and supervised machine learning. Despite many remaining challenges discussed below, this finding suggests not only that LLMs may eventually be able to serve as reliable coding assistants, but that they could also offer scalability (e.g., analyzing larger datasets), flexibility (e.g., easily modifying classification criteria and building on pre-trained models), and efficiency (e.g., dramatically reducing or strategically redirecting the need for hand coding).</p> <hd id="AN0186128775-15">Main Findings</hd> <p>Our tests revealed several key features of prompts and models that should be considered when adopting LLMs for qualitative coding projects. We summarize the lessons from each in turn (prompts and models), but we first reiterate the nature of our data and classification task, which shaped our approach to LLMs in crucial ways. Our units of analysis are long documents (U.S. newsweekly articles from 1980 to 2012), which are relatively understudied in the social science literature on classification analysis with LLMs despite the known strengths of LLMs in thematically and holistically summarizing large corpora ([<reflink idref="bib27" id="ref125">27</reflink>]; [<reflink idref="bib58" id="ref126">58</reflink>]). Within long documents, we sought to identify the multi-faceted concept of economic inequality, which potentially is elaborated piece by piece across multiple sentences and paragraphs (see Figure 2). The documents also contained adjacent economic content, making them "noisy" and challenging for humans to code. In all these ways, our findings should generalize to other qualitative coding projects concerned with concept identification in rich, long-form documents, such as oral history or interview transcripts, organizational documents, and legacy/historical media.</p> <p>Concerning the development of prompts for generative LLMs using these types of data and classification tasks, we highlight four findings. First, even without providing a definition of the concept to be labeled, the baseline test was performed on par with acceptable interrater reliability metrics for hand-coding and accuracy metrics for traditional supervised machine learning. For our use case, which involves classifying English-language newspapers—known to be part of the training data for many LLMs—this means we can effectively prompt LLMs using the same, though much reduced, language we would use to train research assistants and still obtain reliable and accurate classifications. This also implies that LLMs come pre-trained with relevant textual knowledge for our use case, and likely those of many other researchers in the social sciences (though this should be checked whenever possible if open-source models allow it).</p> <p>Second, and to our surprise, the addition of researcher- and LLM-generated definitions to the prompt did not consistently improve the accuracy of the classification tasks over the baseline prompt without a definition. The researcher-generated and LLM-generated definitions were generally comparable in accuracy to the baseline, though the longer, LLM-generated definitions did perform the best for some models and some tests. For those starting a qualitative project without a ground truth, in other words, we cannot provide direction on how detailed of a definition is needed to obtain the highest accuracy, which instead will necessitate a careful, iterative approach ([<reflink idref="bib53" id="ref127">53</reflink>]). Such an approach is best practice in any case, and it should include both hand coding for validation purposes and the calculation of intermodel and interprompt agreement scores to locate potentially easy-to-classify cases and those that are less so (see section "Interprompt and Intermodel Agreement Tests" and the third box of Figure 1).</p> <p>Third, for our use case, the zero-shot prompts performed better than the few-shot learning prompts (note that all LLM-created definitions were embedded in zero-shot prompts). This may result from the small number of articles included in the few-shot learning prompts, which itself is a consequence of the token limit for most of the LLMs we tested. Or it could be a result of the length of our articles in a different way: adding two example articles on top of the article to be classified introduced too much noise into the prompt structure even when the full prompt was within token limits.[<reflink idref="bib14" id="ref128">14</reflink>] Future work should more thoroughly test the impact of varying the number, type, and length of articles submitted in the few-shot learning prompts, particularly with models (such as Llama3.1) that have expanded token limits. At the same time, it may be possible to include all articles and their labels as input to LLMs, in addition to providing the detailed coding guides (as we did), in order to generate a definition (whose length and detail could also vary) from the LLM. This would mimic a supervised machine-learning environment in which a substantial volume of already labeled data is needed to inform/train the model ([<reflink idref="bib59" id="ref129">59</reflink>]).</p> <p>Fourth, and finally, our testing of different prompting strategies suggests both the importance of prompting strategies and the unnerving sensitivity of LLM output to variations in prompt structure. We found, like others (e.g., [<reflink idref="bib53" id="ref130">53</reflink>]), that structured prompts were advantageous. For instance, placing the command for the desired output at the end of the prompt produced more consistent classification output, perhaps due to the volume of textual input in our prompts (see Appendix C in the online supplement). Although non-trivial variation in LLM output across prompts poses a serious challenge when previously hand-coded output is unavailable, iterating the process by varying prompts (e.g., by modifying definitions) and models (e.g., by using multiple LLMs), and assessing agreement across the output classifications, is akin to iterating a codebook and calculating human-rater reliability scores while training research assistants. Using interprompt and intermodel majority voting strategies further mitigates the risk of low-performing outliers, though we emphasize that even when models or prompts produce the same classification, human validation is still needed. In short, the variation in LLM output could be interpreted as mimicking the hand-coding process, and researchers could use this variation for insight into their categories and definitions ([<reflink idref="bib25" id="ref131">25</reflink>]).</p> <p>Concerning our tests across different open- and closed-source models, and models of substantially different sizes, we first highlight two somewhat contradictory findings and then a third finding. First, although the full GPT4 model achieved marginally better accuracy metrics almost across the board, these metrics were not substantially better than the smaller, pre-quantized, open-source models, ranging from 0.01 to 0.03 points (on a 0.0 to 1.0 scale) over the best performing open-source LLM. These findings, combined with both the cost and the extensive methodological, ethical, and environmental problems associated with using foundational LLMs for academic research (discussed in Section "Variation among Generative LLMs"), confirm that academic researchers can and should use open-source LLMs for text-based classification tasks. At the same time, and secondly, GPT4 performed markedly better in the few-shot tests, suggesting that it can better manage larger streams of input data. Moreover, accuracy improvements of even a few points may not be inconsequential for some qualitative researchers and projects.</p> <p>Third, we also observed a potential for unpredictable differences among the models themselves. Llama3, for example, achieved relatively high weighted F1 scores across a wide range of tests, yet Llama3.1, an arguably superior model, performed worse in the first few-shot learning test. And Gemma2 was the highest performing open-source model in many of our zero-shot tests, but all but collapsed in the few-shot tests. Differences in training data and weights and the fine-tuning done by engineers can lead to unpredictable behavior when LLMs are applied to tasks that depart further from the standard benchmark tests ([<reflink idref="bib6" id="ref132">6</reflink>]). In the case of open-source models, researchers have access to the fine-tuning process, but it is not always clear how that fine-tuning impacts model behavior across different types of prompts and tasks, and we return to this point below.</p> <p>Together, our findings suggest that generative LLMs can be a productive and reliable tool for qualitative researchers, especially those without advanced computer programming skills, who want to systematically analyze a collection of texts, either inductively or iteratively, using standard qualitative methodologies. To this end, in Table 4, we highlight three general takeaways that may help researchers consider how to apply these models in their work, given the rapidly changing terrain of LLM development.</p> <p>Table 4. High-Level Summary of Findings.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Guideline</th><th align="left">Key Insight</th><th align="left">Why It Matters</th><th align="left">Practical Application</th></tr></thead><tbody><tr><td>Researcher-LLM-Researcher Workflow</td><td>Use LLMs as part of an iterative process, with researchers designing tasks, LLMs making an initial pass, and humans validating/refining results.</td><td>Ensures LLMs support, rather than replace, qualitative judgment, and redirects researcher efforts in strategic ways.</td><td>Use LLMs for first-pass coding and focus researcher effort on validating clear cases and analyzing ambiguous cases.</td></tr><tr><td>Leverage Interprompt and Intermodel Agreement</td><td>Compare multiple prompts and models to improve consistency and highlight ambiguities.</td><td>Reduces risks from biases and/or inaccuracies in any single LLM or prompt.</td><td>Identify stable classifications by comparing different prompt structures and models, targeting the human-validation step.</td></tr><tr><td>Prioritize Reproducibility and Transparency</td><td>Closed models may perform slightly better, but adopting open-source options offers transparency, reproducibility, and accessibility.</td><td>Promotes research integrity, long-term usability, and accessibility.</td><td>Favor open-source and smaller models, unless accuracy trade-offs are particularly critical.</td></tr></tbody></table> </ephtml> </p> <hd id="AN0186128775-16">Limitations</hd> <p>Our experiments presented both a difficult and a straightforward test case for LLMs, suggesting both benefits and limitations of our work. Our classification task was particularly difficult for generative LLMs because we were instructing LLMs to predict whether there was <emph>any</emph> discussion of inequality in a long document often covering multiple themes, which introduces noise into the classification task. This feature of our work also led to mechanical limitations, as LLMs have strict maximum context windows and will truncate text over those limits. There are also, apparently, computational limitations stemming from our classification task, with accuracy diminishing as the prompts became longer in some Llama3.1 and Gemma2 tests (even though they were within the token limit). Yet, precisely because our classification task was noisy, the acceptable levels of accuracy presented in Table 3 suggest that LLMs can be incorporated into these types of classification tasks.</p> <p>On the other hand, our test case was straightforward for the LLMs because it involved English-language newspapers, which are similar to a large portion of the training data used for many LLMs. In fact, the articles we are classifying may be in the actual training data for the LLMs we tested (though, of course, our classifications of those articles are not). We did not investigate whether our corpus was in fact part of the training data of the models we used, but it should be possible to do so when using open-source models, which is another advantage of such models. The hypothesized alignment between our data and the LLMs' training data may have been one reason we achieved a high F1 score in the baseline test without a definition of our classification concept. It is almost certain that classification tasks based on data or concepts further from the type of training data used in these LLMs would lead to worse, and perhaps unacceptably low, accuracy metrics. The scientific community is increasingly seeking to address this limitation. Qualitative researchers will, of course, need to weigh this limitation against their particular data and classification task, and likely use detailed definitions, fine-tuning, or specialized models for data that is further afield from the standard training data in existing LLMs. Future research may also examine how LLMs can be used for different kinds of interpretive text analysis tasks, such as whether an LLM adopting varying political personas would label vignettes about inequality as being normatively good or bad (e.g., see related work by [<reflink idref="bib30" id="ref133">30</reflink>]; [<reflink idref="bib33" id="ref134">33</reflink>]).</p> <p>Our final limitation is that we did not engage in extensive fine-tuning tests with all the models employed in our main zero-shot and few-shot tests. Fine-tuning can potentially enhance the performance of LLMs by adapting them more specifically to the nuances of the task and data at hand ([<reflink idref="bib2" id="ref135">2</reflink>]). For instance, the accuracy levels in our tests did not exceed those of supervised machine learning (e.g., [<reflink idref="bib43" id="ref136">43</reflink>]), which suggests that supervised machine learning methods, including especially more recent transformer methods, may be a better option if the goal is to strictly apply a pre-determined classification task with the highest accuracy possible ([<reflink idref="bib8" id="ref137">8</reflink>]). We therefore experimented with fine-tuning one of the open-source LLMs, Gemma2, as discussed further in Appendix F in the online supplement. We found that using up to 1000 documents for fine-tuning did not result in higher accuracy rates compared to our main tests in Table 3, and that fine-tuning with fewer documents led to substantially worse accuracy, with accuracy peaking at around 600 documents. In a separate fine-tuning analysis using BERT models, we came to the same conclusion. These results confirm that fine-tuning is not only computationally expensive (see Appendix F in the online supplement) but also can be unpredictable; indeed, it is not yet clear which type of fine-tuning is best for the types of tasks and data we are examining. Future research should continue to assess when, and how much, fine-tuning might be needed for different kinds of models (see, e.g., [<reflink idref="bib11" id="ref138">11</reflink>] for proposed guidelines).</p> <hd id="AN0186128775-17">Conclusion</hd> <p>Despite the fact that the field of generative LLMs and their use in sociology is still in its early moments and presents many challenges, our findings suggest that these models can open new frontiers in qualitative research methodology. They provide a productive blend of automation and nuanced understanding, at least for contemporary English-language text, enabling researchers to analyze vast amounts of text with scalability, flexibility, and efficiency in an interactive way. These advances encourage us both to rethink traditional qualitative inductive and/or iterative coding practices and to integrate them into the workflow of classification using LLMs. As the research community continues to refine and improve LLM capabilities, we are poised to enter an era of qualitative analysis that embraces both the depth of human insight and the breadth of machine learning, without having to leave the comfortable world of natural language behind.</p> <hd id="AN0186128775-18">Supplemental Material</hd> <p>Graph: Supplemental material, sj-docx-1-smr-10.1177_00491241251339188 for Updating "The Future of Coding": Qualitative Coding with Generative Large Language Models by Nga Than, Leanne Fan, Tina Law, Laura K. Nelson and Leslie McCall in Sociological Methods & Research</p> <hd id="AN0186128775-19">Acknowledgments</hd> <p>We are grateful for funding to support this research from the Russell Sage Foundation Visiting Scholars program and grant #83-13-05 and from the PSC-CUNY Research Award #64715-52. We are also grateful for feedback on an earlier version of this paper from participants at the Generative Artificial Intelligence and Sociology workshop at Yale University. Finally, we thank the special issue editors and reviewers for their careful engagement with our work. Their feedback helped clarify and refine our framework and argument. At times, their input felt closer to collaboration than review, in a very productive way.</p> <ref id="AN0186128775-20"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref92" type="bt">1</bibl> <bibtext> The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref17" type="bt">2</bibl> <bibtext> The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the PSC-CUNY Research Award, Russell Sage Foundation (grant number 64715-52, 83-13-05).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref8" type="bt">3</bibl> <bibtext> Nga Than https://orcid.org/0000-0002-6845-6253 Tina Law https://orcid.org/0000-0001-7631-6763 Laura K. Nelson https://orcid.org/0000-0001-8948-300X Leslie McCall https://orcid.org/0000-0002-7700-3969</bibtext> </blist> <blist> <bibl id="bib4" idref="ref81" type="bt">4</bibl> <bibtext> The full text of the news articles is under copyright. The news article metadata (identifying the original news articles), all generated output, and all replication and reproduction materials are available at: https://github.com/lknelson/future-of-coding-revisited#</bibtext> </blist> <blist> <bibl id="bib5" idref="ref4" type="bt">5</bibl> <bibtext> Supplemental material for this article is available https://journals.sagepub.com/doi/suppl/10.1177/00491241251339188.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref27" type="bt">6</bibl> <bibtext> Some examples include tweets and Facebook posts ([11]), summaries of documents or short survey responses ([50]), or sentence-level chunks of longer documents ([15]).</bibtext> </blist> <blist> <bibl id="bib7" idref="ref9" type="bt">7</bibl> <bibtext> Ollama relies on llama.cpp to implement LLMs. Depending on the way the user imports models, Ollama uses quantization to make the models efficient enough to operate without specialized hardware. There has not been enough testing yet to determine whether quantization impacts the performance of the model. In the meantime, anyone can implement these models on more-or-less standard hardware, which helps to make the process more reproducible compared to a situation in which researchers are downloading and implementing models themselves. With an eye toward maximum reproducibility, we thus opted for using the Ollama platform, despite the use of quantization that may impact model performance.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref28" type="bt">8</bibl> <bibtext> We note here that GPT4 is also costly by social science standards, at least at this point in time, even though it is heavily subsidized by OpenAI. Each test we ran with GPT4 cost between $50 and $70 depending on sample sizes. Fine-tuning and other alterations would cost even more (which is why we perform them using open-source models, as discussed below). Open-source models are free to use, if you have access to the relevant hardware and electricity. Cost disparities between GPT and open-source models may change in the future, however.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref56" type="bt">9</bibl> <bibtext> At an earlier stage of experimentation when we were hindered by token limits, we attempted to conduct paragraph-level analyses. However, we found them to be <emph>a priori</emph> infeasible if we were to maintain consistency with the original qualitative coding task, which assessed content holistically over the full span of an article.</bibtext> </blist> <blist> <bibtext> We use F1 scores to measure accuracy using the following terms and equations: the weighted_average_precision = average of precision scores multiplied by the proportion of total rows that are true positives for each category; the weighted_average_recall = average of recall scores multiplied by the proportion of total rows that are true positives for each category; weighted average total F1 = (2 * weighted_average_precision * weighted_average_recall)/(weighted_average_precision + weighted_average_recall). In the prior replication, weighted F1 = 0.74–0.86 for two separate binary schemas across 25 randomized training/test sets ([43], Table 1, also reproduced in Appendix B in the online supplement). Not all tests in this paper are strictly comparable to the tests in [43]. The most comparable F1 scores from that article to the F1 scores we report in this paper are in their Table 1, Column 9 (reproduced in Appendix B) for the first two schemas (relevant vs. irrelevant, median F1=0.83; inequality vs. not inequality, median F1=0.78) in the first row for SML. These are scores obtained for the full corpus, not only for the subset of relevant articles, which is the second step in the sequence of tests followed in this paper.</bibtext> </blist> <blist> <bibtext> New applications to reduce LLMs' sensitivity to natural language prompts are being developed, such as DSPy, but these applications tend to substitute programming for natural language interfaces, which limits their appeal for qualitative researchers ([29]).</bibtext> </blist> <blist> <bibtext> The temperature parameter is designed for users to control the randomness of text output from LLMs, ranging from 0 (more deterministic) to 1 (less deterministic). However, it is currently unclear whether setting the temperature parameter to 0 results in completely deterministic output, as that depends on the specific engineering choices of each model, which are not always clearly documented.</bibtext> </blist> <blist> <bibtext> Some might argue that our meta statement includes our definition of inequality. But note that our tests using Llama3 with no definition and without a meta statement resulted in similar weighted F1 scores, with the highest score being 0.76 (see Appendix C, Table C1, columns 1–3).</bibtext> </blist> <blist> <bibtext> A potential additional reason could be that our tests could not take advantage of memory outside of the instruction window, which may become available in future versions of the models.</bibtext> </blist> <blist> <bibtext> [15]</bibtext> </blist> </ref> <ref id="AN0186128775-21"> <title> References </title> <blist> <bibtext> "A Modest Proposal: Public Policies that Perform." U.S. News & World Report, August 10, 1987.</bibtext> </blist> <blist> <bibtext> Alizadeh Meysam, Kubli Maël, Samei Zeynab, Dehghani Shirin, Zahedivafa Mohammadmasiha, Bermeo Juan Diego, Korobeynikova Maria, Gilardi Fabrizio. 2024. " Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning. " arXiv. <ulink href="http://arxiv.org/abs/2307.02179">http://arxiv.org/abs/2307.02179</ulink>.</bibtext> </blist> <blist> <bibtext> Bail Christopher A.2024. " Can Generative AI Improve Social Science? " Proceedings of the National Academy of Sciences. 121(21):e2314021121.</bibtext> </blist> <blist> <bibtext> Bender Emily M., Gebru Timnit, McMillan-Major Angelina, Shmitchell Shmargaret. 2021. "On the Dangers of Stochastic Parrots: can Language Models be too big?" In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623.</bibtext> </blist> <blist> <bibtext> Bengio Yoshua, Ducharme Réjean, Vincent Pascal, Janvin Christian. 2003. " A Neural Probabilistic Language Model." The Journal of Machine Learning Research. 3:1137-55. via ACM Digital Library.</bibtext> </blist> <blist> <bibtext> Boelaert Julien, Coavoux Samuel, Ollion Etienne, Petev Ivaylo D., Präg Patrick. 2025. " Machine Bias. Generative Large Language Models Have a View of Their Own. " Sociological Methods & Research. doi: https://doi.org/10.1177/00491241251330582</bibtext> </blist> <blist> <bibtext> Bommasani Rishi, Hudson Drew A., Adeli Ehsan, Altman Russ, Arora Simran, von Arx Sydney, Bernstein Michael S., et al.2021. " On the Opportunities and Risks of Foundation Models. " arXiv. doi: https://doi.org/10.48550/ARXIV.2108.07258</bibtext> </blist> <blist> <bibtext> Bonikowski Bart, Luo Yuchen, Stuhler Oscar. 2022. " Politics as Usual? Measuring Populism, Nationalism, and Authoritarianism in U.S. Presidential Campaigns (1952–2020) with Neural Language Models. " Sociological Methods & Research. 51(4):1721-87.</bibtext> </blist> <blist> <bibtext> Brandt Philipp, Timmermans Stefan. 2021. " Abductive Logic of Inquiry for Quantitative Research in the Digital age. " Sociological Science. 8:191-210.</bibtext> </blist> <blist> <bibtext> Brown T., Mann B., Ryder N., Subbiah M., Kaplan J. D., Dhariwal P., Amodei D. 2020. " Language Models are few-Shot Learners. Advances in Neural Information Processing Systems." Conference on Neural Information Processing Systems. 33:1877-901.</bibtext> </blist> <blist> <bibtext> Chae Youngjn, Davidson Thomas. 2025. " Large Language models for Text Classification: From Zero-Shot Learning to Instruction-Tuning. " Sociological Methods & Research. doi: https://doi.org/10.1177/00491241251325</bibtext> </blist> <blist> <bibtext> Chong Dennis, Druckman James N. 2011. "Identifying Frames in Political News." Pp. 238-267 in Sourcebook for Political Communication Research: Methods, Measures, and Analytical Techniques, edited by E. P. Bucy and R. L. Holbert. New York: Routledge.</bibtext> </blist> <blist> <bibtext> Davidson Thomas.2024. " Start Generating: harnessing Generative Artificial Intelligence for Sociological Research. " Socius: Sociological Research for a Dynamic World. 10(January):23780231241259651.</bibtext> </blist> <blist> <bibtext> Deterding Nicole M., Waters Mary C. 2021. " Flexible Coding of In-Depth Interviews: a Twenty-First-Century Approach. " Sociological Methods & Research. 50(2):708-39.</bibtext> </blist> <blist> <bibtext> Do S., Ollion É., Shen R. 2022. " The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy. " Sociological Methods & Research. 53(3):1167-200.</bibtext> </blist> <blist> <bibtext> Dubey Abhimanyu, Jauhri Abhinav, Pandey Abhinav, Kadian Abhishek, Al-Dahle Ahmad, Letman Aiesha, Mathur Akhil, et al.2024. " The Llama 3 Herd of Models. " arXiv. <ulink href="http://arxiv.org/abs/2407.21783">http://arxiv.org/abs/2407.21783</ulink>.</bibtext> </blist> <blist> <bibtext> Ferree Myra Marx, Gamson William Anthony, Gerhards Jurgen, Rucht Dieter. 2002. Shaping Abortion Discourse: Democracy and the Public Sphere in Germany and the United States. NY: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Fleiss J. L.1971. " Measuring Nominal Scale Agreement among Many Raters. " Psychological Bulletin. 76(5):378-82.</bibtext> </blist> <blist> <bibtext> Foster Jacob G., Evans James A. 2024. " Algorithmic Abduction: robots for Alien Reading. " Critical Inquiry. 50(3):375-401. doi: https://doi.org/10.1086/728933</bibtext> </blist> <blist> <bibtext> Gamson William.1992. Talking Politics. New York: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Gilardi Fabrizio, Alizadeh Meysam, Kubli Mael. 2023. " ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks. " PNAS. 120(30):e2305016120.</bibtext> </blist> <blist> <bibtext> Gilens Martin.1999. Why Americans Hate Welfare: Race, Media, and the Politics of Antipoverty Policy. Chicago: University of Chicago Press.</bibtext> </blist> <blist> <bibtext> Grimmer Justin, Roberts Margaret E., Stewart Brandon M. 2022. Text as Data: A new Framework for Machine Learning and the Social Sciences. Princeton: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Griswold Wendy.1987. " A Methodological Framework for the Sociology of Culture. " Sociological Methodology. 17:1-35. doi: https://doi.org/10.2307/271027</bibtext> </blist> <blist> <bibtext> Ibrahim Elida I., Voyer Andrea. 2024. " The Augmented Qualitative Researcher: Using Generative AI in Qualitative Text Analysis. " SocArXiv. January 24. doi: https://doi.org/10.31235/osf.io/gkc8w</bibtext> </blist> <blist> <bibtext> Jensen Jeffrey L., Karell Daniel, Tanigawa-Lau Cole, Habash Nizar, Oudah Mai, Fani Dhia Fairus Shofia. 2022. " Language Models in Sociological Research: an Application to Classifying Large Administrative Data and Measuring Religiosity. " Sociological Methodology. 52(1):30-52.</bibtext> </blist> <blist> <bibtext> Karell Daniel, Shu Matthew, Okura Keitaro, Davidson Thomas. 2024. Artificial Intelligence Summaries of Historical Events Improve Knowledge Compared to Human-Written Summaries. SocArXiv. September 12. doi:10.31235/osf.io/3gsqw</bibtext> </blist> <blist> <bibtext> Kesari Aniket, Kim Jae Yeon, Shah Sono, Brown Taylor, Ventura Tiago, Law Tina. 2023. Training Computational Social Science Ph.D. Students for Academic and Non-Academic Careers.PS: Political Science & Politics.</bibtext> </blist> <blist> <bibtext> Khattab Omar, Singhvi Arnav, Maheshwari Paridhi, Zhang Zhiyuan, Santhanam Keshav, Vardhamanan Sri, Haq Saiful, Sharma Ashutosh, Joshi Thomas T., Moazam Hanna, Miller Heather, Zaharia Matei, Potts Christopher. 2023. " DSPy: Compiling declarative language model calls into self-improving pipelines. " Preprint. https://arxiv.org/abs/2310.03714.</bibtext> </blist> <blist> <bibtext> Kim Junsol, Lee Byungkyu. 2023. " Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys. " arXiv preprint arXiv:2305.09620.</bibtext> </blist> <blist> <bibtext> Knight Carly.2022. " When Corporations are People: agent Talk and the Development of Organizational Actorhood, 1890–1934. " Sociological Methods and Research. 51(4):1634-80.</bibtext> </blist> <blist> <bibtext> Kojima T., Gu S. S., Reid M., Matsuo Y., Iwasawa Y. 2022. " Large Language Models are Zero-Shot Reasoners. " Pp. 22199–213 in Proceedings of the 36th International Neural Information Processing Systems.</bibtext> </blist> <blist> <bibtext> Kozlowski Austin C., Kwon Hyunku, Evans James A. 2024. " In silico sociology: forecasting COVID-19 polarization with large language models. " arXiv preprint arXiv:2407.11190.</bibtext> </blist> <blist> <bibtext> Krippendorff Klaus H.2012. Content Analysis: An Introduction to Its Methodology. Los Angeles; London: Sage Publications, Inc.</bibtext> </blist> <blist> <bibtext> Law Tina.2022. Parsing the Language of Rebellion: Impacts of the 1960s Black-Led Urban Uprisings on American Political and Legal Discourse. Doctoral dissertation, Northwestern University.</bibtext> </blist> <blist> <bibtext> Litterer Benjamin, Jurgens David, Card Dallas. 2024. " Mapping the podcast ecosystem with the structured podcast research corpus. " https://arxiv.org/pdf/2411.07892.</bibtext> </blist> <blist> <bibtext> Madsen Anders Koed, Munk Anders Kristian, Soltoft Johan Irving. 2023. " Friction by Machine: How to slow down reasoning with computational methods. " Ethnographic Praxis in Industry Conference Proceedings. https://<ulink href="http://www.epicpeople.org/friction-by-machine-and-computational-methods/">www.epicpeople.org/friction-by-machine-and-computational-methods/</ulink>.</bibtext> </blist> <blist> <bibtext> Manning Christoper, Schutze Henrich. 1999. Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press.</bibtext> </blist> <blist> <bibtext> McCall Leslie.2013. The Undeserving Rich: American Beliefs About Inequality, Opportunity, and Redistribution. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Mesnard Thomas, Hardin Cassidy, Dadashi Robert, Bhupatiraju Surya, Pathak Shreya, Sifre Laurent, Rivière Morgane, et al.2024. " Gemma: Open Models Based on Gemini Research and Technology. " arXiv. <ulink href="http://arxiv.org/abs/2403.08295">http://arxiv.org/abs/2403.08295</ulink>.</bibtext> </blist> <blist> <bibtext> Nelson Laura K.2020. " Computational Grounded Theory: a Methodological Framework. " Sociological Methods & Research. 49(1):3-42.</bibtext> </blist> <blist> <bibtext> Nelson Laura K.2021. " Cycles of Conflict, a Century of Continuity: the Impact of Persistent Place-Based Political Logics on Social Movement Strategy. " American Journal of Sociology. 127(1):1-59.</bibtext> </blist> <blist> <bibtext> Nelson Laura K., Burk Derek, Knudsen Marcel, McCall Leslie. 2018. " The Future of Coding: a Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods. " Sociological Methods & Research. 50(1):202-37. doi: https://doi.org/10.1177/0049124118769114</bibtext> </blist> <blist> <bibtext> Nelson Laura K., King Brayden G. 2020. " The Meaning of Action: linking Goal Orientations, Tactics, and Strategies in the Environmental Movement. " Mobilization: An International Quarterly. 25(3):315-38.</bibtext> </blist> <blist> <bibtext> O'Connor Cliodhna, Joffe Helene. 2020. " Intercoder Reliability in Qualitative Research: debates and Practical Guidelines. " International Journal of Qualitative Methods. 19. doi:10.1177/1609406919899220</bibtext> </blist> <blist> <bibtext> Ollion Étienne, Shen Rubing, Macanovic Ana, Chatelain Arnault. 2024. " The Dangers of Using Proprietary LLMs for Research. " Nature Machine Intelligence. 6(1):4-5.</bibtext> </blist> <blist> <bibtext> OpenAI. 2023. " GPT-4 Technical Reports. " Preprint. https://cdn.openai.com/papers/gpt-4.pdf.</bibtext> </blist> <blist> <bibtext> Pardo-Guerra Juan Pablo, Pahwa Prithviraj. 2022. " The Extended Computational Case Method: a Framework for Research Design. " Sociological Methods & Research. 51(4):1826-67.</bibtext> </blist> <blist> <bibtext> Reiss Michael.2023. " Testing the reliability of ChatGPT for annotation and classification: A cautionary remark. " Preprint. https://arxiv.org/abs/2304.11085.</bibtext> </blist> <blist> <bibtext> Rytting Christopher Michael, Sorensen Taylor, Argyle Lisa, Busby Ethan, Fulda Nancy, Gubler Joshua, Wingate David. " Towards coding social science datasets with language models. " arXiv preprint arXiv:2306.02177 (2023).</bibtext> </blist> <blist> <bibtext> Spirling Arthur.2023. " Why Open-Source Generative AI Models Are an Ethical Way Forward for Science. " Nature. 616(7957):413-413.</bibtext> </blist> <blist> <bibtext> Stoltz Dustin S., Taylor Marshall A. 2019. " Concept Mover's Distance: Measuring Concept Engagement via Word Embeddings in Texts. " Journal of Computational Social Science. 2:293-313.</bibtext> </blist> <blist> <bibtext> Törnberg Petter.2024. " Best Practices for Text Annotation with Large Language Models. " Sociologica 18(2): 67-85. doi:10.6092/issn.1971-8853/19461</bibtext> </blist> <blist> <bibtext> Voyer Andrea, Kline Zachary D., Danton Madison, Volkova Tatiana. 2022. " From Strange to Normal: computational Approaches to Examining Immigrant Incorporation Through Shifts in the Mainstream. " Sociological Methods and Research. 51(4):1540-79.</bibtext> </blist> <blist> <bibtext> Wagner-Pacifici Robin, Mohr John, Breiger Ronald. 2015. " Ontologies, Methodologies, and new Uses of Big Data in the Social and Cultural Sciences." Big Data & Society. July-Sept:1-11.</bibtext> </blist> <blist> <bibtext> Wu Yu, Prabhumoye S., Min S. Y., Bisk Y., Salakhutdinov R., Azaria A., Mitchell T., Li Y. 2023. " SPRING: Studying the paper and reasoning to play games. " Preprint. https://arxiv.org/pdf/2305.15486.pdf.</bibtext> </blist> <blist> <bibtext> Yang Chengrun, Wang Xuezhi, Lu Yifeng, Liu Hanxiao, Le Quoc V., Zhou Denny, Chen Xinyun. 2023. " Large Language Models as Optimizers. " Preprint. https://arxiv.org/abs/2309.03409.</bibtext> </blist> <blist> <bibtext> Zhang T., Ladhak F., Durmus E., Liang P., McKeown K., Hashimoto T. B. 2024. " Benchmarking Large Language Models for News Summarization." Transactions of the Association for Computational Linguistics. 12:39-57.</bibtext> </blist> <blist> <bibtext> Ziems C., Held W., Shaikh O., Chen J., Zhang Z., Yang D. 2024. " Can Large Language Models Transform Computational Social Science? " Computational Linguistics 50(1): 237-91.</bibtext> </blist> </ref> <aug> <p>By Nga Than; Leanne Fan; Tina Law; Laura K. Nelson and Leslie McCall</p> <p>Reported by Author; Author; Author; Author; Author</p> <p></p> <p>Nga Than is a Research Associate at Stone Center on Socio-Economic Inequality, The Graduate Center, CUNY. Her research has utilized computational social science methods to study public opinion on immigration. She received her Ph.D. in sociology from the Graduate Center, City University of New York. She has published in Ethnic and Racial Studie s, Frontiers in Sociology, Journal of Community & Applied Social Psychology, and Communication Research and Practice.</p> <p>Leanne Fan is a Sociology student at The Graduate Center, City University of New York. Her research interests focus on the conditions that lead to the development of multiracial working-class political coalitions, exploring the intersections of race, class, and social movements through a variety of methodological approaches.</p> <p>Tina Law is an Assistant Professor of Sociology at the University of California, Davis. She studies inequality, race and ethnicity, and democracy and specializes in computational and quantitative methods. She uses computational and quantitative methods to understand the social and political experiences of racially minoritized and low-income residents living in U.S. cities and explores ways to adapt computational methods for sociological research.</p> <p>Laura K. Nelson is an associate professor of sociology at the University of British Columbia, where she also directs the Centre for Computational Social Science. She uses computational methods to study social movements, gender, culture, and institutions, and to advance qualitative computational text analysis methods. She has published in outlets such as American Journal of Sociology, American Sociological Review, and Gender and Society, among others.</p> <p>Leslie McCall is Presidential Professor of Sociology and Political Science and Associate Director of the Stone Center on Socio-Economic Inequality at The Graduate Center, City University of New York. Using a variety of methodological approaches, she studies public opinion and media coverage about inequality, opportunity, and related economic and policy issues; trends in actual earnings and family income inequality; and patterns of intersectional inequality.</p> </aug> <nolink nlid="nl1" bibid="bib23" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib19" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib28" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib38" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib15" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib25" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib13" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib50" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib11" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib21" firstref="ref14"></nolink> <nolink nlid="nl11" bibid="bib46" firstref="ref15"></nolink> <nolink nlid="nl12" bibid="bib49" firstref="ref16"></nolink> <nolink nlid="nl13" bibid="bib59" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib14" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib41" firstref="ref26"></nolink> <nolink nlid="nl16" bibid="bib17" firstref="ref31"></nolink> <nolink nlid="nl17" bibid="bib20" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib22" firstref="ref33"></nolink> <nolink nlid="nl19" bibid="bib35" firstref="ref34"></nolink> <nolink nlid="nl20" bibid="bib42" firstref="ref35"></nolink> <nolink nlid="nl21" bibid="bib44" firstref="ref36"></nolink> <nolink nlid="nl22" bibid="bib29" firstref="ref37"></nolink> <nolink nlid="nl23" bibid="bib56" firstref="ref38"></nolink> <nolink nlid="nl24" bibid="bib57" firstref="ref39"></nolink> <nolink nlid="nl25" bibid="bib51" firstref="ref45"></nolink> <nolink nlid="nl26" bibid="bib37" firstref="ref47"></nolink> <nolink nlid="nl27" bibid="bib12" firstref="ref48"></nolink> <nolink nlid="nl28" bibid="bib24" firstref="ref51"></nolink> <nolink nlid="nl29" bibid="bib34" firstref="ref55"></nolink> <nolink nlid="nl30" bibid="bib55" firstref="ref59"></nolink> <nolink nlid="nl31" bibid="bib31" firstref="ref62"></nolink> <nolink nlid="nl32" bibid="bib43" firstref="ref63"></nolink> <nolink nlid="nl33" bibid="bib52" firstref="ref64"></nolink> <nolink nlid="nl34" bibid="bib54" firstref="ref65"></nolink> <nolink nlid="nl35" bibid="bib26" firstref="ref70"></nolink> <nolink nlid="nl36" bibid="bib16" firstref="ref79"></nolink> <nolink nlid="nl37" bibid="bib40" firstref="ref80"></nolink> <nolink nlid="nl38" bibid="bib39" firstref="ref84"></nolink> <nolink nlid="nl39" bibid="bib36" firstref="ref90"></nolink> <nolink nlid="nl40" bibid="bib48" firstref="ref96"></nolink> <nolink nlid="nl41" bibid="bib10" firstref="ref101"></nolink> <nolink nlid="nl42" bibid="bib47" firstref="ref102"></nolink> <nolink nlid="nl43" bibid="bib53" firstref="ref105"></nolink> <nolink nlid="nl44" bibid="bib32" firstref="ref109"></nolink> <nolink nlid="nl45" bibid="bib45" firstref="ref114"></nolink> <nolink nlid="nl46" bibid="bib58" firstref="ref120"></nolink> <nolink nlid="nl47" bibid="bib18" firstref="ref124"></nolink> <nolink nlid="nl48" bibid="bib27" firstref="ref125"></nolink> <nolink nlid="nl49" bibid="bib30" firstref="ref133"></nolink> <nolink nlid="nl50" bibid="bib33" firstref="ref134"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1475799
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Updating 'The Future of Coding': Qualitative Coding with Generative Large Language Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Nga+Than%22">Nga Than</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6845-6253">0000-0002-6845-6253</externalLink>)<br /><searchLink fieldCode="AR" term="%22Leanne+Fan%22">Leanne Fan</searchLink><br /><searchLink fieldCode="AR" term="%22Tina+Law%22">Tina Law</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7631-6763">0000-0001-7631-6763</externalLink>)<br /><searchLink fieldCode="AR" term="%22Laura+K%2E+Nelson%22">Laura K. Nelson</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8948-300X">0000-0001-8948-300X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Leslie+McCall%22">Leslie McCall</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7700-3969">0000-0002-7700-3969</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2025 54(3):849-888.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 40
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Coding%22">Coding</searchLink><br /><searchLink fieldCode="DE" term="%22Qualitative+Research%22">Qualitative Research</searchLink><br /><searchLink fieldCode="DE" term="%22Cues%22">Cues</searchLink><br /><searchLink fieldCode="DE" term="%22Open+Source+Technology%22">Open Source Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Science+Research%22">Social Science Research</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241251339188
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Over the past decade, social scientists have adapted computational methods for qualitative text analysis, with the hope that they can match the accuracy and reliability of hand coding. The emergence of GPT and open-source generative large language models (LLMs) has transformed this process by shifting from programming to engaging with models using natural language, potentially mimicking the in-depth, inductive, and/or iterative process of qualitative analysis. We test the ability of generative LLMs to replicate and augment traditional qualitative coding, experimenting with multiple prompt structures across four closed- and open-source generative LLMs and proposing a workflow for conducting qualitative coding with generative LLMs. We find that LLMs can perform nearly as well as prior supervised machine learning models in accurately matching hand-coding output. Moreover, using generative LLMs as a natural language interlocutor closely replicates traditional qualitative methods, indicating their potential to transform the qualitative research process, despite ongoing challenges.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1475799
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1475799
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241251339188
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 40
        StartPage: 849
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Coding
        Type: general
      – SubjectFull: Qualitative Research
        Type: general
      – SubjectFull: Cues
        Type: general
      – SubjectFull: Open Source Technology
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Social Science Research
        Type: general
    Titles:
      – TitleFull: Updating 'The Future of Coding': Qualitative Coding with Generative Large Language Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Nga Than
      – PersonEntity:
          Name:
            NameFull: Leanne Fan
      – PersonEntity:
          Name:
            NameFull: Tina Law
      – PersonEntity:
          Name:
            NameFull: Laura K. Nelson
      – PersonEntity:
          Name:
            NameFull: Leslie McCall
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 54
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1