View in EDS HTML Full Text PDF Full Text

From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models

Saved in:
Bibliographic Details
Title: From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models
Language: English
Authors: Oscar Stuhler (ORCID 0000-0001-7391-1743), Cat Dang Ton (ORCID 0009-0000-6872-3588), Etienne Ollion (ORCID 0000-0003-3099-5240)
Source: Sociological Methods & Research. 2025 54(3):794-848.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 55
Publication Date: 2025
Document Type: Journal Articles
Reports - Evaluative
Descriptors: Artificial Intelligence, Sociology, Social Science Research, Natural Language Processing, Information Processing, Biographies, Open Source Technology, Prompting, Cues
DOI: 10.1177/00491241251336794
ISSN: 0049-1241
1552-8294
Abstract: Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into "promptbooks." We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE while also underscoring risks like non-random error patterns that can bias downstream analyses.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1475720
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFBQap907EqCb17jJKR_BZXAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDEGOgPHeKADrIij6LAIBEICBm8T2BypRlviYCmlwl_JxLdgEMyQBvmv6nwY36tvWoPRsC8T6u3lNHFZ-y9SHBnfF0HGqop3NT_sVbpay6cvfMfKMAdHF2udU6GcPPZUe3FUVhq35U20QPZEDXnWf_O8iY-CpWO6LIE1ca6XstpuDJwC5RvbRePSa3R0d6T0U10q02N81u5sH2t4yAlb_m5cjvfhQzTbvwVkqlw01
Text:
  Availability: 1
  Value: <anid>AN0186128772;som01aug.25;2025Jun26.00:54;v2.2.500</anid> <title id="AN0186128772-1">From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models </title> <p>Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into "promptbooks." We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE while also underscoring risks like non-random error patterns that can bias downstream analyses.</p> <p>Keywords: text analysis; information extraction; large language models; generative AI; prompting</p> <hd id="AN0186128772-2">Introduction</hd> <p>Generative artificial intelligence (GenAI) is increasingly recognized as a powerful tool for humanities and social sciences research ([<reflink idref="bib4" id="ref1">4</reflink>]; [<reflink idref="bib22" id="ref2">22</reflink>]). In particular, generative large language models (LLMs)[<reflink idref="bib6" id="ref3">6</reflink>] have been used to simulate human behaviors across a variety of roles and contexts ([<reflink idref="bib34" id="ref4">34</reflink>]; [<reflink idref="bib78" id="ref5">78</reflink>]; [<reflink idref="bib90" id="ref6">90</reflink>]; [<reflink idref="bib13" id="ref7">13</reflink>]; [<reflink idref="bib56" id="ref8">56</reflink>]), such as economic agents in markets ([<reflink idref="bib44" id="ref9">44</reflink>]), subjects in psychological experiments ([<reflink idref="bib1" id="ref10">1</reflink>]), opinion poll respondents ([<reflink idref="bib3" id="ref11">3</reflink>]; [<reflink idref="bib57" id="ref12">57</reflink>]), college applicants ([<reflink idref="bib2" id="ref13">2</reflink>]), or talk show participants ([<reflink idref="bib52" id="ref14">52</reflink>]). More prosaically, but no less importantly, GenAI has been used to aid question generation ([<reflink idref="bib38" id="ref15">38</reflink>]), missing data imputation ([<reflink idref="bib53" id="ref16">53</reflink>]), and content classification ([<reflink idref="bib36" id="ref17">36</reflink>]; [<reflink idref="bib20" id="ref18">20</reflink>]; [<reflink idref="bib22" id="ref19">22</reflink>]; [<reflink idref="bib59" id="ref20">59</reflink>]).</p> <p>Adding to this growing body of work, this paper explores the potential of GenAI, specifically generative LLMs, as a tool for information extraction (IE). IE is an umbrella term for a variety of Natural Language Processing (NLP) methods for finding information in unstructured textual data ([<reflink idref="bib43" id="ref21">43</reflink>]; [<reflink idref="bib51" id="ref22">51</reflink>], Ch. 20). For instance, given a collection of short biographies, an IE model may be trained to extract the subject's date of birth, birthplace, and name to fill a template with this information. IE is a relatively wide field that includes such diverse tasks as temporal tagging, event extraction, named entity recognition, or relation extraction. While sociologists have long sought to systematically extract specific kinds of information from texts (see, e.g., [<reflink idref="bib30" id="ref23">30</reflink>]; [<reflink idref="bib70" id="ref24">70</reflink>]; [<reflink idref="bib87" id="ref25">87</reflink>]), few to date have made use of IE-family approaches (recent exceptions include [<reflink idref="bib54" id="ref26">54</reflink>]; [<reflink idref="bib37" id="ref27">37</reflink>]; [<reflink idref="bib73" id="ref28">73</reflink>]; [<reflink idref="bib83" id="ref29">83</reflink>], [<reflink idref="bib84" id="ref30">84</reflink>], [<reflink idref="bib85" id="ref31">85</reflink>]). This is not only because, thus far, doing IE has required familiarity with a branch of highly specialized NLP methods, but also because IE models may not have aligned with how sociologists want to code their data.</p> <p>In this article, we make the case that the advent of GenAI fundamentally changes this. Specifically, we illustrate the potential of prompt-based IE in a study of newspaper obituaries from which we encode a variety of biographical information. Alongside this empirical analysis, we discuss how social scientists can develop and evaluate prompt architectures for IE, turning codebooks into what we call "promptbooks." We find that the most capable model we tested (Llama 70B Instruct) replicates our manual extractions with a high degree of accuracy for most types of information. Specifically, information that is explicitly stated in the text (e.g., <emph>age</emph> or <emph>cause of death</emph>) and that requires numeric inference (e.g., the <emph>number of children</emph>) was extracted with near-perfect accuracy. However, accuracy declined somewhat for information requiring interpretive competence (<emph>education level</emph>, <emph>origin</emph>, <emph>religion</emph>). In these cases, the model at times adopted an overly literal reading, struggling to integrate contextual cues into broader inferences. When testing different model sizes, we found that a smaller model (Llama 8B Instruct) could perform as well as the larger one for some variables, but not others. Different prompting strategies (chain-of-thought, 0- versus 1-shot, requesting JSON-formatted responses) made little impact on performance.</p> <p>Overall, our analyses showcase some core advantages of prompt-based IE for social scientists but also one major risk. First, the approach is highly <emph>accessible</emph>. In fact, we believe researchers with basic coding skills will be able to build reasonably accurate IE systems. Second, it is highly <emph>flexible</emph>: thus far, social scientists who wanted to do IE have been limited by whether or not computational linguists had built modules that fit their research problem. With prompt-based IE, social scientists can design their own tasks and extract information relevant to their research context and questions. Third, the approach has the potential to enhance <emph>transparency</emph> and <emph>reproducibility</emph> in computational research. In theory, codebooks for content analysis should be sufficient to reproduce the classification they are designed for, but in practice, they often depend on at least some implicitly shared knowledge among the coders. Prompt-based IE forces researchers to explicitly and more rigorously define the qualities they are aiming to measure. Notwithstanding these advantages, our results also suggest that prompt-based IE comes with a major risk for social science: the <emph>non-randomness of errors</emph>. Models tend to provide plausible-sounding responses and, in doing so, sometimes infer them based on semantic patterns learned from their training data, leading to potentially biased predictions. Even when extracted information is highly accurate in the aggregate, this can create serious pitfalls for downstream analyses.</p> <p>The paper is structured as follows: first, we define IE and discuss its relevance and prior applications in sociological research. Second, we introduce our data and argue why obituaries offer a good case study for IE. Third, we define a diverse set of extraction tasks and discuss how we developed a <emph>promptbook</emph> for them. Our goal here is not so much to present the reader with the eventual product, but rather to shed light on our research process. Fourth, we evaluate our approach for different variables and test out the relative performance of different prompting strategies and model sizes. We then summarize our observations and offer guiding thoughts for researchers. Finally, we conclude with a discussion of the opportunities and risks of using prompt-based IE for sociological research.</p> <hd id="AN0186128772-3">What Is Information Extraction, and How Does It Matter for Sociology?</hd> <p>Information extraction refers to the process by which the unstructured information embedded in texts is identified and turned into structured data (for introductions, see [<reflink idref="bib43" id="ref32">43</reflink>]; [<reflink idref="bib51" id="ref33">51</reflink>], Ch. 20). Texts typically contain rich information about events, people, organizations, places, relations, and other kinds of entities. Yet this information remains unusable for systematic analysis unless it is harvested and stored in a database. An IE engine makes this information tractable by parsing unstructured text to identify and extract claims about specific events or relations, following predefined templates.</p> <p>We offer an illustration of this in Figure 1. Imagine, for instance, one would like to study market dynamics based on events reported in business news, but these news reports typically contain a mix of important and relatively redundant information. An IE engine specializing in parsing business news might have a template for acquisition events that includes the elements [acquiring company], [target company], and [deal value]. The engine's task is then to identify such events and fill this template with information from the reports. Similarly, the same engine might also have a template for identifying leadership positions with the elements [person], [role], and [organization].</p> <p>Graph: Figure 1. Illustration of information extraction template filling.</p> <p>Over the past three decades, NLP scholars have built IE systems for a variety of applications (for a review, see [<reflink idref="bib42" id="ref34">42</reflink>]). An IE system often includes subtasks such as named entity recognition, temporal tagging, event extraction, and relation extraction. IE systems are typically built for specific domains and specialize in extracting the events or semantic relations important to that domain. For instance, a system built for business news (e.g., [<reflink idref="bib47" id="ref35">47</reflink>]) might be trained to detect acquisitions, merger events, collaborations (e.g., of the form collaborate{Company A, Company B}), or leadership transitions (e.g., replace{Person A, Person B, Company, Position}). Meanwhile, a system specializing in medical language (e.g., [<reflink idref="bib94" id="ref36">94</reflink>]; [<reflink idref="bib55" id="ref37">55</reflink>]) would detect claims about the effects of specific substances (e.g., cause{Pharmacological substance, Pathological function}).</p> <p>We note three important conceptual clarifications here. First, the term "information" in NLP scholarship is understood relative to the text and carries no claims as to the truth value of what is extracted. Put more precisely, IE's aim is to create formalized representations of information <emph>asserted</emph> in a text. Second, while the exact line between these concepts is not always easy to define, IE is conceptually distinct from <emph>text classification</emph>, where the goal usually is to assign one or more labels (categories) to an entire text based on its overall content or theme. Meanwhile, IE typically involves detecting granular, localized pieces of information scattered throughout the text. Third, IE rests on having clearly defined conceptual templates. These should define among other things the possible values (also referred to as "slot fillers") for each field and the criteria for assigning them. Values can be either text segments (e.g., a company name) or specific concepts that can be inferred from the text and that were prespecified in the template (e.g., a specific organization can be classified as <emph>governmental</emph>).[<reflink idref="bib7" id="ref38">7</reflink>] For a more detailed discussion, we refer to Jurafsky and Martin ([<reflink idref="bib51" id="ref39">51</reflink>], see esp. p. 22–24).</p> <p>IE has evident potential for sociological research. While not using NLP methods, sociologists have long engaged in a similar form of template filling in which textual data are treated as informants on real-world events and relations. The earliest prominent example of this is research that creates event catalogues from newspaper data. [<reflink idref="bib30" id="ref40">30</reflink>] was among the first to propose a subject-action-object coding scheme, which he used to extract information about industrial conflict and protest events from Italian newspapers (also see [<reflink idref="bib31" id="ref41">31</reflink>], [<reflink idref="bib32" id="ref42">32</reflink>]). This approach was later taken up and expanded by [<reflink idref="bib88" id="ref43">88</reflink>], who extracted information on more than 8,000 contentious gathering events from a corpus of 18th-century newspapers.</p> <p>Perhaps the most impressive work in this style stems from the Dynamics of Collective Action Project (see e.g., [<reflink idref="bib92" id="ref44">92</reflink>], [<reflink idref="bib93" id="ref45">93</reflink>]), which created a database of 23,000 protest events in the United States (1965–1995). To do this, researchers examined <emph>The New York Times</emph> for mentions of protest events. When a protest event was mentioned, researchers recorded a range of information, including protest size, location, primary target, tactics used by protestors, duration, police presence, potential violence, and more, if available. Following a similar approach, [<reflink idref="bib12" id="ref46">12</reflink>] used an event coding template that includes bystander reactions to study their effects on xenophobic attacks in Germany (also see [<reflink idref="bib11" id="ref47">11</reflink>]). [<reflink idref="bib66" id="ref48">66</reflink>] code newspaper accounts of lynchings using a template that includes information on place, time, race and sex of the victims, the justification given for the mob's action, the racial makeup of the mob, whether or not an intervention occurred, as well as whether it succeeded, and what kind of actor undertook it.</p> <p>Beyond cataloguing events, sociologists have used similar information templates for recording a variety of information from text, including political claims ([<reflink idref="bib69" id="ref49">69</reflink>]), practices or attributes associated with specific identities (e.g., [<reflink idref="bib70" id="ref50">70</reflink>]; [<reflink idref="bib71" id="ref51">71</reflink>]; [<reflink idref="bib72" id="ref52">72</reflink>]; [<reflink idref="bib67" id="ref53">67</reflink>]; [<reflink idref="bib17" id="ref54">17</reflink>], [<reflink idref="bib18" id="ref55">18</reflink>]), or causal links between events ([<reflink idref="bib7" id="ref56">7</reflink>]; [<reflink idref="bib6" id="ref57">6</reflink>]; [<reflink idref="bib81" id="ref58">81</reflink>]). While these different analyses extract information and fill templates much like NLP-based IE models do, the extraction was primarily done by hand and often required an immense investment of labor and research money (for a more systematic review of this literature, see [<reflink idref="bib84" id="ref59">84</reflink>]).</p> <p>Only more recently have sociologists begun to embrace IE-adjacent computational methods in their research, including a growing excitement about syntax-based parsers. [<reflink idref="bib73" id="ref60">73</reflink>] combined dependency parsers and named entity recognition in their analysis of the National Security Strategy documents. [<reflink idref="bib54" id="ref61">54</reflink>] used dependency parsers to analyze the kinds of actions attributed to corporate entities in early 20th-century newspaper articles. Similarly, Goldenstein and Poschmann extracted subject-verb-object triplets from business news to study corporate responsibility discourse ([<reflink idref="bib37" id="ref62">37</reflink>]). Also relying on dependency parsers, Stuhler has analyzed news coverage of immigration in Germany ([<reflink idref="bib83" id="ref63">83</reflink>]), as well as the relationship between gender and agency in fiction writing ([<reflink idref="bib85" id="ref64">85</reflink>]).</p> <p>Parsers grounded in syntax, however, differ slightly from the kind of IE approach we focus on in this paper, in that they use a general grammar (e.g., actor, action, and recipient) instead of the more specific information templates exemplified in Figure 1. This has the advantage of allowing the extracted content to stay close to the original text without requiring researchers to make rigid <emph>a priori </emph>assumptions. Inductive methods can then be used in a second step to uncover structures in this content. This is important for sociologists who, for good reason, hesitate to adopt predefined coding schemes and put a premium on having patterns emerge inductively from the data. For instance, [<reflink idref="bib83" id="ref65">83</reflink>] analysis of refugee discourse in newspapers involves extracting a series of basic elements (actions, attributes, and possessions) around mentions of refugees, then inductively identifying latent conceptions of refugees using a clustering algorithm. Therefore, one possible reason why sociologists have not embraced existing IE systems is that such systems typically assert rigid coding schemes, when what many sociologists want to do is to investigate shifts or boundaries of established categories. As [<reflink idref="bib89" id="ref66">89</reflink>] noted, event cataloguing schemes are, in effect, theories that make assumptions about what kinds of phenomena exist, with templates imposing categories like "protest," "collective violence," "collective action," "conflict," or "instances of contentious claims-making" — which are by no means uncontroversial. IE systems developed in NLP come with fixed ontologies of relations, events, or entities, and these may often not align with how sociologists would like to partition the world in their research.</p> <p>That said, as is best exemplified by the literature on protest events, there are numerous research scenarios where sociologists may want to assert a more structured coding scheme for the content they study. As we will show below, GenAI offers a highly flexible and easy way for sociologists to design their own IE templates and extract information that is relevant to their research context and questions.</p> <hd id="AN0186128772-4">Studying Obituaries</hd> <p>To examine the potential of GenAI for extracting information, we decided to work on a particular text form: newspaper obituaries. As biographical notes that announce a person's death to a community, obituaries date back to the early days of newspapers. They are a rich source for studying a variety of social science questions ([<reflink idref="bib45" id="ref67">45</reflink>]; [<reflink idref="bib16" id="ref68">16</reflink>]; [<reflink idref="bib10" id="ref69">10</reflink>]). This includes questions concerning the politics of memory and the values and ideologies embedded in whose lives, and which aspects of a life, are considered to be worth commemorating. The content of obituaries provides a rare resource for investigating social norms or social morphology ([<reflink idref="bib28" id="ref70">28</reflink>]; [<reflink idref="bib64" id="ref71">64</reflink>]; [<reflink idref="bib48" id="ref72">48</reflink>]). Which family members are typically construed as survivors? Which kinds of death are admissible, and which are shrouded in secrecy? Is someone's religious affiliation worthy of mention?</p> <p>More importantly for this paper, obituaries are a particularly good case for assessing the ability of generative LLMs to accurately extract information for sociological research. For one, obituaries are fundamentally about people and the social context in which they were embedded. They are thus substantially close to the textual domains sociologists frequently study. Second, obituaries provide a good tradeoff between highly standardized and highly open textual genres. While there are conventions that govern the tone of the writing and the facts reported, we do find considerable variation. Sometimes, obituaries adopt a rather literary style, while at other times, they are more focused on the bare facts. Besides, obituaries also vary significantly in length and in the order in which they discuss different aspects of the deceased person's life and achievements. Third, and most importantly, obituaries gave us an opportunity to test prompt-based IE on a large and diverse set of information that are typically scattered across the text. For instance, we extract numeric (e.g., age at death in years), geographic (e.g., the place lived last), and biographical information (e.g., higher education institutions attended). Some variables have a rather limited set of outcomes (e.g., gender), and measuring them can thus also be considered a classification problem.[<reflink idref="bib8" id="ref73">8</reflink>] For others, the model is required to extract literal strings of text from the obituaries (e.g., cause of death), while again others require reasoning because the information is not literally stated in the text (e.g., number of children). Some variables, depending on the context, can require the model to summarize information (e.g., we ask for phrases that best summarize the deceased person's occupation).</p> <p>For our data, we compiled a corpus of all obituaries published in <emph>The New York Times</emph> between 1980 and March 2024. We disregarded paid death notices and commentaries, limiting ourselves to only texts that were written by the <emph>Times</emph>' obituary department, sometimes dubbed the "death beat" (see, e.g., [<reflink idref="bib5" id="ref74">5</reflink>]). We also removed duplicate documents and correction statements, leading to a final count of approximately 80,000 obituaries. The average obituary is 706 tokens long, typically including titles that summarize what the person was known for and how old they were, such as "Jane Doe, Art Historian Who Led Restoration of Historic Frescoes, dies at 89."[<reflink idref="bib9" id="ref75">9</reflink>] The obituaries typically focus on people who were in the public eye, or who were prominent in a specific field or industry. Indeed, the <emph>Times</emph> described their selection criteria as follows: "people who made a difference on a large stage — people who, we think, will command the broadest interest. If you made news in life, chances are your death is news, too" ([<reflink idref="bib75" id="ref76">75</reflink>]). Their declared aim is to "report deaths and to sum up lives, illuminating why, in our judgment, those lives were significant. The justification for the obituary is in the story it tells" ([<reflink idref="bib75" id="ref77">75</reflink>]).</p> <hd id="AN0186128772-5">From Codebook to Promptbook</hd> <p></p> <hd id="AN0186128772-6">Defining Variables</hd> <p>In order to test the potential of generative LLMs for social science-oriented IE tasks, we sought to define a series of variables that are both substantively important but also heterogeneous in terms of their format and the kinds of capabilities they require of the model. In Table 1, we provide an overview of the 12 variables on which we eventually converged.</p> <p>Table 1. Variables for Information Extraction.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Variable</th><th align="left">Task type</th><th align="left">Format</th><th align="left">Example response</th><th align="left">Notes</th></tr></thead><tbody><tr><td>Age in years</td><td>String extraction</td><td>Integer</td><td>"94"</td><td>Age is usually stated in the title of an obituary. However, the format of these titles varies. In some cases, the information is more deeply embedded in the text. In a few cases, the information must be inferred from the birthdate and publication date.</td></tr><tr><td>Cause of death</td><td>String extraction</td><td>Text string</td><td>"injuries sustained in a car accident"</td><td>Most obituaries state a cause of death but these causes are not standardized. We extract the phrase that specifies the cause of death. Usually, these appear in the first few sentences of the obituary but in a few cases, we also saw them occur later in the text.</td></tr><tr><td>Institutions of higher education attended</td><td>String extraction, some inference</td><td>Text string</td><td>"Rutgers University, University of Michigan"</td><td>For this variable, we extract the names of all institutions of higher education that the person attended. This requires string extraction, but can also require inference, given that sometimes such institutions are mentioned but the person didn't actually attend them, or there are statements that imply attending such an institution (e.g., "He was a quarterback at Michigan.")</td></tr><tr><td>Religious affiliation</td><td>String extraction, inference</td><td>Text string</td><td>"catholic"</td><td>This variable records the religious affiliation of the deceased person if and only if it was explicitly mentioned. We do not make inferences based on name, origin, or other other categorical attributes of the deceased person. Only a person's voluntary association with religious organizations or lifestyles (e.g., donated to a specific church or active roles at places of worship) are sufficient to code religion. Applying these criteria, only about one in ten obituaries mention the deceased person's religious affiliation.</td></tr><tr><td>Origin</td><td>String extraction, inference, exogenous knowledge</td><td>Preformatted text string</td><td>"Queens, NY"</td><td>This variable records the municipality where the person grew up, along with a state code or country name. Often, though not always, this place is mentioned somewhere in the text. The variable requires inference, however, because an obituary typically mentions numerous places where the person lived or worked at different points in their life. At times, multiple places are mentioned in connection to the person's youth, and a selection needs to be made on what is the most significant one. Furthermore, municipalities and states are not always mentioned and need to be inferred by the model based on exogenous knowledge (e.g., "grew up in the East Village" would imply New York, NY).</td></tr><tr><td>Place lived last</td><td>String extraction, inference, exogenous knowledge</td><td>Preformatted text string</td><td>"Paris, France"</td><td>Much of what applies to the <italic>origin</italic> variable also applies here. Places lived last are typically mentioned. When obituaries do not mention such a place, but instead mention where the person died, we record this place.</td></tr><tr><td>Gender</td><td>Inference</td><td>Categorical</td><td>"female", "male", "other"</td><td>This categorical variable captures the gender of the deceased person. Measurement requires inference on the part of the model, but this inference is relatively simple. Effectively, this is a classification problem with three possible outcomes.</td></tr><tr><td>Military service</td><td>Inference</td><td>Binary</td><td>"Yes", "Not mentioned"</td><td>The fact that a person served in the military can appear anywhere in the text. It is sometimes stated explicitly ("During WWII, Smith served in the Navy [...]"), but at other times only mentioned in passing ("the former platoon leader said ..."). Note that an obituary not mentioning that the person served does not necessarily mean that the person actually did not serve.</td></tr><tr><td>Number of children</td><td>Inference</td><td>Integer</td><td>"4"</td><td>This variable requires making an inference about the number of children of the deceased person based on the children listed and other information that may be scattered throughout the text. The number of children is typically not stated literally in the text. </td></tr><tr><td>Highest level of education</td><td>Inference</td><td>Categorical</td><td>"Less than high school", "High school", "Some college", "College", "Masters, PhD, or equivalent", "not inferable"</td><td>This variable requires making an inference about the person's highest level of education. It is effectively a form of zero-shot classification, for there are only five valid options the model must choose from.</td></tr><tr><td>Survivors</td><td>Summarization</td><td>Preformatted text string</td><td>"1 wife, 2 sons, 2 daughters, 1 sister, 6 grandchildren"</td><td>Most obituaries contain sentences that list survivors at the end (e.g., "Besides her husband Dennis of Tampa, FL, Jane Smith is survived by [...] "). We encode this information in a string that contains the family roles of the people mentioned, along with a count. This requires the model to summarize information that is stated in a different format.</td></tr><tr><td>Occupation</td><td>Summarization</td><td>Text string</td><td>"politician, lawyer"</td><td>In this variable, we record phrases that best summarize the occupation of the deceased person. Usually, this is a single word or phrase, but in some instances where a person had multiple significant occupations, we record up to two occupation phrases. This task often requires the model to summarize content, as the phrases may not be mentioned directly. For instance, the obituary of a politician may not use the word "politician." Furthermore, this requires distinguishing significant occupations from those that are merely a minor feature of the narrative (e.g., "At age 13, he worked as a liftboy").</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note:</emph> This table gives a summary of the different pieces of information we coded. For more detailed information on the variables, we refer to the promptbook in the Appendix.</p> <p>The variables are ordered by task type. This is not a formal typology of IE tasks, but rather our summary of what each task entails. The variables <emph>age in years</emph> and <emph>cause of death</emph>, for instance, typically require identifying a specific number or phrase directly stated in the text and converting them to a uniform format. Similarly, we also extracted the <emph>institutions of higher education attended</emph> by the deceased person and their <emph>religious affiliation</emph>, but these variables also required somewhat more complex inference. For instance, simply extracting all institutions of higher education using named entity recognition is insufficient because many obituaries discuss scholars and where they taught; meanwhile, a person's religious affiliation is often not outright stated as an attribute, but provided indirectly through a description of their participation in places of worship (see the notes-column for more details).</p> <p>Recording the <emph>origin</emph>, that is, the municipality where a person mainly grew up and the <emph>place where they lived last</emph>, can also involve string extraction and inference, given that obituaries typically mention a variety of places, including, for instance, where a person worked or died. Besides, recording this information also often requires exogenous knowledge because municipalities, states, and countries may need to be inferred from place names(e.g., "grew up in the East Village" should be coded as "New York, NY"). Four of our variables require inferring information and assigning a code that is typically not literally stated in the text. In the case of <emph>gender</emph>, this means making a selection between the values "male," "female," and "other," which we made based on pronouns, names, and honorifics (all cases we encountered in the sample used either male or female pronouns). Similarly, we record whether the obituary mentions that the deceased <emph>served in the military</emph>, but this information can be expressed in a variety of ways. The <emph>highest level of education</emph> must be inferred based on different pieces of information. It is worth noting that these three variables are categorical and can thus be treated as classification problems that require selecting among a set of predefined values. Finally, the <emph>number of children</emph> is typically not mentioned in an obituary, but must also be inferred based on information that is often scattered across the text.</p> <p>Beyond these, two of our variables (decedent's <emph>survivors</emph>, decedent's <emph>occupation</emph>) are best described as a summarization task. Most obituaries list <emph>survivors</emph> at the end (e.g., "She is survived by her wife Jane Smith of Tampa, FL; her daughter Mary Johnson of [...]"). These sentences, however, can vary considerably in their form. For instance, some survivors may be mentioned in antepositions ("in addition to his wife, Smith is survived by [...]") or different forms of enumeration ("among his survivors are his nephews John, Joe, and Jack."). Second, we also code a term or phrase that best summarizes the deceased person's primary <emph>occupation</emph><emph>(s)</emph>. This is not a mere string extraction, as obituaries would often describe a person's professional achievements without using a term that denotes the profession itself. Second, obituaries often mention a variety of positions and job titles, some of which may be peripheral to the person's career (e.g., "At age 13, he worked as a liftboy"), while others may be too specific (e.g., "worked as assistant manager for inventory control"). While there are obituaries that contain appropriate phrases for this variable, more often than not, this task requires summarizing and weighing various pieces of information in the obituary against each other.</p> <p>An important conceptual difference between IE and discourse-analytic text coding concerns the distinction between the empirical facts of a person's life and the discourse about them represented by obituaries. Our variables ultimately target the latter, not the former. For instance, the absence of a reference to military service does not imply that a person did not serve in the military; and neither does mention of such service technically prove that the person indeed served. Similarly, names, places of origin, and ethnic backgrounds might strongly suggest a particular religious affiliation, but we only code such an affiliation if it is directly indicated in the forms described above. Therefore, the target value for our variable is not the actual religious affiliation of the person, but the one conveyed in obituary discourse. This is perhaps a subtle distinction, but we think it is an important one to keep in mind. Beyond the discussion here, the Notes column in Table 1 contains additional information for each variable. A detailed overview of all variables is given in the promptbook provided in the Appendix, the development of which we will discuss in the next section.</p> <hd id="AN0186128772-7">Developing a Codebook</hd> <p>We began our analysis by drawing a stratified random sample of 300 obituaries, ensuring that our data are equally distributed across the 44-year time span. This means we assigned 3,600 unique values for the 12 different variables. We used these cases to develop our variables, come up with clear definitions and coding guidelines, run initial tests of the model, and develop our prompts. We will refer to these data as our <emph>development set</emph>, though we recognize that in classic machine learning, this term typically denotes data used for hyperparameter tuning, and it is debatable to what extent prompt modification is a direct analog of that. Furthermore, note that we chose to structure our paper by first introducing the variables (see previous section), but that some of these variables emerged as we engaged more deeply with the data.</p> <p>Our coding of the obituaries aligns with the standard procedures for annotation in content analysis and supervised machine learning projects (for extensive discussions of this, see [<reflink idref="bib58" id="ref78">58</reflink>]; [<reflink idref="bib40" id="ref79">40</reflink>], Ch. 18). First, the three authors independently dove into subsets of the data with minimal written rules on how to code each variable. During this process, we would discover cases that were challenging to encode information from based on the instructions we had. This led us to expand our codebook, adding new rules or specify existing ones. As readers who have done this kind of coding will know, it usually takes a certain number of cases to establish conventions among coders, even if the challenges that arise are relatively mundane. For instance, if an obituary mentions that someone "took classes" at a college, should this person's highest level of education be coded as "College" or "Some college"? (We decided on the latter). If an obituary states where a person was born but discusses how they spent their youth at a different place, should the first or the second be coded as <emph>origin</emph>? Similarly, when comparing codes for doubly annotated cases, we discussed instances in which we had set diverging values for a variable. Especially for those variables that depend on string extraction, this would require revising our codebook so that we would record the information in the same format. At a certain point, we got the sense that we had sufficient generalizability, in the sense that new cases would not take on forms unaccounted for in the codebook, as well as reliability, in that we would come to the same conclusions.</p> <hd id="AN0186128772-8">Choosing a Model</hd> <p>Choosing an appropriate model can be difficult in an increasingly disorienting environment where new models are released on a daily basis and typically advertised with grandiose claims. Additionally, we observe that state-of-the-art models are increasingly available in a multiplicity of variants with different numbers of parameters, levels of quantization, or degrees of fine-tuning—Meta fittingly describes its recent releases as "the Llama 3 herd of models" ([<reflink idref="bib24" id="ref80">24</reflink>]). A detailed assessment of the tradeoffs of different models is outside of the scope of this paper, and we believe that lasting advice regarding model choice is, in any case, not feasible. That said, three broad considerations guided our selection. All three of these have recently received a more comprehensive treatment by [<reflink idref="bib20" id="ref81">20</reflink>], which is why we will keep our discussion relatively short.</p> <p>First, we argue that social science research should use <emph>open-source models</emph>, that is, models with publicly available and downloadable weights. As has been pointed out by others (e.g., [<reflink idref="bib82" id="ref82">82</reflink>]; [<reflink idref="bib22" id="ref83">22</reflink>]), this is important to ensure reproducibility, as models that are only accessible through an API may be subject to changes in availability, training data, model weights, and other parameters beyond researchers' supervision. Additionally, API-based models would necessitate sharing data with a third party, which may be problematic for researchers who are working with proprietary or sensitive data. Besides, recent developments in large language modeling indicate that the performance gap between open-source models and proprietary models like GPT is closing ([<reflink idref="bib24" id="ref84">24</reflink>]). While there may be scenarios that provide an exception, open-source models on appropriate servers should be the default choice for social science research (for discussion of this, see [<reflink idref="bib77" id="ref85">77</reflink>]).</p> <p>Second, the kind of prompt-based IE that we do here, as well as few- and zero-shot learning in general, work best with models that have been <emph>fine-tuned to follow instructions</emph>. This fine-tuning, also known as instruction tuning, is typically achieved through a process referred to as reinforcement learning with human feedback (RLHF) in which humans rate the quality of model responses, though in practice, this process also heavily relies on synthetic augmentations of human data (see, e.g., [<reflink idref="bib24" id="ref86">24</reflink>], 17–18). Model variants that have undergone this process are typically labeled as "-instruct" versions, and popular open-source foundation model families like Llama ([<reflink idref="bib24" id="ref87">24</reflink>]) or Mixtral ([<reflink idref="bib49" id="ref88">49</reflink>]) typically release a "base" and an "-instruct" version. For most prompt-based tasks, social scientists are likely to be best served by models fine-tuned to follow instructions.</p> <p>Third, model choice is limited by the available hardware. We, like most other researchers, are limited in terms of our resources. Additionally, considering ours as a model project that others might follow with a similar setup, it seemed counteractive to us to use a model that many researchers would be unable to run. While the precise computational costs and hardware requirements of running a model depend on various factors, the two main parameters to consider are model size (i.e., the number of parameters), and the level of quantization. Quantization ([<reflink idref="bib39" id="ref89">39</reflink>]) is a technique that reduces the precision of the model weights by converting them from higher bit depths (e.g., float32) to lower ones (e.g., four- or eight-bit integers). This considerably lowers the memory demands of running a model and speeds up inference, typically with only minimal loss in performance (for a detailed discussion of quantization techniques and the broader topic of model compression, see [<reflink idref="bib97" id="ref90">97</reflink>]).</p> <p>With these three considerations in mind, we decided to test two instruction-tuned versions of Meta's Llama 3.0, one with 8 billion parameters (8B) and the other with 70 billion parameters (70B). For both of these, we used five-bit quantized versions in GGUF format[<reflink idref="bib10" id="ref91">10</reflink>] and ran inference using the llama.cpp library ([<reflink idref="bib35" id="ref92">35</reflink>]). The 70B variant is about 50 GB in disk size and was run using two NVIDIA V100 PCIe GPUs with 32 GB each. This is hardware that few researchers will have themselves, but that may be accessible to many via their university's high-performance computing (HPC) resources (which is how we ran our models), or that can be rented from most cloud computing providers. The 8B variant is about 6 GB in disk size and was run on a single V100 GPU on our HPC cluster, but could be run on interactive notebook services like Google Colab as well.</p> <p>Finally, for all our applications, we follow the precedent set by recent work using GenAI in the social science applications and set the model temperature—a hyperparameter controlling the randomness of a language model's outputs—to 0 (see, e.g., [<reflink idref="bib20" id="ref93">20</reflink>]; [<reflink idref="bib15" id="ref94">15</reflink>]). This effectively constrains the model to output the most likely tokens (i.e., words with the highest probability to appear next, based on patterns commonly found in the model's training data). We note, however, that the effects of temperature on the quality of models' inference is an active research area. A recent paper found that for a variety of problem-solving tasks and prompting approaches, varying temperature from 0 to 1 did not have a significant effect on model performance ([<reflink idref="bib80" id="ref95">80</reflink>]).</p> <hd id="AN0186128772-9">Developing a Promptbook</hd> <p>We use the term <emph>promptbook</emph> here in analogy to codebook. By promptbook, we simply mean the complete set of prompts used in a project that builds on prompt-based inference to extract information or label data. In this section, we outline our method of developing our promptbook and discuss some of the difficulties we encountered, which we expect others may also face when following a similar path.</p> <p>We began developing a promptbook by taking our codebook as the starting point. While it is certainly worth familiarizing oneself with prompting strategies (see, e.g., [<reflink idref="bib96" id="ref96">96</reflink>]), we think that this should typically not be the starting point for social scientific measurement as it risks conceptual slippage. Instead, as we exemplify here, the starting point should be to develop an intersubjectively shared understanding of the quality to be measured, codified in a codebook. As a first step, we therefore simply prompted the model using the codebook we had established to annotate our data. For each variable, we created one prompt based on the codebook instructions, with minimal changes.</p> <p>For instance, we added a sentence at the beginning and the end of the prompt to introduce the obituary text, and changed the language from asking a coder to "insert" a value to asking the model to "respond" with a value (see Table 2). We also added some formatting-related instructions, to ensure consistency and prevent verbose responses (i.e., to give answers like "[<reflink idref="bib94" id="ref97">94</reflink>]" instead of "Certainly! Here is the age: 94"). We also used a system prompt to instruct the model to be rigorous, concise, and to closely adhere to our instructions (see documentation in the Appendix). For simpler variables, such as <emph>age in years</emph> or <emph>military service</emph>, these minimal adjustments improved the accuracy of responses in our development set. However, other variables proved more challenging.</p> <p>Table 2. Codebook and Prompt Instruction for age in Years and Military Service.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Codebook instructions</th><th align="left">Prompt instructions</th></tr></thead><tbody><tr><td>Based on both the title, publication date, and the obituary text, infer the person's age at death in years and insert it as a numeric value. If age is not inferable, leave this field blank.</td><td><italic>Below I will provide an obituary of a deceased person.</italic>Based on both the title, publication date, and the obituary text, infer the person's age at death in years. <italic>You should respond with a numeric value.</italic> If the person's age is not inferable, <italic>respond with "9999"</italic>.<italic>-</italic><italic>Please format your response in plain text, inside quotation marks, like this: "<insert your response>"</italic><italic>Here is the obituary date, title, and text: {insert obituary text}</italic></td></tr><tr><td>Does the obituary mention that the person served in the army? If so, put "yes." Otherwise, put "not mentioned".If a person served in a foreign (that is non-US) army, also put "yes." Advisory roles do not count as having served in the army.</td><td>Below I will provide an obituary of a deceased person.Does the obituary mention that the person served in the military? If so, <italic>respond with </italic>"yes". Otherwise, <italic>respond with</italic> "not mentioned".If a person served in a foreign (that is non-US) military, also respond with "yes". <italic>Please limit your response to only one of these two codes: "yes", "not mentioned".</italic>Advisory roles do not count as having served in the military.-<italic>Please format your response in plain text, inside quotation marks, like this: "<insert your response>"</italic><italic>Here is the obituary date, title, and text: {insert obituary text}</italic></td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note: Changes are highlighted in italics.</emph></p> <p>The general procedure we followed is best described as one of iterative optimization. We'll also refer to this as our <emph>development phase,</emph> in which we adjust the prompt to improve model performance. During this phase, we worked with the 70B variant of the model. First, we would run a prompt against our data and retrieve a set of values for a given variable. We would then compare these values with the manually coded labels in our development set. Third, based on this comparison, we made tweaks to the prompt by adding conditions or revising the wording. In making the revisions, we paid special attention to avoiding conceptual "drift," that is, adding language that shifts the meaning of the target concept we intend to measure. We would then repeat this procedure until we either achieved performance we deemed satisfactory or couldn't get much improvement through revisions. Through this process, we noticed some general patterns.</p> <p>First, prompts for LLMs need to be more <emph>explicit</emph> than codebooks for human annotators typically are. Ideally, a codebook should spell out coding instructions in such detail and clarity that two untrained coders could use it to come to the same conclusions about any case. However, in practice, most codebooks are likely to rely on a certain level of implicit understanding of what a task is about and what ends it serves. This is especially likely if annotations are not crowd-sourced, but done by a group of researchers who interact with each other, as will be true for most projects. For instance, when coding the <emph>highest level of education</emph> variable, we had an unarticulated, shared understanding that we would code this variable based on what the obituary says about a person's educational trajectory. Yet when using our codebook instructions for prompting, we quickly noticed that the model drew inferences based on other pieces of information. People with certain occupations, or who were high-achieving professionally, were often coded as having a college education, even if no such education was explicitly mentioned in the text. In retrospect, the wording in our codebook ("Based on the entire obituary, to the best of your ability, infer [...]") does indeed not preclude coding the variable in this way and can perhaps even be read as encouraging such inference. Meanwhile, by "infer" (as far as we can realistically recollect), we had primarily meant to draw conclusions about which of the (potentially many) educational degrees listed was the highest. This then led us to revise the language in the prompt and to add an explicit instruction for the model not to make such inferences (see Table 3). RLHF is, of course, precisely about inducing (unstated) anticipations about user preferences into models, but our experience suggests that models need things to be spelled out that human coders typically do not. On the upside, as a third party devoid of research-specific contextual knowledge, models can help researchers identify unarticulated assumptions that perhaps should have been spelled out in the codebook in the first place.</p> <p>Table 3. Codebook and Prompt Instruction for the Highest Level of Education.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Codebook instructions</th><th align="left">Prompt instructions</th></tr></thead><tbody><tr><td>Based on the entire obituary, to the best of your ability, infer and record the highest levelof education of the deceased person as one of the following options:Less than high school, High school, Some college, College, Masters, PhD, or equivalent, Not inferable[abbreviated]</td><td><italic>Below I will provide an obituary of a deceased person.</italic><italic>Based on what the text says about this person's education, record</italic> the highest level of education of the deceased person as <italic>only</italic> one of the following codes:"Less than high school", "High school", "Some college", "College", "Masters, PhD, or equivalent", "not inferable"<italic>When giving your response, consider the following rules:</italic><italic>1) Generally, do not infer this person's level of education from their occupation without explicit statements in the text about the degrees they obtained. The only exceptions to this</italic><italic>are people who teach at universities, people who practice law, or those who practice medicine. In those cases, respond with "Masters, PhD, or equivalent".</italic>[abbreviated]</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note:</emph> Changes are highlighted in italics. Texts were abbreviated and the full prompt is provided in the Appendix.</p> <p>We also noticed that models can be persistently prone to specific, often simple kinds of errors. For instance, when coding <emph>institutions of higher education</emph> that the deceased person attended, the model would regularly include educational institutions that were not postsecondary (e.g., high schools), institutions that were mentioned but attended by others, and institutions where the person taught as a professor, but that they did not attend as a student. We have no theory as to why the model would make these particular errors, but we can say that nothing in our initial codebook-based prompt can be read as encouraging or permitting these kinds of behaviors. We addressed this problem by adding extensive explicit instructions asking the model not to make a particular kind of error (see Table 4). This improved performance on some cases, but it still did not solve the issue, leading us to our next observation.</p> <p>Table 4. Codebook and Prompt Instruction for Attended Institutions of Higher Education.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Codebook instructions</th><th align="left">Prompt instructions</th></tr></thead><tbody><tr><td>Record all institutions of higher education that were attended by the person (i.e., universities and colleges, or graduate & professional schools). These should be recorded even if the person did not complete their degree.Institutions should be typed exactly as they appear in the text.</td><td><italic>Below I will provide an obituary of a deceased person.</italic>Record all institutions of higher education that the person obtained a degree from (i.e., universities, colleges, or graduate & professional schools), exactly as written in the text. <italic>If the text indicates that this person attended some institution as a student, but did not complete their degree, record this institution as well. When giving your response, consider the following rules:</italic><list list-type="Bullet"><list-item><p><italic>1) Do not include high schools or college preparatory schools.</italic></p></list-item><list-item><p><italic>2) Do not include institutions that the person's friends, family, coworkers or partners attended, unless the deceased person also attended them.</italic></p></list-item><list-item><p><italic>3) Obituaries may describe decedents who were employed at academic institutions, such as instructors, scientists, university administrators and coaches. You must distinguish higher education</italic><italic>institutions that this person studied at from those that this person worked at. Only institutions where the person studied should be considered in your response. Do not record higher education</italic><italic>institutions only because the person worked, taught, or held a job there. For example, if the text says "after transferring from University 1 to study mathematics at University 2, he eventually got a</italic><italic>master's degree from University 3. He became a head coach at University 4 and taught sports science at University 5", your response should only include Universities 1, 2 and 3, but not University</italic><italic>4.</italic></p></list-item></list><italic>If the text does not mention any institutions of higher education that the person attended, simply respond with "none". Otherwise, your answer should include a rationale, as well as quotes from the text as</italic><italic>evidence. Your response should be formatted as a JSON file that follows this template:</italic><italic>{</italic><italic>"evidence": ["<insert quote 1>", "<insert quote n>"],</italic><italic>"rationale": "<insert your rationale>",</italic><italic>"higher_education_institutions": ["<insert institution 1>", "<insert institution 2>", "<insert institution 3>"],</italic><italic>}</italic><italic>Here is the obituary date, title, and text: {insert obituary text}</italic></td></tr></tbody></table> </ephtml> </p> <p>4 <emph>Note:</emph> Changes are highlighted in italics.</p> <p>Researchers who define their own IE templates will notice that some tasks, and not always those one would expect, are simply hard for a model. In our case, even after adding instructions, the model would still regularly respond with educational institutions that it shouldn't respond with. To tackle this, we asked the model to provide not just the value, but to first provide relevant evidence from the obituary and then a rationale for its eventual response (see Table 4). Such strategies, discussed under the label <emph>chain-of-thought prompting</emph>, have been shown to have potential for improving response behavior ([<reflink idref="bib95" id="ref98">95</reflink>]), including in social science applications (see, e.g., [<reflink idref="bib25" id="ref99">25</reflink>]; [<reflink idref="bib91" id="ref100">91</reflink>]). The general idea behind chain-of-thought prompting is that the model generates intermediate reasoning steps (e.g., selecting relevant evidence) that allow it to break down complex problems into smaller, more manageable parts before arriving at the final response. As we will show in the next section, this technique indeed elevated the model's performance on some tasks, but not on others.</p> <p>In this context, we note two challenges we encountered when developing our prompts. Firstly, prompts can be tweaked in nearly infinite ways. In classic machine learning, there is typically a limited, if large, number of possible hyperparameter configurations. Various techniques exist for systematically searching this space for ideal parameter combinations. Prompt tuning ([<reflink idref="bib61" id="ref101">61</reflink>]; [<reflink idref="bib60" id="ref102">60</reflink>])—the unsupervised optimization of prompts for specific tasks—may be a promising equivalent for prompt-based IE. When engineering prompts manually as we did here, however, it is difficult to navigate the space of possible and promising alterations, especially while guarding against conceptual drift. Adding to that, we caution against a temptation to overengineer prompts under the assumption that a model can handle an inference task if we just prompt it well enough, as we do with human coding assistants. For example, even after we added explicit instructions not to record the university that someone worked at for the variable described above, the model would still make this error. Despite our best efforts to revise our language and add additional reminders, we eventually concluded that we couldn't reliably prevent the model from making this mistake. Although we cannot exclude the possibility that some prompt does exist to resolve the issue, our experience showcases the challenges to reliably preventing model errors.</p> <p>This then points to a more general dilemma: how can researchers develop a sense of whether their prompts are "good enough," or when they've reached the model's limit such that further tweaking will not improve performance? Put differently, how do we know when to stop prompt engineering? We cannot offer a definitive solution, but we will describe the heuristic we followed in the hope that it proves useful. Generally, when evaluating model responses against our development set, we noticed two types of model errors: systematic and unsystematic. We focused our attention on the former. For instance, when coding military service, we noticed that the model would, strangely, code people who served in the Navy or the Air Force as "not mentioned." Upon checking our prompt and codebook, we realized that we had used the term "army" with the colloquial understanding that it refers to the military as a whole. Substituting it for "military" in our prompt improved the model's performance significantly. We stopped prompt engineering once the model's remaining mistakes seemed unsystematic and could not be directly addressed by tweaking the prompt further, or when systematic errors remained, but these could not be fixed by introducing explicit instructions to avoid them. We recognize, of course, that whether or not something appears "systematic" under scrutiny depends on its prevalence, sample size, and the researchers' capacity to recognize this systematicity. The concrete implications of our heuristic will thus depend on the context and the researcher.</p> <hd id="AN0186128772-10">Evaluation</hd> <p></p> <hd id="AN0186128772-11">Test Set and Model Scenarios</hd> <p>All evaluations presented in this section are made on a new, double-coded stratified random sample of 200 cases. We refer to this as our <emph>test set</emph>. We also blinded the obituaries to prevent the model from responding based on exogenous knowledge it might have learned about the deceased people from its training data.[<reflink idref="bib11" id="ref103">11</reflink>] We did this by replacing all first and last names with generic names. Men's first names were replaced with "John," and subsequent middle names were replaced with a single middle name, "Michael." Women's respective names became "Jane" and "Mary." The last names were replaced with "Smith" and "Johnson."</p> <p>We assess the performance of a set of different prompting strategies. Specifically, for all twelve IE variables, we examine the effect of <emph>chain-of-thought prompting</emph>, as opposed to just asking for a response directly. We also examine whether asking the model to structure its response in JSON format or just asking it for plain text makes a difference. Additionally, our discussion of the promptbook so far has focused on zero-shot prompting, that is, we ask the model for a response to a single obituary alone, without giving it any prior examples. Going beyond this, we also implemented a one-shot prompting scenario, giving the model one example and one simulated correct answer, then providing the actual obituary to label (see Appendix D for details).[<reflink idref="bib12" id="ref104">12</reflink>] The idea behind this strategy is that examples allow the model to recognize expected behavior. The evidence on whether this technique improves model performance is inconclusive, suggesting that it is effective for some tasks but not for others ([<reflink idref="bib14" id="ref105">14</reflink>]; [<reflink idref="bib20" id="ref106">20</reflink>]).[<reflink idref="bib13" id="ref107">13</reflink>]</p> <p>The results we present below are based on 19,200 distinct model executions. That is, for each of the 200 cases in our test set, we prompted the model 96 times, covering twelve variables, four prompting strategies, and two differently sized models. Below, we provide a curated summary of our results, focusing on the key patterns we consider to be most relevant. The full results are presented in Table 5 (70B model) and Table 6 (8B model).</p> <p>Table 5. Accuracy of Information Extraction for Scenarios with 70B Model.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" colspan="6">Model scenario</th></tr></thead><tbody><tr><td colspan="2">Size</td><td>70B</td><td>70B</td><td>70B</td><td>70B</td></tr><tr><td colspan="2">Requested JSON format</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td colspan="2">Chain-of-thought applied</td><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td></tr><tr><td colspan="2">Shots</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td><bold>Task</bold></td><td><bold>Measure</bold></td><td /><td /><td /><td /></tr><tr><td rowspan="3">Gender (categorical)</td><td>Acc</td><td><bold>1</bold></td><td><bold>1</bold></td><td><bold>1</bold></td><td><bold>1</bold></td></tr><tr><td>Acc. CIs</td><td>[1, 1]</td><td>[1, 1]</td><td>[1, 1]</td><td>[.99, 1]</td></tr><tr><td>F1</td><td>1</td><td>1</td><td>1</td><td>.67</td></tr><tr><td rowspan="2">Age in years (integer)</td><td>Acc.</td><td>.99</td><td>.99</td><td>.99</td><td><bold>1</bold></td></tr><tr><td>Acc. CIs</td><td>[.98, 1]</td><td>[.98, 1]</td><td>[.98, 1]</td><td>[.99, 1]</td></tr><tr><td rowspan="3">Military (binary)</td><td>Acc.</td><td><bold>.99</bold></td><td><bold>.99</bold></td><td>.98</td><td>.98</td></tr><tr><td>Acc. CIs</td><td>[.98, 1]</td><td>[.98, 1]</td><td>[.96, 1]</td><td>[.97, 1]</td></tr><tr><td>F1</td><td>.98</td><td>.98</td><td>.97</td><td>.98</td></tr><tr><td rowspan="2">Children (integer)</td><td>Acc.</td><td>.97</td><td>.98</td><td><bold>1</bold></td><td>.98</td></tr><tr><td>Acc. CIs</td><td>[.95,.99]</td><td>[.95, 1]</td><td>[.99, 1]</td><td>[.97, 1]</td></tr><tr><td rowspan="2">Cause of death (text string)</td><td>Acc.</td><td>.96</td><td>.96</td><td><bold>.98</bold></td><td>.97</td></tr><tr><td>Acc. CIs</td><td>[.94,.99]</td><td>[.94,.99]</td><td>[.97, 1]</td><td>[.95,.99]</td></tr><tr><td rowspan="3">Survivors (preformatted text string)</td><td>Acc.</td><td>.93</td><td><bold>.96</bold></td><td><bold>.96</bold></td><td>.94</td></tr><tr><td>Acc. CIs</td><td>[.89,.97]</td><td>[.93,.99]</td><td>[.93,.99]</td><td>[.91,.97]</td></tr><tr><td>Acc. partial</td><td>.96</td><td>.97</td><td>.98</td><td>.96</td></tr><tr><td rowspan="2">Place lived last (preformatted text string)</td><td>Acc.</td><td>.92</td><td>.93</td><td><bold>.94</bold></td><td>.<bold>94</bold></td></tr><tr><td>Acc. CIs</td><td>[.88,.95]</td><td>[.89,.97]</td><td>[.91,.97]</td><td>[.90,.97]</td></tr><tr><td rowspan="3">Occupation (text string)</td><td>Acc.</td><td>.92</td><td><bold>.93</bold></td><td>.92</td><td>.<bold>93</bold></td></tr><tr><td>Acc. CIs</td><td>[.88,.95]</td><td>[.89,.97]</td><td>[.88,.96]</td><td>[.89,.97]</td></tr><tr><td>Acc. partial</td><td>.98</td><td>.98</td><td>.99</td><td>.97</td></tr><tr><td rowspan="3">Institutions of higher education attended (text string)</td><td>Acc.</td><td><bold>.96</bold></td><td>.90</td><td>.88</td><td>.92</td></tr><tr><td>Acc. CIs</td><td>[.93,.98]</td><td>[.86,.95]</td><td>[.84,.93]</td><td>[.89,.96]</td></tr><tr><td>Acc. partial</td><td>.97</td><td>.94</td><td>.92</td><td>.95</td></tr><tr><td rowspan="3">Religious affiliation (text string)</td><td>Acc.</td><td>.86</td><td>.90</td><td><bold>.94</bold></td><td>.92</td></tr><tr><td>Acc. CIs</td><td>[.81,.9]</td><td>[.85,.94]</td><td>[.90,.97]</td><td>[.88,.95]</td></tr><tr><td>Acc. partial</td><td>.86</td><td>.90</td><td>.94</td><td>.92</td></tr><tr><td rowspan="3">Highest level of education (categorical)</td><td>Acc.</td><td><bold>.88</bold></td><td>.86</td><td>.86</td><td>.86</td></tr><tr><td>Acc. CIs</td><td>[.83,.93]</td><td>[.82,.91]</td><td>[.81,.91]</td><td>[.81,.90]</td></tr><tr><td>F1</td><td>.82</td><td>.83</td><td>.73</td><td>.73</td></tr><tr><td rowspan="2">Origin (preformatted text string)</td><td>Acc.</td><td>.83</td><td>.82</td><td><bold>.90</bold></td><td>.84</td></tr><tr><td>Acc. CIs</td><td>[.78,.88]</td><td>[.77,.88]</td><td>[.86,.94]</td><td>[.79,.89]</td></tr></tbody></table> </ephtml> </p> <p>5 <emph>Note:</emph> The table reports the accuracy of information extraction for the four scenarios with the 70B Llama Instruct model. Obituaries were blinded by replacing the first, middle, and last names of the person discussed in the obituary. This includes replacing the names of their partners, heirs, siblings, and others with same names. Additionally, we report the macro F1 scores for categorical outcomes. For some tasks, we provide a measure of partial accuracy (Acc. partial), which includes cases where the model's response partially matches the human response. For instance, for extracting survivors, incorrect responses were typically at least partially correct but missing one survivor, sometimes two. Partial accuracy, by definition, is higher than our main accuracy measure. All results are based on a time stratified random sample of 200 obituaries. To reflect the uncertainty of our accuracy values, we use normal approximation to compute 95% confidence intervals (CIs) for each accuracy via <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="normal">Acc</mi></mrow><mo>.</mo><mo>±</mo><mrow><mi mathvariant="normal">z</mi></mrow><msqrt><mrow><mfrac><mrow><mrow><mi mathvariant="normal">Acc</mi></mrow><mo>.</mo><mo stretchy="false">(</mo><mrow><mn>1</mn><mo>−</mo><mrow><mi mathvariant="normal">Acc</mi></mrow><mo>.</mo></mrow><mo stretchy="false">)</mo></mrow><mn>200</mn></mfrac></mrow></msqrt></math> </ephtml> . Note that this approach can lead to unrealistic values (e.g., an upper bound of 1) for those tasks with perfect or near perfect accuracy in our sample.</p> <p>Table 6. Accuracy of Information Extraction for Scenarios with 8B Model.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="left" colspan="7">Model scenario</th></tr></thead><tbody><tr><td colspan="2">Size</td><td>8B</td><td>8B</td><td>8B</td><td>8B</td><td>8B</td></tr><tr><td colspan="2">Requested JSON format</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td colspan="2">Chain-of-thought applied</td><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td><td>No</td></tr><tr><td colspan="2">Shots</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td colspan="2">System prompt given</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Task</td><td>Measure</td><td /><td /><td /><td /><td /></tr><tr><td rowspan="3">Gender (categorical)</td><td>Acc</td><td>.96</td><td>.96</td><td><bold>1</bold></td><td>.98</td><td><bold>1</bold></td></tr><tr><td>Acc. CIs</td><td>[.93,.98]</td><td>[.93,.98]</td><td>[1, 1]</td><td>[.96, 1]</td><td>[1, 1]</td></tr><tr><td>F1</td><td>.65</td><td>.65</td><td>1</td><td>.66</td><td>1</td></tr><tr><td rowspan="2">Age in years (integer)</td><td>Acc.</td><td>.98</td><td>.98</td><td>.99</td><td><bold>1</bold></td><td>.99</td></tr><tr><td>Acc. CIs</td><td>[.97, 1]</td><td>[.97, 1]</td><td>[.98, 1]</td><td>[1, 1]</td><td>[.98, 1]</td></tr><tr><td rowspan="3">Military (binary)</td><td>Acc.</td><td>.76</td><td>.68</td><td><bold>.86</bold></td><td>.85</td><td>.82</td></tr><tr><td>Acc. CIs</td><td>[.70,.82]</td><td>[.62,.74]</td><td>[.81,.91]</td><td>[.80,.90]</td><td>[.77,.88]</td></tr><tr><td>F1</td><td>.72</td><td>.65</td><td>.80</td><td>.81</td><td>.78</td></tr><tr><td rowspan="2">Children (integer)</td><td>Acc.</td><td>.86</td><td>.<bold>88</bold></td><td><bold>.88</bold></td><td>.64</td><td>.72</td></tr><tr><td>Acc. CIs</td><td>[.81,.91]</td><td>[.84,.93]</td><td>[.84,.93]</td><td>[.58,.71]</td><td>[.65,.78]</td></tr><tr><td rowspan="2">Cause of death (text string)</td><td>Acc.</td><td>.95</td><td>.95</td><td><bold>.96</bold></td><td>.96</td><td>.<bold>96</bold></td></tr><tr><td>Acc. CIs</td><td>[.92,.98]</td><td>[.92,.98]</td><td>[.94,.99]</td><td>[.93,.99]</td><td>[.94,.99]</td></tr><tr><td rowspan="3">Survivors (preformatted text string)</td><td>Acc.</td><td>.8</td><td>.78</td><td>.78</td><td><bold>.84</bold></td><td>.76</td></tr><tr><td>Acc. CIs</td><td>[.74,.86]</td><td>[.73,.84]</td><td>[.73,.84]</td><td>[.79,.89]</td><td>[.70,.82]</td></tr><tr><td>Acc. partial</td><td>.91</td><td>.88</td><td>.90</td><td>.92</td><td>.92</td></tr><tr><td rowspan="2">Place lived last (preformatted text string)</td><td>Acc.</td><td>.84</td><td>.86</td><td>.88</td><td><bold>.87</bold></td><td>.83</td></tr><tr><td>Acc. CIs</td><td>[.79,.90]</td><td>[.82,.91]</td><td>[.83,.93]</td><td>[.82,.92]</td><td>[.78,.88]</td></tr><tr><td rowspan="3">Occupation (text string)</td><td>Acc.</td><td>.88</td><td>.90</td><td><bold>.94</bold></td><td>.91</td><td>.93</td></tr><tr><td>Acc. CIs</td><td>[.83,.93]</td><td>[.86,.94]</td><td>[.90,.97]</td><td>[.87,.95]</td><td>[.89,.97]</td></tr><tr><td>Acc. partial</td><td>.98</td><td>.96</td><td>.98</td><td>.97</td><td>.98</td></tr><tr><td rowspan="3">Institutions of higher education attended (text string)</td><td>Acc.</td><td>.74</td><td>.73</td><td>.79</td><td>.79</td><td>.<bold>80</bold></td></tr><tr><td>Acc. CIs</td><td>[.68,.81]</td><td>[.67,.79]</td><td>[.73,.85]</td><td>[.73,.85]</td><td>[.74,.85]</td></tr><tr><td>Acc. partial</td><td>.80</td><td>.80</td><td>.86</td><td>.85</td><td>.88</td></tr><tr><td rowspan="3">Religious affiliation (text string)</td><td>Acc.</td><td>.72</td><td>.71</td><td><bold>.94</bold></td><td>.81</td><td>.86</td></tr><tr><td>Acc. CIs</td><td>[.65,.78]</td><td>[.65,.77]</td><td>[.91,.98]</td><td>[.76,.86]</td><td>[.81,.91]</td></tr><tr><td>Acc. partial</td><td>.72</td><td>.72</td><td>.95</td><td>.81</td><td>.87</td></tr><tr><td rowspan="3">Highest level of education (categorical)</td><td>Acc.</td><td>.58</td><td>.56</td><td>.40</td><td><bold>.72</bold></td><td>.50</td></tr><tr><td>Acc. CIs</td><td>[.51,.64]</td><td>[.49,.62]</td><td>[.33,.47]</td><td>[.66,.78]</td><td>[.43,.56]</td></tr><tr><td>F1</td><td>.41</td><td>.42</td><td>.25</td><td>.58</td><td>.35</td></tr><tr><td rowspan="2">Origin (preformatted text string)</td><td>Acc.</td><td>.72</td><td>.<bold>76</bold></td><td><bold>.76</bold></td><td>.74</td><td>.<bold>76</bold></td></tr><tr><td>Acc. CIs</td><td>[.66,.79]</td><td>[.71,.82]</td><td>[.70,.82]</td><td>[.68,.81]</td><td>[.70,.82]</td></tr></tbody></table> </ephtml> </p> <p>6 <emph>Note:</emph> The table reports the accuracy of information extraction for the four scenarios with the 8B Llama Instruct model. For details, see the description of Table 5. Additionally, upon the suggestion of one reviewer, we tested one strategy where we dropped the system prompt.</p> <hd id="AN0186128772-12">Results by IE Tasks</hd> <p>The most important question we want to answer is whether the generative large language models can accurately extract the kinds of information we discussed above. Therefore, before getting to the differences between different models and prompting strategies, we first discuss the results of our largest and best-performing model (Llama 70B Instruct), broken down by variable. Figure 2 shows the accuracy of this model for each of our tasks (red). <emph>Accuracy</emph> (Acc.) is the share of cases for which the model's response corresponded to the value we assigned when coding the test set. When assessing this, we used minimal post-processing for the model responses as detailed in Appendix E. The accuracy scores in Figure 2 are averages across the four different prompting strategies (see Table 5 description for details). We emphasize that researchers who want to extract information for downstream tasks will have to conduct their own evaluations, and that what counts as sufficient performance will depend on such aspects as sample size and specific analytic goals.</p> <p>Graph: Figure 2. Accuracy across information extraction tasks.</p> <p>The 70B model achieved almost perfect accuracy for <emph>gender</emph> (<reflink idref="bib1" id="ref108">1</reflink>), <emph>age in years</emph> (.99), <emph>military service</emph> (.99), and the <emph>number of children</emph> (.98).[<reflink idref="bib14" id="ref109">14</reflink>] The first two variables may be seen as relatively easy targets: gender can be identified by linguistic features such as the prevalence of certain honorifics ("Mr.," "Mrs.," "Ms.," etc.) or gendered pronouns ("she," "her," "he," "him," "his") that most conventional machine learning classifiers could have picked up on. Age, as we discussed above, is often mentioned directly in the title or early in the text, in relatively standardized wording (e.g., "died at the age of [...]"). Both military service and gender are binary (in our test set) and have roughly similar imbalances (with 22.5% women and 19% people with stated military service). But whereas gender typically permeates the whole obituary via pronouns and names, military service is typically mentioned in passing somewhere deeply embedded in the text. It is impressive that the model succeeds at picking this up, because conventional supervised learning methods can struggle with this kind of sparse-signal information. Finally, unlike the previous variables, extracting the number of children required identifying information that is not literally stated in the text. The model's strong performance on this task highlights the effectiveness of our prompt-based approach in tackling an IE task that involves summarizing and inferring information from relatively long texts.</p> <p>Our models also did well when it came to extracting the <emph>cause of death</emph> (acc. of.97). This shows that the 70B model can successfully identify and extract relevant text strings. The variable that records <emph>survivors</emph> is best described as a summarization task. Survivors are usually mentioned in a subsection at the end of an obituary, but as we discussed above, the lexico-syntactic heterogeneity in these expressions makes the extraction of survivors non-trivial. Like for the <emph>number of children</emph>, the model must not only detect references to surviving kin, but also enumerate them and distinguish them from kin mentioned in passing. The fact that the model reached an accuracy of.95 on this task attests to its ability to perform several inferences at a time. Besides, incorrect responses were often partially correct, missing only one or two survivors; we report these results in Table 5, which average.97.</p> <p>The model performed slightly worse on extracting geographic information: <emph>place lived last</emph> (acc. of.93) and <emph>origin</emph> (.85). These tasks require string extraction, inference, and, at times, exogenous knowledge. In particular, identifying a person's <emph>origin</emph> can require piecing together and weighing scattered information (e.g., school attendance, parent's place of work, events that marked the decedent's childhood, etc.). For both tasks, the errors primarily stemmed from ambiguity in the texts. Also, to the model's credit, it was able to find relevant information sprinkled in various places of the obituaries, for example, by inferring the deceased person's last place of residence via the location where their partner lived.</p> <p>When asked to summarize the main professional activities (<emph>occupation</emph>), the model gave an identical or highly similar answer to human coders in most of the cases (.93). By highly similar, we mean minor variations in wording. While one could use an exact string matching criterion (which would imply lower accuracy), this seemed to us to go against the intent of the task. The partial accuracy for this variable (see Table 5), suggests that for another 5% of the cases, the agreement was at least partial: the model aptly identified one profession but not the other(s). It is worth noting that obituaries published in the Times typically give a great deal of attention to a person's professional achievement, so the error rate might have been higher had this not been the case.</p> <p>Two tasks that the model struggled with more are <emph>institutions of higher education attended</emph> (acc. of.92) and <emph>highest level of education</emph> (acc. of.87 to.9; average macro-F1 of.78). Concerning the former, we have already indicated that we could not reliably prevent the model from listing institutions that were mentioned for reasons other than the person having attended them, such as the person having taught there. Concerning the latter, the model was generally too strict in recognizing academic achievements: phrases stating that a person "attended" or "went to" a specific university are commonly understood as indicative of degree completion and were treated by us as sufficient for assigning the value "College." Despite our instructions for the model to behave in the same way, however, it would regularly assign the code "Some college." Other errors came from ambiguities regarding the level of an institution, mostly between college and advanced degrees.</p> <p>Finally, the results for <emph>religious affiliation</emph> merit special attention. On the surface, accuracy (.9) doesn't look too bad (though this is in good part driven by 88% of cases not mentioning the person's religion). However, what is noteworthy is the nature of the errors the model made, which we had already noticed when working with our development set. Despite our best efforts, we could not prevent the model from imputing religious affiliations from information that was not explicitly about religious practice, even though our prompt instructed the model not to infer religion based on identity markers including nationality, name, origin, and such. This indicates a real limitation and risk of using prompt-based IE in social science research, the implications of which we will discuss in more detail below.</p> <hd id="AN0186128772-13">Results by Model Size</hd> <p>As we mentioned above, running a 70-billion parameter model and analyzing large corpora with it involves significant computational and logistical costs, which may be prohibitive for many sociologists. It is, therefore, worth testing whether a smaller model like the 8-billion parameter Llama 3 Instruct can handle our IE tasks similarly well. In Table 6, we report the performance of our 8-billion parameter model. Previously mentioned Figure 2 also contains the average model performances and for the smaller model (blue).</p> <p>Generally, we find that the smaller model performs worse in all tasks and scenarios. However, there is some interesting heterogeneity. For gender[<reflink idref="bib15" id="ref110">15</reflink>] (acc. of.97), <emph>age</emph> (.99), <emph>cause of death</emph> (.96), and <emph>occupation</emph> (.91), extraction is almost as accurate as for the 70B model. However, for <emph>military</emph> and <emph>survivors</emph>, where the 70B model had accuracies of.95 or higher, performance dropped sharply (−.20, −.16, and −.15, respectively). We also see declines concerning <emph>place lived last</emph> (−.14), <emph>origin</emph> (−.07), <emph>institutions of higher education</emph> (−.10), and <emph>religion</emph> (−.11). Finally, the poorest performance of the 8B model concerns <emph>level of education</emph> (acc. of.56) where the average accuracy dropped by.30.</p> <p>Beyond this, we note that the 8B model had more difficulties adhering to the requested output format. First, we noticed that in a few cases, the model would be verbose by adding statements like "Here is the response," before giving the actual response. In Appendix E, we provide a systematic analysis of this behavior, showing that for the 8B model this occurs in 0%‒4% of the cases (depending on the variable), while for the 70B model, we only found.1% such cases. Second, while the 8B model also responded with JSON style answers when prompted to do so, unlike for the 70B model, these frequently did not strictly adhere to JSON format (leaving out, for instance, quotation marks around "rationale" items). While it was straightforward to extract the model's responses, our experience suggests that researchers working with the 8B model cannot expect to reliably receive JSON-formatted responses.</p> <p>Overall, these results lead us to draw a mixed conclusion regarding model size. For some tasks, the smaller model reached similarly high accuracy as the 70B, which would likely be sufficient for many downstream tasks. Especially with large corpora and many cases to annotate, this suggests the potential to save a substantial portion of the computational costs. On the other hand, our analysis also shows that for some tasks, the smaller model fell considerably short of the larger one. One caveat here is that we worked with the 70B model when we developed our prompts, so we can't exclude the possibility that some of the differences we find between models are a consequence of the specific prompts we used. To the extent that performance differences are not conditional on prompts, however, our results suggest that getting good performance on some tasks may indeed require using a larger model. What distinguishes such tasks from those where we saw few or no differences? It appears that the smallest performance differences concerned information that is highly present throughout the obituaries (<emph>gender</emph> and <emph>occupation</emph>) or reliably stated in more or less formulaic form right at the beginning (<emph>age</emph>, and <emph>cause of death</emph>). Meanwhile, some of the most significant performance drops concerned information that is sparse and akin to a needle in a haystack (<emph>military</emph>, <emph>education level</emph>) and/or that requires numeric inference (<emph>number of children</emph>, <emph>survivors, education level</emph>). We will reflect on this observation in our discussion below.</p> <hd id="AN0186128772-14">Prompting Strategies</hd> <p>As indicated above, we also tested four different prompting strategies.[<reflink idref="bib16" id="ref111">16</reflink>] Our main observation is that none of them performs universally better across the tasks. This holds true for both the 70B model and the 8B model. Especially for the tasks with the highest performance, approaches fared almost equally well, and the differences typically rest on only a few cases. For the tasks where the models performed worse, the variation between approaches is larger, but at least some of this heterogeneity is likely to be driven by chance. In Figure 3, we present accuracy estimates for each strategy for both models. These estimates and the corresponding 95% confidence intervals were obtained by fitting eight independent random-effects meta-analysis models that treat the results for each of the 12 tasks as separate studies. Overall, the differences between the strategies are quite modest.</p> <p>Graph: Figure 3. Comparison between prompting strategies. Note. The figure presents accuracy estimates for each strategy for both models. These estimates and the corresponding 95% confidence intervals were obtained by fitting eight independent random-effects meta-analysis models that treat the results for each of the 12 tasks as separate studies (n = 200, respectively). In this way, each result is based on 2,400 model executions.</p> <p>Even so, one approach stands out slightly: the scenario without chain-of-thought prompting generally performed best, independent of model size. For both the 70B and the 8B models, avoiding this technique led to the best accuracy for eight of our 12 variables. A qualitative inspection of the results suggests that initiating a chain-of-thought can make models less likely to state that the information is absent. Instead, models select vaguely relevant textual evidence and proceed to make rather creative interpretations (this was especially the case for <emph>origin</emph>). This aligns with prior findings that generative LLMs have a predisposition to give false positives and to hallucinate when information simply is not present ([<reflink idref="bib65" id="ref112">65</reflink>]). On the other hand, for at least one task (identifying <emph>institutions of higher education</emph>) the model performed markedly better with this mode of prompting. Taking up our prior discussion of this variable, this might suggest that chain-of-thought prompting is a good strategy for forcing the model to comply with explicitly stated rules if it otherwise doesn't. However, our results suggest that it is by no means a universally superior strategy, which aligns with some recent findings by Burnham ([<reflink idref="bib15" id="ref113">15</reflink>]: 11).</p> <hd id="AN0186128772-15">Baseline Approaches</hd> <p>Overall, our results point to GenAI's potential for sociological inquiry, showing that especially the 70B model, but also the 8B model, can serve as an accessible tool for IE. However, at least for some of the tasks we discussed above, there is reason to suspect that simpler, more conventional approaches could tackle them reasonably well. In addition to the different models and prompting strategies described, we created a set of <emph>baseline</emph> approaches that rely on more conventional methods for a subset of our tasks. Our findings are summarized in Table 7.</p> <p>Table 7. Evaluation of Baseline Approaches Versus Generative Models.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /><col align="center" /></colgroup><thead><tr><th align="center" colspan="2" rowspan="2" /><th align="center" rowspan="2">70B Model</th><th align="center" rowspan="2">8B Model</th><th align="center" colspan="2">Baseline</th></tr><tr><th align="center">Longformer</th><th align="center">Naïve Bayes</th></tr></thead><tbody><tr><td rowspan="2">Gender (categorical)</td><td>Acc.</td><td><bold>1</bold></td><td>.97</td><td>.99</td><td>.98</td></tr><tr><td>F1</td><td>.92</td><td>.79</td><td>.<bold>99</bold></td><td>.97</td></tr><tr><td rowspan="2">Military (binary)</td><td>Acc.</td><td>.<bold>99</bold></td><td>.8</td><td>.83</td><td>.81</td></tr><tr><td>F1</td><td>.<bold>98</bold></td><td>.75</td><td>.72</td><td>.55</td></tr><tr><td rowspan="2">Highest level of education (categorical)</td><td>Acc.</td><td>.<bold>87</bold></td><td>.55</td><td>.5</td><td>.51</td></tr><tr><td>F1</td><td>.<bold>78</bold></td><td>.40</td><td>.29</td><td>.29</td></tr><tr><td>Age in years (integer)</td><td>Acc.</td><td>.<bold>99</bold></td><td>.<bold>99</bold></td><td colspan="2">.95</td></tr><tr><td>Cause of death (text string)</td><td>Acc.</td><td>.<bold>97</bold></td><td>.96</td><td colspan="2">.76</td></tr></tbody></table> </ephtml> </p> <p>7 <emph>Note:</emph> The table shows the average accuracy and macro-F1 scores for our baseline approaches. For direct comparison, we also show the average accuracy and F1-scores averaged across prompting strategies for the 70B and the 8B model<emph>.</emph></p> <p>First, <emph>gender</emph>, <emph>military</emph>, and the <emph>highest level of education</emph> have a fixed number of outcomes and can be considered conventional classification problems. We implement two alternative approaches to tackle this task. First, fine-tuning smaller language models has been shown to be a potent strategy for classification problems, even with small numbers of annotated cases (see, e.g., [<reflink idref="bib9" id="ref114">9</reflink>]; [<reflink idref="bib23" id="ref115">23</reflink>]; [<reflink idref="bib20" id="ref116">20</reflink>]). We fine-tune Longformer models ([<reflink idref="bib8" id="ref117">8</reflink>]). Second, as an even simpler approach, we implemented Naïve Bayes classifiers that uses the raw document-term matrix. Both approaches are evaluated on our test set via 10-fold cross-validation (see Appendix C for details). We find that for <emph>gender</emph>, the two conventional classifiers perform just as good as the 70B model and even slightly better than the 8B model. However, for <emph>military</emph> and especially for <emph>level of education</emph>, both baseline models did markedly worse than our 70B model. It seems likely that both baselines would improve with more labeled data. Overall, our results suggest that more conventional classification approaches can be an economical alternative to prompt-based extraction if the information is highly present in the text. For sparse-signal information that is more akin to a needle in a haystack, like <emph>military service or details about education</emph>, prompting may be preferable.</p> <p>Beyond this, we created regular expressions to extract the <emph>age</emph> of the deceased person. When working with our development set, we noticed that age is usually stated at the beginning of obituaries and only rarely requires inference. While we initially tried more complex string-matching approaches, we eventually realized that we could get surprisingly reliable results by simply extracting the first two- or three-digit number that appears in an obituary. With this approach we reach an accuracy of.95, which is only slightly worse than both generative models (.99).</p> <p>Finally, we noticed that the language around <emph>causes of death</emph> was highly patterned. We systematically looked at obituaries that mention causes of death in our development set and built a system of regular expressions that extract phrases that follow things like "died of," "died from," "the cause was," and so forth (see Appendix C for details). We created this system using only our development set. When running this against out test set, we reach an accuracy of.76. Though we spent considerable time developing this baseline, we note that the approach could be improved by examining more cases and expanding the set of indicative phrases. Nonetheless, this result suggests that even for information embedded in a highly patterned lexical environment, achieving comparable extraction performance to a prompt-based approach might be hard.</p> <p>Other variables did not seem particularly amenable to this kind of approach, as they required inference, or the language around the relevant pieces of information was more varied (e.g., occupation). This does not mean that one could not build systems for measuring them that don't depend on GenAI, say, by extracting noun phrases. The baselines aimed to test whether, for at least some IE tasks, researchers can get similar results with simpler and more economical tools. This is indeed the case, which we'll reflect on in our summary.</p> <hd id="AN0186128772-16">Summary and Guiding Thoughts</hd> <p>In this section, we summarize our findings and discuss their practical implications. Overall, we consider our results encouraging. They demonstrate that two moderately-sized open-source generative LLMs that scholars can run locally (i.e., not through an API) successfully handled relatively complex inference tasks when instructed well with prompts.</p> <p>How can researchers know whether their task can be successfully tackled with this approach? The short answer is that they can't know a priori, but our results provide some useful insights. We found that the 70B model had near-perfect accuracy for extracting information that is more or less explicitly stated in the text (<emph>gender</emph>, <emph>age</emph>, <emph>military service</emph>, <emph>cause of death</emph>). It also did well for variables that require precise numeric inference (<emph>number of children</emph>, <emph>survivors</emph>). Meanwhile, the model's ability was somewhat more limited when it came to extracting information that required what one might best describe as interpretive competence (<emph>education level</emph>, <emph>origin</emph>, <emph>religion</emph>): despite our instructions, the model would treat such phrases as that someone "went to" a university as insufficient for assigning the value "College" — an error human readers would be unlikely to make. Along similar lines, inferring where a person grew up requires reading cues and competently weighing different pieces of information. That is to say, the model sometimes approached the texts with a perhaps overly literal reading, struggling to integrate cues into broader inferences. In a rapidly evolving environment where new and more capable models are frequently released, it is unclear to what extent these particular observations will remain valid. Thus, rather than giving definite predictions on which tasks models will do well on, our goal is more modest: by exploring a wide array of tasks and variables, we hope to open readers' imagination as to the different kinds of information they <emph>might</emph> now be able to extract from text at scale. The true and only possible answer to the initial question is best summarized with Grimmer and Stewart's old mantra: "Validate, validate, validate" ([<reflink idref="bib41" id="ref118">41</reflink>]: 271).</p> <p>Another important insight from our results is that for some tasks, there may be much more economical tools than generative LLMs. In the case of <emph>age</emph>, for example, a simple regular expression performed almost as well as prompting our 70 billion parameter model. While it may be tempting to use prompting as a one-stop-shop for IE tasks, this might amount to a dramatic waste of resources when data sets are large. We recommend that researchers wanting to do IE carefully consider how the information they would like to extract is embedded in their texts. Sometimes, this embedding will be patterned in ways that lend themselves to other forms of extraction. In our case, it was our close reading of the text in the development phase that eventually led us to recognize how we could reliably extract the age of a deceased person. Similarly, we found that for one of our tasks (<emph>gender</emph>), fine-tuning a small language model led to similarly accurate results as prompting a large one (resonating findings by [<reflink idref="bib65" id="ref119">65</reflink>]; [<reflink idref="bib20" id="ref120">20</reflink>]). Therefore, even when prompting generative LLMs yields highly accurate results, this approach need not be the first choice.</p> <p>Along similar lines, we found that our smaller model could handle some IE tasks just as or almost as well as the large one. Smaller models may therefore be another way to save both computational and logistical costs. Our analyses of twelve different IE tasks indicate that performance differences between models of different sizes don't follow a simple scaling function: some tasks that were "difficult" for the small model were "easy" for the large model. Others were similarly easy or difficult for both. The observed performance differences appear to depend on qualities of the tasks, such as whether the information is highly salient or more akin to a needle in a haystack. However, these observations remain qualitative and should not be interpreted as reliable predictions as to what tasks a small model can or cannot handle. Instead, we recommend that researchers take an exploratory approach, beginning with smaller models. Our results suggest there is a good chance that these might have sufficient accuracy for a project's analytic goals.</p> <p>When it comes to strategies for improving model output, the most significant choice was whether or not to use chain-of-thought prompting. The impact of this strategy, however, varied: it improved performance on a task where the model struggled to follow explicit instructions but led to declines for others. Asking for a specific output format or applying 1-shot prompting did not have much of an effect on our tasks. While beyond the scope of this paper, it is worth pointing out that there are a variety of other techniques and prompting strategies that researchers seeking to improve their results can explore. For instance, [<reflink idref="bib14" id="ref121">14</reflink>] found positive effects when increasing the number of examples given to the model in few-shot prompting beyond one. Researchers might also want to consider logit biasing, a technique for adjusting the likelihood of specific tokens during text generation, effectively restricting or promoting certain model outputs. In a recent paper, [<reflink idref="bib15" id="ref122">15</reflink>] found that this improved model performance on a social science classification task. Alternatively, or in addition, one could use thresholds for output tokens. That is, one might consider a piece of information absent from the text unless the model's response tokens have a probability above a certain threshold. This might be an effective way to reduce false positives and to mitigate hallucinations (see, e.g., [<reflink idref="bib79" id="ref123">79</reflink>]). Prompt tuning ([<reflink idref="bib61" id="ref124">61</reflink>]; [<reflink idref="bib60" id="ref125">60</reflink>]), as discussed above, might be another way of improving model performance.</p> <p>Finally, the overall high accuracy for extracting a variety of pieces of information should not distract from the fact that our results also uncovered a concerning weakness: models sometimes rely on semantic patterns learned from their training data to infer probable answers that have no justification in the text. Our most clear-cut example of this concerned <emph>religion</emph>, where even through explicit instruction, we could not reliably prevent models from making inferences based on identity markers like nationality, name, or origin. While this is a drastic case, it is easy to imagine how this tendency might affect other tasks, especially when it comes to extracting demographic and identity-related information. This not only raises ethical concerns (see, e.g., [<reflink idref="bib50" id="ref126">50</reflink>]) but implies that the error in measures derived with a prompting approach will likely be non-random. This has major implications for downstream analyses. As Egami and colleagues ([<reflink idref="bib27" id="ref127">27</reflink>], [<reflink idref="bib26" id="ref128">26</reflink>]) recently demonstrated with regard to text classification, even if models produce labels that are highly accurate in the aggregate, non-random errors can lead to highly biased estimates in subsequent analyses. Therefore, we recommend that researchers carefully examine the kinds of errors that their models make and test whether errors are correlated with variables that are of theoretical interest. We also recommend exploring statistical tools for systematically addressing this challenging problem, at least as it relates to obtaining de-biased estimators in downstream analyses, such as the Design-based Supervised Learning (DSL) framework (see [<reflink idref="bib27" id="ref129">27</reflink>], [<reflink idref="bib26" id="ref130">26</reflink>]).</p> <hd id="AN0186128772-17">Discussion</hd> <p>Having extracted a variety of pieces of information with mixed but overall high accuracy, we see three core advantages of prompt-based IE over previous approaches relating to <emph>accessibility</emph>, <emph>flexibility</emph>, and <emph>transparency and reproducibility</emph>. However, based on our results, we also highlight a major risk of this approach, which concerns the likely <emph>non-randomness of prediction errors</emph>.</p> <p>First, and perhaps most importantly, prompt-based IE is highly <emph>accessible</emph>. In a pre-GenAI era, doing IE required tedious human labor or would have required sociologists to familiarize themselves with a branch of highly specialized and often niche NLP methods. In retrospect, we think that there may have been real potential for sociological analysis in some of the specialized IE tools developed in NLP. To some extent, however, GenAI gives sociologists the chance to leapfrog over such tools in a manner that requires relatively few special skills. Most of the work we did for this study involved conceptually developing our variables and annotating, as well as editing prompts and examining results to steer the model toward producing desired outcomes. While it is worth familiarizing oneself with basic prompting strategies, prompt development is not a high art. As we mentioned above, we think that developing prompts from an existing codebook is relatively straightforward, and researchers can build on prompting patterns from other published work or prompt catalogues (see, e.g., [<reflink idref="bib96" id="ref131">96</reflink>]).</p> <p>Furthermore, running inference with the models typically involves only a few lines of code. Perhaps the most difficult technical aspect of a project like ours is setting models up on an appropriate compute infrastructure. For a few researchers, it will make sense to purchase the hardware necessary to run inference on even a moderately sized open-source generative LLM like the one we used. Therefore, running inference will typically require using an HPC cluster or other cloud compute environment. Increasing adoption of inference with generative LLMs will likely lead research institutions to provide guides and instructions for doing so, as some already have. Some may even provide graphical user interfaces that further reduce technical hurdles. That said, it is also important to point out that access to technical support and the hardware necessary for this kind of project will not be distributed equally. Researchers at some institutions will have clear advantages over others, especially when it comes to projects that require inference on a large number of documents. Nonetheless, we think that overall, GenAI makes running IE much more accessible for many sociologists, especially for those without specialized skills in NLP.</p> <p>Leveraging GenAI for IE via prompts offers another key advantage over previous approaches, especially from the point of view of social science research: it is highly <emph>flexible</emph>. Rather than having to build on IE templates and ontologies developed by others, sociologists now have the opportunity to tailor coding schemes to their project and research question. Rather than adapting their work to a tool just because it is available, researchers will ideally be able to align models with their respective codebooks. As Davidson recently put it, GenAI can "facilitate sophisticated approaches to content analysis that avoid sacrificing the interpretative nuances that are easily lost when using conventional computational techniques" ([<reflink idref="bib22" id="ref132">22</reflink>]: 5).</p> <p>There is, of course, a limit here. As we illustrated in Figure 1 and as we noted in our discussion of sociologists' usage of event catalogues, IE templates are often more complex. Here, with the exception of listing survivors, we focused on extracting single pieces of information. Advanced IE tasks may involve retrieving multiple events from a text, encoding temporal or other meta information about these events, or identifying linked entities and tagging the relationship. It appears likely that such tasks would have proven more difficult for our model. However, it is also not certain that they would have provided an insurmountable challenge, for one can disaggregate most tasks into a series of subtasks, such as, say, identifying the sentence that relates two entities, and then running a series of prompts on that sentence to fill a template. We anticipate that future studies will more thoroughly explore these possibilities than we could here.</p> <p>Third, we think that the use of prompts provides an opportunity for increasing <emph>transparency</emph> in the annotation of texts. This argument may appear surprising, as others have rightly pointed out that generative LLMs pose some challenges to replicability ([<reflink idref="bib76" id="ref133">76</reflink>]; [<reflink idref="bib82" id="ref134">82</reflink>]). In particular, proprietary models can be modified or removed by third parties without notice, thereby thwarting any future replication efforts. We share these concerns, but we also think that GenAI offers some unique opportunities when it comes to being transparent about how a concept was measured. As discussed above, codebooks ideally provide complete and exhaustive instructions for anyone to replicate the codes used in a project. However, the reality is that most data annotation in social science projects occurs in groups, and such groups operate to some extent on implicitly shared understandings that may not be spelled out. Furthermore, there is effectively no way for readers or reviewers to assess the extent to which a codebook's instructions were correctly implemented on the training data unless they want to engage in data labeling themselves (which they rarely do). Meanwhile, data labeling and extracting information via prompts largely depend on two things, both of which ideally are publicly available: the prompt and the model.</p> <p>The combination of an open-source model and what we call a promptbook goes a long way in giving anyone a clear sense of the data labeling process that underlies a study. Not only can readers easily examine the promptbook and assess its quality (which we invite the readers to do; see Appendix A), but this also opens up exciting new possibilities for prompt-sharing and building on other people's work. While previously, researchers could build on others' codebooks, this hasn't exactly been a common practice in sociology, and as indicated above, we have some doubts as to what extent most codebooks would suffice to truly replicate a concept. Meanwhile, promptbooks make it very easy to implement another researcher's measurement strategy for a certain concept simply by running their prompt against the model they chose. This might facilitate a new level of dialogue among researchers, where one project's conception of a complex concept (whether "populist rhetoric," "politeness," or "anger," to name a few examples) can easily be replicated in another. The point then isn't so much that GenAI is <emph>necessary</emph> for replication, but that it makes it <emph>easier</emph>, and therefore more <emph>likely</emph> to actually happen.</p> <p>Fourth, despite these positive aspects, we also want to warn researchers that the likely <emph>non-randomness of prediction errors</emph> implies a major risk of this approach. Social scientists will typically extract information with generative LLMs to perform downstream analyses. When errors in the extracted information are correlated with theoretically relevant variables, they can introduce significant biases into downstream estimates. This risk remains present even when the extracted information is highly accurate overall ([<reflink idref="bib26" id="ref135">26</reflink>], [<reflink idref="bib27" id="ref136">27</reflink>]). To be clear, this problem is not unique to variables generated with a prompting approach. Social scientists have long overlooked the fact that non-random errors in machine-generated predictions may bias downstream analysis ([<reflink idref="bib26" id="ref137">26</reflink>], [<reflink idref="bib27" id="ref138">27</reflink>]). However, the risks of doing so appear significantly greater with a new family of models that can draw on semantic patterns learned from exogenous data to infer responses. A model that infers religion based on a name despite being instructed not to do so effectively draws on its own prejudices. Researchers need to be aware that even if they have a largely accurate IE model, this accuracy may vary across subsections of their data, and, consequently, that extracted data may reflect the model's biases. This is not an easy problem to address and we commend recent efforts to give social scientists statistical tools for systematically tackling it ([<reflink idref="bib26" id="ref139">26</reflink>], [<reflink idref="bib27" id="ref140">27</reflink>]).</p> <p>Nonetheless, overall, our analyses demonstrate that generative LLMs can be turned into flexible and (for many tasks) useful IE engines. While it is good to be cautious in the context of a general hype around GenAI, we do think that this further opens the door for a revival of a sociological paradigm that treats texts as informants about events, people, relations, organizations, places, and other kinds of entities. Many of the highlights of this paradigm (e.g., [<reflink idref="bib92" id="ref141">92</reflink>]; [<reflink idref="bib88" id="ref142">88</reflink>]; [<reflink idref="bib70" id="ref143">70</reflink>]; [<reflink idref="bib69" id="ref144">69</reflink>]) required large research collaborations and/or immense investments of labor and research money. With prompt-based IE, sociologists now have a comparatively affordable and scalable tool for collecting structured data from unstructured corpora. This could open new frontiers across a variety of research areas like studying methods used in scientific articles ([<reflink idref="bib21" id="ref145">21</reflink>]), characteristics attributed to players in sports broadcasting ([<reflink idref="bib29" id="ref146">29</reflink>]), definitions of relationships in interpersonal communication ([<reflink idref="bib68" id="ref147">68</reflink>]), and many others.</p> <hd id="AN0186128772-18">Conclusion</hd> <p>In theater, the term "promptbook" refers to the document that records all the information needed to produce a show. Details relating to script, acting, lighting, or sound are documented in order to leave as few uncertainties as possible as to how a performance should unfold. In the wake of the development of generative AI and the increasing adoption of prompt-based techniques in sociology, it seems fitting to us to borrow this term. Adapted to the social sciences, a promptbook should document the tasks involved in conducting an annotation or IE project.</p> <p>Along with open-source models, the use and sharing of promptbooks could considerably increase the transparency of social scientific content analysis. It gives readers a direct chance to examine the authors' conceptual choices. In other words, it makes explicit what sometimes remains implicit in a researcher's work. Yet promptbooks have another important function: More informative than the raw annotations used to train models, and usually more explicit than the codebooks provided to research assistants, they allow us to <emph>share</emph> research operations involved in a process that has traditionally often remained hidden to others. Just as code-sharing allows researchers working with quantitative data to imitate, reflect, and improve on the solutions designed by their colleagues, promptbooks allow us to make the other dimensions of research more visible and exchangeable.</p> <hd id="AN0186128772-19">Supplemental Material</hd> <p>Graph: Supplemental material, sj-docx-1-smr-10.1177_00491241251336794 for From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models by Oscar Stuhler, Cat Dang Ton and Etienne Ollion in Sociological Methods & Research</p> <hd id="AN0186128772-20">Acknowledgments</hd> <p>The authors are grateful to Émilien Schultz for helping them with advanced data manipulation. We also thank the participants of the NLP and Social Sciences seminar at the Institut Polytechnique de Paris and those of the Social Science Colloquium at the University of Stuttgart for their insightful comments.</p> <ref id="AN0186128772-21"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref10" type="bt">1</bibl> <bibtext> The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref13" type="bt">2</bibl> <bibtext> This research was supported by Svenska Vetenskapsrådet, Agence Nationale de la Recherche Labex (grant number Mining for Meaning 2018-05170, ANR-11-LABX-0047, ANR-19-P3IA-0001).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref11" type="bt">3</bibl> <bibtext> Oscar Stuhler https://orcid.org/0000-0001-7391-1743 Cat Dang Ton https://orcid.org/0009-0000-6872-3588 Etienne Ollion https://orcid.org/0000-0003-3099-5240</bibtext> </blist> <blist> <bibl id="bib4" idref="ref1" type="bt">4</bibl> <bibtext> The replication materials for this paper can be found under the corresponding OSF repository ([86]).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref74" type="bt">5</bibl> <bibtext> Supplemental material for this article is available online.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref3" type="bt">6</bibl> <bibtext> We emphasize that LLM stands for large language model only, and that there are LLMs that are not generative.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref38" type="bt">7</bibl> <bibtext> For an impressive example of what defining such templates can look like, we refer to the collected ACE Tasks and Specifications codebooks by the [62].</bibtext> </blist> <blist> <bibl id="bib8" idref="ref73" type="bt">8</bibl> <bibtext> This is a consequence of us focusing on categorical pieces of information that are, by definition, already linked to one and only one entity (the deceased person). For this paper, we wanted to work with a relatively bounded medium of text that would allow us to test extraction for a large variety of different kinds of information. As we note in our <emph>Discussion</emph> section, IE templates can be designed to be more complex than what we did here. For instance, a more complex example of IE would be to extract all names and corresponding genders from one newspaper issue's entire obituary section.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref75" type="bt">9</bibl> <bibtext> For this paper, we decided to anonymize and modify text from obituaries. While obituaries are public, we suspect people might object to having their deceased relatives' obituaries used for illustrating a methodological point in a research paper. We also think that this choice makes no substantive difference to our arguments.</bibtext> </blist> <blist> <bibtext> Specifically, we used the quantized version of Llama 3 70B Instruct made available on the Model Hub at huggingface.co under "Bartowski/Meta-Llama-3-70B-Instruct-Q5_K_M.gguf" and "Bartowski/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf."</bibtext> </blist> <blist> <bibtext> We cannot exclude the possibility that the model might still be able to infer the true identities of the people based on other pieces of information in the obituary. There is unfortunately no way to fully blind these data, but this will likely also be true for many other social science applications. Still, for some research applications, the model's capacity to draw on exogenous knowledge can be advantageous. This is somewhat exemplified in our variables about <emph>origin</emph> and <emph>last place lived</emph> where the model drew from general geographical knowledge. However, it is arguably not true for our tasks at large, given that, as we stated above, our variables are meant to encode aspects of the discourse represented in obituaries, and not the actual lives of people portrayed in them.</bibtext> </blist> <blist> <bibtext> Not least because obituaries are quite long, we don't explore few-shot prompting (i.e., giving the model more than one example) and limit ourselves to one-shot prompting.</bibtext> </blist> <blist> <bibtext> Considering the high number of tasks we evaluate here, we limit ourselves to four scenarios instead of testing all eight possible combinations of these choices. Our default scenario asks for JSON-formatted responses, uses chain-of-thought prompting, and is structured as 0-shot. The other three scenarios respectively alter one parameter in this default scenario.</bibtext> </blist> <blist> <bibtext> The few errors for <emph>age</emph> stemmed from cases where the information was not present, but rather than stating this, the model would make dubious inferences.</bibtext> </blist> <blist> <bibtext> It might be surprising that the model would miss cases on a seemingly easy task like gender. Our prompt included the categories "male," "female," and "other," while, at least based on pronouns, all cases in our test set could be classified as either "male" or "female." The errors for gender almost exclusively stemmed from the model being reluctant to assign either "male" or "female" and instead choosing "other." It seems likely that accuracy in our test set would have been higher had we used a prompt with binary options, but this would have come at the cost of excluding an option for non-binary people.</bibtext> </blist> <blist> <bibtext> Additionally, on the suggestion of one reviewer, we tested whether removing custom system prompts can elevate smaller models' performance. The 8B Llama Instruct model's capacity to tackle our IE tasks was not impacted by this (see Table 6).</bibtext> </blist> <blist> <bibtext> "\\s+(\\w+\\s+){0,3}" is a regex pattern that stands for "0, 1, 2 or 3 words". A search string like "died\\s+(\\w\\s+){0.3}of" would include phrases like "died of", "died suddenly of", "died on Thursday of", and "died in Cedar Rapids, of".</bibtext> </blist> <blist> <bibtext> [18]</bibtext> </blist> </ref> <ref id="AN0186128772-22"> <title> References </title> <blist> <bibtext> Aher Gati V., Arriaga Rosa I., Kalai Adam Tauman. 2023. "Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies." Pp. 337–71 in Proceedings of the 40th International Conference on Machine Learning. Honolulu, Hawaii, USA: PMLR.</bibtext> </blist> <blist> <bibtext> Alvero A. J., Lee Jinsook, Regla-Vargas Alejandra, Kizilcec René, Joachims Thorsten, Antonio Anthony Lising. 2024. "Large Language Models, Social Demography, and Hegemony: Comparing Authorship in Human and Synthetic Text." SocArxiv. doi: 10.31235/osf.io/qfx4a.</bibtext> </blist> <blist> <bibtext> Argyle Lisa P., Busby Ethan C., Fulda Nancy, Gubler Joshua R., Rytting Christopher, Wingate David. 2023. " Out of One, Many: Using Language Models to Simulate Human Samples." Political Analysis. 31(3):337–51. doi: 10.1017/pan.2023.2.</bibtext> </blist> <blist> <bibtext> Bail Christopher A. 2024. " Can Generative AI Improve Social Science? " Proceedings of the National Academy of Sciences. 121(21):e2314021121. doi: 10.1073/pnas.2314021121.</bibtext> </blist> <blist> <bibtext> Baranick Alana, Sheeler Jim, Miller Stephen. 2005. Life on the Death Beat: A Handbook for Obituary Writers. Oak Park, IL: Marion Street Press.</bibtext> </blist> <blist> <bibtext> Bearman Peter, Faris Robert, Moody James. 1999. " Blocking the Future: New Solutions for Old Problems in Historical Social Science." Social Science History. 23(4):501–33. doi: 10.1017/S0145553200021854.</bibtext> </blist> <blist> <bibtext> Bearman Peter S., Stovel Katherine. 2000. " Becoming a Nazi: A Model for Narrative Networks." Poetics. 27(2):69–90. doi: 10.1016/S0304-422X(99)00022-4.</bibtext> </blist> <blist> <bibtext> Beltagy Iz, Peters Matthew E., Cohan Arman. 2020. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150.</bibtext> </blist> <blist> <bibtext> Bonikowski Bart, Luo Yuchen, Stuhler Oscar. 2022. " Politics as Usual? Measuring Populism, Nationalism, and Authoritarianism in U.S. Presidential Campaigns (1952–2020) with Neural Language Models." Sociological Methods & Research. 51(4):1721–87.</bibtext> </blist> <blist> <bibtext> Bourdieu Pierre. 1988. Homo Academicus. Cambridge, UK: Polity Press.</bibtext> </blist> <blist> <bibtext> Braun Robert, Koopmans Ruud. 2010. " The Diffusion of Ethnic Violence in Germany: the Role of Social Similarity." European Sociological Review. 26(1):111–23. doi: 10.1093/esr/jcp056.</bibtext> </blist> <blist> <bibtext> Braun Robert, Koopmans Ruud. 2014. " Watch the Crowd: Bystander Responses, Trickle-Down Politics, and Xenophobic Mobilization." Comparative Political Studies. 47(4):631–58. doi: 10.1177/0010414013488544.</bibtext> </blist> <blist> <bibtext> Broska David, Howes Michael, van Loon Austin. 2024. The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations. OSF preprint: https://doi.org/10.31235/osf.io/j3bnt_v3.</bibtext> </blist> <blist> <bibtext> Brown Tom B., Mann Benjamin, Ryder Nick, Subbiah Melanie, Kaplan Jared, Dhariwal Prafulla, Neelakantan Arvind, Shyam Pranav, Sastry Girish, Askell Amanda, Agarwal Sandhini, Herbert-Voss Ariel, Krueger Gretchen, Henighan Tom, Child Rewon, Ramesh Aditya, Ziegler Daniel M., Wu Jeffrey, Winter Clemens, Hesse Christopher, Chen Mark, Sigler Eric, Litwin Mateusz, Gray Scott, Chess Benjamin, Clark Jack, Berner Christopher, McCandlish Sam, Radford Alec, Sutskever Ilya, Amodei Dario. 2020. "Language Models Are Few-Shot Learners." arXiv:2005.14165.</bibtext> </blist> <blist> <bibtext> Burnham Michael. 2024. " Stance Detection: a Practical Guide to Classifying Political Beliefs in Text." Political Science Research and Methods: 1–18. doi: https://doi.org/10.1017/psrm.2024.35.</bibtext> </blist> <blist> <bibtext> Bytheway Bill, Johnson Julia. 1996. " Valuing Lives? Obituaries and the Life Course." Mortality. 1(2):219–34. doi: 10.1080/713685833.</bibtext> </blist> <blist> <bibtext> Carley Kathleen. 1993. " Coding Choices for Textual Analysis: A Comparison of Content Analysis and Map Analysis." Sociological Methodology. 23:75–126. doi: 10.2307/271007.</bibtext> </blist> <blist> <bibtext> Carley Kathleen. 1994. " Extracting Culture Through Textual Analysis." Poetics. 22(4):291–312. doi: 10.1016/0304-422X(94)90011-6.</bibtext> </blist> <blist> <bibtext> CDC. 2001. " HIV and AIDS — United States, 1981–2000." Morbidity and Mortality Weekly Report. 50(21):429–56.</bibtext> </blist> <blist> <bibtext> Chae Youngjin, Davidson Thomas. 2024. "Large Language Models for Text Classification. From Zero-Shot Learning to Instruction-Tuning". Working paper.</bibtext> </blist> <blist> <bibtext> Dagdelen John, Dunn Alexander, Lee Sanghoon, Walker Nicholas, Rosen Andrew S., Ceder Gerbrand, Persson Kristin A., Jain Anubhav. 2024. " Structured Information Extraction from Scientific Text with Large Language Models." Nature Communications. 15(1):1418.</bibtext> </blist> <blist> <bibtext> Davidson Thomas. 2024. " Start Generating: Harnessing Generative Artificial Intelligence for Sociological Research." Socius: Sociological Research for a Dynamic World. 10:23780231241259651. doi: 10.1177/23780231241259651.</bibtext> </blist> <blist> <bibtext> Do Salomé, Ollion Étienne, Shen Rubing. 2022. " The Augmented Social Scientist: Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy." Sociological Methods & Research. 53(3):1167–200.</bibtext> </blist> <blist> <bibtext> Dubey Abhimanyu, Jauhri Abhinav, Pandey Abhinav, Kadian Abhishek, Al-Dahle Ahmad, Letman Aiesha, Mathur Akhil, et al.2024. "The Llama 3 Herd of Models." arXiv. arXiv:2407.21783.</bibtext> </blist> <blist> <bibtext> Dunivin Zackary. 2024. "Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks." arXiv. arXiv:2401.15170.</bibtext> </blist> <blist> <bibtext> Egami Naoki, Hinck Musashi, Stewart Brandon, Wei Hanying. 2023. "Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models." Proceedings of the 37th International Conference on Neural Information Processing Systems, Article 3000, 68589-68601. arXiv:2306.04746.</bibtext> </blist> <blist> <bibtext> Egami Naoki, Hinck Musashi, Stewart Brandon M., Wei Hanying. 2024. "Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses." Working paper. https://naokiegami.com/paper/dsl_ss.pdf.</bibtext> </blist> <blist> <bibtext> Fowler Bridget, Bielsa Esperança. 2007. " The Lives We Choose to Remember: A Quantitative Analysis of Newspaper Obituaries." The Sociological Review. 55(2):203–26. doi: 10.1111/j.1467-954X.2007.00702.x.</bibtext> </blist> <blist> <bibtext> Foy Steven L., Ray Rashawn. 2019. " Skin in the Game: Colorism and the Subtle Operation of Stereotypes in Men's College Basketball." American Journal of Sociology. 125(3):730–85.</bibtext> </blist> <blist> <bibtext> Franzosi Roberto. 1989. " From Words to Numbers: A Generalized and Linguistics-Based Coding Procedure for Collecting Textual Data." Sociological Methodology. 19(1):263–98.</bibtext> </blist> <blist> <bibtext> Franzosi Roberto. 1990. " Computer-assisted Coding of Textual Data: An Application to Semantic Grammars." Sociological Methods & Research. 19(2):225–57.</bibtext> </blist> <blist> <bibtext> Franzosi Roberto. 1994. " From Words to Numbers: a set Theory Framework for the Collection, Organization, and Analysis of Narrative Data." Sociological Methodology. 24:105–36.</bibtext> </blist> <blist> <bibtext> Franzosi Roberto. 1998. " Narrative as Data: Linguistic and Statistical Tools for the Quantitative Study of Historical Events." International Review of Social History. 43:81–104.</bibtext> </blist> <blist> <bibtext> Gao Chen, Lan Xiaochong, Lu Zhihong, Mao Jinzhu, Piao Jinghua, Wang Huandong, Jin Depeng, Li Yong. 2023. "S3: Social-Network Simulation System with Large Language Model-Empowered Agents." arXiv. arXiv:2307.14984.</bibtext> </blist> <blist> <bibtext> Gerganov Georgi. 2024. Llama.cpp: Inference of Meta's LLaMA model (and others) in pure C/C++. https://github.com/ggerganov/llama.cpp.</bibtext> </blist> <blist> <bibtext> Gilardi Fabrizio, Alizadeh Meysam, Kubli Maël. 2023. " ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks." Proceedings of the National Academy of Sciences. 120(30):e2305016120. doi:10.1073/pnas.2305016120.</bibtext> </blist> <blist> <bibtext> Goldenstein Jan, Poschmann Philipp. 2019. " Analyzing Meaning in Big Data: Performing a Map Analysis Using Grammatical Parsing and Topic Modeling." Sociological Methodology. 49(1):83–131. doi: 10.1177/0081175019852762.</bibtext> </blist> <blist> <bibtext> Götz Friedrich M., Maertens Rakoen, Loomba Sahil, van der Linden Sander. 2024. " Let the Algorithm Speak: How to Use Neural Networks for Automatic Item Generation in Psychological Scale Development." Psychological Methods. 29(3):494–518.doi: https://doi.org/10.1037/met0000540.</bibtext> </blist> <blist> <bibtext> Gray Robert, Neuhoff David L. 1998. " Quantization." IEEE Transactions on Information Theory. 44(6):2325–83.</bibtext> </blist> <blist> <bibtext> Grimmer Justin, Roberts Margaret E., Stewart Brandon M. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Grimmer Justin, Stewart Brandon M. 2013. " Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts." Political Analysis. 21(3):267–97.</bibtext> </blist> <blist> <bibtext> Grishman Ralph. 2019. " Twenty-Five Years of Information Extraction." Natural Language Engineering. 25(6):677–92. doi: 10.1017/S1351324919000512.</bibtext> </blist> <blist> <bibtext> Grishman Ralph. 2022 "Information Extraction." Pp. 934–50 in The Oxford Handbook of Computational Linguistics, edited by Mitkov R. London: Oxford University Press.</bibtext> </blist> <blist> <bibtext> Horton John J. 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?" National Bureau of Economic Research. NEBR Working Papers. Working Paper 31122. doi: 10.3386/w31122.</bibtext> </blist> <blist> <bibtext> Hume Janice. 2000. Obituaries in American Culture. Jackson: University Press of Mississippi.</bibtext> </blist> <blist> <bibtext> Hwang Jackelyn, Naik Nikhil. 2023. " Systematic Social Observation at Scale: Using Crowdsourcing and Computer Vision to Measure Visible Neighborhood Conditions." Sociological Methodology. 53(2):183–216. doi: 10.1177/00811750231160781.</bibtext> </blist> <blist> <bibtext> Jacobs Gilles, Hoste Véronique. 2022. " SENTiVENT: Enabling Supervised Information Extraction of Company-Specific Events in Economic and Financial News." Language Resources and Evaluation. 56(1):225–57. doi: 10.1007/s10579-021-09562-4.</bibtext> </blist> <blist> <bibtext> Jallinoja Riitta. 2011 "Obituaries as Family Assemblages." Pp. 78–91 in Families and Kinship in Contemporary Europe: Rules and Practices of Relatedness, edited by Jallinoja R., Widmer E. D. London: Palgrave Macmillan UK.</bibtext> </blist> <blist> <bibtext> Jiang Albert, Sablayrolles Alexandre, Roux Antoine, Mensch Arthur, Savary Blanche, Bamford Chris, Chaplot Devendra Singh, et al.2024. "Mixtral of experts." arXiv. arXiv:2401.04088.</bibtext> </blist> <blist> <bibtext> Joyce Kelly, Smith-Doerr Laurel, Alegria Sharla, Bell Susan, Cruz Taylor, Hoffman Steve G., Noble Safiya Umoja, Shestakofsky Benjamin. 2021. " Toward a Sociology of Artificial Intelligence: A Call for Research on Inequalities and Structural Change." Socius: Sociological Research for a Dynamic World. 7:2378023121999581. doi: 10.1177/2378023121999581.</bibtext> </blist> <blist> <bibtext> Jurafsky Daniel, Martin James H. 2025. "Information Extraction: Relations, Events, and Time." Pp. 435–60 in Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, third edition. Edited by D. Jurafsky, J. H. Martin https://web.stanford.edu/∼jurafsky/slp3/20.pdf.</bibtext> </blist> <blist> <bibtext> Karell Daniel, Sachs Jeffrey, Barrett Ryan. 2023. "Synthetic Duality: A Framework for Analyzing Natural Language Generation's Representation of Social Reality." SocArXiv. https://osf.io/htxa4.</bibtext> </blist> <blist> <bibtext> Kim Junsol, Lee Byungkyu. 2024. "AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction." arXiv. arXiv:2305.09620.</bibtext> </blist> <blist> <bibtext> Knight Carly. 2022. " When Corporations Are People: Agent Talk and the Development of Organizational Actorhood, 1890–1934." Sociological Methods & Research. 51(4):1634–80. doi: 10.1177/00491241221122528.</bibtext> </blist> <blist> <bibtext> Koleck Theresa A., Dreisbach Caitlin, Bourne Philip E., Bakken Suzanne. 2019. " Natural Language Processing of Symptoms Documented in Free-Text Narratives of Electronic Health Records: A Systematic Review." Journal of the American Medical Informatics Association. 26(4):364–79. doi: 10.1093/jamia/ocy173.</bibtext> </blist> <blist> <bibtext> Kozlowski Austin, Evans James A. 2024. "Simulating Subjects: The Promise and Peril of AI Stand-Ins for Social Agents and Interactions." OSF preprint: https://doi.org/10.31235/osf.io/vp3j2.</bibtext> </blist> <blist> <bibtext> Kozlowski Austin, Kwon Hyunku, Evans James A. 2024. "In Silico Sociology: Forecasting COVID-19 Polarization with Large Language Models." ArXiv preprint: arXiv:2407.11190.</bibtext> </blist> <blist> <bibtext> Krippendorff Klaus. 2019. Content Analysis: An Introduction to Its Methodology. CA: Sage Publications, Inc.</bibtext> </blist> <blist> <bibtext> Law Tina, Roberto Elizabeth. 2024. "Generative Multimodal Models for Social Science: an Application with Satellite and Streetscape Imagery." SocArXiv preprint: https://doi.org/10.31235/osf.io/6jq32.</bibtext> </blist> <blist> <bibtext> Lester Brian, Al-Rfou Rami, Constant Noah. 2021. "The power of scale for parameter-efficient prompt tuning." arXiv preprint arXiv:2104.08691.</bibtext> </blist> <blist> <bibtext> Li Xiang Lisa, Liang Percy. 2021. "Prefix-tuning: Optimizing continuous prompts for generation." arXiv preprint arXiv:2101.00190. Available at: https://<ulink href="http://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v5.6.6.pdf">www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v5.6.6.pdf</ulink></bibtext> </blist> <blist> <bibtext> Linguistic Data Consortium. 2024. "Annotation Tasks and Specifications." Available at: https://<ulink href="http://www.ldc.upenn.edu/collaborations/past-projects/ace/annotation-tasks-and-specifications">www.ldc.upenn.edu/collaborations/past-projects/ace/annotation-tasks-and-specifications</ulink></bibtext> </blist> <blist> <bibtext> Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Man- dar, Chen Danqi, Levy Omer, Lewis Mike, Zettlemoyer Luke, Stoyanov Veselin. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint, abs/1907.11692.</bibtext> </blist> <blist> <bibtext> Long Gary L. 1987. " Organizations and Identity: Obituaries 1856-1972." Social Forces. 65(4):964. doi: 10.2307/2579019.</bibtext> </blist> <blist> <bibtext> Ma Yubo, Cao Yixin, Hong Yong Ching, Sun Aixin. 2023. "Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 10572–10601.</bibtext> </blist> <blist> <bibtext> Makovi Kinga, Hagen Ryan, Bearman Peter. 2016. " The Course of Law: State Intervention in Southern Lynch Mob Violence 1882–1930." Sociological Science. 3:860–88. doi: 10.15195/v3.a37.</bibtext> </blist> <blist> <bibtext> Martin John Levi. 2000. " What Do Animals Do All Day?: The Division of Labor, Class Bodies, and Totemic Thinking in the Popular Imagination." Poetics. 27(2–3):195–231. doi: 10.1016/S0304-422X(99)00025-X.</bibtext> </blist> <blist> <bibtext> McFarland Daniel A., Wolff Tom. 2022. " Writing into Relationships." Social Networks. 71:96–114.</bibtext> </blist> <blist> <bibtext> Mische Ann, Pattison Philippa. 2000. " Composing a Civic Arena: Publics, Projects, and Social Settings." Poetics. 27(2):163–94. doi:10.1016/S0304-422X(99)00024-8.</bibtext> </blist> <blist> <bibtext> Mohr John W. 1994. " Soldiers, Mothers, Tramps and Others: Discourse Roles in the 1907 New York City Charity Directory." Poetics. 22(4):327–57. doi: 10.1016/0304-422X(94)90013-2.</bibtext> </blist> <blist> <bibtext> Mohr John W., Duquenne Vincent. 1997. " The Duality of Culture and Practice: Poverty Relief in New York City, 1888-1917." Theory and Society. 26(2/3):305–56.</bibtext> </blist> <blist> <bibtext> Mohr John W., Lee Helene K. 2000. " From Affirmative Action to Outreach: Discourse Shifts at the University of California." Poetics. 28(1):47–71. doi: 10.1016/S0304-422X(00)00024-3.</bibtext> </blist> <blist> <bibtext> Mohr John W., Wagner-Pacifici Robin, Breiger Ronald L., Bogdanov Petko. 2013. " Graphing the Grammar of Motives in National Security Strategies: Cultural Interpretation, Automated Text Analysis and the Drama of Global Politics." Poetics. 41(6):670–700. doi: 10.1016/j.poetic.2013.08.003.</bibtext> </blist> <blist> <bibtext> Naik Nikhil, Kominers Scott Duke, Raskar Ramesh, Glaeser Edward L., Hidalgo César A. 2017. " Computer Vision Uncovers Predictors of Physical Urban Change." Proceedings of the National Academy of Sciences. 114(29):7571–76. doi: 10.1073/pnas.1619003114.</bibtext> </blist> <blist> <bibtext> New York Times. 2022. "How does the New York Times decide who gets an obituary?". https://<ulink href="http://www.nytimes.com/article/new-york-times-obituary-process.html">www.nytimes.com/article/new-york-times-obituary-process.html</ulink>.</bibtext> </blist> <blist> <bibtext> Ollion Étienne, Shen Rubing, Macanovic Ana, Chatelain Arnault. 2024. " The Dangers of Using Proprietary LLMs for Research." Nature Machine Intelligence. 6(1):4–5. doi: 10.1038/s42256-023-00783-6.</bibtext> </blist> <blist> <bibtext> Palmer Alexis, Smith Noah A., Spirling Arthur. 2024. " Using Proprietary Language Models in Academic Research Requires Explicit Justification." Nature Computational Science. 4(1):2–3. doi: 10.1038/s43588-023-00585-1</bibtext> </blist> <blist> <bibtext> Park Joon Sung, O'Brien Joseph, Cai Carrie Jun, Morris Meredith Ringel, Liang Percy, Bernstein Michael S. 2023 "Generative Agents: Interactive Simulacra of Human Behavior." Pp. 1–22 in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST '23. New York, NY, USA: Association for Computing Machinery.</bibtext> </blist> <blist> <bibtext> Plauth Benjamin, Nguyen Khanh Xuan, Trinh Tu. 2024. "Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A." arXiv:2402.13213v2.</bibtext> </blist> <blist> <bibtext> Renze Matthew, Guven Erhan. 2024. "The effect of sampling temperature on problem solving in large language models." arXiv preprint arXiv:2402.05201.</bibtext> </blist> <blist> <bibtext> Smith Tammy. 2007. " Narrative Boundaries and the Dynamics of Ethnic Conflict and Conciliation." Poetics. 35(1):22–46. doi: 10.1016/j.poetic.2006.11.001.</bibtext> </blist> <blist> <bibtext> Spirling Arthur. 2023. " Why Open-Source Generative AI Models Are an Ethical Way Forward for Science." Nature. 616(7957):413–413. doi:10.1038/d41586-023-01295-4.</bibtext> </blist> <blist> <bibtext> Stuhler Oscar. 2021. " What's in a Category? A New Approach to Discourse Role Analysis." Poetics. 88:101568. doi:10.1016/j.poetic.2021.101568.</bibtext> </blist> <blist> <bibtext> Stuhler Oscar. 2022. " Who Does What to Whom? Making Text Parsers Work for Sociological Inquiry." Sociological Methods & Research. 51(4):1580–633.</bibtext> </blist> <blist> <bibtext> Stuhler Oscar. 2024. " The Gender Agency Gap in Fiction Writing (1850 to 2010)." Proceedings of the National Academy of Sciences. 121(29):e2319514121. doi: 10.1073/pnas.2319514121.</bibtext> </blist> <blist> <bibtext> Stuhler Oscar, Ollion Etienne, Schultz Émilien, Ton Cat Dang. 2025. "Replication Materials for: From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models." SocArXiv. doi: 10.17605/OSF.IO/HWUVS.</bibtext> </blist> <blist> <bibtext> Tilly Charles. 1995. Popular Contention in Great Britain, 1758-1834. Cambridge, MA: Harvard University Press.</bibtext> </blist> <blist> <bibtext> Tilly Charles. 1997. " Parliamentarization of Popular Contention in Great Britain, 1758-1834." Theory and Society. 26(2/3):245–73.</bibtext> </blist> <blist> <bibtext> Tilly Charles. 2002. " Event Catalogs as Theories." Sociological Theory. 20(2):248–54. doi: 10.1111/1467-9558.00161.</bibtext> </blist> <blist> <bibtext> Törnberg Petter. 2023. "ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning." arXiv. arXiv:2304.06588.</bibtext> </blist> <blist> <bibtext> Underwood Ted. 2023. "Using GPT-4 to Measure the Passage of Time in Fiction." Using Large Digital Libraries to Advance Literary History (blog). March 19. https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/.</bibtext> </blist> <blist> <bibtext> Wang Dan J., Soule Sarah A. 2012. " Social Movement Organizational Collaboration: Networks of Learning and the Diffusion of Protest Tactics, 1960-1995." American Journal of Sociology. 117(6):1674–1722. doi: https://doi.org/10.1086/664685.</bibtext> </blist> <blist> <bibtext> Wang Dan J., Soule Sarah A. 2016. " Tactical Innovation in Social Movements: The Effects of Peripheral and Multi-Issue Protest." American Sociological Review. 81(3):517–48.</bibtext> </blist> <blist> <bibtext> Wang Yanshan, Wang Liwei, Rastegar-Mojarad Majid, Moon Sungrim, Shen Feichen, Afzal Naveed, Liu Sijia, Zeng Yuqun, Mehrabi Saeed, Sohn Sunghwan, Liu Hongfang. 2018. " Clinical Information Extraction Applications: A Literature Review." Journal of Biomedical Informatics. 77:34–49. doi: 10.1016/j.jbi.2017.11.011.</bibtext> </blist> <blist> <bibtext> Wei Jason, Wang Xuezhi, Schuurmans Dale, Bosma Maarten, Xia Fei, Chi Ed, Le Quoc V., Zhou Denny. 2022. " Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems. 35:24824–37. arXiv:2201.11903.</bibtext> </blist> <blist> <bibtext> White Jules, Fu Quchen, Hays Sam, Sandborn Michael, Olea Carlos, Gilbert Henry, Elnashar Ashraf, Spencer-Smith Jesse, Schmidt Douglas C. 2023. "A prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT." arXiv:2302.11382.</bibtext> </blist> <blist> <bibtext> Zhu Xunyu, Li Jian, Liu Yong, Ma Can, Wang Weiping. 2024. "A Survey on Model Compression for Large Language Models." arXiv:2308.07633.</bibtext> </blist> </ref> <aug> <p>By Oscar Stuhler; Cat Dang Ton and Etienne Ollion</p> <p>Reported by Author; Author; Author</p> <p></p> <p>Oscar Stuhler is an Assistant Professor of Sociology at Northwestern University. He studies discourse with computational methods.</p> <p>Cat Dang Ton is a PhD student in Sociology at Northwestern University. She studies culture, computational methods, and death, particularly the afterlives of bodies, belongings, and data.</p> <p>Étienne Ollion is a Senior Researcher at CNRS and a Professor of Sociology at l'École polytechnique. His work focuses on the transformations of political fields in Europe and on integrating AI into social science.</p> </aug> <nolink nlid="nl1" bibid="bib22" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib34" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib78" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib90" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib13" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib56" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib44" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib57" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib52" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib38" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib53" firstref="ref16"></nolink> <nolink nlid="nl12" bibid="bib36" firstref="ref17"></nolink> <nolink nlid="nl13" bibid="bib20" firstref="ref18"></nolink> <nolink nlid="nl14" bibid="bib59" firstref="ref20"></nolink> <nolink nlid="nl15" bibid="bib43" firstref="ref21"></nolink> <nolink nlid="nl16" bibid="bib51" firstref="ref22"></nolink> <nolink nlid="nl17" bibid="bib30" firstref="ref23"></nolink> <nolink nlid="nl18" bibid="bib70" firstref="ref24"></nolink> <nolink nlid="nl19" bibid="bib87" firstref="ref25"></nolink> <nolink nlid="nl20" bibid="bib54" firstref="ref26"></nolink> <nolink nlid="nl21" bibid="bib37" firstref="ref27"></nolink> <nolink nlid="nl22" bibid="bib73" firstref="ref28"></nolink> <nolink nlid="nl23" bibid="bib83" firstref="ref29"></nolink> <nolink nlid="nl24" bibid="bib84" firstref="ref30"></nolink> <nolink nlid="nl25" bibid="bib85" firstref="ref31"></nolink> <nolink nlid="nl26" bibid="bib42" firstref="ref34"></nolink> <nolink nlid="nl27" bibid="bib47" firstref="ref35"></nolink> <nolink nlid="nl28" bibid="bib94" firstref="ref36"></nolink> <nolink nlid="nl29" bibid="bib55" firstref="ref37"></nolink> <nolink nlid="nl30" bibid="bib31" firstref="ref41"></nolink> <nolink nlid="nl31" bibid="bib32" firstref="ref42"></nolink> <nolink nlid="nl32" bibid="bib88" firstref="ref43"></nolink> <nolink nlid="nl33" bibid="bib92" firstref="ref44"></nolink> <nolink nlid="nl34" bibid="bib93" firstref="ref45"></nolink> <nolink nlid="nl35" bibid="bib12" firstref="ref46"></nolink> <nolink nlid="nl36" bibid="bib11" firstref="ref47"></nolink> <nolink nlid="nl37" bibid="bib66" firstref="ref48"></nolink> <nolink nlid="nl38" bibid="bib69" firstref="ref49"></nolink> <nolink nlid="nl39" bibid="bib71" firstref="ref51"></nolink> <nolink nlid="nl40" bibid="bib72" firstref="ref52"></nolink> <nolink nlid="nl41" bibid="bib67" firstref="ref53"></nolink> <nolink nlid="nl42" bibid="bib17" firstref="ref54"></nolink> <nolink nlid="nl43" bibid="bib18" firstref="ref55"></nolink> <nolink nlid="nl44" bibid="bib81" firstref="ref58"></nolink> <nolink nlid="nl45" bibid="bib89" firstref="ref66"></nolink> <nolink nlid="nl46" bibid="bib45" firstref="ref67"></nolink> <nolink nlid="nl47" bibid="bib16" firstref="ref68"></nolink> <nolink nlid="nl48" bibid="bib10" firstref="ref69"></nolink> <nolink nlid="nl49" bibid="bib28" firstref="ref70"></nolink> <nolink nlid="nl50" bibid="bib64" firstref="ref71"></nolink> <nolink nlid="nl51" bibid="bib48" firstref="ref72"></nolink> <nolink nlid="nl52" bibid="bib75" firstref="ref76"></nolink> <nolink nlid="nl53" bibid="bib58" firstref="ref78"></nolink> <nolink nlid="nl54" bibid="bib40" firstref="ref79"></nolink> <nolink nlid="nl55" bibid="bib24" firstref="ref80"></nolink> <nolink nlid="nl56" bibid="bib82" firstref="ref82"></nolink> <nolink nlid="nl57" bibid="bib77" firstref="ref85"></nolink> <nolink nlid="nl58" bibid="bib49" firstref="ref88"></nolink> <nolink nlid="nl59" bibid="bib39" firstref="ref89"></nolink> <nolink nlid="nl60" bibid="bib97" firstref="ref90"></nolink> <nolink nlid="nl61" bibid="bib35" firstref="ref92"></nolink> <nolink nlid="nl62" bibid="bib15" firstref="ref94"></nolink> <nolink nlid="nl63" bibid="bib80" firstref="ref95"></nolink> <nolink nlid="nl64" bibid="bib96" firstref="ref96"></nolink> <nolink nlid="nl65" bibid="bib95" firstref="ref98"></nolink> <nolink nlid="nl66" bibid="bib25" firstref="ref99"></nolink> <nolink nlid="nl67" bibid="bib91" firstref="ref100"></nolink> <nolink nlid="nl68" bibid="bib61" firstref="ref101"></nolink> <nolink nlid="nl69" bibid="bib60" firstref="ref102"></nolink> <nolink nlid="nl70" bibid="bib14" firstref="ref105"></nolink> <nolink nlid="nl71" bibid="bib65" firstref="ref112"></nolink> <nolink nlid="nl72" bibid="bib23" firstref="ref115"></nolink> <nolink nlid="nl73" bibid="bib41" firstref="ref118"></nolink> <nolink nlid="nl74" bibid="bib79" firstref="ref123"></nolink> <nolink nlid="nl75" bibid="bib50" firstref="ref126"></nolink> <nolink nlid="nl76" bibid="bib27" firstref="ref127"></nolink> <nolink nlid="nl77" bibid="bib26" firstref="ref128"></nolink> <nolink nlid="nl78" bibid="bib76" firstref="ref133"></nolink> <nolink nlid="nl79" bibid="bib21" firstref="ref145"></nolink> <nolink nlid="nl80" bibid="bib29" firstref="ref146"></nolink> <nolink nlid="nl81" bibid="bib68" firstref="ref147"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1475720
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Oscar+Stuhler%22">Oscar Stuhler</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7391-1743">0000-0001-7391-1743</externalLink>)<br /><searchLink fieldCode="AR" term="%22Cat+Dang+Ton%22">Cat Dang Ton</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0000-6872-3588">0009-0000-6872-3588</externalLink>)<br /><searchLink fieldCode="AR" term="%22Etienne+Ollion%22">Etienne Ollion</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-3099-5240">0000-0003-3099-5240</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2025 54(3):794-848.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 55
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Sociology%22">Sociology</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Science+Research%22">Social Science Research</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Information+Processing%22">Information Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Biographies%22">Biographies</searchLink><br /><searchLink fieldCode="DE" term="%22Open+Source+Technology%22">Open Source Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Prompting%22">Prompting</searchLink><br /><searchLink fieldCode="DE" term="%22Cues%22">Cues</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241251336794
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into "promptbooks." We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE while also underscoring risks like non-random error patterns that can bias downstream analyses.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1475720
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1475720
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241251336794
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 55
        StartPage: 794
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Sociology
        Type: general
      – SubjectFull: Social Science Research
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Information Processing
        Type: general
      – SubjectFull: Biographies
        Type: general
      – SubjectFull: Open Source Technology
        Type: general
      – SubjectFull: Prompting
        Type: general
      – SubjectFull: Cues
        Type: general
    Titles:
      – TitleFull: From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Oscar Stuhler
      – PersonEntity:
          Name:
            NameFull: Cat Dang Ton
      – PersonEntity:
          Name:
            NameFull: Etienne Ollion
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 54
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1