View in EDS HTML Full Text PDF Full Text

Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery

Saved in:
Bibliographic Details
Title: Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery
Language: English
Authors: Tina Law (ORCID 0000-0001-7631-6763), Elizabeth Roberto (ORCID 0000-0001-7667-6953)
Source: Sociological Methods & Research. 2025 54(3):889-932.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 44
Publication Date: 2025
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Artificial Intelligence, Visual Aids, Open Source Technology, Social Science Research, Neighborhood Integration, Data Analysis, Research Problems
DOI: 10.1177/00491241251339673
ISSN: 0049-1241
1552-8294
Abstract: Although there is growing social science research examining how generative AI models can be effectively and systematically applied to text-based tasks, whether and how these models can be used to analyze images remain open questions. In this article, we introduce a framework for analyzing images with generative multimodal models, which consists of three core tasks: curation, discovery, and measurement and inference. We demonstrate this framework with an empirical application that uses OpenAI's GPT-4o model to analyze satellite and streetscape images (n = 1,101) to identify built environment features that contribute to contemporary residential segregation in U.S. cities. We find that when GPT-4o is provided with well-defined image labels, the model labels images with high validity compared to expert labels. We conclude with thoughts for other use cases and discuss how social scientists can work collaboratively to ensure that image analysis with generative multimodal models is rigorous, reproducible, ethical, and sustainable.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1475739
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGsoX5Cnn0X26brRyopW9JRAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCY3RA_YMZAqZ9T3-wIBEICBm033unxtNP4_Xe2AF2mAt2Zg2ClnauUpL7w5jcD8WXlmRg1UHy9Ij836zHHiHyPteWweOp__lzfFn87UDT6Ncd6i-9lMNuxf71d4-uqitVQlKZRKSkMAg3T9cthqN7lDk_Pmm6N7Q2Tbnuzd6qNaYtSxOBiEEB4veDmv2o2BwQ2lUBP-ca_EcTMtbBG2k4WQwi0f_tLMIbPRXKOP
Text:
  Availability: 1
  Value: <anid>AN0186128776;som01aug.25;2025Jun26.00:54;v2.2.500</anid> <title id="AN0186128776-1">Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery </title> <p>Although there is growing social science research examining how generative AI models can be effectively and systematically applied to text-based tasks, whether and how these models can be used to analyze images remain open questions. In this article, we introduce a framework for analyzing images with generative multimodal models, which consists of three core tasks: curation, discovery, and measurement and inference. We demonstrate this framework with an empirical application that uses OpenAI's GPT-4o model to analyze satellite and streetscape images (n = 1,101) to identify built environment features that contribute to contemporary residential segregation in U.S. cities. We find that when GPT-4o is provided with well-defined image labels, the model labels images with high validity compared to expert labels. We conclude with thoughts for other use cases and discuss how social scientists can work collaboratively to ensure that image analysis with generative multimodal models is rigorous, reproducible, ethical, and sustainable.</p> <p>Keywords: artificial intelligence; generative multimodal models; image as data; computer vision; prompt development</p> <hd id="AN0186128776-2">Introduction</hd> <p>In March 2023, OpenAI publicly released GPT-4o ([<reflink idref="bib1" id="ref1">1</reflink>]), a generative multimodal model that is capable of understanding not just text but also images. There is rapidly growing research on how generative large language models (LLMs) can be applied for text-based tasks in social science research (e.g., [<reflink idref="bib3" id="ref2">3</reflink>]; [<reflink idref="bib48" id="ref3">48</reflink>]; [<reflink idref="bib86" id="ref4">86</reflink>]). However, there has been, to date, only exploratory assessments of generative multimodal models and their applicability for image-based tasks in the social sciences ([<reflink idref="bib4" id="ref5">4</reflink>]; [<reflink idref="bib19" id="ref6">19</reflink>]; [<reflink idref="bib96" id="ref7">96</reflink>]). As a result, it is currently unclear whether and how generative multimodal models can be used to accurately analyze images and in particular, whether these models can overcome longstanding technical challenges in computer vision that have limited large-scale image analysis ([<reflink idref="bib11" id="ref8">11</reflink>]; [<reflink idref="bib25" id="ref9">25</reflink>]), especially in the social sciences.</p> <p>In this article, we begin to examine the opportunities and challenges of using generative multimodal models (MMs) for large-scale image analysis in social science research. We start by drawing on previous research in computer science and the social sciences to orient social scientists, who may have varying levels of familiarity with computer vision, to key developments and challenges in large-scale image analysis. We then introduce a framework that provides guidance for analyzing images with generative MMs, which extends and adapts an existing framework for computational text analysis ([<reflink idref="bib34" id="ref10">34</reflink>]). We demonstrate this framework with an empirical application that uses OpenAI's GPT-4o model to analyze satellite and streetscape images (<emph>n</emph> = 1,101 images) from the Google Maps API and the Google Cloud Platform, with the aim of providing new insight into the role of the built environment in contributing to contemporary residential segregation in the United States. Overall, we find that when it comes to analyzing satellite and streetscape images, GPT-4o generates labels that are reasonably valid compared to expert-generated labels, though the model output is still not completely deterministic. We conclude with thoughts for other use cases as well as how researchers can ensure that large-scale image analysis in the generative AI era is rigorous, reproducible, ethical, and sustainable.</p> <hd id="AN0186128776-3">Prior Research on Large-Scale Image Analysis</hd> <p>In this section, we review previous research on large-scale image analysis. We begin by providing a brief overview of the technical innovations and challenges that organize computer vision research, and then we proceed to discuss implications for applications in social science research. Our goal in this section is to sufficiently contextualize why recent innovations in generative multimodal models present new opportunities for social scientists to engage in large-scale image analysis.</p> <p>In the field of computer science, vision has been and continues to be one of the most challenging human abilities to computationally emulate ([<reflink idref="bib11" id="ref11">11</reflink>]; see also [<reflink idref="bib64" id="ref12">64</reflink>]; [<reflink idref="bib87" id="ref13">87</reflink>]; [<reflink idref="bib25" id="ref14">25</reflink>]). It has been technically difficult to mimic the embodied and contextual manner in which humans understand what they see ([<reflink idref="bib11" id="ref15">11</reflink>]). Moreover, even when computer scientists have been able to translate an aspect of human sight into a specific computational technique, they traditionally require significant computing resources in order to train models on images, which are exceedingly high-dimensional ([<reflink idref="bib11" id="ref16">11</reflink>]).</p> <p>Research on computer vision has long focused on optimizing task-specific engineering, or training models to accurately, reliably, and efficiently carry out specific information extraction tasks. Table 1 summarizes the most common computer vision tasks to date (for reviews, see [<reflink idref="bib11" id="ref17">11</reflink>]; [<reflink idref="bib66" id="ref18">66</reflink>]; [<reflink idref="bib80" id="ref19">80</reflink>]), with examples of existing and potential applications in social science analyses of satellite and streetscape images. Social scientists often carry out multiple tasks in order to analyze a set of images; for example, researchers may want to first identify objects of interest within a neighborhood, such as cars, and then classify detected objects in order to predict neighborhood socioeconomic characteristics based on the types of cars that are present ([<reflink idref="bib28" id="ref20">28</reflink>]).</p> <p>Table 1. Common Computer Vision Tasks.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Task</th><th align="left">Goal</th><th align="left">Applied Example</th></tr></thead><tbody><tr><td>Object recognition</td><td>To label an image as containing (or not containing) an object of interest or multiple objects of interest</td><td>Classifying the level of trash present in a neighborhood (<xref ref-type="bibr" rid="bibr39">Hwang et al. 2023</xref>)</td></tr><tr><td>Object detection</td><td>To label all objects within an image</td><td>Identifying the presence of cars in a neighborhood (<xref ref-type="bibr" rid="bibr28">Gebru et al. 2017</xref>)</td></tr><tr><td>Facial recognition</td><td>To label an image as containing (or not containing) a person of interest or multiple persons of interest</td><td>Identifying presence of individuals in parks as measure of local park use</td></tr><tr><td>Facial detection</td><td>To label all persons within an image</td><td>Identifying individuals on the street in a neighborhood as measure of local daytime population density</td></tr><tr><td>Image segmentation</td><td>To identify the outline of an object/person of interest or multiple objects/persons of interest</td><td>Identifying built environment features theorized to be associated with local crime (<xref ref-type="bibr" rid="bibr38">Hipp et al. 2025</xref>)</td></tr><tr><td>Human activity recognition</td><td>To identify an activity that humans are engaged in within a video</td><td>Identifying racial avoidance among pedestrians in a neighborhood (<xref ref-type="bibr" rid="bibr22">Dietrich and Sands 2023</xref>)</td></tr><tr><td>Human pose estimation</td><td>To identify human body postures within a video</td><td>Identifying body postures of cyclists as measure of safety of local roads for cyclists</td></tr><tr><td>Optical character recognition</td><td>To extract text in a machine-readable format from within an image</td><td>Extracting names of stores in a neighborhood</td></tr></tbody></table> </ephtml> </p> <p>In 2009, a team of computer scientists developed ImageNet ([<reflink idref="bib21" id="ref21">21</reflink>]), a hierarchical image database consisting of over three million manually labeled and high-quality images mapped to over 5,000 concepts. This was a significant development that, coupled with a contemporaneous increase in availability of graphics processing units (GPUs) for model training (e.g., [<reflink idref="bib70" id="ref22">70</reflink>]), enabled computer vision researchers to shift from task-specific engineering to developing deep learning models ([<reflink idref="bib11" id="ref23">11</reflink>]; see also [<reflink idref="bib56" id="ref24">56</reflink>]; [<reflink idref="bib91" id="ref25">91</reflink>]). Deep learning models differ from conventional machine learning models in how they read and learn images. While conventional machine learning models read and learn images by extracting and transforming features into a representation that has been carefully engineered by humans and therefore requires considerable dimension reduction, deep learning models can use raw images to "discover the representations needed for detection or classification" ([<reflink idref="bib56" id="ref26">56</reflink>]: 436). Because of their complex architecture, deep learning models can obtain robust and generalizable knowledge about images after pre-training, and can then be efficiently adapted to learn new image datasets through fine-tuning ([<reflink idref="bib56" id="ref27">56</reflink>]; [<reflink idref="bib91" id="ref28">91</reflink>]).</p> <p>There are several different types of deep learning vision models, with convolutional neural networks being used most commonly by social scientists.[<reflink idref="bib6" id="ref29">6</reflink>] Convolutional neural networks (CNNs) ([<reflink idref="bib51" id="ref30">51</reflink>]; [<reflink idref="bib57" id="ref31">57</reflink>]; [<reflink idref="bib58" id="ref32">58</reflink>]) are deep learning models that have a distinctive architecture that enables them to quickly and efficiently learn data containing multiple arrays, such as images, each of which may contain hundreds or thousands of pixels for three different matrices corresponding to the colors red, green, and blue (for reviews, see [<reflink idref="bib56" id="ref33">56</reflink>]; [<reflink idref="bib88" id="ref34">88</reflink>]; [<reflink idref="bib93" id="ref35">93</reflink>]). CNNs, which have been available since the early 1990s, became more widely used for computer vision tasks starting in 2012 and have quickly become "the dominant approach for almost all recognition and detection tasks" over the past decade ([<reflink idref="bib56" id="ref36">56</reflink>]; [<reflink idref="bib91" id="ref37">91</reflink>]).</p> <p>CNNs have facilitated increased engagement in large-scale image analysis among social scientists in recent years, particularly among computational social scientists who leverage computational methods to pursue social science research questions ([<reflink idref="bib23" id="ref38">23</reflink>]; [<reflink idref="bib55" id="ref39">55</reflink>]; [<reflink idref="bib77" id="ref40">77</reflink>]; see also [<reflink idref="bib45" id="ref41">45</reflink>]). As [<reflink idref="bib93" id="ref42">93</reflink>] explain in their excellent introductory text on CNNs, the high predictive accuracy of pre-trained CNNs typically makes it unnecessary for social scientists to train a CNN from scratch and they can instead fine-tune a pre-trained CNN, many of which are available in open-source machine learning libraries such as Pytorch's torchvision ([<reflink idref="bib62" id="ref43">62</reflink>]). Indeed, "[w]hereas the original algorithm may have been the product of many months of effort using millions of labeled examples, this <emph>fine-tuning</emph> or <emph>transfer learning</emph> can produce remarkably accurate results using a much smaller training set of images (as few as 100 in some cases)" ([<reflink idref="bib93" id="ref44">93</reflink>]: 9, italics in original). Social scientists have used CNNs to study neighborhood change and inequality ([<reflink idref="bib40" id="ref45">40</reflink>]; [<reflink idref="bib39" id="ref46">39</reflink>]; [<reflink idref="bib47" id="ref47">47</reflink>]), crime ([<reflink idref="bib38" id="ref48">38</reflink>]), collective action ([<reflink idref="bib97" id="ref49">97</reflink>]), and election fraud ([<reflink idref="bib14" id="ref50">14</reflink>]), as well as used CNNs to develop new methodological techniques for analyzing large image ([<reflink idref="bib98" id="ref51">98</reflink>]) and video ([<reflink idref="bib7" id="ref52">7</reflink>]; [<reflink idref="bib29" id="ref53">29</reflink>]) datasets.[<reflink idref="bib7" id="ref54">7</reflink>]</p> <p>Although pre-trained large computer vision models have substantially decreased the amount of manually labeled images, computing power, and training time that social scientists must invest in order to analyze images, these models have not been widely adopted. This is partly because even fine-tuning a pre-trained CNN requires a nontrivial amount of technical expertise. For example, in order to fine-tune a pre-trained CNN, researchers need to first experiment with hyperparameter values, such as the learning rate and the number of iterations and epochs, and then retrain model parameters—often with several different pre-trained models or the same pre-trained model with different numbers of layers ([<reflink idref="bib93" id="ref55">93</reflink>]). Fine-tuning a pre-trained CNN also requires researchers to preprocess images, which includes resizing images and normalizing color intensities, as the performance of CNNs is contingent on how objects of interest are positioned and contrasted in images ([<reflink idref="bib88" id="ref56">88</reflink>]; [<reflink idref="bib93" id="ref57">93</reflink>]). Moreover, fine-tuned CNNs still struggle to accurately and reliably classify images when the differences between classes are more subtle (e.g., distinguishing between diverse types of protests) or involve more complex reasoning (e.g., identifying evoked emotions) ([<reflink idref="bib88" id="ref58">88</reflink>]; [<reflink idref="bib93" id="ref59">93</reflink>]).</p> <p>Recently developed generative artificial intelligence models, also known as foundation models, offer new possibilities for large-scale image analysis in the social sciences. Foundation models are models that are "trained on broad data (generally using self-supervision at scale) that can be adapted (fine-tuned) to a wide range of downstream tasks" ([<reflink idref="bib8" id="ref60">8</reflink>]:3). While foundation models rely on deep learning, they differ from earlier deep learning models in that they are trained on substantially more data, can be programmed to understand non-technical input, and have generative capabilities (for an in-depth overview, see [<reflink idref="bib8" id="ref61">8</reflink>]; see also [<reflink idref="bib43" id="ref62">43</reflink>] for an accessible primer).</p> <p>There are currently two types of foundation models. Generative large language models (LLMs) are one type of foundation model that can understand text input and generate text output, and they can also receive input in natural language via chatbot interfaces and prompts. One of the first generative LLMs made widely available was OpenAI's GPT-3 model ([<reflink idref="bib10" id="ref63">10</reflink>]), which was released in 2020 and available for access via ChatGPT in 2022.[<reflink idref="bib8" id="ref64">8</reflink>] There are now many generative LLMs available, including proprietary models, such as OpenAI's GPT-4 model (released in 2023), and open-source models, such as Meta's Llama 3 model (released in 2024).</p> <p>Another type of foundation model is generative multimodal models (MMs). Generative MMs can interact with multiple modalities (e.g., text, image, audio), understanding input in one or more modalities and generating output in one modality. In this article, we focus on generative MMs that can understand text and image input and generate text output, though we note that there are a growing number of generative MMs with other capabilities.[<reflink idref="bib9" id="ref65">9</reflink>] Compared to generative LLMs, generative MMs are still "early-stage," but computer scientists have obtained promising results thus far in their assessments of generative MMs and their ability to complete traditional computer vision tasks and more complex computer vision tasks involving generalizability—often matching or exceeding the performance of fully supervised models ([<reflink idref="bib17" id="ref66">17</reflink>]; [<reflink idref="bib36" id="ref67">36</reflink>]; [<reflink idref="bib37" id="ref68">37</reflink>]; [<reflink idref="bib69" id="ref69">69</reflink>]; [<reflink idref="bib72" id="ref70">72</reflink>]; see also [<reflink idref="bib11" id="ref71">11</reflink>]). These promising early results, along with strong commercial incentives to invest in continued development of these models, suggest that generative MMs may eventually "reduce dependence on explicit annotations," which will not only make traditional computer vision tasks more efficient and more generalizable but also "lead to progress on essential cognitive skills (e.g., commonsense reasoning) which have proven difficult in the current, fully-supervised paradigm" ([<reflink idref="bib11" id="ref72">11</reflink>]: 29).</p> <p>Compared to generative LLMs, there are fewer generative MMs, though more are likely to be available in the near future given growing investments in generative MM development. Table 2 summarizes four leading generative MMs: Google Gemini 1.0, Anthropic Claude 3, OpenAI GPT-4o, and Meta Llama 3.2. These models vary in terms of their architecture (specifically how many modalities they are designed to understand and how they understand modalities in relation to one another), whether they are closed or open-source, and whether they allow for fine-tuning. Each of these four models also comes in multiple versions that vary in parameter size. In terms of performance, generative MMs are often evaluated in computer science using MMMU ([<reflink idref="bib95" id="ref73">95</reflink>]), which is a benchmark that assesses the ability of generative MMs to correctly answer a series of multimodal college-level problems spanning multiple disciplines. Based on self-reported MMMU benchmarking, GPT-4o, which is based on GPT-4 ([<reflink idref="bib1" id="ref74">1</reflink>]), currently performs best when it comes to multimodal question answering.[<reflink idref="bib10" id="ref75">10</reflink>]</p> <p>Table 2. Leading Generative Multimodal Models.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Model</th><th align="left">Initial Release Date</th><th align="left">Input Mode</th><th align="left">Output Mode</th><th align="left">Self-Reported MMMU</th><th align="left">Closed vs. Open Source</th><th align="left">Fine-Tuning Options</th></tr></thead><tbody><tr><td>Google Gemini 1.0</td><td>December 13, 2023</td><td>Text, image, audio, video</td><td>Text</td><td>59.4%(Ultra model)</td><td>Closed</td><td>No</td></tr><tr><td>Anthropic Claude 3</td><td>March 13, 2024</td><td>Text, images</td><td>Text</td><td>59.4%(Opus model)</td><td>Closed</td><td>No</td></tr><tr><td>OpenAI GPT-4o</td><td>May 13, 2024</td><td>Text, images</td><td>Text</td><td>69.1%</td><td>Closed</td><td>Yes</td></tr><tr><td>Meta Llama 3.2</td><td>September 25, 2024</td><td>Text, images</td><td>Text</td><td>60.3%(90B model)</td><td>Open</td><td>Yes</td></tr></tbody></table> </ephtml> </p> <p>While newly developed generative MMs appear promising, these models remain empirically untested in the social sciences. Although there is growing work in the social sciences to examine how generative LLMs can be used to analyze text data (e.g., [<reflink idref="bib3" id="ref76">3</reflink>]; [<reflink idref="bib48" id="ref77">48</reflink>]; [<reflink idref="bib86" id="ref78">86</reflink>]), there have been, to date, only exploratory assessments of generative MMs and their image analysis capabilities ([<reflink idref="bib4" id="ref79">4</reflink>]; [<reflink idref="bib19" id="ref80">19</reflink>]; [<reflink idref="bib96" id="ref81">96</reflink>]). As a result, there are still many unanswered questions about generative MMs and their applicability for image analysis tasks in social science research.</p> <hd id="AN0186128776-4">Satellite and Streetscape Imagery as Social Science Data</hd> <p>Newly developed generative MMs have the potential to make large-scale image analysis much more efficient and accessible, but these models still need to be systematically tested for applications in the social sciences. We test one potential application: using generative MMs to analyze satellite and streetscape images to identify built environment features that contribute to social and spatial division in U.S. cities. This application focuses on an important and understudied social science topic for which images and generative MMs can potentially offer new insight: the role of the built environment in shaping contemporary residential segregation in the United States (see [<reflink idref="bib2" id="ref82">2</reflink>]; [<reflink idref="bib5" id="ref83">5</reflink>]; [<reflink idref="bib32" id="ref84">32</reflink>], [<reflink idref="bib33" id="ref85">33</reflink>]; [<reflink idref="bib73" id="ref86">73</reflink>]; [<reflink idref="bib74" id="ref87">74</reflink>]; [<reflink idref="bib49" id="ref88">49</reflink>]).</p> <p>Satellite and streetscape imagery contain rich information about many topics that are central to the social sciences. For example, streetscape images from sources such as Google Streetview have been used in previous studies to examine neighborhood inequality and change ([<reflink idref="bib41" id="ref89">41</reflink>]; [<reflink idref="bib40" id="ref90">40</reflink>]; [<reflink idref="bib39" id="ref91">39</reflink>]; [<reflink idref="bib47" id="ref92">47</reflink>]), housing affordability and homelessness ([<reflink idref="bib26" id="ref93">26</reflink>]; [<reflink idref="bib60" id="ref94">60</reflink>]), social determinants of health ([<reflink idref="bib67" id="ref95">67</reflink>]), voting behavior ([<reflink idref="bib28" id="ref96">28</reflink>]), and surveillance ([<reflink idref="bib81" id="ref97">81</reflink>]). To a lesser extent, satellite images have been used to investigate social science topics such as population estimation ([<reflink idref="bib76" id="ref98">76</reflink>]) and climate change ([<reflink idref="bib85" id="ref99">85</reflink>]). To date, social scientists have analyzed streetscape and satellite images using conventional machine learning models, which require task-specific training that is computationally-, cost-, and time-intensive, or convolutional neural networks, which often only require fine-tuning but still demand considerable technical expertise.</p> <p>Generative MMs equipped with more robust and generalizable computer vision capabilities, therefore, present new opportunities for social scientists to use satellite and streetscape images as data. Although tech firms have generally not disclosed their training data for developing generative MMs (even for open-source models), it is likely that satellite and streetscape images are included in training data since they are a readily available type of large image data.[<reflink idref="bib11" id="ref100">11</reflink>] There are also strong commercial interests to train generative MMs to analyze satellite and streetscape imagery to advance lucrative applications such as self-driving cars ([<reflink idref="bib11" id="ref101">11</reflink>]). Because of these factors, satellite and streetscape imagery may be particularly amenable to analysis with generative MMs, compared to other types of images of interest to social scientists, such as images of protest activities shared on social media (e.g., [<reflink idref="bib15" id="ref102">15</reflink>]; [<reflink idref="bib97" id="ref103">97</reflink>]) and images of political leaders (e.g., [<reflink idref="bib79" id="ref104">79</reflink>]).</p> <p>In the following section, we introduce a framework for analyzing satellite and streetscape imagery with OpenAI's GPT-4o model, which is currently the best-performing generative MM among leading models based on multimodal question answering ability. Our framework extends and adapts a familiar framework for computational text analysis ([<reflink idref="bib34" id="ref105">34</reflink>]) to provide social scientists with guidance for conducting computational image analysis, specifically with generative MMs.</p> <hd id="AN0186128776-5">A Social Science Framework for Image Analysis with Generative Multimodal Models</hd> <p>Social scientists increasingly recognize that although computational methods developed by computer scientists can be useful for studying social, political, and economic phenomena, their research goals differ in important ways from those of computer scientists and, therefore, distinct frameworks are needed to guide the use of computational methods in social science research ([<reflink idref="bib9" id="ref106">9</reflink>]; [<reflink idref="bib45" id="ref107">45</reflink>]; [<reflink idref="bib77" id="ref108">77</reflink>]). To date, one of the most widely used computational social science frameworks is [<reflink idref="bib34" id="ref109">34</reflink>] "agnostic approach" to text analysis. The approach defines four core tasks in computational text analysis: selection and representation, discovery, measurement, and inference. These tasks are guided by an underlying assumption of modeling agnosticism—that because "there are no "true" values for us to target with text-as-data methods, or no one best model that can be used for all applications of a corpus," researchers should identify the most useful organization of text data based on their research question and then validate their approach based on its ability to capture this organization ([<reflink idref="bib34" id="ref110">34</reflink>]: 18). This agnostic approach to text analysis extends [<reflink idref="bib50" id="ref111">50</reflink>]: 29, italics in original) foundational framework for content analysis, which posits that texts do not contain inherent meaning but instead must be interpreted by researchers to "<emph>have meanings</emph> [which are] <emph>relative to particular contexts, discourses, or purposes.</emph>"</p> <p>We argue that although [<reflink idref="bib34" id="ref112">34</reflink>] agnostic approach was designed for computational text analysis, it can be extended to provide a framework for conducting computational <emph>image</emph> analysis. Like text, images also do not contain inherent meaning but instead must be interpreted to have meanings that are necessarily context-specific. Arguably, images are more open to interpretation compared to text because they are comprised of more "latent" content and less "manifest" content ([<reflink idref="bib50" id="ref113">50</reflink>]); for example, while the building block of a text document (a word) may carry latent and manifest content, the building block of an image (a pixel) is entirely latent in content. As such, an agnostic approach, where researchers define a useful organization of data and then validate an approach based on its ability to obtain this organization, is perhaps even more relevant when analyzing images.</p> <p>With some modifications, their four core tasks can also be used to organize computational image analysis. The need to first define research questions and gather relevant data, captured in the selection and representation task, also applies to any computational image analysis project. The discovery, measurement, and inference tasks are also applicable to any computational image analysis project, though the specific steps involved may differ depending on the type of computer vision method that is used. For example, the steps involved in fine-tuning a pre-trained deep learning vision model are distinct from the steps involved in using a generative MM. Although there is guidance for social scientists on measuring concepts of interest and making inferences with deep learning vision models, similar guidance does not currently exist for how to carry out such tasks with generative MMs.</p> <p>Accordingly, we extend [<reflink idref="bib34" id="ref114">34</reflink>] agnostic approach to provide a social science framework for computational image analysis, specifically with generative MMs. Our framework, which is visualized in Figure 1, consists of three core tasks: (<reflink idref="bib1" id="ref115">1</reflink>) curation, (<reflink idref="bib2" id="ref116">2</reflink>) discovery, and (<reflink idref="bib3" id="ref117">3</reflink>) measurement and inference. We introduce this framework using our empirical application, which uses OpenAI's GPT-4o model to analyze satellite and streetscape images.</p> <p>Graph: Figure 1. A social science framework for image analysis with generative multimodal models.</p> <hd id="AN0186128776-6">Curation</hd> <p>As with any research project, social scientists should begin a computational image analysis project by defining their research question and collecting relevant data. Given that image data are highly diverse and new sources of image data are increasingly available (for reviews, see [<reflink idref="bib88" id="ref118">88</reflink>]; [<reflink idref="bib93" id="ref119">93</reflink>]), we offer some general guidance for collecting image data. In particular, we emphasize that social scientists should collect image data in a systematic and principled manner.</p> <p>Images introduce unique considerations in terms of <emph>systematic</emph> data collection. Namely, we echo advice from previous research involving large-scale image analysis ([<reflink idref="bib40" id="ref120">40</reflink>]; [<reflink idref="bib39" id="ref121">39</reflink>]; [<reflink idref="bib93" id="ref122">93</reflink>]) that social scientists should ensure that their images are consistent in size, scale, and resolution. Application programming interfaces (APIs) can make image data collection more systematic (and perhaps also more reproducible), though images collected in this way should always be manually screened for legibility (e.g., blurriness, obstructed views) and errors. However, the ability of researchers to collect images in a systematic manner may be limited in certain cases, such as when researchers collect images from social media or archival sources. In these cases, researchers will want to carefully consider potential biases they are inheriting in their collections.</p> <p>Images also raise new considerations for <emph>principled</emph> data collection. Here, we argue that [<reflink idref="bib34" id="ref123">34</reflink>] principles for corpus construction can also be relevant for image collection curation. Within the context of collecting images, these principles recognize that: (<reflink idref="bib1" id="ref124">1</reflink>) a collection must be useful for answering the research question, (<reflink idref="bib2" id="ref125">2</reflink>) curating images implies ethical obligations, (<reflink idref="bib3" id="ref126">3</reflink>) there may be multiple ways to represent images, and (<reflink idref="bib4" id="ref127">4</reflink>) validation is necessary. In particular, the second principle takes on increased importance when working with image data. Although text data may present ethical concerns in terms of institutional representation and omission, racial and gender bias, and privacy and informed consent, these concerns are more acute when working with image data—especially images of humans—given well-documented and pervasive biases in large image datasets as well as their greater susceptibility to being re-identified and misused for harmful applications ([<reflink idref="bib12" id="ref128">12</reflink>]; [<reflink idref="bib13" id="ref129">13</reflink>]).</p> <p>For our empirical application, our research question is: what types of built environment features contribute to social and spatial division in U.S. neighborhoods? Based on this research question, we collected streetscape and satellite images that capture the built environment of socially and spatially divided sites located within U.S. cities. We curated our image dataset through three main steps. First, we selected five cities that vary in demographics (e.g., population size, racial composition) and spatial characteristics: Chicago, IL, Cincinnati, OH, Hartford, CT, Miami, FL, and Seattle, WA (see Table 3).</p> <p>Table 3. Summary of Sites.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left">City</th><th align="left">Population<sup>a</sup></th><th align="left">Prop. Black</th><th align="left">Prop. Hispanic</th><th align="left">Prop. White</th><th align="left">All Potential Sites</th><th align="left">Selected Sites<xref ref-type="table-fn" rid="tfn2">b</xref></th><th align="left">Selected Sites With Images</th></tr></thead><tbody><tr><td>Chicago</td><td>2,694,971</td><td>0.324</td><td>0.289</td><td>0.317</td><td>5,496</td><td>158</td><td>138</td></tr><tr><td>Cincinnati</td><td>304,286</td><td>0.440</td><td>0.028</td><td>0.487</td><td>3,174</td><td>116</td><td>96</td></tr><tr><td>Hartford</td><td>126,785</td><td>0.352</td><td>0.431</td><td>0.162</td><td>477</td><td>32</td><td>22</td></tr><tr><td>Miami</td><td>398,507</td><td>0.163</td><td>0.700</td><td>0.119</td><td>835</td><td>29</td><td>24</td></tr><tr><td>Seattle</td><td>608,617</td><td>0.077</td><td>0.066</td><td>0.663</td><td>4,755</td><td>91</td><td>87</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: <sups>a</sups>Demographic information comes from the 2010 decennial census ([<reflink idref="bib89" id="ref130">89</reflink>]). Census categories for race and ethnicity are used to define the three race groups used in the analysis: Black (non-Hispanic), Hispanic (any race), and White (non-Hispanic).</p> <p>2 We were unable to obtain two streetscape images for 57 of the sites and two sites had images that could not be properly encoded across multiple runs. As a result, our final dataset consists of 367 sites.</p> <p>Second, we used [<reflink idref="bib75" id="ref131">75</reflink>] counterfactual road networks (CRN) method along with population and geographic data from the 2010 decennial census ([<reflink idref="bib89" id="ref132">89</reflink>], [<reflink idref="bib90" id="ref133">90</reflink>]) to identify sites in these cities that are divided spatially and socially. The CRN method first characterizes the existing road network using a variety of features, namely, the maximum straight-line distance (i.e., the maximum distance between two intersections directly connected by a road—using the language of graph theory, the intersections are the nodes and the road segments are the edges in the network), shadow angle threshold (i.e., the typical level of co-linearity between connected nodes), and neighbor angle threshold (i.e., the typical angles in connected v-shaped motifs). CRN uses these features, along with road classifications, to constrain the set of new edge "candidates" (i.e., road segments that are not currently in the network but could plausibly exist). After removing redundant edge candidates, CRN assesses the utility of each edge in terms of how much connectivity it contributes to the local network. [<reflink idref="bib75" id="ref134">75</reflink>] show that the unexpected disconnectivity in a city's road network due to these missing road segments is associated with greater differences in racial composition between spatially proximate areas and is associated with higher levels of segregation in the local areas of missing road segments and at the city level.</p> <p>Using the CRN method, we derived a set of candidate edges, or counterfactual road segments, for each of the five cities. We selected a subset of these counterfactual road segments that are most likely to be associated with social and spatial division using two criteria: (<reflink idref="bib1" id="ref135">1</reflink>) potential to improve the connectivity of residential roads[<reflink idref="bib12" id="ref136">12</reflink>] and (<reflink idref="bib2" id="ref137">2</reflink>) connects areas with different racial compositions.[<reflink idref="bib13" id="ref138">13</reflink>] These counterfactual road segments—more specifically, the geographic coordinates of where these roads would be located within a city—serve as our sites of spatial and social division. Figure 2 shows a stylized example of a counterfactual road segment.</p> <p>Graph: Figure 2. Stylized example of a counterfactual road segment.</p> <p>Third, using geographic coordinates for the sites of spatial and social division, we collected satellite and streetscape images in 2024 from the Google Maps API and the Google Cloud Platform ([<reflink idref="bib30" id="ref139">30</reflink>], [<reflink idref="bib31" id="ref140">31</reflink>]). We collected images of built environments in two different visual representations (satellite and streetscape) because we hypothesize that satellite and streetscape images offer different insights about the relationship between the built environment and segregation; for example, whereas satellite images may show larger features that obstruct connectivity for residents, such as highways, streetscape images may show smaller features that may be similarly obstructive, such as fencing. For each site, we collected one satellite image centered on where the counterfactual road segment would be located and two streetscape images from the vantage point of each endpoint of the counterfactual road segment and looking toward the other endpoint. We added red lines to the satellite images to indicate where counterfactual road segments would be present.[<reflink idref="bib14" id="ref141">14</reflink>]Figure 3 shows the satellite and streetscape images for a single site in Chicago. Our final dataset (see Table 3) consists of 1,101 images from 367 sites that are consistent in size and image resolution (the maximum allowed by the APIs).</p> <p>Graph: Figure 3. Example of satellite and streetscape images for a site in Chicago, IL.</p> <hd id="AN0186128776-7">Discovery</hd> <p>After curating a relevant image collection, social scientists should select a method for analyzing images and proceed to develop image labels and model prompts. This part of the computational image analysis framework focuses on operationalizing key concepts, which will require social scientists to "simplify the highly complex world that we live in to study one or two specific aspects of it" ([<reflink idref="bib34" id="ref142">34</reflink>]: 16). With text analysis, researchers often operationalize concepts as specific words that capture specific topics or sentiments. With image analysis, researchers can operationalize concepts as specific objects, which may be present or absent, contain particular characteristics, vary in size or quantity, and/or appear with other objects. For example, Hwang and colleagues used the presence and amount of trash in neighborhoods to examine neighborhood conditions ([<reflink idref="bib40" id="ref143">40</reflink>]; [<reflink idref="bib39" id="ref144">39</reflink>]), while [<reflink idref="bib28" id="ref145">28</reflink>] used the make, model, and year of cars to examine local voting behavior.</p> <p>As we previously discussed, social scientists conducting computational image analysis projects currently have three main options for methods: (<reflink idref="bib1" id="ref146">1</reflink>) training a fully-supervised machine learning model, (<reflink idref="bib2" id="ref147">2</reflink>) fine-tuning a pre-trained CNN or other type of deep learning vision model, or (<reflink idref="bib3" id="ref148">3</reflink>) applying a generative MM. For social scientists interested in training a fully-supervised machine learning model or fine tuning a pre-trained deep learning vision model, there are several resources that provide excellent methodological guidance (e.g., [<reflink idref="bib14" id="ref149">14</reflink>]; [<reflink idref="bib39" id="ref150">39</reflink>]; [<reflink idref="bib80" id="ref151">80</reflink>]; [<reflink idref="bib88" id="ref152">88</reflink>]; [<reflink idref="bib93" id="ref153">93</reflink>]). For those interested in applying a generative MM, we outline two key sub-tasks—developing image labels and model prompts—and provide guidance for carrying out these sub-tasks.</p> <p>The first sub-task is to create and test image labels with human coders and generative MMs. Social scientists should begin by developing a theory-driven list of labels that can be suitably applied to their image collection, drawing on labels used in previous research if available. Social scientists can then use generative MMs to identify additional labels. This inductive use of generative MMs may be particularly useful for computational image analysis projects focused on comprehensively identifying features related to an understudied topic. Social scientists should then review and select a final set of labels, which are tested by human coders and revised as needed.</p> <p>In our empirical application, we operationalized the built environment as a set of physical features. We identified 18 physical features (see Table 4) through a multi-step process that involved human coders and ChatGPT Plus. We first developed a preliminary list of 31 labels based on features identified in previous research as physical barriers in cities (e.g., multi-lane highway or freeway, railroad tracks) ([<reflink idref="bib42" id="ref154">42</reflink>]; [<reflink idref="bib63" id="ref155">63</reflink>]; [<reflink idref="bib78" id="ref156">78</reflink>]; [<reflink idref="bib84" id="ref157">84</reflink>]) and features identified by one author's manual coding of a subset of satellite images from three cities. Then we compiled a preliminary list of labels generated by ChatGPT Plus[<reflink idref="bib15" id="ref158">15</reflink>] via an open-ended analysis of a subset of satellite images from three cities. We found that while there was considerable overlap between the manually generated labels and the ChatGPT Plus-generated labels, ChatGPT Plus generated some new labels for features we did not initially consider examining (e.g., parking facilities).[<reflink idref="bib16" id="ref159">16</reflink>] Note that we opted to use ChatGPT Plus to compile model-generated labels because there was limited information available about the GPT-4 series of multimodal models at the time of our analysis and we wanted to test image analysis capabilities with the chatbot first before purchasing access to the API. We would now encourage researchers to use API calls to test labels (if budgets and API rate limits allow) because they are easier to document and compare.</p> <p>Table 4. Built Environment Features.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /></colgroup><tbody><tr><td><list list-type="Bullet"><list-item><p>Multi-lane highway or freeway</p></list-item><list-item><p>Local or residential road</p></list-item><list-item><p>Railroad track</p></list-item><list-item><p>Other transportation infrastructure</p></list-item><list-item><p>Physical barrier (includes guardrail, bollard, fencing, wall, or other physical barrier)</p></list-item><list-item><p>Street sign indicating no passage</p></list-item><list-item><p>Vegetation</p></list-item><list-item><p>Residential buildings and property</p></list-item><list-item><p>Community buildings and property</p></list-item><list-item><p>Industrial buildings and property</p></list-item><list-item><p>Other buildings and property</p></list-item><list-item><p>Recreational areas</p></list-item><list-item><p>Parking facility</p></list-item><list-item><p>Cemeteries</p></list-item><list-item><p>Industrial area that is not a building</p></list-item><list-item><p>Undeveloped land</p></list-item><list-item><p>Water body or waterway</p></list-item><list-item><p>Topographical feature</p></list-item></list></td></tr></tbody></table> </ephtml> </p> <p>We selected a set of labels from these two sets of preliminary labels and worked with five paid undergraduate, graduate, and postdoctoral researchers (hereafter "human coders") to finalize this list of labels. Human coders were trained to use [<reflink idref="bib20" id="ref160">20</reflink>], which is a qualitative analysis software program that can be used to analyze images. To ensure consistency in coding, we first worked with the human coders to practice labeling a small number of satellite and streetscape images and then compared and discussed applied labels. After this practice, human coders proceeded to label all satellite and streetscape images in our reliability and validity test set (discussed in the next section). We revised the list of labels based on feedback from the human coders and re-coded images with updated labels as needed. The final list of labels from this manual labeling process serves as our list of labels for the study.</p> <p>The second sub-task is to create and test model prompts. With generative LLMs, the natural language interface allows for "almost infinite variation in prompts," with no clear way to determine a priori what the best prompt will be for a particular task and corpus ([<reflink idref="bib86" id="ref161">86</reflink>]:4). As such, there is a rapidly growing body of research on prompt development for generative LLMs (e.g., [<reflink idref="bib16" id="ref162">16</reflink>]; [<reflink idref="bib46" id="ref163">46</reflink>]; [<reflink idref="bib82" id="ref164">82</reflink>]; [<reflink idref="bib86" id="ref165">86</reflink>]; [<reflink idref="bib94" id="ref166">94</reflink>]; [<reflink idref="bib99" id="ref167">99</reflink>]; [<reflink idref="bib100" id="ref168">100</reflink>]). Generative MMs also use natural language interfaces and therefore similarly offer a multitude of options for prompt design, but there is currently very limited guidance on how to develop effective prompts for generative MMs (for exception, see [<reflink idref="bib92" id="ref169">92</reflink>]).</p> <p>To address this gap, we offer some general guidance about prompt development with generative MMs. Social scientists should begin by creating and testing prompts with different wording to ascertain their core prompt language—that is, the phrasing of their task that will generate optimal model output. Depending on the image collection and task, effective core prompt language may include some context about the images and tasks and/or a list of pre-defined labels. After social scientists have obtained their core prompt language, they should decide whether to use prompt parameters and capabilities, namely model personas and "few-shot" learning where examples of manually labeled images are provided to the model. Throughout this prompt development process, social scientists should document their tests so that they can report results to support learning and reproducibility.</p> <p>For our empirical application, we began by testing different prompts with ChatGPT Plus that would allow us to computationally label satellite images of sites that we identified as being socially and spatially divided. That is, we first focused on identifying effective prompt language for the simplest version of our task (labeling a satellite image for each site) and then adapted and scaled the prompt language to carry out our full task (labeling one satellite image and two streetscape images for each site).</p> <p>We performed some informal tests with satellite images of two different sites to first assess whether ChatGPT Plus can detect built environment features and visual markers in images. These tests were necessary because at the time of our analysis, ChatGPT Plus was newly released and there was limited information available about the image analysis capabilities of GPT-4 (the underlying model for ChatGPT Plus). Results from these tests indicated that ChatGPT Plus can recognize visual markers and does not need definitions of built environment features in order to detect them.[<reflink idref="bib17" id="ref170">17</reflink>]</p> <p>We then used a satellite image from one of the two sites to more systematically test six different prompts. The six prompts and their output are available in Appendix A in the online supplement. Through these tests, we learned that prompts perform better when they supply some context about image origins and they provide clear directions on what the model should and should not do. We ascertained the following prompt language:</p> <p></p> <ulist> <item> This is a [image type] image from [city, state]. This image contains a [visual marker]. Identify built environment features that specifically overlap with the [visual marker] in this image. Do not include built environment features that do not overlap with the [visual marker].</item> </ulist> <p>Our core prompt language resulted in mostly satisfactory output, so we decided not to add a model persona to our prompt. Because the output was acceptable and because it was unclear at the time whether generative MMs were capable of image recall, we also decided to use the model in a "zero-shot" training format. However, we decided to include our list of labels (developed as part of the first sub-task) in our core prompt language in order to standardize output because we learned that ChatGPT Plus tends to use synonyms to describe the same type of feature (e.g., "residential buildings," "residential houses," "residential structures"). Appendix B in the online supplement provides examples of five satellite images and compares output that are generated for these images using prompts with and without pre-defined labels.</p> <p>We then adapted the core prompt language to carry out our full task. Specifically, we added stepped instructions for analyzing three images for each site, in a manner mimicking how a human coder might approach this task. Our final prompt is provided below:</p> <p></p> <ulist> <item> System prompt:</item> <p></p> <item> You will be provided with three images of the same location. The first image is a satellite image and the second and third images are street-level images. Follow these instructions in order:</item> <p></p> <item> 1. For the satellite image, list one or more built environment features that specifically overlap with the red line in the center of the image only if the feature is in this list:</item> <p></p> <item> [list of 18 pre-defined built environment feature labels]</item> <p></p> <item> Do not list features that do not overlap with the red line.</item> <p></p> <item> 2. For the street-level images, identify any features from the list that you did not identify in the satellite image, if there are any.</item> <p></p> <item> Summarize unique features across the three images in one line of text with each feature separated by a comma. You do not need to provide summaries of each image.</item> <p></p> <item> User prompt:</item> <p></p> <item> [set of images for a city, with one satellite image and two streetscape images for each site]</item> </ulist> <p>Again, we opted to use ChatGPT Plus to develop prompts because there was limited information available about GPT-4 at the time of our analysis, but we now recommend that researchers use API calls so that prompt development tests can be more readily documented and compared. To ascertain our core prompt language, we used two sites so that we could more easily compare ChatGPT Plus output from several different prompts, though we encourage researchers to use more development sites when working with API calls. These development sites were excluded from our reliability and validity test set, though we obtained nearly identical results when they were included (see Appendix E in the online supplement). We suspect that the results were almost exactly the same because our development image set and validity image set are very, very small compared to GPT-4o's training data, though additional research is needed to systematically examine whether separate subsets of images are needed for prompt development and validation when using generative MMs in a zero-shot format and to provide guidance on optimal development and validation data splits.</p> <hd id="AN0186128776-8">Measurement and Inference</hd> <p>Social scientists can then measure their concepts of interest once they are operationalized and use these measures to make inferences after they are validated. In the context of computational image analysis, this part of the framework focuses on prompting generative MMs to label a collection of images, validating model-generated image labels, and using validated labels to make inferences. For this article, we focus on providing general guidance for measurement tasks with generative MMs—that is, how to obtain and validate model-generated image labels. We leave for future research to consider how principles for making predictions and causal inferences with validated text labels (e.g., [<reflink idref="bib34" id="ref171">34</reflink>]) can be adapted for image analysis, including how non-random error from generative multimodal models can be addressed to de-bias or minimize bias in downstream analyses, which we will return to later in our discussion about the limitations of GPT-4o for image analysis.</p> <p>After developing image labels and model prompts, social scientists should use an appropriate generative MM to label their image collection. As more generative MMs become available, researchers may want to compare results using multiple models, particularly results from proprietary versus open-source models. Because generative AI models behave in non-deterministic ways, we strongly recommend that researchers conduct multiple model runs to ensure labeling consistency. Social scientists should then assess their model-generated labels for reliability and validity.</p> <p>For our empirical application, we implemented our prompt using gpt-4o-2024-05-13, which was the latest version of GPT-4o at the time of our analysis.[<reflink idref="bib18" id="ref172">18</reflink>] We batched images of sites by city and implemented one API call per city, following previous research that finds that cities are visually distinct and city-specific analyses work best for deep learning vision analysis of streetscape images ([<reflink idref="bib40" id="ref173">40</reflink>]; [<reflink idref="bib39" id="ref174">39</reflink>]). For all API calls, we set the temperature parameter to 0 to minimize output randomness and set a seed and recorded system fingerprints to support reproducibility of our work. Although it is currently unclear whether it is possible for generative AI models to produce completely deterministic output, we nonetheless recommend setting the temperature parameter to 0 to minimize output randomness as much as possible. We repeated each city-specific API call three times, resulting in three sets of model-generated image labels for each city. Our computing time and costs are available in Appendix C in the online supplement.</p> <p>We obtained GPT-generated labels for all 367 sites (<emph>n</emph> = 1,101 satellite and streetscape images). To ensure consistency in model-generated labels, we measured the agreement between labels generated in three different model runs (or API calls). There was high agreement among pairwise model run comparisons for all sites, ranging from 94 percent to 98 percent (see Table 5). Pairwise agreement was similarly high for sites in the reliability and validity test set, ranging from 93 to 99 percent. Agreement measured using Cohen's κ and Krippendorff's α ([<reflink idref="bib27" id="ref175">27</reflink>]; [<reflink idref="bib35" id="ref176">35</reflink>]) was also high for all sites and the test set (see Table 5).[<reflink idref="bib19" id="ref177">19</reflink>] These results show that although GPT-4o does not behave in a completely deterministic manner, the model was able to generate highly consistent labels across multiple runs.</p> <p>Table 5. GPT Labeling Consistency.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="char" char="." /><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="left" /><col align="char" char="." /></colgroup><thead><tr><th align="left" /><th align="left" /><th align="left" colspan="3">Test Set (<italic>n</italic> = 49 sites)</th><th align="left" colspan="3">All Sites (<italic>n</italic> = 367 sites)</th></tr><tr><th align="left">City</th><th align="left">Comparison</th><th align="left">Agreement</th><th align="left"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">κ</mi></math></p></th><th align="left"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">α</mi></math></p></th><th align="left">Agreement</th><th align="left"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">κ</mi></math></p></th><th align="left"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">α</mi></math></p></th></tr></thead><tbody><tr><td>Chicago</td><td>Run 1 vs Run 2</td><td>93.3</td><td>.800</td><td /><td>96.3</td><td>.885</td><td /></tr><tr><td /><td>Run 1 vs Run 3</td><td>96.7</td><td>.902</td><td /><td>97.2</td><td>.914</td><td /></tr><tr><td /><td>Run 2 vs Run 3</td><td>95.6</td><td>.867</td><td /><td>96.2</td><td>.881</td><td /></tr><tr><td /><td>Overall</td><td>92.8</td><td /><td>.857</td><td>94.8</td><td /><td>.893</td></tr><tr><td>Cincinnati</td><td>Run 1 vs Run 2</td><td>98.3</td><td>.944</td><td /><td>97.9</td><td>.931</td><td /></tr><tr><td /><td>Run 1 vs Run 3</td><td>98.3</td><td>.942</td><td /><td>98.0</td><td>.936</td><td /></tr><tr><td /><td>Run 2 vs Run 3</td><td>98.9</td><td>.961</td><td /><td>97.7</td><td>.926</td><td /></tr><tr><td /><td>Overall</td><td>97.8</td><td /><td>.949</td><td>96.8</td><td /><td>.931</td></tr><tr><td>Hartford</td><td>Run 1 vs Run 2</td><td>95.7</td><td>.871</td><td /><td>97.7</td><td>.930</td><td /></tr><tr><td /><td>Run 1 vs Run 3</td><td>96.3</td><td>.888</td><td /><td>95.7</td><td>.872</td><td /></tr><tr><td /><td>Run 2 vs Run 3</td><td>96.9</td><td>.906</td><td /><td>95.5</td><td>.864</td><td /></tr><tr><td /><td>Overall</td><td>94.4</td><td /><td>.888</td><td>94.4</td><td /><td>.889</td></tr><tr><td>Miami</td><td>Run 1 vs Run 2</td><td>95.0</td><td>.857</td><td /><td>95.4</td><td>.870</td><td /></tr><tr><td /><td>Run 1 vs Run 3</td><td>96.7</td><td>.905</td><td /><td>95.4</td><td>.871</td><td /></tr><tr><td /><td>Run 2 vs Run 3</td><td>98.3</td><td>.952</td><td /><td>96.0</td><td>.887</td><td /></tr><tr><td /><td>Overall</td><td>95.0</td><td /><td>.905</td><td>93.5</td><td /><td>.876</td></tr><tr><td>Seattle</td><td>Run 1 vs Run 2</td><td>97.8</td><td>.929</td><td /><td>97.2</td><td>.910</td><td /></tr><tr><td /><td>Run 1 vs Run 3</td><td>97.2</td><td>.910</td><td /><td>98.4</td><td>.947</td><td /></tr><tr><td /><td>Run 2 vs Run 3</td><td>98.3</td><td>.946</td><td /><td>97.0</td><td>.904</td><td /></tr><tr><td /><td>Overall</td><td>96.7</td><td /><td>.929</td><td>96.3</td><td /><td>.920</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note</emph>: All values of Cohen's <emph>κ</emph> are significant at p <.05.</p> <p>Table 6 shows the proportion of sites with predicted built environment features. Notably, some features that have been commonly studied in research on residential segregation (e.g., highways or freeways) are indeed present in our sites of social and spatial division, but they are not as prevalent as other features that are less well-known (e.g., local or residential roads, vegetation). In other words, our large-scale image analysis with GPT-4o highlighted built environment features that merit more attention among residential segregation researchers. We also conducted a separate analysis in which we prompted GPT-4o to label a small subset of satellite images without visual markers and then validated these labels, the results of which are available in Appendix D in the online supplement.</p> <p>Table 6. Proportion of Sites with Predicted Built Environment Features.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /></colgroup><thead><tr><th align="left">Label</th><th align="left">Prop. of Sites(<italic>n</italic> = 367 sites)</th></tr></thead><tbody><tr><td>Cemeteries</td><td>0.00</td></tr><tr><td>Community buildings and property</td><td>0.06</td></tr><tr><td>Industrial area that is not a building</td><td>0.01</td></tr><tr><td>Industrial buildings and property</td><td>0.23</td></tr><tr><td>Local or residential road</td><td>0.47</td></tr><tr><td>Multi-lane highway or freeway</td><td>0.36</td></tr><tr><td>Other buildings and property</td><td>0.07</td></tr><tr><td>Other transportation infrastructure</td><td>0.03</td></tr><tr><td>Parking facility</td><td>0.20</td></tr><tr><td>Physical barrier</td><td>0.35</td></tr><tr><td>Railroad track</td><td>0.30</td></tr><tr><td>Recreational areas</td><td>0.06</td></tr><tr><td>Residential buildings and property</td><td>0.52</td></tr><tr><td>Street sign indicating no passage</td><td>0.02</td></tr><tr><td>Topographical feature</td><td>0.00</td></tr><tr><td>Undeveloped land</td><td>0.03</td></tr><tr><td>Vegetation</td><td>0.88</td></tr><tr><td>Water body or waterway</td><td>0.06</td></tr></tbody></table> </ephtml> </p> <p>To assess GPT-generated labels, we selected a random subset of sites (49 sites total, nine for Hartford and 10 each for the other cities)[<reflink idref="bib20" id="ref178">20</reflink>] to serve as our reliability and validity test set (hereafter "test set"). All images from these sites were labeled by our team of human coders, with two coders assigned to each city so that all images for a city were labeled by the same set of coders to enable comparisons between coders. Figure 4 provides an example of an image labeled by human coders using Dedoose. In addition, one of the authors and a coder (who was not otherwise assigned to labeling tasks) labeled all images in the test set to provide a set of "expert labels." The expert labels represent a close approximation of the ground truth of features that can be identified in the images.</p> <p>Graph: Figure 4. Example of a manually labeled satellite image for a site in Miami, FL.</p> <hd id="AN0186128776-9">Reliability</hd> <p>To assess that our labels capture well-defined built environment features, we measured the agreement between human coders' labels for each city. We first measured pairwise agreement as the percent of labels that match in presence (or absence) for each pair of coders. To account for the possibility of chance agreement, we also measured pairwise agreement using Cohen's κ ([<reflink idref="bib27" id="ref179">27</reflink>]; [<reflink idref="bib35" id="ref180">35</reflink>]). Cohen's κ is calculated as the ratio of what two coders agree on minus what we would expect them to agree on, and how much agreement is possible beyond chance: <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>κ</mi><mo>=</mo><mstyle displaystyle="true" scriptlevel="0"><mrow><mfrac><mrow><msub><mi>P</mi><mi>o</mi></msub><mo>−</mo><msub><mi>P</mi><mi>e</mi></msub></mrow><mrow><mn>1</mn><mo>−</mo><msub><mi>P</mi><mi>e</mi></msub></mrow></mfrac></mrow><mrow><mspace width="0.25em" /></mrow><mo>,</mo></mstyle></math> </ephtml></p> <p>Graph</p> <p>where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>P</mi><mi>o</mi></msub></math> </ephtml> is the observed proportion of agreement between two coders and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>P</mi><mi>e</mi></msub></math> </ephtml> is the expected proportion of agreement due to chance ([<reflink idref="bib18" id="ref181">18</reflink>]).[<reflink idref="bib21" id="ref182">21</reflink>]</p> <p>For sites in our test set, we found high agreement among pair comparisons for coders, ranging from 85 percent to 88 percent (see Table 7). Interrater reliability for pairs of coders was similarly high when measured using Cohen's κ, with values for coders ranging from.471 to.620, representing moderate to substantial agreement. These results show that our list of built environment feature labels is well-defined.</p> <p>Table 7. Reliability of Labeling by Human Coders for Sites in the Test Set.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left">City</th><th align="left">Comparison</th><th align="left">Agreement</th><th align="left">Cohen's <p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi xmlns="">κ</mi></math></p></th></tr></thead><tbody><tr><td>Chicago</td><td>Coder 1 vs Coder 2</td><td>86.7</td><td>.471</td></tr><tr><td>Cincinnati</td><td>Coder 1 vs Coder 2</td><td>88.3</td><td>.620</td></tr><tr><td>Hartford</td><td>Coder 1 vs Coder 2</td><td>84.6</td><td>.486</td></tr><tr><td>Miami</td><td>Coder 1 vs Coder 2</td><td>86.7</td><td>.557</td></tr><tr><td>Seattle</td><td>Coder 1 vs Coder 2</td><td>88.3</td><td>.583</td></tr></tbody></table> </ephtml> </p> <p>4 <emph>Note</emph>: All values of Cohen's <emph>κ</emph> are significant at p <.05.</p> <hd id="AN0186128776-10">Validity</hd> <p>To assess the validity of our model-generated labels, we use labels that were generated by GPT in at least two out of three runs to serve as our model-generated labels and we use the expert-generated labels to serve as our gold-standard labels. Using GPT- and expert-generated labels for the 49 sites in our test set (<emph>n</emph> = 147 images), we calculated four metrics commonly used in computational analyses of text and images to measure validity: recall, precision, F1-score, and accuracy (see [<reflink idref="bib34" id="ref183">34</reflink>] for text analysis and [<reflink idref="bib93" id="ref184">93</reflink>] for image analysis). Accuracy is a global measure that summarizes the overall proportion of images or documents that are correctly labeled. Precision and recall are label-specific measures, where precision summarizes the proportion of images or documents that are correctly labeled among true and false positives and recall summarizes the proportion of images or documents that are correctly labeled among true positives and false negatives. F1-score is also a label-specific measure and captures the harmonic mean of precision and recall values, formally specified as <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>F</mi><mn>1</mn><mo>=</mo><mn>2</mn><mo>×</mo><mrow><mfrac><mrow><mrow><mi mathvariant="normal">precision</mi><mspace width="0.25em" /></mrow><mo>×</mo><mrow><mi mathvariant="normal">recall</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">precision</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">recall</mi></mrow></mrow></mfrac></mrow></math> </ephtml> .</p> <p>Overall, we obtained a high accuracy score (0.86). Our accuracy score is comparable to those obtained in previous studies that use computational methods to analyze streetscape imagery; for example, [<reflink idref="bib39" id="ref185">39</reflink>] obtained a cross-validated accuracy score of 87% when training a deep learning vision model to predict the presence of trash in Google Streetview images from three U.S. cities, while [<reflink idref="bib65" id="ref186">65</reflink>] obtained an accuracy score of 78% when training a support vector model (SVM) to predict perceived safety in Google Streetview images in 21 U.S. cities. Although we are not able to provide a direct comparison of labeling accuracy for generative MMs and other computational methods for image analysis, our findings suggest that at least with streetscape images, researchers are likely to obtain the most accurate labels when using a fully trained or fine-tuned deep learning vision model, likely because these models are trained on diverse streetscape images and therefore possess robust knowledge about this specific type of image data. Yet, our findings also suggest that researchers may be able to obtain more accurate labels for streetscape images when using a generative MM in a zero-shot format compared to using a fully trained SVM, likely because SVMs have relatively less complex model architectures. In other words, researchers seeking to label streetscape images may be able obtain more accurate labels in a more efficient and cost-effective way by using generative MMs instead of traditional supervised ML models. Our finding is consistent with growing research in computer science showing that generative MMs can often match or exceed the performance of fully supervised models when for computer vision tasks (e.g., [<reflink idref="bib11" id="ref187">11</reflink>]).</p> <p>However, accuracy is a very general measure of model performance and is less useful when evaluating the ability of models to classify imbalanced datasets. Table 8 shows the proportion of sites containing each built environment feature label based on our expert-generated labels. Table 8 shows that our test set is indeed imbalanced; four labels are present in at least 60% of sites (local or residential road, physical barrier, vegetation, other transportation infrastructure), nine labels are present in less than 20% of sites (community buildings and property, industrial areas, industrial buildings and property, multi-lane highway or freeway, topographic feature, waterbody or waterway, undeveloped land, recreational areas, street signs indicating no passage), and one label does not appear in any sites (cemeteries). Given our imbalanced test set, it is more useful to rely on our label-specific metrics, which do account for imbalances in data.</p> <p>Table 8. Proportion of Sites with Expert-Labeled Built Environment Features.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /></colgroup><thead><tr><th align="left">Label</th><th align="left">Prop. of Sites(<italic>n</italic> = 49 sites)</th></tr></thead><tbody><tr><td>Cemeteries</td><td>0.00</td></tr><tr><td>Community buildings and property</td><td>0.08</td></tr><tr><td>Industrial area that is not a building</td><td>0.18</td></tr><tr><td>Industrial buildings and property</td><td>0.10</td></tr><tr><td>Local or residential road</td><td>0.86</td></tr><tr><td>Multi-lane highway or freeway</td><td>0.08</td></tr><tr><td>Other buildings and property</td><td>0.22</td></tr><tr><td>Other transportation infrastructure</td><td>0.62</td></tr><tr><td>Parking facility</td><td>0.30</td></tr><tr><td>Physical barrier</td><td>0.62</td></tr><tr><td>Railroad track</td><td>0.34</td></tr><tr><td>Recreational areas</td><td>0.62</td></tr><tr><td>Residential buildings and property</td><td>0.32</td></tr><tr><td>Street sign indicating no passage</td><td>0.08</td></tr><tr><td>Topographical feature</td><td>0.08</td></tr><tr><td>Undeveloped land</td><td>0.14</td></tr><tr><td>Vegetation</td><td>0.76</td></tr><tr><td>Water body or waterway</td><td>0.06</td></tr></tbody></table> </ephtml> </p> <p>Table 9 presents precision, recall, and F1-scores for all labels. Average precision, recall, and F1-score for model-generated labels were 0.86, 0.60, and 0.66, respectively. Our precision and recall scores are generally comparable to those obtained in previous studies using deep learning vision models to analyze streetscape imagery, though our precision score is higher and our recall score is lower. [<reflink idref="bib39" id="ref188">39</reflink>] obtained average precision and recall scores of 74.3% and 87%, respectively, for a deep learning vision model trained for binary classification of Google Streetview images. Notably, whereas deep learning vision models used in previous studies performed better on recall compared to precision, GPT in our study performed better on precision compared to recall. This means that when GPT identified a built environment feature label, the labels tended to be correct, but GPT missed labels that were actually present in sites at a non-trivial rate.</p> <p>Table 9. Validity of GPT-Generated Labels Compared to Expert-Generated Labels.</p> <p>Graph</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="left" /><col align="left" /><col align="left" /></colgroup><thead><tr><th align="left">Label</th><th align="left">Precision</th><th align="left">Recall</th><th align="left">F1</th></tr></thead><tbody><tr><td>Cemeteries</td><td>–</td><td>–</td><td>–</td></tr><tr><td>Community buildings and property</td><td>1.00</td><td>0.50</td><td>0.67</td></tr><tr><td>Industrial area that is not a building</td><td>No pred</td><td>0.00</td><td>Undef</td></tr><tr><td>Industrial buildings and property</td><td>0.36</td><td>1.00</td><td>0.53</td></tr><tr><td>Local or residential road</td><td>1.00</td><td>0.67</td><td>0.81</td></tr><tr><td>Multi-lane highway or freeway</td><td>0.44</td><td>1.00</td><td>0.62</td></tr><tr><td>Other buildings and property</td><td>1.00</td><td>0.45</td><td>0.63</td></tr><tr><td>Other transportation infrastructure</td><td>1.00</td><td>0.10</td><td>0.18</td></tr><tr><td>Parking facility</td><td>1.00</td><td>0.60</td><td>0.75</td></tr><tr><td>Physical barrier</td><td>1.00</td><td>0.45</td><td>0.62</td></tr><tr><td>Railroad track</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>Recreational areas</td><td>1.00</td><td>1.00</td><td>1.00</td></tr><tr><td>Residential buildings and property</td><td>0.64</td><td>1.00</td><td>0.78</td></tr><tr><td>Street sign indicating no passage</td><td>1.00</td><td>0.25</td><td>0.40</td></tr><tr><td>Topographical feature</td><td>No pred</td><td>0.00</td><td>Undef</td></tr><tr><td>Undeveloped land</td><td>1.00</td><td>0.14</td><td>0.25</td></tr><tr><td>Vegetation</td><td>0.86</td><td>1.00</td><td>0.93</td></tr><tr><td>Water body or waterway</td><td>0.60</td><td>1.00</td><td>0.75</td></tr></tbody></table> </ephtml> </p> <p>5 <emph>Note</emph>: Cemeteries were not present in the expert- and GPT-generated labels. For two labels (industrial area and topographic feature), there were no true or false positives from GPT, resulting in no values for precision and undefined values for F1-score.</p> <p>For our purposes, the F1-score is the most relevant metric because we are interested in both avoiding false positives (GPT identifying a built environment feature that is not actually present in a site) and avoiding false negatives (GPT misses a built environment feature that is actually present in a site). The average F1-score is quite high (0.66) and the F1-scores for labels are generally high. F1-scores are 0.75 and higher for seven labels, some of which appear relatively rarely in the images (e.g., water body or waterway, multi-lane highway or freeway). The magnitude of these scores suggests that the training data for GPT-4o likely include satellite and streetscape images of cities. Three labels had exceedingly low F1-scores (other transportation infrastructure, undeveloped land, street sign indicating no passage), which we suspect is due to label ambiguity. Appendix F in the online supplement discusses potential reasons why these labels performed poorly compared to other labels.</p> <hd id="AN0186128776-11">Limitations of GPT-4o for Image Analysis</hd> <p>We encountered several limitations while using GPT-4o that we think are important to highlight. First, when identifying built environment features in satellite and streetscape images, GPT-4o simply lists features and cannot provide any information about the certainty of its labels. We suspect that this limitation would apply to other types of images as well as other generative MMs. This is a real disadvantage of using generative MMs to label images compared to using other computational methods that can provide probabilities of label certainty. We attempted to prompt GPT to provide estimates of label certainty, but this effort was unsuccessful. It is possible that generative MMs will be able to provide measures of label certainty in the future as models are trained to engage in more complex reasoning.</p> <p>Second, GPT-4o cannot generate bounding boxes for predicted built environment features in satellite and streetscape images. Again, we suspect this limitation would apply to other types of images as well as other generative MMs. Bounding boxes for predicted labels, which other computational methods can provide, are useful in image analysis because they provide researchers with additional information for checking validity and refining labels. It seems entirely possible that generative MMs will be able to generate bounding boxes for analyzed images in the near future, given that there are already generative MMs that can output visual features (e.g., OpenAI DALL-E).</p> <p>Third, at the time of our analysis, there was no fine-tuning capability available for GPT-4o. OpenAI has since added fine-tuning capability for GPT-4o, as of October 2024, though fine-tuning capability generally remains limited among leading generative MMs (see Table 2). The ability to fine-tune a generative MM is important for researchers, particularly those working with images that are unlikely to be included as part of a training dataset (e.g., non-digital images). Relatedly, it is currently difficult for researchers to determine whether they will need to fine-tune a generative MM until after they have conducted initial tests due to opacity about training datasets. Future research should examine how fine-tuning affects the performance of generative MMs in labeling different image datasets.</p> <p>Fourth, our validity metrics show that while GPT-4o is quite accurate, the model exhibits non-random error. For example, GPT-4o was more successful at accurately detecting the presence (or absence) of specific built environment features, and this accuracy varied across cities. Although we did not explicitly test for this, it is also likely that GPT-4o's accuracy varied across neighborhoods within cities. This suggests that while GPT-4o may be trained to analyze satellite and streetscape imagery, it may be optimized to detect certain features (e.g., local or residential roads) over others (e.g., fences). There may also be variation in the extent of its training with images across a range of contexts and conditions, such as the extent of tree cover and roofing, amount of sunlight, and type of pavement, each of which may affect the visibility and the hue and brightness of images. Because of this non-random error, researchers who use image labels from GPT-4o and other generative MMs in downstream analyses will likely obtain biased results—even if the image labels are highly accurate.</p> <p>For instance, a generative MM may be trained on image data for recreation fields that have well-maintained appearances—such as those with green grass, clearly marked boundary lines, and visible goals or equipment. If so, it may be better at recognizing recreation fields that conform to that standard and struggle to identify recreation fields that are less manicured: those with patchy grass or dirt, faded or missing boundary lines, or lacking visible equipment. This would introduce systematic bias, where recreational amenities in under-resourced neighborhoods may be undercounted or misclassified, not because they are absent, but because they do not match the aesthetic or structural conventions emphasized in the training data. A generative MM may recognize this space as undeveloped land rather than as a recreational area.</p> <p>To make predictions and causal inferences with image labels from generative MMs, researchers must therefore address these models' propensity for non-random error. This is particularly important because ignoring non-random prediction errors can lead to substantial bias, invalid confidence intervals, and inaccurate <emph>p</emph>-values when prediction errors are correlated with observed and unobserved variables in downstream analyses ([<reflink idref="bib24" id="ref189">24</reflink>]:2). A key challenge in assessing non-random error is that documentation for generative MMs, including open-source generative MMs, rarely includes information about training data. Fortunately, there are new methods for addressing this issue, such as [<reflink idref="bib61" id="ref190">61</reflink>] recent extension of design-based supervised learning (DSL) ([<reflink idref="bib24" id="ref191">24</reflink>]) to debias image labels for downstream analyses.</p> <p>Lastly, although we did not encounter instances where GPT-4o applied racially and gender biased labels in our application, this is likely because of the specific type of images we analyzed (satellite and streetscape images) rather than evidence of the absence of model bias. Extensive research shows that computer vision techniques exhibit racial and gender biases ([<reflink idref="bib12" id="ref192">12</reflink>]; [<reflink idref="bib13" id="ref193">13</reflink>]; [<reflink idref="bib71" id="ref194">71</reflink>]; [<reflink idref="bib79" id="ref195">79</reflink>]), as do large language models ([<reflink idref="bib6" id="ref196">6</reflink>]). Future social science research that analyzes images with generative MMs should consistently and carefully document model biases and provide recommendations on how to address these biases.</p> <hd id="AN0186128776-12">Conclusion</hd> <p>The advent of generative AI models has introduced new possibilities and considerations for social science research, particularly for research involving text-based tasks. Until now, there has been limited work to systematically examine the potential utility of generative AI models, which are increasingly evolving to include multimodal functionality, for image-based tasks in social science research. In this article, we introduced a framework that provides guidance for computational analysis of images using generative MMs, which extends and adapts an existing framework for computational text analysis ([<reflink idref="bib34" id="ref197">34</reflink>]). This framework consists of three core tasks: curation, discovery, and measurement and inference. We demonstrated this framework with an empirical application in which we use a newly developed generative MM, OpenAI's GPT-4o model, to analyze satellite and streetscape images to identify built environment features that contribute to contemporary residential segregation in U.S. cities.</p> <p>Overall, we find that when it comes to analyzing satellite and streetscape images, GPT can generate labels that are reasonably valid compared to expert-generated labels. When provided with well-defined image labels (as measured by between-research assistant agreement), GPT is able to apply labels to images with high validity, with an accuracy score of 0.86 and average precision, recall, and F1 scores of 0.86, 0.60, and 0.66, respectively. Our accuracy, precision, and recall scores are comparable to those obtained in previous studies that use deep learning models and traditional machine learning models to classify streetscape imagery ([<reflink idref="bib39" id="ref198">39</reflink>]; [<reflink idref="bib65" id="ref199">65</reflink>]). Although we are not able to make direct comparisons between labels generated by GPT and other computational image analysis methods, this is a promising finding regarding the ability of GPT to label satellite and streetscape imagery. In terms of specific labels, we find that GPT is able to accurately detect and classify many built environment features of interest to social scientists (e.g., residential buildings and property, local or residential roads), including features that are relatively rare, which suggests that satellite and streetscape imagery are likely included in GPT-4o's training data. Future research should examine why, as our analysis suggests, generative MMs appear to perform better in terms of precision instead of recall, compared to other computational methods for large-scale image analysis. Future research should also clarify aspects of image labels, as well as model prompts, that may not be ambiguous to humans but may be ambiguous to a generative MM, which is increasingly capable of understanding images and natural language but, as we observe, still faces limitations.</p> <p>Our finding that GPT is not just capable of but is, in fact, quite adept at analyzing satellite and streetscape imagery in a zero-shot training format potentially suggests a new horizon for large-scale image analysis in social science research. Researchers have traditionally analyzed images with manual coding and, more recently, with fully supervised machine learning models and fine-tuned deep learning vision models—approaches that are time-consuming and resource-intensive and that often require substantial technical expertise. The results of our empirical application suggest that with generative MMs, researchers may be able to not only analyze more satellite and streetscape images, but also carry out this analysis with increased complexity, precision, and speed and decreased cost. This is a very promising, and perhaps even unprecedented, development for researchers who are interested in using satellite and streetscape imagery to shed light on topics that are central to the social sciences, such as neighborhood change and inequality, housing affordability and homelessness, and health disparities.</p> <p>As generative MMs continue to evolve and improve, researchers will need to determine whether these models <emph>can</emph> be used for scientific purposes. In this article, we found that GPT can be used to systematically conduct large-scale analysis of satellite and streetscape imagery, but further work is needed to determine whether GPT and other generative MMs can analyze other types of images. Perhaps more important, we emphasize that researchers will also need to collaboratively determine when and how generative MMs <emph>should</emph> be used for image analysis. In particular, researchers should consider identifying use cases where image analysis with generative MMs may pose ethical challenges and develop and adopt practices to proactively address these challenges. For example, there is robust research in computer science cautioning against the use of computer vision to analyze images of humans due to well-documented racial and gender biases, as well as high risk of harmful real-world applications (e.g., [<reflink idref="bib12" id="ref200">12</reflink>]; [<reflink idref="bib13" id="ref201">13</reflink>]; [<reflink idref="bib71" id="ref202">71</reflink>]). Another important consideration for researchers moving forward will be to determine whether there should be greater development of and investment in open-source generative MMs, given that currently, all leading generative MMs except for one (Llama 3.2) are proprietary and therefore not conducive to traditional quantitative social science paradigms regarding benchmarking and reproducibility ([<reflink idref="bib4" id="ref203">4</reflink>]; [<reflink idref="bib68" id="ref204">68</reflink>]; [<reflink idref="bib83" id="ref205">83</reflink>]; [<reflink idref="bib86" id="ref206">86</reflink>].; see also [<reflink idref="bib53" id="ref207">53</reflink>]). The rapid advancement of generative AI models—in terms of natural language processing ability and now computer vision ability—is indeed promising but will require researchers to organize apace to develop shared goals and practices to ensure that the future of large-scale image analysis is not only rigorous and reproducible but also ethical and sustainable.</p> <hd id="AN0186128776-13">Supplemental Material</hd> <p>Graph: Supplemental material, sj-docx-1-smr-10.1177_00491241251339673 for Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery by Tina Law and Elizabeth Roberto in Sociological Methods & Research</p> <hd id="AN0186128776-14">Acknowledgements</hd> <p>We would like to thank Alyssa Boerst, Carolina Escobar, Brendan Frizzell, Jaleh Jalili, Caroline Wolski, and Belinda Zhu for their research assistance. We would also like to thank participants of the 2024 Generative Artificial Intelligence and Sociology Workshop at Yale University and the 2024 American Sociological Association Meeting and the anonymous reviewers for their feedback on earlier versions of this manuscript. In addition, this manuscript benefitted from helpful conversations with Nga Than, Joscha Legewie, and Taylor Brown. This research was supported in part by a Rice University BRIDGE Seed Grant.</p> <ref id="AN0186128776-15"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref82" type="bt">2</bibl> <bibtext> The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by a Rice University BRIDGE Seed Grant.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref2" type="bt">3</bibl> <bibtext> Sample coder and resources for implementing the methodological framework introduced in this article is available on Harvard Dataverse ([54]. We are unable to distribute the image data used in this article due to copyright, but these data are accessible to researchers through the Google Maps API and the Google Cloud Platform.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> Tina Law https://orcid.org/0000-0001-7631-6763 Elizabeth Roberto https://orcid.org/0000-0001-7667-6953</bibtext> </blist> <blist> <bibl id="bib5" idref="ref83" type="bt">5</bibl> <bibtext> Supplemental material for this article is available online.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref29" type="bt">6</bibl> <bibtext> Other types of deep learning vision models include Boltzmann family models and stacked autoencoders (for a review, see [91]).</bibtext> </blist> <blist> <bibl id="bib7" idref="ref52" type="bt">7</bibl> <bibtext> Social science research using CNNs builds upon previous research that applies other computer vision techniques and tools, such as edge detection ([59]) and commercial autotaggers ([79]; [44]), as well as research that uses crowdsourcing to label images (e.g., [15]; [41]).</bibtext> </blist> <blist> <bibl id="bib8" idref="ref60" type="bt">8</bibl> <bibtext> Although OpenAI's earlier GPT models also have "generative" in their names, these earlier models are quite distinct from GPT-3 in that they are unable to generate text and understand natural language prompts and questions.[8] argue that while these earlier models contributed to the development of GPT-3, GPT-1 (released in 2018) most closely resembles a word embeddings model and GPT-2 (released in 2019) is most similar to a BERT or transformer-based natural language processing model.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref65" type="bt">9</bibl> <bibtext> For example, there are recently developed generative audio-text MMs, such as OpenAI's Whisper model which can understand audio input and generate text output and Meta's Audiobox which can understand audio and text inputs and generate audio output. There are also generative text-image MMs, such as OpenAI's DALL-E model, which can understand text input and generate image output.</bibtext> </blist> <blist> <bibtext> OpenAI's model evaluation results for GPT-4o are available at: https://openai.com/index/hello-gpt-4o/.</bibtext> </blist> <blist> <bibtext> For example, the model card for open-source Llama 3.2 (available at https://github.com/meta-llama/llama-models/blob/main/models/llama3%5f2/MODEL%5fCARD%5fVISION.md) states only that its training data "includes publicly available vision datasets, as well as over 3M synthetically generated examples."</bibtext> </blist> <blist> <bibtext> We measured improved connectivity using Roberto et al.'s (Forthcoming) utility score, which computes a value for each counterfactual road segment based on the average reduction in the shortest path length in the local area of the potential road segment if it is added to the residential road network, and then normalizes this value by the length of the counterfactual road segment. Residential roads include secondary, local, and private roads, service drives, alleys, and parking lot roads, and exclude highways, ramps, and pedestrian paths. For this application, we selected counterfactual road segments that were above the 25% percentile of utility scores.</bibtext> </blist> <blist> <bibtext> Using the census categories for race and ethnicity from the 2010 decennial census ([89]), we defined three race groups: Black, non-Hispanic ("Black"); Hispanic, any race ("Hispanic"); and White, non-Hispanic ("White"). We selected sites where there is a nonzero population of Black, Hispanic/Latino, or White residents and the absolute value of the difference in the proportion of any racial group between the endpoints of the missing road segments is.2 or greater. We use each endpoint's local environment population, which is measured using the Spatial Proximity and Connectivity method ([73]). This method aggregates the population of nodes within.5 km road distance of each endpoint (without the counterfactual road segment in the network).</bibtext> </blist> <blist> <bibtext> We initially marked the satellite images with a dashed, partially transparent, purple line as a visual marker for our counterfactual road segments. However, this marker was less visible in the images for Miami, which tend to have lighter colors compared to images for other cities. We found that a solid, partially transparent, red line was more visible across a range of color palettes and worked well in all five cities.</bibtext> </blist> <blist> <bibtext> ChatGPT Plus is a subscriber-based chatbot that provides access to OpenAI's GPT-4 series of multimodal models. At the time of our analysis, ChatGPT Plus provided access to the GPT-4 model.</bibtext> </blist> <blist> <bibtext> We also found that ChatGPT Plus was not more likely to accurately apply image labels that it generated, compared to researcher-generated labels. For example, "parking facility" is a label that we identified using ChatGPT Plus and its F1 score is 0.75, which is slightly below average compared to our other labels (see Table 9).</bibtext> </blist> <blist> <bibtext> We considered using ChatGPT Plus and GPT-4 to analyze street maps, in addition to satellite and streetscape images, but we learned early on that ChatGPT Plus struggles to detect built environment features in street maps. We suspect this is likely because unlike satellite and streetscape images, street maps contain words (e.g., street names), which can be noisy to generative MMs when they are prompted to label images that contain words and pixels (see [95]).</bibtext> </blist> <blist> <bibtext> Since our analysis, OpenAI has released two additional versions of GPT-4o: gpt-4o-2024-08-06 and gpt-4o-2024-11-20.</bibtext> </blist> <blist> <bibtext> Krippendorff's α is calculated as a ratio between observed disagreement ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>o</mi></msub></math> </ephtml> ) and expected disagreement ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>D</mi><mi>e</mi></msub></math> </ephtml> ) due to chance ([50]): <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi><mo>=</mo>1<mo>−</mo><mstyle displaystyle="true" scriptlevel="0"><mrow><mfrac><mrow><msub><mi>D</mi><mi>o</mi></msub></mrow><mrow><msub><mi>D</mi><mi>e</mi></msub></mrow></mfrac></mrow><mrow><mspace width="0.25em" /></mrow><mo>.</mo></mstyle></math> </ephtml></bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext> We selected 10 sites for Hartford but later removed one site that was used in prompt development. The results that we present here are robust to whether this site is included or excluded from the test set. Reliability and validity metrics for all 50 sites in the test set are available in Appendix E in the online supplement.</bibtext> </blist> <blist> <bibtext> [52] suggest interpreting the scale of agreement for Cohen's κ as poor (0.00 or less), slight (0.01-0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (0.81–1.00).</bibtext> </blist> <blist> <bibtext> [22]Availability Statement included at the end of the article</bibtext> </blist> </ref> <ref id="AN0186128776-16"> <title> References </title> <blist> <bibtext> Achiam Josh, Adler Steven, Agarwal Sandhini, Ahmad Lama, Akkaya Ilge, Aleman Florencia Leoni, Almeida Diogo, et al.2023. "Gpt-4 Technical Report." arXiv preprint arXiv:2303.08774. https://arxiv.org/abs/2303.08774</bibtext> </blist> <blist> <bibtext> Archer Deborah N. 2020. " "White Men's Roads Through Black Men's Homes": Advancing Racial Equity Through Highway Reconstruction." Vanderbilt Law Review. 73(5):1259–330.</bibtext> </blist> <blist> <bibtext> Argyle Lisa P., Busby Ethan C., Fulda Nancy, Gubler Joshua R., Rytting Christopher, Wingate David. 2023. " Out of one, Many: Using Language Models to Simulate Human Samples." Political Analysis. 31(3):337–51.</bibtext> </blist> <blist> <bibtext> Bail Christopher A. 2024. " Can Generative AI Improve Social Science? " Proceedings of the National Academy of Sciences. 121(21):e2314021121.</bibtext> </blist> <blist> <bibtext> Bayor Ronald H. 1988. " Roads to Racial Segregation." Journal of Urban History. 15(1):3–21. https://doi.org/10.1177/009614428801500101</bibtext> </blist> <blist> <bibtext> Bender Emily M., Gebru Timnit, McMillan-Major Angelina, Shmitchell Shmargaret. 2021. "On the Dangers of Stochastic Parrots: Can Language Models be Too Big?" In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623.</bibtext> </blist> <blist> <bibtext> Bernasco Wim, Hoeben Evelien M., Koelma Dennis, Liebst Lasse Suonperä, Thomas Josephine, Appelman Joska, Snoek Cees GM, Lindegaard Marie Rosenkrantz. 2023. " Promise into Practice: Application of Computer Vision in Empirical Research on Social Distancing." Sociological Methods & Research. 52(3):1239–87.</bibtext> </blist> <blist> <bibtext> Bommasani Rishi, Hudson Drew A., Adeli Ehsan, Altman Russ, Arora Simran, von Arx Sydney, Bernstein Michael S., et al.2022. "On the Opportunities and Risks of Foundation Models." arXiv preprint arXiv:2108.07258.</bibtext> </blist> <blist> <bibtext> Bonikowski Bart, Nelson Laura K. 2022. " From Ends to Means: The Promise of Text Analysis for Theoretically Driven Sociological Research." Sociological Methods & Research. 51(4):1469–83.</bibtext> </blist> <blist> <bibtext> Brown Tom, Mann Benjamin, Ryder Nick, Subbiah Melanie, Kaplan Jared D., Dhariwal Prafulla, Neelakantan Arvind, et al.2020. "Language Models are Few-shot Learners." arXiv preprint arXiv:2005.14165.</bibtext> </blist> <blist> <bibtext> Buch Shyamal, Hudson Drew A., Rong Frieda, Tamkin Alex, Zhang Xikun, Wu Bohan, Adeli Ehsan, et al.2022. "Vision." Pp. 28-33 in "On the Opportunities and Risks of Foundation Models." Written by Bommasani, Rishi et al. arXiv preprint arXiv:2108.07258.</bibtext> </blist> <blist> <bibtext> Buolamwini Joy. 2023. Unmasking AI: My Mission to Protect What is Human in a World of Machines. New York, NY: Penguin Random House.</bibtext> </blist> <blist> <bibtext> Buolamwini Joy, Gebru Timnit. 2018. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." In Conference on Fairness, Accountability and Transparency, pp. 77-91.</bibtext> </blist> <blist> <bibtext> Cantú Francisco. 2019. " The Fingerprints of Fraud: Evidence from Mexico's 1988 Presidential Election." American Political Science Review. 113(3):710–26.</bibtext> </blist> <blist> <bibtext> Casas Andreu, Webb Williams Nora. 2018. " Images That Matter: Online Protests and the Mobilizing Role of Pictures." Political Research Quarterly. 72(2):360–75.</bibtext> </blist> <blist> <bibtext> Chae Youngjn, Davidson Thomas. 2024. "Large Language models for Text Classification: From Zero-Shot Learning to Instruction-Tuning." SocArXiv. August 24. doi:10.31235/osf.io/sthwk.</bibtext> </blist> <blist> <bibtext> Chen Ting, Kornblith Simon, Norouzi Mohammad, Hinton Geoffrey. 2020. "A Simple Framework for Contrastive Learning of Visual Representations." In International Conference on Machine Learning (ICML), pp. 1597-1607.</bibtext> </blist> <blist> <bibtext> Cohen Jacob. 1960. " A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement. 20(1):37–46. https://doi.org/10.1177/001316446002000104</bibtext> </blist> <blist> <bibtext> Davidson Thomas. 2024. " Start Generating: Harnessing Generative Artificial Intelligence for Sociological Research." Socius. 10:23780231241259651.</bibtext> </blist> <blist> <bibtext> Dedoose. 2024. "Version 9.2.012, Cloud Application for Managing, Analyzing, and Presenting Qualitative and Mixed Method Research Data."</bibtext> </blist> <blist> <bibtext> Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, Fei-Fei Li. 2009 "ImageNet: A Large-Scale Hierarchical Image Database." In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 248–255.</bibtext> </blist> <blist> <bibtext> Dietrich Brian J., Sands Melissa L. 2023. " Seeing Racial Avoidance on New York City Streets." Nature Human Behavior. 7:1275–81.</bibtext> </blist> <blist> <bibtext> Edelmann Achim, Wolff Tom, Montagne Danielle, Bail Christopher A. 2020. " Computational Social Science and Sociology." Annual Review of Sociology. 46(1):61–81.</bibtext> </blist> <blist> <bibtext> Egami Naoki, Hinck Musahi, Stewart Brandon, Wei Hanying. 2024. "Using Large Language Model Annotations for the Social Sciences: A General Framework for Using Predicted Variables in Downstream Analyses." Working paper. https://naokiegami.com/paper/dsl_ss.pdf</bibtext> </blist> <blist> <bibtext> Fei-Fei Li, Iyer Asha, Koch Christof, Perona Pietro. 2007. " What Do We Perceive in a Glance of a Real-World Scene? " Journal of Vision. 7(1):10. 1–29.</bibtext> </blist> <blist> <bibtext> Finnigan Ryan. 2021. " The Growth and Shifting Spatial Distribution of Tent Encampments in Oakland, California." The ANNALS of the American Academy of Political and Social Science. 693(1):284–300. https://doi.org/10.1177/0002716221994459</bibtext> </blist> <blist> <bibtext> Gamer Matthias, Lemon Jim, Fellows Ian, Singh Puspendra. 2019. "irr: Various Coefficients of Interrater Reliability and Agreement." R package version 0.84.1. doi: 10.32614/CRAN.package.irr</bibtext> </blist> <blist> <bibtext> Gebru Timnit, Krause Jonathan, Wang Yilun, Chen Duyun, Deng Jia, Aiden Erez Lieberman, Fei-Fei Li. 2017. " Using Deep Learning and Google Street View to Estimate the Demographic Makeup of Neighborhoods Across the United States." Proceedings of the National Academy of Sciences. 114(50):13108–13.</bibtext> </blist> <blist> <bibtext> Goldstein Yoav, Legewie Nicolas M., Shiffer-Sebba Doron. 2023. " 3D Social Research: Analysis of Social Interaction Using Computer Vision." Sociological Methods & Research. 52(3):1201–38.</bibtext> </blist> <blist> <bibtext> Google.2024a. "Maps Static API."</bibtext> </blist> <blist> <bibtext> Google.2024b. "Street View Static API."</bibtext> </blist> <blist> <bibtext> Grannis Rick. 1998. " The Importance of Trivial Streets: Residential Streets and Residential Segregation." American Journal of Sociology. 103(6):1530–64. https://doi.org/10.1086/231400</bibtext> </blist> <blist> <bibtext> Grannis Rick. 2005. " T-communities: Pedestrian Street Networks and Residential Segregation in Chicago, Los Angeles, and New York." City and Community. 4(3):295–321.</bibtext> </blist> <blist> <bibtext> Grimmer Justin, Roberts Margaret E., Stewart Brandon M. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Hallgren Kevin A. 2012. " Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial." Tutorials in Quantitative Methods for Psychology. 8(1):23–4.</bibtext> </blist> <blist> <bibtext> He Kaiming, Fan Haoqi, Wu Yuxin, Xie Saining, Girshick Ross. 2019. "Momentum Contrast for Unsupervised Visual Representation Learning." arXiv preprint arXiv:1911.05722.</bibtext> </blist> <blist> <bibtext> Hénaff Olivier J., Koppula Skanda, Alayrac Jean-Baptiste, Van den Oord Aaron, Vinyals Oriol, Carreira Joao. 2021. "Efficient Visual Pretraining with Contrastive Detection." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10086-10096.</bibtext> </blist> <blist> <bibtext> Hipp John R., Lee Sugie, Ki Dong Hwan, Kim Jae Hong. 2025. " How Concentrated Disadvantage Moderates the Built Environment and Crime Relationship on Street Segments in Los Angeles." Criminology & Criminal Justice. 25(2):501–29.</bibtext> </blist> <blist> <bibtext> Hwang Jackleyn, Dahir Nima, Sarukkai Mayuka, Wright Gabby. 2023. " Curating Training Data for Reliable Large-Scale Visual Data Analysis: Lessons from Identifying Trash in Street View Imagery." Sociological Methods & Research. 52(3):1155–200. https://doi.org/10.1177/00491241231171945</bibtext> </blist> <blist> <bibtext> Hwang Jackleyn, Naik Nikhil. 2023. " Systematic Social Observation at Scale: Using Crowdsourcing and Computer Vision to Measure Visible Neighborhood Conditions." Sociological Methodology. 53(2):183–216. https://doi.org/10.1177/00811750231160781</bibtext> </blist> <blist> <bibtext> Hwang Jackelyn, Sampson Robert J. 2014. " Divergent Pathways of Gentrification: Racial Inequality and the Social Order of Renewal in Chicago Neighborhoods." American Sociological Review. 79(4):726–51. https://doi.org/10.1177/0003122414535774</bibtext> </blist> <blist> <bibtext> Jackson Kenneth T. 1985. Crabgrass Frontier: The Suburbanization of the United States. New York, NY: Oxford University Press.</bibtext> </blist> <blist> <bibtext> Jones Elliot. 2023. "Explainer: What is a Foundation Model?" Ada Lovelace Institute. Retrieved December 5, 2024. https://<ulink href="http://www.adalovelaceinstitute.org/resource/foundation-models-explainer/">www.adalovelaceinstitute.org/resource/foundation-models-explainer/</ulink>.</bibtext> </blist> <blist> <bibtext> Karrell Daniel, Freedman Michael, Gidron Noam. 2023. " Analyzing Text and Images in Digital Communication: The Case of Securitization in American White Supremacist Online Discourse." Socius. 9. doi: https://doi.org/10.1177/23780231231161049.</bibtext> </blist> <blist> <bibtext> Kesari Aniket, Kim Jae Yeon, Shah Sono, Brown Taylor, Ventura Tiago, Law Tina. 2024. " Training Computational Social Science PhD Students for Academic and non-Academic Careers." PS: Political Science & Politics. 57(1):101–6.</bibtext> </blist> <blist> <bibtext> Khattab Omar, Singhvi Arnav, Maheshwari Paridhi, Zhang Zhiyuan, Santhanam Keshav, Vardhamanan Sri, Haq Saiful, et al.2023. "Dspy: Compiling Declarative Language Model Calls into Self-improving Pipelines." arXiv preprint arXiv:2310.03714. https://arxiv.org/abs/2310.03714</bibtext> </blist> <blist> <bibtext> Kim Jae Hong, Ki Donghwan, Osutei Nene, Lee Sugie, Hipp John R. 2024. " Beyond Visual Inspection: Capturing Neighborhood Dynamics with Historical Google Street View and Deep Learning-Based Semantic Segmentation." Journal of Geographical Systems. 26(4):541–564.</bibtext> </blist> <blist> <bibtext> Kim Junsol, Lee Byungkyu. N.d."AI-Augmented Surveys: Leverage Large Language Models and Surveys for Opinion Prediction." Working paper.</bibtext> </blist> <blist> <bibtext> Korver-Glenn Elizabeth, Roberto Elizabeth, Binkovitz Leah, Mayorga Sarah. 2024. " Barriers and Boundaries: Built Environments, Meaning-Making, and Segregation." Sociological Perspectives. 67(4–6):261–88. https://doi.10.1177/07311214241274159</bibtext> </blist> <blist> <bibtext> Krippendorff Klaus. 2019. Content Analysis: An Introduction to Its Methodology. Thousand Oaks, CA: Sage Publications.</bibtext> </blist> <blist> <bibtext> Krizehvsky Alex, Sutskever Illya, Hinton Geoffrey E. 2012. " ImageNet Classification with Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems. 25.</bibtext> </blist> <blist> <bibtext> Landis Richard J., Koch Gary G. 1977. " The Measurement of Observer Agreement for Categorical Data." Biometrics. 33(1):159–74. doi: https://doi.org/10.2307/2529310.</bibtext> </blist> <blist> <bibtext> Law Tina, McCall Leslie. 2024. " Artificial Intelligence Policymaking: An Agenda for Sociological Research." Socius. 10:23780231241261596.</bibtext> </blist> <blist> <bibtext> Law Tina, Roberto Elizabeth. 2025. " Replication Data for: Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery." Harvard Dataverse. V1. doi: https://doi.org/10.7910/DVN/6QGZOA.</bibtext> </blist> <blist> <bibtext> Lazer David, Pentland Alex, Adamic Lada, Aral Sinan, Barabási Albert-László, Brewer Devon, Christakis Nicholas, et al.2009. " Computational Social Science." Science. 323(5915):721–3.</bibtext> </blist> <blist> <bibtext> LeCun Yann, Bengio Yoshua, Hinton Geoffrey. 2015. " Deep Learning." Nature. 521:436–44.</bibtext> </blist> <blist> <bibtext> LeCun Yann, Boser Bernhard, Denker John, Henderson Donnie, Howard R., Hubbard Wayne, Jackel Lawrence. 1990. "Handwritten Digit Recognition with a Back-propagation Network." In Proceeding Advances in Neural Information Processing Systems, pp. 396-404.</bibtext> </blist> <blist> <bibtext> LeCun Yann, Bottou Léon, Bengio Yoshua, Haffner Patrick. 1998. " Gradient-based Learning Applied to Document Recognition." Proceedings of the IEEE. 86(11):2278–324. doi: https://doi.org/10.1109/5.726791.</bibtext> </blist> <blist> <bibtext> Legewie Joscha, Schaeffer Merlin. 2016. " Contested Boundaries: Explaining Where Ethnoracial Diversity Provokes Neighborhood Conflict." American Journal of Sociology. 122(1):125–61.</bibtext> </blist> <blist> <bibtext> López Ochoa Esteban A., Zhai Wei. 2024. " Housing Perceptions and Code Enforcement: An Assessment of Demolition Orders Using Street View Imagery and Machine Intelligence." Journal of Planning Education and Research. doi: https://doi.org/10.1177/0739456X241233976.</bibtext> </blist> <blist> <bibtext> Maranca Rister Portinari, Alessandra Jihoon Chung, Hinck Musahi, Wolsky Adam D., Egami Naoki, Stewart Brandon M. 2025. "Correcting the Measurement Errors of AI-assisted Labeling in Image Analysis Using Design-based Supervised Learning." Working paper. https://naokiegami.com/paper/dsl_image.pdf</bibtext> </blist> <blist> <bibtext> Marcel Sébastien, Rodriguez Yann. 2010. "Torchvision the Machine-Vision Package of Torch." In Proceedings of the 18th ACM International Conference on Multimedia, pp. 1485-1488.</bibtext> </blist> <blist> <bibtext> Mohl Raymond A. 2008. " The Interstates and the Cities: The U.S. Department of Transportation and the Freeway Revolt, 1966–1973." Journal of Policy History. 20(2):193–226. doi: https://doi.org/10.1353/jph.0.0014</bibtext> </blist> <blist> <bibtext> Moravec Hans. 1988. Mind Children: The Future of Robot and Human Intelligence. Cambridge, MA: Harvard University Press.</bibtext> </blist> <blist> <bibtext> Naik Nikhil, Philipoom Jade, Raskar Ramesh, Hidalgo César. 2014. "Streetscore-predicting the Perceived Safety of One Million Streetscapes." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 779-785.</bibtext> </blist> <blist> <bibtext> Nassauer Anne, Legewie Nicolas M. 2021. " Video Data Analysis: A Methodological Frame for a Novel Research Trend." Sociological Methods & Research. 50(1):135–74.</bibtext> </blist> <blist> <bibtext> Nguyen Quynh C., Huang Yuru, Kumar Abhinav, Duan Haoshu, Keralis Jessica M., Dwivedi Pallavi, Meng Hsien-Wen, Brunisholz Kimberly D., Jay Jonathan, Javanmardi Mehran, Tasdizen Tolga. 2020. " Using 164 Million Google Street View Images to Derive Built Environment Predictors of COVID-19 Cases." International Journal of Environmental Research and Public Health. 17(17):6359.</bibtext> </blist> <blist> <bibtext> Palmer Alexis, Smith Noah A., Spirling Arthur. 2024. " Using Proprietary Language Models in Academic Research Requires Explicit Justification." Nature Computational Science. 4(1):2–3. doi: https://doi.org/10.1038/s43588-023-00585-1</bibtext> </blist> <blist> <bibtext> Radford Alec, Kim Jong Wook, Hallacy Chris, Ramesh Aditya, Goh Gabriel, Agarwal Sandhini, Sastry Girish, Askell Amanda, Mishkin Pamela, Clark Jack, et al.2021. "Learning Transferable Visual Models from Natural Language Supervision." In International Conference on Machine Learning, pp. 8748-8763.</bibtext> </blist> <blist> <bibtext> Raina Rajat, Madhavan Anand, Ng Andrew Y. 2009. "Large-scale Deep Unsupervised Learning Using Graphics Processors." In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 873-880.</bibtext> </blist> <blist> <bibtext> Raji Inioluwa Deborah, Gebru Timnit, Mitchell Margaret, Buolamwini Joy, Lee Joonseok, Denton Emily. 2020. "Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing." In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 145-151.</bibtext> </blist> <blist> <bibtext> Ramesh Aditya, Pavlov Mikhail, Goh Gabriel, Gray Scott, Voss Chelsea, Radford Alec, Chen Mark, Sutskever Ilya. 2021. "Zero-shot Text-to-image Generation." In International Conference on Machine Learning, pp. 8821-8831.</bibtext> </blist> <blist> <bibtext> Roberto Elizabeth. 2018. " The Spatial Proximity and Connectivity Method for Measuring and Analyzing Residential Segregation." Sociological Methodology. 48(1):182–224. doi: https://doi.org/10.1080/00045608.2012.685049</bibtext> </blist> <blist> <bibtext> Roberto Elizabeth, Korver-Glenn Elizabeth. 2021. " The Spatial Structure and Local Experience of Residential Segregation." Spatial Demography. 9(3):277–307. doi: https://doi.org/10.1007/s40980-021-00086-7</bibtext> </blist> <blist> <bibtext> Roberto Elizabeth, Zhu Yu, Segarra Santiago, Jalili Jaleh. Forthcoming. "Constructing Segregation: Examining Social and Spatial Division in Road Networks." Sociological Methodology.</bibtext> </blist> <blist> <bibtext> Robinson Caleb, Hohman Fred, Dilkina Bistra. 2017. "A Deep Learning Approach for Population Estimation from Satellite Imagery." In Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities, pp. 47-54.</bibtext> </blist> <blist> <bibtext> Salganik Matthew J. 2017. Bit by Bit: Social Research in the Digital Age. Princeton, NJ: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Schindler Sarah B. 2015. " Architectural Exclusion: Discrimination and Segregation Through Physical Design of the Built Environment." The Yale Law Journal. 124(6):1934–2024.</bibtext> </blist> <blist> <bibtext> Schwemmer Carsten, Knight Carly, Bello-Pardo Emily D., Oklobdzija Stan, Schoonvelde Martijn, Lockhart Jeffrey W. 2020. " Diagnosing Gender Bias in Image Recognition Systems." Socius. 6:2378023120967171.</bibtext> </blist> <blist> <bibtext> Schwemmer Carsten, Unger Saïd, Heiberger Raphael. 2023 "Automated Image Analysis for Studying Online Behaviour." Pp. 278–91 in Research Handbook on Digital Sociology, edited by J. Skopek. Cheltenham, UK: Edward Elgar Publishing.</bibtext> </blist> <blist> <bibtext> Sheng Hao, Yao Keniel, Goel Sharad. 2021. "Surveilling Surveillance: Estimating the Prevalence of Surveillance Cameras with Street View Data." In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 221-230.</bibtext> </blist> <blist> <bibtext> Sorensen Taylor, Robinson Joshua, Rytting Christopher Michael, Shaw Alexander Glenn, Rogers Kyle Jeffrey, Delorey Alexia Pauline, Khalil Mahmoud, Fulda Nancy, Wingate David. 2022. "An information-theoretic Approach to Prompt Engineering Without Ground Truth Labels." arXiv preprint arXiv:2203.11364. https://arxiv.org/abs/2203.11364</bibtext> </blist> <blist> <bibtext> Spirling Arthur. 2023. " Why Open-Source Generative AI Models are an Ethical way Forward for Science." Nature. 616(7957):413–413.</bibtext> </blist> <blist> <bibtext> Sugrue Thomas J. 2005. The Origins of the Urban Crisis: Race and Inequality in Postwar Detroit. Princeton, NJ: Princeton University Press.</bibtext> </blist> <blist> <bibtext> Tellman Beth, Sullivan Jonathan A., Kuhn Catherine, Kettner Albert J., Doyle Colin S., Robert Brakenridge G., Erickson Tyler A., Slayback Daniel A. 2021. " Satellite Imaging Reveals Increased Proportion of Population Exposed to Floods." Nature. 596(7870):80–6.</bibtext> </blist> <blist> <bibtext> Than Nga, Fan Leanne, Law Tina, Nelson Laura, McCall Leslie. 2025. "Updating "The Future of Coding:" Qualitative Coding with Generative Large Language Models." Working paper. https://osf.io/preprints/socarxiv/wg82k_v1</bibtext> </blist> <blist> <bibtext> Thorpe Simon, Fize Denis, Marlot Catherine. 1996. " Speed of Processing in the Human Visual System." Nature. 381(6582):520–2. https://doi.org/10.1038/381520a0</bibtext> </blist> <blist> <bibtext> Torres Michelle, Cantú Francisco. 2022. " Learning to See: Convolutional Neural Networks for the Analysis of Social Science Data." Political Analysis. 30(1):113–31. https://doi.org/10.1017/pan.2021.9</bibtext> </blist> <blist> <bibtext> U.S. Census Bureau. 2011. "2010 Census Summary File 1—United States."</bibtext> </blist> <blist> <bibtext> U.S. Census Bureau. 2012. "2010 TIGER/Line Shapefiles."</bibtext> </blist> <blist> <bibtext> Voulodimos Athanasios, Doulamis Nikolaos, Doulamis Anastasios, Protopapadakis Eftychios. 2018. " Deep Learning for Computer Vision: A Brief Review." Computational Intelligence and Neuroscience. 2018(1):7068349. doi: https://doi.org/10.1155/2018/7068349</bibtext> </blist> <blist> <bibtext> Wang Jiaqi, Liu Zhengliang, Zhao Lin, Wu Zihao, Ma Chong, Yu Sigang, Dai Haixing, et al.2023. " Review of Large Vision Models and Visual Prompt Engineering." Meta-Radiology. 100047.</bibtext> </blist> <blist> <bibtext> Webb Williams Nora, Casas Andreu, Wilkerson John D. 2020. Images as Data for Social Science Research: An Introduction to Convolutional Neural Nets for Image Classification. of Elements in Quantitative and Computational Methods for the Social Sciences. Cambridge: Cambridge University Press.</bibtext> </blist> <blist> <bibtext> White Jules, Fu Quchen, Hays Sam, Sandborn Michael, Olea Carlos, Gilbert Henry, Elnashar Ashraf, Spencer-Smith Jesse, Schmidt Douglas C. 2023. "A Prompt Pattern Catalog to Enhance Prompt Engineering with Chatgpt." arXiv preprint arXiv:2302.11382. https://arxiv.org/abs/2302.11382</bibtext> </blist> <blist> <bibtext> Yue Xiang, Ni Yuansheng, Zhang Kai, Zheng Tianyu, Liu Ruoqi, Zhang Ge, Stevens Samuel, et al.2024. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567.</bibtext> </blist> <blist> <bibtext> Zhang Yongjun. 2023. "Generative AI has Lowered the Barriers to Computational Social Sciences." arXiv preprint arXiv:2311.10833. https://arxiv.org/abs/2311.10833</bibtext> </blist> <blist> <bibtext> Zhang Han, Pan Jennifer. 2019. " Casm: A Deep-Learning Approach for Identifying Collective Action Events with Text and Image Data from Social media." Sociological Methodology. 49(1):1–57.</bibtext> </blist> <blist> <bibtext> Zhang Han, Peng Yilang. 2022. " Image Clustering: An Unsupervised Approach to Categorize Visual Data in Social Science Research." Sociological Methods & Research. 53(3):1534–87.</bibtext> </blist> <blist> <bibtext> Zhou Yongchao, Muresanu Andrei Ioan, Han Ziwen, Paster Keiran, Pitis Silviu, Chan Harris, Ba Jimmy. 2023. "Large Language Models Are Human-level Prompt Engineers." arXiv preprint arXiv:2211.01910. https://arxiv.org/abs/2211.01910</bibtext> </blist> <blist> <bibtext> Ziems Caleb, Held William, Shaikh Omar, Chen Jiaao, Zhang Zhehao, Yang Diyi. 2024. " Can Large Language Models Transform Computational Social Science? " Computational Linguistics. 50(1):237–91.</bibtext> </blist> </ref> <aug> <p>By Tina Law and Elizabeth Roberto</p> <p>Reported by Author; Author</p> <p></p> <p>Tina Law is an Assistant Professor of Sociology at the University of California, Davis. She studies inequality, race and ethnicity, democracy, and computational methodology. She uses computational and quantitative methods to understand the social and political experiences of racially minoritized and low-income Americans, particularly how they define and advance their goals for housing, safety, and political self-determination in cities that are often highly unequal. She also develops approaches for using AI and natural language processing to analyze text and image data.</p> <p>Elizabeth Roberto is an Assistant Professor in the Department of Sociology and the founding Co-Director of the Center for Computational Insight on Inequality and Society at Rice University. She has broad research interests in social and spatial inequality, a substantive focus on residential segregation, and methodological expertise in computational social science and quantitative methods. Her research uses innovative methods to examine the complex relationship between the social and built environment of cities. Her research has been supported by the James S. McDonnell Foundation, the American Sociological Association, and the National Academies.</p> </aug> <nolink nlid="nl1" bibid="bib48" firstref="ref3"></nolink> <nolink nlid="nl2" bibid="bib86" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib19" firstref="ref6"></nolink> <nolink nlid="nl4" bibid="bib96" firstref="ref7"></nolink> <nolink nlid="nl5" bibid="bib11" firstref="ref8"></nolink> <nolink nlid="nl6" bibid="bib25" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib34" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib64" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib87" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib66" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib80" firstref="ref19"></nolink> <nolink nlid="nl12" bibid="bib28" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib21" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib70" firstref="ref22"></nolink> <nolink nlid="nl15" bibid="bib56" firstref="ref24"></nolink> <nolink nlid="nl16" bibid="bib91" firstref="ref25"></nolink> <nolink nlid="nl17" bibid="bib51" firstref="ref30"></nolink> <nolink nlid="nl18" bibid="bib57" firstref="ref31"></nolink> <nolink nlid="nl19" bibid="bib58" firstref="ref32"></nolink> <nolink nlid="nl20" bibid="bib88" firstref="ref34"></nolink> <nolink nlid="nl21" bibid="bib93" firstref="ref35"></nolink> <nolink nlid="nl22" bibid="bib23" firstref="ref38"></nolink> <nolink nlid="nl23" bibid="bib55" firstref="ref39"></nolink> <nolink nlid="nl24" bibid="bib77" firstref="ref40"></nolink> <nolink nlid="nl25" bibid="bib45" firstref="ref41"></nolink> <nolink nlid="nl26" bibid="bib62" firstref="ref43"></nolink> <nolink nlid="nl27" bibid="bib40" firstref="ref45"></nolink> <nolink nlid="nl28" bibid="bib39" firstref="ref46"></nolink> <nolink nlid="nl29" bibid="bib47" firstref="ref47"></nolink> <nolink nlid="nl30" bibid="bib38" firstref="ref48"></nolink> <nolink nlid="nl31" bibid="bib97" firstref="ref49"></nolink> <nolink nlid="nl32" bibid="bib14" firstref="ref50"></nolink> <nolink nlid="nl33" bibid="bib98" firstref="ref51"></nolink> <nolink nlid="nl34" bibid="bib29" firstref="ref53"></nolink> <nolink nlid="nl35" bibid="bib43" firstref="ref62"></nolink> <nolink nlid="nl36" bibid="bib10" firstref="ref63"></nolink> <nolink nlid="nl37" bibid="bib17" firstref="ref66"></nolink> <nolink nlid="nl38" bibid="bib36" firstref="ref67"></nolink> <nolink nlid="nl39" bibid="bib37" firstref="ref68"></nolink> <nolink nlid="nl40" bibid="bib69" firstref="ref69"></nolink> <nolink nlid="nl41" bibid="bib72" firstref="ref70"></nolink> <nolink nlid="nl42" bibid="bib95" firstref="ref73"></nolink> <nolink nlid="nl43" bibid="bib32" firstref="ref84"></nolink> <nolink nlid="nl44" bibid="bib33" firstref="ref85"></nolink> <nolink nlid="nl45" bibid="bib73" firstref="ref86"></nolink> <nolink nlid="nl46" bibid="bib74" firstref="ref87"></nolink> <nolink nlid="nl47" bibid="bib49" firstref="ref88"></nolink> <nolink nlid="nl48" bibid="bib41" firstref="ref89"></nolink> <nolink nlid="nl49" bibid="bib26" firstref="ref93"></nolink> <nolink nlid="nl50" bibid="bib60" firstref="ref94"></nolink> <nolink nlid="nl51" bibid="bib67" firstref="ref95"></nolink> <nolink nlid="nl52" bibid="bib81" firstref="ref97"></nolink> <nolink nlid="nl53" bibid="bib76" firstref="ref98"></nolink> <nolink nlid="nl54" bibid="bib85" firstref="ref99"></nolink> <nolink nlid="nl55" bibid="bib15" firstref="ref102"></nolink> <nolink nlid="nl56" bibid="bib79" firstref="ref104"></nolink> <nolink nlid="nl57" bibid="bib50" firstref="ref111"></nolink> <nolink nlid="nl58" bibid="bib12" firstref="ref128"></nolink> <nolink nlid="nl59" bibid="bib13" firstref="ref129"></nolink> <nolink nlid="nl60" bibid="bib89" firstref="ref130"></nolink> <nolink nlid="nl61" bibid="bib75" firstref="ref131"></nolink> <nolink nlid="nl62" bibid="bib90" firstref="ref133"></nolink> <nolink nlid="nl63" bibid="bib30" firstref="ref139"></nolink> <nolink nlid="nl64" bibid="bib31" firstref="ref140"></nolink> <nolink nlid="nl65" bibid="bib42" firstref="ref154"></nolink> <nolink nlid="nl66" bibid="bib63" firstref="ref155"></nolink> <nolink nlid="nl67" bibid="bib78" firstref="ref156"></nolink> <nolink nlid="nl68" bibid="bib84" firstref="ref157"></nolink> <nolink nlid="nl69" bibid="bib16" firstref="ref159"></nolink> <nolink nlid="nl70" bibid="bib20" firstref="ref160"></nolink> <nolink nlid="nl71" bibid="bib46" firstref="ref163"></nolink> <nolink nlid="nl72" bibid="bib82" firstref="ref164"></nolink> <nolink nlid="nl73" bibid="bib94" firstref="ref166"></nolink> <nolink nlid="nl74" bibid="bib99" firstref="ref167"></nolink> <nolink nlid="nl75" bibid="bib100" firstref="ref168"></nolink> <nolink nlid="nl76" bibid="bib92" firstref="ref169"></nolink> <nolink nlid="nl77" bibid="bib18" firstref="ref172"></nolink> <nolink nlid="nl78" bibid="bib27" firstref="ref175"></nolink> <nolink nlid="nl79" bibid="bib35" firstref="ref176"></nolink> <nolink nlid="nl80" bibid="bib65" firstref="ref186"></nolink> <nolink nlid="nl81" bibid="bib24" firstref="ref189"></nolink> <nolink nlid="nl82" bibid="bib61" firstref="ref190"></nolink> <nolink nlid="nl83" bibid="bib71" firstref="ref194"></nolink> <nolink nlid="nl84" bibid="bib68" firstref="ref204"></nolink> <nolink nlid="nl85" bibid="bib83" firstref="ref205"></nolink> <nolink nlid="nl86" bibid="bib53" firstref="ref207"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1475739
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Tina+Law%22">Tina Law</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7631-6763">0000-0001-7631-6763</externalLink>)<br /><searchLink fieldCode="AR" term="%22Elizabeth+Roberto%22">Elizabeth Roberto</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7667-6953">0000-0001-7667-6953</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Sociological+Methods+%26+Research%22"><i>Sociological Methods & Research</i></searchLink>. 2025 54(3):889-932.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 44
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Visual+Aids%22">Visual Aids</searchLink><br /><searchLink fieldCode="DE" term="%22Open+Source+Technology%22">Open Source Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Science+Research%22">Social Science Research</searchLink><br /><searchLink fieldCode="DE" term="%22Neighborhood+Integration%22">Neighborhood Integration</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Problems%22">Research Problems</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/00491241251339673
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0049-1241<br />1552-8294
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Although there is growing social science research examining how generative AI models can be effectively and systematically applied to text-based tasks, whether and how these models can be used to analyze images remain open questions. In this article, we introduce a framework for analyzing images with generative multimodal models, which consists of three core tasks: curation, discovery, and measurement and inference. We demonstrate this framework with an empirical application that uses OpenAI's GPT-4o model to analyze satellite and streetscape images (n = 1,101) to identify built environment features that contribute to contemporary residential segregation in U.S. cities. We find that when GPT-4o is provided with well-defined image labels, the model labels images with high validity compared to expert labels. We conclude with thoughts for other use cases and discuss how social scientists can work collaboratively to ensure that image analysis with generative multimodal models is rigorous, reproducible, ethical, and sustainable.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1475739
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1475739
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/00491241251339673
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 44
        StartPage: 889
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Visual Aids
        Type: general
      – SubjectFull: Open Source Technology
        Type: general
      – SubjectFull: Social Science Research
        Type: general
      – SubjectFull: Neighborhood Integration
        Type: general
      – SubjectFull: Data Analysis
        Type: general
      – SubjectFull: Research Problems
        Type: general
    Titles:
      – TitleFull: Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Tina Law
      – PersonEntity:
          Name:
            NameFull: Elizabeth Roberto
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0049-1241
            – Type: issn-electronic
              Value: 1552-8294
          Numbering:
            – Type: volume
              Value: 54
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Sociological Methods & Research
              Type: main
ResultId 1