Balancing Large Language Model Alignment and Algorithmic Fidelity in Social Science Research
Saved in:
| Title: | Balancing Large Language Model Alignment and Algorithmic Fidelity in Social Science Research |
|---|---|
| Language: | English |
| Authors: | Alex Lyman (ORCID |
| Source: | Sociological Methods & Research. 2025 54(3):1110-1155. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 46 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Artificial Intelligence, Computer Simulation, Open Source Technology, Social Science Research, Models, Fidelity, Algorithms |
| DOI: | 10.1177/00491241251342008 |
| ISSN: | 0049-1241 1552-8294 |
| Abstract: | Generative artificial intelligence (AI) has the potential to revolutionize social science research. However, researchers face the difficult challenge of choosing a specific AI model, often without social science-specific guidance. To demonstrate the importance of this choice, we present an evaluation of the effect of alignment, or human-driven modification, on the ability of large language models (LLMs) to simulate the attitudes of human populations (sometimes called "silicon sampling"). We benchmark aligned and unaligned versions of six open-source LLMs against each other and compare them to similar responses by humans. Our results suggest that model alignment impacts output in predictable ways, with implications for prompting, task completion, and the substantive content of LLM-based results. We conclude that researchers must be aware of the complex ways in which model training affects their research and carefully consider model choice for each project. We discuss future steps to improve how social scientists work with generative AI tools. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1475767 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Generative artificial intelligence (AI) has the potential to revolutionize social science research. However, researchers face the difficult challenge of choosing a specific AI model, often without social science-specific guidance. To demonstrate the importance of this choice, we present an evaluation of the effect of alignment, or human-driven modification, on the ability of large language models (LLMs) to simulate the attitudes of human populations (sometimes called "silicon sampling"). We benchmark aligned and unaligned versions of six open-source LLMs against each other and compare them to similar responses by humans. Our results suggest that model alignment impacts output in predictable ways, with implications for prompting, task completion, and the substantive content of LLM-based results. We conclude that researchers must be aware of the complex ways in which model training affects their research and carefully consider model choice for each project. We discuss future steps to improve how social scientists work with generative AI tools. |
|---|---|
| ISSN: | 0049-1241 1552-8294 |
| DOI: | 10.1177/00491241251342008 |
