Comparing Zero-Shot Large Language Model Prompting with Human Coding of Theory Concepts in Student Essays
Saved in:
| Title: | Comparing Zero-Shot Large Language Model Prompting with Human Coding of Theory Concepts in Student Essays |
|---|---|
| Language: | English |
| Authors: | Shelley Keith, Philip I. Pavlik, Kristen L. Stives, Laura Jean Kerr |
| Source: | Journal of Educational Data Mining. 2026 18(1):286-317. |
| Availability: | International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM |
| Peer Reviewed: | Y |
| Page Count: | 32 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research Tests/Questionnaires |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Artificial Intelligence, Natural Language Processing, Prompting, Writing Evaluation, Essays, Scoring, Automation, Computer Assisted Testing, Coding, Accuracy, College Students, Error Patterns, Bias, Criminology, Required Courses, Majors (Students), Computation, Correlation |
| ISSN: | 2157-2100 |
| Abstract: | Recent studies have explored the cost and time benefits of using artificial intelligence (AI), particularly large language models (LLMs), in coding student essays. While these models show promise, not enough is understood about the factors that affect how their qualitative coding performance compares to human coding. This study examines coding accuracy for content errors in college student essays on criminological theories by comparing human-coded results with outputs from four LLMs. We evaluated human-AI correlations, AI error, and AI bias across four LLMs, five prompt types, three theory content coding dimensions, and four criminological theories. Results indicate that LLM choice significantly influenced human-AI correspondence, with Claude Sonnet 4 exhibiting the best overall performance and GPT 4.1 Mini the worst. Prompt type had minimal impact on performance. Across models, error rates were lowest when identifying whether students listed a concept, and highest when assessing whether definitions were correct. LLMs performed better on concise theories than on more complex ones. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1506614 |
| Database: | ERIC |
| Abstract: | Recent studies have explored the cost and time benefits of using artificial intelligence (AI), particularly large language models (LLMs), in coding student essays. While these models show promise, not enough is understood about the factors that affect how their qualitative coding performance compares to human coding. This study examines coding accuracy for content errors in college student essays on criminological theories by comparing human-coded results with outputs from four LLMs. We evaluated human-AI correlations, AI error, and AI bias across four LLMs, five prompt types, three theory content coding dimensions, and four criminological theories. Results indicate that LLM choice significantly influenced human-AI correspondence, with Claude Sonnet 4 exhibiting the best overall performance and GPT 4.1 Mini the worst. Prompt type had minimal impact on performance. Across models, error rates were lowest when identifying whether students listed a concept, and highest when assessing whether definitions were correct. LLMs performed better on concise theories than on more complex ones. |
|---|---|
| ISSN: | 2157-2100 |
