Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods
Saved in:
| Title: | Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods |
|---|---|
| Language: | English |
| Authors: | Ya Wang (ORCID |
| Source: | Language Testing. 2026 43(2):141-164. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 24 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Second Language Learning, Listening Comprehension Tests, Foreign Countries, Undergraduate Students, Psychometrics, Artificial Intelligence, Multiple Choice Tests, English (Second Language), Technology Uses in Education, Language Tests, Expertise, Test Construction |
| Geographic Terms: | China |
| DOI: | 10.1177/02655322251400375 |
| ISSN: | 0265-5322 1477-0946 |
| Abstract: | This study explores the integration of generative artificial intelligence (GenAI) with human experts to improve the quality of distractors in multiple-choice questions (MCQs) for second language (L2) listening tests. A psychometric analysis of responses from 2267 EFL Chinese undergraduates, using the two-parameter logistic nested logit model (2PLNLM), identified problematic items and distractors. Guided by established distractor design principles, GenAI was applied iteratively to refine these distractors, and GenAI was iteratively used to revise these distractors, with human experts providing ongoing feedback throughout the process. The revised versions were then evaluated by expert judgment and NLP-based cosine similarity analysis. The results indicate that GenAI effectively enhanced distractor quality by maintaining content and structural alignment and ensuring semantic independence. However, it struggled to fully capture listening miscomprehension patterns and contextualized language use. These preliminary findings suggest that GenAI revisions, guided by principle-based prompts and supervised by humans, tend to effectively improve the quality of distractors. This study offers practical insights into the potential and limitations of GenAI in improving L2 listening tests. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1501926 |
| Database: | ERIC |
| Abstract: | This study explores the integration of generative artificial intelligence (GenAI) with human experts to improve the quality of distractors in multiple-choice questions (MCQs) for second language (L2) listening tests. A psychometric analysis of responses from 2267 EFL Chinese undergraduates, using the two-parameter logistic nested logit model (2PLNLM), identified problematic items and distractors. Guided by established distractor design principles, GenAI was applied iteratively to refine these distractors, and GenAI was iteratively used to revise these distractors, with human experts providing ongoing feedback throughout the process. The revised versions were then evaluated by expert judgment and NLP-based cosine similarity analysis. The results indicate that GenAI effectively enhanced distractor quality by maintaining content and structural alignment and ensuring semantic independence. However, it struggled to fully capture listening miscomprehension patterns and contextualized language use. These preliminary findings suggest that GenAI revisions, guided by principle-based prompts and supervised by humans, tend to effectively improve the quality of distractors. This study offers practical insights into the potential and limitations of GenAI in improving L2 listening tests. |
|---|---|
| ISSN: | 0265-5322 1477-0946 |
| DOI: | 10.1177/02655322251400375 |
