View in EDS

Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods

Saved in:
Bibliographic Details
Title: Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods
Language: English
Authors: Ya Wang (ORCID 0000-0001-5612-2486), Yaru Meng (ORCID 0000-0003-4154-3776)
Source: Language Testing. 2026 43(2):141-164.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 24
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Second Language Learning, Listening Comprehension Tests, Foreign Countries, Undergraduate Students, Psychometrics, Artificial Intelligence, Multiple Choice Tests, English (Second Language), Technology Uses in Education, Language Tests, Expertise, Test Construction
Geographic Terms: China
DOI: 10.1177/02655322251400375
ISSN: 0265-5322
1477-0946
Abstract: This study explores the integration of generative artificial intelligence (GenAI) with human experts to improve the quality of distractors in multiple-choice questions (MCQs) for second language (L2) listening tests. A psychometric analysis of responses from 2267 EFL Chinese undergraduates, using the two-parameter logistic nested logit model (2PLNLM), identified problematic items and distractors. Guided by established distractor design principles, GenAI was applied iteratively to refine these distractors, and GenAI was iteratively used to revise these distractors, with human experts providing ongoing feedback throughout the process. The revised versions were then evaluated by expert judgment and NLP-based cosine similarity analysis. The results indicate that GenAI effectively enhanced distractor quality by maintaining content and structural alignment and ensuring semantic independence. However, it struggled to fully capture listening miscomprehension patterns and contextualized language use. These preliminary findings suggest that GenAI revisions, guided by principle-based prompts and supervised by humans, tend to effectively improve the quality of distractors. This study offers practical insights into the potential and limitations of GenAI in improving L2 listening tests.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1501926
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1501926
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ya+Wang%22">Ya Wang</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5612-2486">0000-0001-5612-2486</externalLink>)<br /><searchLink fieldCode="AR" term="%22Yaru+Meng%22">Yaru Meng</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4154-3776">0000-0003-4154-3776</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Language+Testing%22"><i>Language Testing</i></searchLink>. 2026 43(2):141-164.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Listening+Comprehension+Tests%22">Listening Comprehension Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Multiple+Choice+Tests%22">Multiple Choice Tests</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Expertise%22">Expertise</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22China%22">China</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/02655322251400375
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0265-5322<br />1477-0946
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study explores the integration of generative artificial intelligence (GenAI) with human experts to improve the quality of distractors in multiple-choice questions (MCQs) for second language (L2) listening tests. A psychometric analysis of responses from 2267 EFL Chinese undergraduates, using the two-parameter logistic nested logit model (2PLNLM), identified problematic items and distractors. Guided by established distractor design principles, GenAI was applied iteratively to refine these distractors, and GenAI was iteratively used to revise these distractors, with human experts providing ongoing feedback throughout the process. The revised versions were then evaluated by expert judgment and NLP-based cosine similarity analysis. The results indicate that GenAI effectively enhanced distractor quality by maintaining content and structural alignment and ensuring semantic independence. However, it struggled to fully capture listening miscomprehension patterns and contextualized language use. These preliminary findings suggest that GenAI revisions, guided by principle-based prompts and supervised by humans, tend to effectively improve the quality of distractors. This study offers practical insights into the potential and limitations of GenAI in improving L2 listening tests.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1501926
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501926
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/02655322251400375
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 141
    Subjects:
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Listening Comprehension Tests
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: Expertise
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: China
        Type: general
    Titles:
      – TitleFull: Optimizing Distractor Quality in a Locally Developed Second Language Listening Test: Integrating Generative AI and Psychometric Methods
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ya Wang
      – PersonEntity:
          Name:
            NameFull: Yaru Meng
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0265-5322
            – Type: issn-electronic
              Value: 1477-0946
          Numbering:
            – Type: volume
              Value: 43
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Language Testing
              Type: main
ResultId 1