View in EDS

Finding Words Associated with DIF: Predicting Differential Item Functioning Using LLMs and Explainable AI

Saved in:
Bibliographic Details
Title: Finding Words Associated with DIF: Predicting Differential Item Functioning Using LLMs and Explainable AI
Language: English
Authors: Hotaka Maeda (ORCID 0009-0000-9498-786X), Yikai Lu (ORCID 0000-0003-4410-2589)
Source: Journal of Educational Measurement. 2025 62(4):883-906.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 24
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Secondary Education
Descriptors: Artificial Intelligence, Prediction, Test Bias, Test Items, Vocabulary, Language Arts, Mathematics, Summative Evaluation, Elementary Secondary Education, Test Content
DOI: 10.1111/jedm.70017
ISSN: 0022-0655
1745-3984
Abstract: We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to identify specific words associated with the DIF prediction. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction R[superscript 2] ranged from 0.04 to 0.32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor subdomains included in the test blueprint by design, rather than construct-irrelevant content that may need to be removed from assessments. This may explain why qualitative reviews of DIF items often yield inconclusive results. Our approach can be used to (1) screen words associated with DIF during the item-writing process for immediate revision to reduce preventable adverse DIF, (2) assist traditional DIF item reviews by highlighting key words, or (3) use DIF prediction as an alternative when obtaining sufficient sample size for traditional DIF analyses is impossible. Extensions of this research can enhance the assessment fairness, especially programs that lack resources to build high-quality items, and among smaller subpopulations with insufficient sample sizes for traditional DIF analyses.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1491342
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1491342
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Finding Words Associated with DIF: Predicting Differential Item Functioning Using LLMs and Explainable AI
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hotaka+Maeda%22">Hotaka Maeda</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0000-9498-786X">0009-0000-9498-786X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Yikai+Lu%22">Yikai Lu</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4410-2589">0000-0003-4410-2589</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2025 62(4):883-906.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Vocabulary%22">Vocabulary</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Arts%22">Language Arts</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics%22">Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22Summative+Evaluation%22">Summative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Content%22">Test Content</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.70017
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We fine-tuned and compared several encoder-based Transformer large language models (LLM) to predict differential item functioning (DIF) from the item text. We then applied explainable artificial intelligence (XAI) methods to identify specific words associated with the DIF prediction. The data included 42,180 items designed for English language arts and mathematics summative state assessments among students in grades 3 to 11. Prediction R[superscript 2] ranged from 0.04 to 0.32 among eight focal and reference group pairs. Our findings suggest that many words associated with DIF reflect minor subdomains included in the test blueprint by design, rather than construct-irrelevant content that may need to be removed from assessments. This may explain why qualitative reviews of DIF items often yield inconclusive results. Our approach can be used to (1) screen words associated with DIF during the item-writing process for immediate revision to reduce preventable adverse DIF, (2) assist traditional DIF item reviews by highlighting key words, or (3) use DIF prediction as an alternative when obtaining sufficient sample size for traditional DIF analyses is impossible. Extensions of this research can enhance the assessment fairness, especially programs that lack resources to build high-quality items, and among smaller subpopulations with insufficient sample sizes for traditional DIF analyses.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1491342
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1491342
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.70017
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 883
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Vocabulary
        Type: general
      – SubjectFull: Language Arts
        Type: general
      – SubjectFull: Mathematics
        Type: general
      – SubjectFull: Summative Evaluation
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Test Content
        Type: general
    Titles:
      – TitleFull: Finding Words Associated with DIF: Predicting Differential Item Functioning Using LLMs and Explainable AI
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hotaka Maeda
      – PersonEntity:
          Name:
            NameFull: Yikai Lu
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 62
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1