View in EDS

Evaluating General-Purpose Multimodal AI for Q-Matrix Generation from Math Items: A Cognitive Diagnostic Modeling Exploration

Saved in:
Bibliographic Details
Title: Evaluating General-Purpose Multimodal AI for Q-Matrix Generation from Math Items: A Cognitive Diagnostic Modeling Exploration
Language: English
Authors: Kang Xue (ORCID 0000-0003-2161-6931), James J. Appleton
Source: Journal of Educational Measurement. 2026 63(1).
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 24
Publication Date: 2026
Document Type: Journal Articles
Reports - Evaluative
Education Level: Elementary Secondary Education
Secondary Education
Descriptors: Cognitive Measurement, Models, Artificial Intelligence, Mathematics Tests, Test Items, Matrices, Achievement Tests, Elementary Secondary Education, Mathematics Achievement, Foreign Countries, International Assessment, Secondary School Students
Assessment and Survey Identifiers: Trends in International Mathematics and Science Study, Program for International Student Assessment
DOI: 10.1111/jedm.70028
ISSN: 0022-0655
1745-3984
Abstract: Cognitive Diagnostic Models (CDMs) provide fine-grained diagnostic feedback, but their central component--the Q-matrix--remains costly and labor-intensive to construct. This study explores the automated generation of Q-matrices using general-purpose AI, including ChatGPT-4o, Gemini-2.5-pro, and Claude-sonnet-4. We evaluated two prompting strategies (all-at-once and one-by-one) across TIMSS 2007, TIMSS 2011, and PISA 2012 mathematics assessments. Results show that AI-generated Q-matrices approximate human baselines with competitive model fitting performance (AIC, BIC, log-likelihood, and SRMSR) and acceptable classification discrepancies. While AI predictions for larger and more complicated assessments (TIMSS 07 and 11) were generally sparser than human-generated Q-matrices, they still achieved equal or better fit statistics under most CDMs. In contrast, for the smaller and less complicated PISA 2012 assessment, AI-generated Q-matrices matched human density and fitting quality. Importantly, chatbot-human matching accuracy remained high across models, with Gemini benefiting from all-at-once prompting, ChatGPT-4o maintaining stable performance under both strategies, and Claude showing sensitivity to prompt structure. These findings highlight both the promise and current limitations of automated Q-matrix generation, underscoring opportunities for integrating LLMs into scalable diagnostic assessment practices.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1501282
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1501282
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluating General-Purpose Multimodal AI for Q-Matrix Generation from Math Items: A Cognitive Diagnostic Modeling Exploration
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Kang+Xue%22">Kang Xue</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2161-6931">0000-0003-2161-6931</externalLink>)<br /><searchLink fieldCode="AR" term="%22James+J%2E+Appleton%22">James J. Appleton</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2026 63(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Cognitive+Measurement%22">Cognitive Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Matrices%22">Matrices</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Trends+in+International+Mathematics+and+Science+Study%22">Trends in International Mathematics and Science Study</searchLink><br /><searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.70028
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Cognitive Diagnostic Models (CDMs) provide fine-grained diagnostic feedback, but their central component--the Q-matrix--remains costly and labor-intensive to construct. This study explores the automated generation of Q-matrices using general-purpose AI, including ChatGPT-4o, Gemini-2.5-pro, and Claude-sonnet-4. We evaluated two prompting strategies (all-at-once and one-by-one) across TIMSS 2007, TIMSS 2011, and PISA 2012 mathematics assessments. Results show that AI-generated Q-matrices approximate human baselines with competitive model fitting performance (AIC, BIC, log-likelihood, and SRMSR) and acceptable classification discrepancies. While AI predictions for larger and more complicated assessments (TIMSS 07 and 11) were generally sparser than human-generated Q-matrices, they still achieved equal or better fit statistics under most CDMs. In contrast, for the smaller and less complicated PISA 2012 assessment, AI-generated Q-matrices matched human density and fitting quality. Importantly, chatbot-human matching accuracy remained high across models, with Gemini benefiting from all-at-once prompting, ChatGPT-4o maintaining stable performance under both strategies, and Claude showing sensitivity to prompt structure. These findings highlight both the promise and current limitations of automated Q-matrix generation, underscoring opportunities for integrating LLMs into scalable diagnostic assessment practices.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1501282
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501282
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.70028
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
    Subjects:
      – SubjectFull: Cognitive Measurement
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Matrices
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Trends in International Mathematics and Science Study
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: Evaluating General-Purpose Multimodal AI for Q-Matrix Generation from Math Items: A Cognitive Diagnostic Modeling Exploration
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kang Xue
      – PersonEntity:
          Name:
            NameFull: James J. Appleton
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 63
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1