A comparison of the psychometric properties of GPT-4 versus human novice and expert authors of clinically complex MCQs in a mock examination of Australian medical students.
Saved in:
| Title: | A comparison of the psychometric properties of GPT-4 versus human novice and expert authors of clinically complex MCQs in a mock examination of Australian medical students. |
|---|---|
| Authors: | Wu, Hannah1,2 (AUTHOR) hannah.wu@adelaide.edu.au, Lee, Daniel3 (AUTHOR), Zerner, Toby2 (AUTHOR), Court-Kowalski, Stefan1,2 (AUTHOR), Devitt, Peter2 (AUTHOR), Palmer, Edward3 (AUTHOR) |
| Source: | Medical Teacher. Jan2026, Vol. 48 Issue 1, p74-84. 11p. |
| Subject Terms: | *Generative artificial intelligence, *Data analysis, *Undergraduate programs, *Educational tests & measurements, *Medical students, *Clinical competence, *Comparative studies, Cronbach's alpha, Research evaluation, Fisher exact test, Descriptive statistics, Psychometrics, Analysis of variance, Statistics, Data analysis software |
| Geographic Terms: | Australia |
| Abstract: | Purpose: Creating clinically complex Multiple Choice Questions (MCQs) for medical assessment can be time-consuming. Large language models such as GPT-4, a type of generative artificial intelligence (AI), are a potential MCQ design tool. Evaluating the psychometric properties of AI-generated MCQs is essential to ensuring quality. Methods: A 120-item mock examination was constructed, containing 40 human-generated MCQs at novice item-writer level, 40 at expert level, and 40 AI-generated MCQs. int. All examination items underwent panel review to ensure they tested higher order cognitive skills and met a minimum acceptable standard. The online mock examination was administered to Australian medical students, who were blinded to each item's author. Results: 234 medical students completed the examination. Analysis showed acceptable reliability (Cronbach's 0.836). There were no differences in item difficulty or discrimination between AI, Novice, and Expert items. The mean item difficulty was 'easy' and mean item discrimination 'fair' across all groups. AI items had lower distractor efficiency (39%) compared to Novice items (55%, p = 0.035), but no difference to Expert items (48%, p = 0.382). Conclusions: The psychometric properties of AI-generated MCQs are comparable to human-generated MCQs at both novice and expert level. Item quality can be improved across all author groups. AI-generated items should undergo human review to enhance distractor efficiency. [ABSTRACT FROM AUTHOR] |
| Copyright of Medical Teacher is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Purpose: Creating clinically complex Multiple Choice Questions (MCQs) for medical assessment can be time-consuming. Large language models such as GPT-4, a type of generative artificial intelligence (AI), are a potential MCQ design tool. Evaluating the psychometric properties of AI-generated MCQs is essential to ensuring quality. Methods: A 120-item mock examination was constructed, containing 40 human-generated MCQs at novice item-writer level, 40 at expert level, and 40 AI-generated MCQs. int. All examination items underwent panel review to ensure they tested higher order cognitive skills and met a minimum acceptable standard. The online mock examination was administered to Australian medical students, who were blinded to each item's author. Results: 234 medical students completed the examination. Analysis showed acceptable reliability (Cronbach's 0.836). There were no differences in item difficulty or discrimination between AI, Novice, and Expert items. The mean item difficulty was 'easy' and mean item discrimination 'fair' across all groups. AI items had lower distractor efficiency (39%) compared to Novice items (55%, p = 0.035), but no difference to Expert items (48%, p = 0.382). Conclusions: The psychometric properties of AI-generated MCQs are comparable to human-generated MCQs at both novice and expert level. Item quality can be improved across all author groups. AI-generated items should undergo human review to enhance distractor efficiency. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 0142159X |
| DOI: | 10.1080/0142159X.2025.2513418 |
