ChatGPT vs. DeepSeek: A Comparative Psychometric Evaluation of AI Tools in Generating Multiple-Choice Questions
Saved in:
| Title: | ChatGPT vs. DeepSeek: A Comparative Psychometric Evaluation of AI Tools in Generating Multiple-Choice Questions |
|---|---|
| Language: | English |
| Authors: | Ceylan Gündeger Kilci (ORCID |
| Source: | International Journal of Assessment Tools in Education. 2025 12(4):1055-1079. |
| Availability: | International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate |
| Peer Reviewed: | Y |
| Page Count: | 25 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Psychometrics, Multiple Choice Tests, Artificial Intelligence, Natural Language Processing, Test Items, Item Analysis, Content Validity, Difficulty Level, Statistical Significance, Scores, Undergraduate Students, Schools of Education, Technology Uses in Education, Test Reliability, Test Validity, Test Construction, Foreign Countries, Expertise, Feedback (Response) |
| Geographic Terms: | Turkey |
| ISSN: | 2148-7456 |
| Abstract: | This study examined the psychometric quality of multiple-choice questions generated by two AI tools, ChatGPT and DeepSeek, within the context of an undergraduate Educational Measurement and Evaluation course. Guided by ten learning outcomes (LOs) aligned with Bloom's Taxonomy, each tool was prompted to generate one five-option multiple-choice item per LO. Following expert review (Kendall's "W" = 0.58); revisions were made, and the finalized test was administered to 120 students. Item analyses revealed no statistically significant differences between the two AI models regarding item difficulty, discrimination, variance, or reliability. A few items--two from ChatGPT and one from DeepSeek--had suboptimal discrimination indices. Tetrachoric correlation analyses of item pairs generated by the two AI tools for the same LO revealed that only one pair showed a non-significant association, whereas all other pairs demonstrated statistically significant and generally moderate correlations. KR-20 and split-half reliability coefficients reflected acceptable internal consistency for a classroom-based assessment, with the DeepSeek-generated half showing a slightly stronger correlation with total scores. Expert feedback indicated that while AI tools generally produced valid stems and correct answers, most revisions focused on improving distractor quality, highlighting the need for human refinement. Generalizability and Decision studies confirmed consistency in expert ratings and recommended a minimum of seven experts for reliable evaluations. In conclusion, both AI tools demonstrated the capacity to generate psychometrically comparable items, highlighting their potential to support educators and test developers in test construction. The study concludes with practical recommendations for effectively incorporating AI into test development workflows. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1491386 |
| Database: | ERIC |
| Abstract: | This study examined the psychometric quality of multiple-choice questions generated by two AI tools, ChatGPT and DeepSeek, within the context of an undergraduate Educational Measurement and Evaluation course. Guided by ten learning outcomes (LOs) aligned with Bloom's Taxonomy, each tool was prompted to generate one five-option multiple-choice item per LO. Following expert review (Kendall's "W" = 0.58); revisions were made, and the finalized test was administered to 120 students. Item analyses revealed no statistically significant differences between the two AI models regarding item difficulty, discrimination, variance, or reliability. A few items--two from ChatGPT and one from DeepSeek--had suboptimal discrimination indices. Tetrachoric correlation analyses of item pairs generated by the two AI tools for the same LO revealed that only one pair showed a non-significant association, whereas all other pairs demonstrated statistically significant and generally moderate correlations. KR-20 and split-half reliability coefficients reflected acceptable internal consistency for a classroom-based assessment, with the DeepSeek-generated half showing a slightly stronger correlation with total scores. Expert feedback indicated that while AI tools generally produced valid stems and correct answers, most revisions focused on improving distractor quality, highlighting the need for human refinement. Generalizability and Decision studies confirmed consistency in expert ratings and recommended a minimum of seven experts for reliable evaluations. In conclusion, both AI tools demonstrated the capacity to generate psychometrically comparable items, highlighting their potential to support educators and test developers in test construction. The study concludes with practical recommendations for effectively incorporating AI into test development workflows. |
|---|---|
| ISSN: | 2148-7456 |
