Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude.
Saved in:
| Title: | Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude. |
|---|---|
| Authors: | Yapıcı Coşkun, Zuhal1,2 (AUTHOR) zuhalyapici@yahoo.com, Kıyak, Yavuz Selim3 (AUTHOR), Coşkun, Özlem3 (AUTHOR), Budakoğlu, Işıl İrem3 (AUTHOR), Özdemir, Özhan4 (AUTHOR) |
| Source: | Medical Teacher. Nov2025, Vol. 47 Issue 11, p1767-1771. 5p. |
| Subject Terms: | *Generative artificial intelligence, *Medical education, *Undergraduates, *Educational tests & measurements, *Medical students, Medical logic, Cross-sectional method, Scale analysis (Psychology), Primary health care, Descriptive statistics, Gynecology, Chatbots, Obstetrics |
| Abstract: | Objective: To evaluate the performance of large language models (ChatGPT-4o and Claude 3.5 Sonnet) to generate script concordance test (SCT) items for assessing clinical reasoning in obstetrics and gynecology. Methods: This cross-sectional study involved the generation of SCT items for five common diagnostic topics in obstetrics and gynecology in primary care settings. A total of 16 panelists evaluated the AI-generated SCT items against 11 predefined criteria. Descriptive statistics were used to compare the models' performance across criteria. Results: ChatGPT-4o had an overall agreement rate of 90.57% for SCT items meeting the quality criteria, while Claude 3.5 Sonnet achieved 91.48%. The criterion with the lowest scores was "The scenario is of appropriate difficulty for medical students," with ChatGPT-4o rated at 71.25% and Claude 3.5 Sonnet at 76.25%. Conclusion: Large language models can generate SCT items that effectively assess clinical reasoning; however, further refinement is required to ensure the appropriate level of difficulty for medical students. These findings highlight the potential of AI to enhance the efficiency of SCT generation in obstetrics and gynecology within primary care settings. [ABSTRACT FROM AUTHOR] |
| Copyright of Medical Teacher is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Objective: To evaluate the performance of large language models (ChatGPT-4o and Claude 3.5 Sonnet) to generate script concordance test (SCT) items for assessing clinical reasoning in obstetrics and gynecology. Methods: This cross-sectional study involved the generation of SCT items for five common diagnostic topics in obstetrics and gynecology in primary care settings. A total of 16 panelists evaluated the AI-generated SCT items against 11 predefined criteria. Descriptive statistics were used to compare the models' performance across criteria. Results: ChatGPT-4o had an overall agreement rate of 90.57% for SCT items meeting the quality criteria, while Claude 3.5 Sonnet achieved 91.48%. The criterion with the lowest scores was "The scenario is of appropriate difficulty for medical students," with ChatGPT-4o rated at 71.25% and Claude 3.5 Sonnet at 76.25%. Conclusion: Large language models can generate SCT items that effectively assess clinical reasoning; however, further refinement is required to ensure the appropriate level of difficulty for medical students. These findings highlight the potential of AI to enhance the efficiency of SCT generation in obstetrics and gynecology within primary care settings. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 0142159X |
| DOI: | 10.1080/0142159X.2025.2497888 |
