View in EDS

The Performance of ChatGPT and Other Large Language Models on Multiple-Choice Questions in Biomedical Disciplines: A Meta-Analysis

Saved in:
Bibliographic Details
Title: The Performance of ChatGPT and Other Large Language Models on Multiple-Choice Questions in Biomedical Disciplines: A Meta-Analysis
Language: English
Authors: Colleen M. Cheverko (ORCID 0000-0002-9065-1768), Volodymyr Mavrych, Olena Bolgova (ORCID 0009-0002-9496-9754), Fathima Raahima Riyas Moham, Jennifer Westrick (ORCID 0000-0002-3606-3957), Lorena Juarez, Emily Rush, Kathryn A. Solka, Alison F. Doubleday, Jessica N. Byram (ORCID 0000-0001-7097-8352), Robert Becker, Victoria Gomez, Brenda K. Anak Ganeng, Leslie A. Hoffman, Victoria A. Roach (ORCID 0000-0003-4020-2224), Kirsten M. Brown, Nicole DeVaul, Colleen N. Garnett (ORCID 0000-0002-9426-9304), Hannah L. Herriott (ORCID 0000-0002-5410-0360), Rebecca S. Lufler, Jason C. Mussell (ORCID 0000-0003-4123-3356), Joy Y. Balta, Michael A. Pascoe, Jason W. Middleton, Sharon Duffy, Georgina C. Stephens (ORCID 0000-0001-9695-7592), Adam B. Wilson
Source: Anatomical Sciences Education. 2026 19(9):1556-1569.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 14
Publication Date: 2026
Document Type: Journal Articles
Information Analyses
Reports - Research
Descriptors: Artificial Intelligence, Technology Uses in Education, Natural Language Processing, Computer Assisted Testing, Multiple Choice Tests, Biomedicine, Meta Analysis, Accuracy, Supplementary Education, Medical Education
DOI: 10.1002/ase.70262
ISSN: 1935-9772
1935-9780
Abstract: While large language models (LLMs) have shown promise as learning tools for medical education, their reported accuracy on multiple-choice questions (MCQs) varies widely across studies, necessitating synthesis. This meta-analysis synthesizes LLM accuracy on text-based MCQs from biomedical disciplines and USMLE Step 1-level content and explores study characteristics that moderate LLM performance. Studies published between January 1, 2022, and August 5, 2025, were identified using seven databases. Titles and abstracts were screened against eligibility criteria, which included testing LLM performance on text-based MCQs in English. Extracted data included accuracy rates (correct responses/total questions) and study characteristics. A random-effects proportional meta-analysis was used to calculate the pooled accuracy of LLM versions across studies, and a meta-regression tested the effects of study characteristics on accuracy. The final analysis included 41 articles comprising 207 proportions. Newer OpenAI models (GPT-4o through GPT o1 = 90%; GPT-4 = 82%) demonstrated significantly higher accuracy than earlier versions (GPT-3/3.5 = 59%; p < 0.001). Model version explained 56% of the total variance (p < 0.001). The pooled accuracy of newer Anthropic (Claude 3 through 3.7 = 86%) and DeepSeek (R1 = 86%) versions fell within the range of performance for the newer OpenAI versions, while newer Google (Gemini series = 76%) and Microsoft (Copilot = 69%) versions demonstrated lower performance. A discipline-level analysis demonstrated that OpenAI's GPT-4 through GPT-o1 models achieved pooled accuracy rates between 83 and 87% for most disciplines, including the anatomical sciences. These findings demonstrate that newer LLMs are achieving higher accuracy than older models on biomedical and Step 1-level MCQs, supporting their potential integration as supplementary study tools in foundational biomedical education.
Abstractor: As Provided
Entry Date: 20260101
Accession Number: EJ1517238
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1517238
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Performance of ChatGPT and Other Large Language Models on Multiple-Choice Questions in Biomedical Disciplines: A Meta-Analysis
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Colleen+M%2E+Cheverko%22&quot;&gt;Colleen M. Cheverko&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0002-9065-1768&quot;&gt;0000-0002-9065-1768&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Volodymyr+Mavrych%22&quot;&gt;Volodymyr Mavrych&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Olena+Bolgova%22&quot;&gt;Olena Bolgova&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0009-0002-9496-9754&quot;&gt;0009-0002-9496-9754&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Fathima+Raahima+Riyas+Moham%22&quot;&gt;Fathima Raahima Riyas Moham&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Jennifer+Westrick%22&quot;&gt;Jennifer Westrick&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0002-3606-3957&quot;&gt;0000-0002-3606-3957&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Lorena+Juarez%22&quot;&gt;Lorena Juarez&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Emily+Rush%22&quot;&gt;Emily Rush&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Kathryn+A%2E+Solka%22&quot;&gt;Kathryn A. Solka&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Alison+F%2E+Doubleday%22&quot;&gt;Alison F. Doubleday&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Jessica+N%2E+Byram%22&quot;&gt;Jessica N. Byram&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0001-7097-8352&quot;&gt;0000-0001-7097-8352&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Robert+Becker%22&quot;&gt;Robert Becker&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Victoria+Gomez%22&quot;&gt;Victoria Gomez&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Brenda+K%2E+Anak+Ganeng%22&quot;&gt;Brenda K. Anak Ganeng&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Leslie+A%2E+Hoffman%22&quot;&gt;Leslie A. Hoffman&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Victoria+A%2E+Roach%22&quot;&gt;Victoria A. Roach&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0003-4020-2224&quot;&gt;0000-0003-4020-2224&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Kirsten+M%2E+Brown%22&quot;&gt;Kirsten M. Brown&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Nicole+DeVaul%22&quot;&gt;Nicole DeVaul&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Colleen+N%2E+Garnett%22&quot;&gt;Colleen N. Garnett&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0002-9426-9304&quot;&gt;0000-0002-9426-9304&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hannah+L%2E+Herriott%22&quot;&gt;Hannah L. Herriott&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0002-5410-0360&quot;&gt;0000-0002-5410-0360&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Rebecca+S%2E+Lufler%22&quot;&gt;Rebecca S. Lufler&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Jason+C%2E+Mussell%22&quot;&gt;Jason C. Mussell&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0003-4123-3356&quot;&gt;0000-0003-4123-3356&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Joy+Y%2E+Balta%22&quot;&gt;Joy Y. Balta&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Michael+A%2E+Pascoe%22&quot;&gt;Michael A. Pascoe&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Jason+W%2E+Middleton%22&quot;&gt;Jason W. Middleton&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Sharon+Duffy%22&quot;&gt;Sharon Duffy&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Georgina+C%2E+Stephens%22&quot;&gt;Georgina C. Stephens&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0001-9695-7592&quot;&gt;0000-0001-9695-7592&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Adam+B%2E+Wilson%22&quot;&gt;Adam B. Wilson&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;SO&quot; term=&quot;%22Anatomical+Sciences+Education%22&quot;&gt;&lt;i&gt;Anatomical Sciences Education&lt;/i&gt;&lt;/searchLink&gt;. 2026 19(9):1556-1569.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley &amp; Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 14
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles&lt;br /&gt;Information Analyses&lt;br /&gt;Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Artificial+Intelligence%22&quot;&gt;Artificial Intelligence&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Technology+Uses+in+Education%22&quot;&gt;Technology Uses in Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Natural+Language+Processing%22&quot;&gt;Natural Language Processing&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Computer+Assisted+Testing%22&quot;&gt;Computer Assisted Testing&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Multiple+Choice+Tests%22&quot;&gt;Multiple Choice Tests&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Biomedicine%22&quot;&gt;Biomedicine&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Meta+Analysis%22&quot;&gt;Meta Analysis&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Accuracy%22&quot;&gt;Accuracy&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Supplementary+Education%22&quot;&gt;Supplementary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Medical+Education%22&quot;&gt;Medical Education&lt;/searchLink&gt;
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/ase.70262
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1935-9772&lt;br /&gt;1935-9780
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: While large language models (LLMs) have shown promise as learning tools for medical education, their reported accuracy on multiple-choice questions (MCQs) varies widely across studies, necessitating synthesis. This meta-analysis synthesizes LLM accuracy on text-based MCQs from biomedical disciplines and USMLE Step 1-level content and explores study characteristics that moderate LLM performance. Studies published between January 1, 2022, and August 5, 2025, were identified using seven databases. Titles and abstracts were screened against eligibility criteria, which included testing LLM performance on text-based MCQs in English. Extracted data included accuracy rates (correct responses/total questions) and study characteristics. A random-effects proportional meta-analysis was used to calculate the pooled accuracy of LLM versions across studies, and a meta-regression tested the effects of study characteristics on accuracy. The final analysis included 41 articles comprising 207 proportions. Newer OpenAI models (GPT-4o through GPT o1 = 90%; GPT-4 = 82%) demonstrated significantly higher accuracy than earlier versions (GPT-3/3.5 = 59%; p &lt; 0.001). Model version explained 56% of the total variance (p &lt; 0.001). The pooled accuracy of newer Anthropic (Claude 3 through 3.7 = 86%) and DeepSeek (R1 = 86%) versions fell within the range of performance for the newer OpenAI versions, while newer Google (Gemini series = 76%) and Microsoft (Copilot = 69%) versions demonstrated lower performance. A discipline-level analysis demonstrated that OpenAI&#39;s GPT-4 through GPT-o1 models achieved pooled accuracy rates between 83 and 87% for most disciplines, including the anatomical sciences. These findings demonstrate that newer LLMs are achieving higher accuracy than older models on biomedical and Step 1-level MCQs, supporting their potential integration as supplementary study tools in foundational biomedical education.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 20260101
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1517238
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1517238
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/ase.70262
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 14
        StartPage: 1556
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Multiple Choice Tests
        Type: general
      – SubjectFull: Biomedicine
        Type: general
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Supplementary Education
        Type: general
      – SubjectFull: Medical Education
        Type: general
    Titles:
      – TitleFull: The Performance of ChatGPT and Other Large Language Models on Multiple-Choice Questions in Biomedical Disciplines: A Meta-Analysis
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Colleen M. Cheverko
      – PersonEntity:
          Name:
            NameFull: Volodymyr Mavrych
      – PersonEntity:
          Name:
            NameFull: Olena Bolgova
      – PersonEntity:
          Name:
            NameFull: Fathima Raahima Riyas Moham
      – PersonEntity:
          Name:
            NameFull: Jennifer Westrick
      – PersonEntity:
          Name:
            NameFull: Lorena Juarez
      – PersonEntity:
          Name:
            NameFull: Emily Rush
      – PersonEntity:
          Name:
            NameFull: Kathryn A. Solka
      – PersonEntity:
          Name:
            NameFull: Alison F. Doubleday
      – PersonEntity:
          Name:
            NameFull: Jessica N. Byram
      – PersonEntity:
          Name:
            NameFull: Robert Becker
      – PersonEntity:
          Name:
            NameFull: Victoria Gomez
      – PersonEntity:
          Name:
            NameFull: Brenda K. Anak Ganeng
      – PersonEntity:
          Name:
            NameFull: Leslie A. Hoffman
      – PersonEntity:
          Name:
            NameFull: Victoria A. Roach
      – PersonEntity:
          Name:
            NameFull: Kirsten M. Brown
      – PersonEntity:
          Name:
            NameFull: Nicole DeVaul
      – PersonEntity:
          Name:
            NameFull: Colleen N. Garnett
      – PersonEntity:
          Name:
            NameFull: Hannah L. Herriott
      – PersonEntity:
          Name:
            NameFull: Rebecca S. Lufler
      – PersonEntity:
          Name:
            NameFull: Jason C. Mussell
      – PersonEntity:
          Name:
            NameFull: Joy Y. Balta
      – PersonEntity:
          Name:
            NameFull: Michael A. Pascoe
      – PersonEntity:
          Name:
            NameFull: Jason W. Middleton
      – PersonEntity:
          Name:
            NameFull: Sharon Duffy
      – PersonEntity:
          Name:
            NameFull: Georgina C. Stephens
      – PersonEntity:
          Name:
            NameFull: Adam B. Wilson
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 1935-9772
            – Type: issn-electronic
              Value: 1935-9780
          Numbering:
            – Type: volume
              Value: 19
            – Type: issue
              Value: 9
          Titles:
            – TitleFull: Anatomical Sciences Education
              Type: main
ResultId 1