View in EDS HTML Full Text PDF Full Text

Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation.

Saved in:
Bibliographic Details
Title: Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation.
Authors: Huppertz, Marc Sebastian1 (AUTHOR) mhuppertz@ukaachen.de, Siepmann, Robert1 (AUTHOR), Topp, David1 (AUTHOR), Nikoubashman, Omid2 (AUTHOR), Yüksel, Can1 (AUTHOR), Kuhl, Christiane Katharina1 (AUTHOR), Truhn, Daniel1 (AUTHOR), Nebelung, Sven1 (AUTHOR)
Source: European Radiology. Mar2025, Vol. 35 Issue 3, p1111-1121. 11p.
Subjects: Generative pre-trained transformers, Language models, Magnetic resonance imaging, Medical personnel, Differential diagnosis
Abstract: Objectives: ChatGPT-4 Vision (GPT-4V) is a state-of-the-art multimodal large language model (LLM) that may be queried using images. We aimed to evaluate the tool's diagnostic performance when autonomously assessing clinical imaging studies. Materials and methods: A total of 206 imaging studies (i.e., radiography (n = 60), CT (n = 60), MRI (n = 60), and angiography (n = 26)) with unequivocal findings and established reference diagnoses from the radiologic practice of a large university hospital were accessed. Readings were performed uncontextualized, with only the image provided, and contextualized, with additional clinical and demographic information. Responses were assessed along multiple diagnostic dimensions and analyzed using appropriate statistical tests. Results: With its pronounced propensity to favor context over image information, the tool's diagnostic accuracy improved from 8.3% (uncontextualized) to 29.1% (contextualized, first diagnosis correct) and 63.6% (contextualized, correct diagnosis among differential diagnoses) (p ≤ 0.001, Cochran's Q test). Diagnostic accuracy declined by up to 30% when 20 images were re-read after 30 and 90 days and seemed unrelated to the tool's self-reported confidence (Spearman's ρ = 0.117 (p = 0.776)). While the described imaging findings matched the suggested diagnoses in 92.7%, indicating valid diagnostic reasoning, the tool fabricated 258 imaging findings in 412 responses and misidentified imaging modalities or anatomic regions in 65 images. Conclusion: GPT-4V, in its current form, cannot reliably interpret radiologic images. Its tendency to disregard the image, fabricate findings, and misidentify details, especially without clinical context, may misguide healthcare providers and put patients at risk. Key Points: QuestionCan Generative Pre-trained Transformer 4 Vision (GPT-4V) interpret radiologic images—with and without clinical context? FindingsGPT-4V performed poorly, demonstrating diagnostic accuracy rates of 8% (uncontextualized), 29% (contextualized, most likely diagnosis correct), and 64% (contextualized, correct diagnosis among differential diagnoses). Clinical relevanceThe utility of commercial multimodal large language models, such as GPT-4V, in radiologic practice is limited. Without clinical context, diagnostic errors and fabricated findings may compromise patient safety and misguide clinical decision-making. These models must be further refined to be beneficial. [ABSTRACT FROM AUTHOR]
Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 183110192
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Huppertz%2C+Marc+Sebastian%22">Huppertz, Marc Sebastian</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> mhuppertz@ukaachen.de</i><br /><searchLink fieldCode="AR" term="%22Siepmann%2C+Robert%22">Siepmann, Robert</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Topp%2C+David%22">Topp, David</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Nikoubashman%2C+Omid%22">Nikoubashman, Omid</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Yüksel%2C+Can%22">Yüksel, Can</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Kuhl%2C+Christiane+Katharina%22">Kuhl, Christiane Katharina</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Truhn%2C+Daniel%22">Truhn, Daniel</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Nebelung%2C+Sven%22">Nebelung, Sven</searchLink><relatesTo>1</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22European+Radiology%22">European Radiology</searchLink>. Mar2025, Vol. 35 Issue 3, p1111-1121. 11p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Generative+pre-trained+transformers%22">Generative pre-trained transformers</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Magnetic+resonance+imaging%22">Magnetic resonance imaging</searchLink><br /><searchLink fieldCode="DE" term="%22Medical+personnel%22">Medical personnel</searchLink><br /><searchLink fieldCode="DE" term="%22Differential+diagnosis%22">Differential diagnosis</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectives: ChatGPT-4 Vision (GPT-4V) is a state-of-the-art multimodal large language model (LLM) that may be queried using images. We aimed to evaluate the tool's diagnostic performance when autonomously assessing clinical imaging studies. Materials and methods: A total of 206 imaging studies (i.e., radiography (n = 60), CT (n = 60), MRI (n = 60), and angiography (n = 26)) with unequivocal findings and established reference diagnoses from the radiologic practice of a large university hospital were accessed. Readings were performed uncontextualized, with only the image provided, and contextualized, with additional clinical and demographic information. Responses were assessed along multiple diagnostic dimensions and analyzed using appropriate statistical tests. Results: With its pronounced propensity to favor context over image information, the tool's diagnostic accuracy improved from 8.3% (uncontextualized) to 29.1% (contextualized, first diagnosis correct) and 63.6% (contextualized, correct diagnosis among differential diagnoses) (p ≤ 0.001, Cochran's Q test). Diagnostic accuracy declined by up to 30% when 20 images were re-read after 30 and 90 days and seemed unrelated to the tool's self-reported confidence (Spearman's ρ = 0.117 (p = 0.776)). While the described imaging findings matched the suggested diagnoses in 92.7%, indicating valid diagnostic reasoning, the tool fabricated 258 imaging findings in 412 responses and misidentified imaging modalities or anatomic regions in 65 images. Conclusion: GPT-4V, in its current form, cannot reliably interpret radiologic images. Its tendency to disregard the image, fabricate findings, and misidentify details, especially without clinical context, may misguide healthcare providers and put patients at risk. Key Points: QuestionCan Generative Pre-trained Transformer 4 Vision (GPT-4V) interpret radiologic images—with and without clinical context? FindingsGPT-4V performed poorly, demonstrating diagnostic accuracy rates of 8% (uncontextualized), 29% (contextualized, most likely diagnosis correct), and 64% (contextualized, correct diagnosis among differential diagnoses). Clinical relevanceThe utility of commercial multimodal large language models, such as GPT-4V, in radiologic practice is limited. Without clinical context, diagnostic errors and fabricated findings may compromise patient safety and misguide clinical decision-making. These models must be further refined to be beneficial. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=183110192
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s00330-024-11115-6
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 1111
    Subjects:
      – SubjectFull: Generative pre-trained transformers
        Type: general
      – SubjectFull: Language models
        Type: general
      – SubjectFull: Magnetic resonance imaging
        Type: general
      – SubjectFull: Medical personnel
        Type: general
      – SubjectFull: Differential diagnosis
        Type: general
    Titles:
      – TitleFull: Revolution or risk?—Assessing the potential and challenges of GPT-4V in radiologic image interpretation.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Huppertz, Marc Sebastian
      – PersonEntity:
          Name:
            NameFull: Siepmann, Robert
      – PersonEntity:
          Name:
            NameFull: Topp, David
      – PersonEntity:
          Name:
            NameFull: Nikoubashman, Omid
      – PersonEntity:
          Name:
            NameFull: Yüksel, Can
      – PersonEntity:
          Name:
            NameFull: Kuhl, Christiane Katharina
      – PersonEntity:
          Name:
            NameFull: Truhn, Daniel
      – PersonEntity:
          Name:
            NameFull: Nebelung, Sven
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Text: Mar2025
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 09387994
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: European Radiology
              Type: main
ResultId 1