View in EDS HTML Full Text PDF Full Text

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.

Saved in:
Bibliographic Details
Title: Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.
Authors: Kuo, Michael D.1,2 mikedkuo@gmail.com, Chiu, Keith W. H.1, Wang, David S.3, Larici, Anna Rita4,5, Poplavskiy, Dmytro2, Valentini, Adele6, Napoli, Alessandro7, Borghesi, Andrea8, Ligabue, Guido9,10, Fang, Xin Hao B.11, Wong, Hing Ki C.12, Zhang, Sailong1, Hunter, John R.3, Mousa, Abeer13, Infante, Amato5,14, Elia, Lorenzo4,5, Golemi, Salvatore8, Yu, Leung Ho P.15, Hui, Christopher K. M.16,17, Erickson, Bradley J.13
Source: European Radiology. Jan2023, Vol. 33 Issue 1, p23-33. 11p. 1 Diagram, 1 Chart, 3 Graphs.
Subjects: Respiratory infections, COVID-19, Radiography, Disease prevalence
Geographic Terms: United States
Abstract: Objectives: While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR. Methods: A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases. Results: RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78–0.80) on an independent test cohort of 5,894 patients. Delong's test showed statistical differences in model performance across patients from different regions (p < 0.01), disease severity (p < 0.001), gender (p < 0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar's test showed the model has higher sensitivity (p < 0.001) but lower specificity (p < 0.001). Conclusion: An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR. Key Points: • An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model's NPV is greater than 98.5% at any prevalence below 4.5%. [ABSTRACT FROM AUTHOR]
Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 160821799
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Kuo%2C+Michael+D%2E%22&quot;&gt;Kuo, Michael D.&lt;/searchLink&gt;&lt;relatesTo&gt;1,2&lt;/relatesTo&gt;&lt;i&gt; mikedkuo@gmail.com&lt;/i&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Chiu%2C+Keith+W%2E+H%2E%22&quot;&gt;Chiu, Keith W. H.&lt;/searchLink&gt;&lt;relatesTo&gt;1&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Wang%2C+David+S%2E%22&quot;&gt;Wang, David S.&lt;/searchLink&gt;&lt;relatesTo&gt;3&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Larici%2C+Anna+Rita%22&quot;&gt;Larici, Anna Rita&lt;/searchLink&gt;&lt;relatesTo&gt;4,5&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Poplavskiy%2C+Dmytro%22&quot;&gt;Poplavskiy, Dmytro&lt;/searchLink&gt;&lt;relatesTo&gt;2&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Valentini%2C+Adele%22&quot;&gt;Valentini, Adele&lt;/searchLink&gt;&lt;relatesTo&gt;6&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Napoli%2C+Alessandro%22&quot;&gt;Napoli, Alessandro&lt;/searchLink&gt;&lt;relatesTo&gt;7&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Borghesi%2C+Andrea%22&quot;&gt;Borghesi, Andrea&lt;/searchLink&gt;&lt;relatesTo&gt;8&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Ligabue%2C+Guido%22&quot;&gt;Ligabue, Guido&lt;/searchLink&gt;&lt;relatesTo&gt;9,10&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Fang%2C+Xin+Hao+B%2E%22&quot;&gt;Fang, Xin Hao B.&lt;/searchLink&gt;&lt;relatesTo&gt;11&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Wong%2C+Hing+Ki+C%2E%22&quot;&gt;Wong, Hing Ki C.&lt;/searchLink&gt;&lt;relatesTo&gt;12&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Zhang%2C+Sailong%22&quot;&gt;Zhang, Sailong&lt;/searchLink&gt;&lt;relatesTo&gt;1&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hunter%2C+John+R%2E%22&quot;&gt;Hunter, John R.&lt;/searchLink&gt;&lt;relatesTo&gt;3&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Mousa%2C+Abeer%22&quot;&gt;Mousa, Abeer&lt;/searchLink&gt;&lt;relatesTo&gt;13&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Infante%2C+Amato%22&quot;&gt;Infante, Amato&lt;/searchLink&gt;&lt;relatesTo&gt;5,14&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Elia%2C+Lorenzo%22&quot;&gt;Elia, Lorenzo&lt;/searchLink&gt;&lt;relatesTo&gt;4,5&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Golemi%2C+Salvatore%22&quot;&gt;Golemi, Salvatore&lt;/searchLink&gt;&lt;relatesTo&gt;8&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Yu%2C+Leung+Ho+P%2E%22&quot;&gt;Yu, Leung Ho P.&lt;/searchLink&gt;&lt;relatesTo&gt;15&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hui%2C+Christopher+K%2E+M%2E%22&quot;&gt;Hui, Christopher K. M.&lt;/searchLink&gt;&lt;relatesTo&gt;16,17&lt;/relatesTo&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Erickson%2C+Bradley+J%2E%22&quot;&gt;Erickson, Bradley J.&lt;/searchLink&gt;&lt;relatesTo&gt;13&lt;/relatesTo&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%22European+Radiology%22&quot;&gt;European Radiology&lt;/searchLink&gt;. Jan2023, Vol. 33 Issue 1, p23-33. 11p. 1 Diagram, 1 Chart, 3 Graphs.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Respiratory+infections%22&quot;&gt;Respiratory infections&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22COVID-19%22&quot;&gt;COVID-19&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Radiography%22&quot;&gt;Radiography&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Disease+prevalence%22&quot;&gt;Disease prevalence&lt;/searchLink&gt;
– Name: SubjectGeographic
  Label: Geographic Terms
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22United+States%22&quot;&gt;United States&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Objectives: While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR. Methods: A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases. Results: RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78–0.80) on an independent test cohort of 5,894 patients. Delong&#39;s test showed statistical differences in model performance across patients from different regions (p &lt; 0.01), disease severity (p &lt; 0.001), gender (p &lt; 0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar&#39;s test showed the model has higher sensitivity (p &lt; 0.001) but lower specificity (p &lt; 0.001). Conclusion: An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR. Key Points: • An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model&#39;s NPV is greater than 98.5% at any prevalence below 4.5%. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: &lt;i&gt;Copyright of European Radiology is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder&#39;s express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.&lt;/i&gt; (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=160821799
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s00330-022-08969-z
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 23
    Subjects:
      – SubjectFull: Respiratory infections
        Type: general
      – SubjectFull: COVID-19
        Type: general
      – SubjectFull: Radiography
        Type: general
      – SubjectFull: Disease prevalence
        Type: general
      – SubjectFull: United States
        Type: general
    Titles:
      – TitleFull: Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kuo, Michael D.
      – PersonEntity:
          Name:
            NameFull: Chiu, Keith W. H.
      – PersonEntity:
          Name:
            NameFull: Wang, David S.
      – PersonEntity:
          Name:
            NameFull: Larici, Anna Rita
      – PersonEntity:
          Name:
            NameFull: Poplavskiy, Dmytro
      – PersonEntity:
          Name:
            NameFull: Valentini, Adele
      – PersonEntity:
          Name:
            NameFull: Napoli, Alessandro
      – PersonEntity:
          Name:
            NameFull: Borghesi, Andrea
      – PersonEntity:
          Name:
            NameFull: Ligabue, Guido
      – PersonEntity:
          Name:
            NameFull: Fang, Xin Hao B.
      – PersonEntity:
          Name:
            NameFull: Wong, Hing Ki C.
      – PersonEntity:
          Name:
            NameFull: Zhang, Sailong
      – PersonEntity:
          Name:
            NameFull: Hunter, John R.
      – PersonEntity:
          Name:
            NameFull: Mousa, Abeer
      – PersonEntity:
          Name:
            NameFull: Infante, Amato
      – PersonEntity:
          Name:
            NameFull: Elia, Lorenzo
      – PersonEntity:
          Name:
            NameFull: Golemi, Salvatore
      – PersonEntity:
          Name:
            NameFull: Yu, Leung Ho P.
      – PersonEntity:
          Name:
            NameFull: Hui, Christopher K. M.
      – PersonEntity:
          Name:
            NameFull: Erickson, Bradley J.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan2023
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 09387994
          Numbering:
            – Type: volume
              Value: 33
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: European Radiology
              Type: main
ResultId 1