Thinking back on the last two years, what were the dominant topics of discussion in radiology? Certainly, artificial intelligence (AI) in radiology has sparked a lot of interest and enthusiasm in radiology, and COVID-19, which was a topic nobody could avoid. So, it comes as no surprise that the combination of both topics – i.e. using AI to detect COVID on chest CTs – was obvious to many, academic researchers and commercial vendors alike. Consequently, a number of commercial solutions were developed and offered to radiologists around the world. Interestingly, quite early during the pandemic, the World Health Organization (WHO) noted that CT should not be used to screen for COVID-19 due to its low specificity [1]. So, could AI – and especially commercially available AI solutions – help overcome the low specificity issue?
A recently-published article shows that generally speaking, AI is probably not yet ready to help radiologists more accurately detect COVID-19 in chest CTs [2]. In their study, the authors included 50 PCR-proven COVID-positive cases as well as 450 CT scans from before the pandemic and, therefore, COVID-negative. The COVID-negative cases were selected randomly, apart from 50 cases with ground-glass opacities, to enrich the dataset with studies in which lung lesions were present that could mimic COVID-19. Unsurprisingly, just as human radiologists would be unable to tell ground-glass opacities unrelated to COVID from those caused by it, the included commercial AI solutions struggled to reliably differentiate COVID-19 from other conditions with similar appearance in CT (specificities were between 0.31 and 0.8 and positive predictive values 0.19 to 0.25).
That said, it is without a doubt that AI will surely play a major role in the radiology of the future. But, however promising an interesting AI solution to a diagnostic problem may seem, caution should be advised. As suggested in the ECLAIR guidelines, an important question to ask before the adoption of AI software in clinical routine is: “How has performance been evaluated?” [2]. And just as new drugs have to prove their efficacy and safety in trials, AI software should ideally be carefully and independently evaluated – just as the authors of this paper did for some of the commercially available COVID-19 detection software.
Read more about this in “Commercial AI solutions in detecting COVID‐19 pneumonia in chest CT: not yet ready for clinical implementation?” in the link below.
[1] World Health Organization (2020) Use of chest imaging in COVID-19: a rapid advice guide. World Health Organization. Available via https://www.who.int/publications/i/item/use-of- chest-imaging-in-covid-19. Accessed 13 Jun 2021
[2] Omoumi P, Ducarouge A, Tournier A, et al (2021) To buy or not to buy—evaluating commercial AI solutions in radiology (the ECLAIR guidelines) Eur Radiol https://doi.org/10.1007/s00330-020-07684-x
Key points
- Commercial AI solutions achieved a sensitivity and specificity ranging from 62 to 96% and from 31 to 80%, respectively, in identifying patients suspicious for COVID-19 in a clinical dataset.
- Sensitivity remained within the same range, while specificity was even lower in subgroup analysis of CT studies with ground-glass opacities, and interrater agreement between the commercial AI solutions was minimal to nonexistent.
- Thus, commercial AI solutions have the potential to be integrated as alert tools for the detection of patients with lung changes suspicious for COVID-19 pneumonia in a clinical routine workflow, if further improvement is made.
Authors: Florian Jungmann, Lukas Müller, Felix Hahn, Maximilian Weustenfeld, Ann-Kathrin Dapper, Aline Mähringer-Kunz, Dirk Graafen, Christoph Düber, Darius Schafigh, Daniel Pinto dos Santos , Peter Mildenberger & Roman Kloeckner
