Limitations of using artificial intelligence services to analyze chest x-ray imaging
- Authors: Vasilev Y.A.1, Vladzymyrskyy A.V.1, Arzamasov K.M.1, Shulkin I.M.1, Astapenko E.V.1, Pestrenin L.D.1
-
Affiliations:
- Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
- Issue: Vol 5, No 3 (2024)
- Pages: 407-420
- Section: Original Study Articles
- URL: https://journal-vniispk.ru/DD/article/view/310027
- DOI: https://doi.org/10.17816/DD626310
- ID: 310027
Cite item
Full Text
Abstract
BACKGROUND: Chest X-ray examination is one of the first radiology areas that started applying artificial intelligence, and it is still used to the present. However, when interpreting X-ray scans using artificial intelligence, radiologists still experience several routine restrictions that should be considered in issuing a medical report and require the attention of artificial intelligence developers to further improve the algorithms and increase their efficiency.
AIM: To identify restrictions of artificial intelligence services for analyzing chest X-ray images and assesses the clinical significance of these restrictions.
MATERIALS AND METHODS: A retrospective analysis was performed for 155 cases of discrepancies between the conclusions of artificial intelligence services and medical reports when analyzing chest X-ray images. All cases included in the study were obtained from the Unified Radiological Information Service of the Unified Medical Information and Analytical System of Moscow.
RESULTS: Of the 155 analyzed difference cases, 48 (31.0%) were false-positive and 78 (50.3%) were false-negative cases. The remaining 29 (18.7%) cases were removed from further studies because they were true positive (27) or true negative (2) in the expert review. Most (93.8%) of the 48 false-positive cases were due to the artificial intelligence service mistaking normal chest anatomy (97.8% of cases) or catheter shadow (2.2% of cases) for pneumothorax signs. Overlooked clinically significant pathologies accounted for 22.0% of false-negative scans. Nearly half of these cases (44.4%) were overlooked lung nodules. Lung calcifications (60.9%) were the most common clinically insignificant pathology.
CONCLUSIONS: Artificial intelligence services demonstrate a tendency toward over diagnosis. All false-positive cases were associated with erroneous detection of clinically significant pathology: pneumothorax, lung nodules, and pulmonary consolidation. Among false-negative cases, the rate of overlooked clinically significant pathology was low, which accounted for less than one-fourth.
Full Text
##article.viewOnOriginalSite##About the authors
Yuriy A. Vasilev
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Email: npcmr@zdrav.mos.ru
ORCID iD: 0000-0002-5283-5961
SPIN-code: 4458-5608
MD, Cand. Sci. (Medicine)
Russian Federation, MoscowAnton V. Vladzymyrskyy
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Email: a.vladzimirsky@npcmr.ru
ORCID iD: 0000-0002-2990-7736
SPIN-code: 3602-7120
MD, Dr. Sci. (Medicine), Professor
Russian Federation, MoscowKirill M. Arzamasov
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Email: ArzamasovKM@zdrav.mos.ru
ORCID iD: 0000-0001-7786-0349
SPIN-code: 3160-8062
MD, Cand. Sci. (Medicine)
Russian Federation, MoscowIgor M. Shulkin
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Email: i.shulkin@npcmr.ru
ORCID iD: 0000-0002-7613-5273
SPIN-code: 5266-0618
Russian Federation, Moscow
Elena V. Astapenko
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Author for correspondence.
Email: AstapenkoEV1@zdrav.mos.ru
ORCID iD: 0009-0006-6284-2088
SPIN-code: 7362-8553
Russian Federation, Moscow
Lev D. Pestrenin
Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies
Email: PestreninLD@zdrav.mos.ru
ORCID iD: 0000-0002-1786-4329
SPIN-code: 7193-7706
Russian Federation, Moscow
References
- Çallı E, Sogancioglu E, van Ginneken B, et al. Deep learning for chest X-ray analysis: A survey. Medical Image Analysis. 2021;72:102125. doi: 10.1016/j.media.2021.102125
- Vasilev YuA, Tyrov IA, Vladzymyrskyy AV, et al. A New Model of Organizing Mass Screening Based on Stand-Alone Artificial Intelligence Used for Fluorography Image Triage. Zdorov’e Naseleniya i Sreda Obitaniya. 2023;31(11):23–32. (In Russ.) doi: 10.35627/2219-5238/2023-31-11-23-32
- Akhter Y, Singh R, Vatsa M. AI-based radiodiagnosis using chest X-rays: A review. Frontiers in Big Data. 2023;6:1120989. doi: 10.3389/fdata.2023.1120989
- Fanni SC, Marcucci A, Volpi F, et al. Artificial Intelligence-Based Software with CE Mark for Chest X-ray Interpretation: Opportunities and Challenges. Diagnostics (Basel). 2023;13(12):2020. doi: 10.3390/diagnostics13122020
- Gusev AV, Vladzymyrskyy AV, Sharova DE, et al. Evolution of research and development in the field of artificial intelligence technologies for healthcare in the Russian Federation: results of 2021. Digital Diagnostics. 2022;3(3):178–194. doi: 10.17816/DD107367
- Kim J, Kim KH. Role of chest radiographs in early lung cancer detection. Translational Lung Cancer Research. 2020;9(3):522–531. doi: 10.21037/tlcr.2020.04.02
- Golubev NA, Ogryzko EV, Tyurina EM, et al. Features of the development of the radiology diagnostic service in the Russian Federation for 2014-2019. Current problems of health care and medical statistics. 2021;(2):356–376. doi: 10.24412/2312-2935-2021-2-356-376
- Wu JT, Wong KCL, Gur Y, et al. Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents. JAMA Network Open. 2020;3(10):e2022779. doi: 10.1001/jamanetworkopen.2020.22779
- Miró Catalina Q, Fuster-Casanovas A, Solé-Casals J, Vidal-Alaball J. Developing an Artificial Intelligence Model for Reading Chest X-rays: Protocol for a Prospective Validation Study. JMIR Research Protocols. 2022;11(11):e39536. doi: 10.2196/39536
- Plesner LL, Müller FC, Nybing JD, et al. Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact. Radiology. 2023;307(3):e222268. doi: 10.1148/radiol.222268
- Vasilev Yu, Vladzymyrskyy A, Omelyanskaya O, et al. AI-Based CXR First Reading: Current Limitations to Ensure Practical Value. Diagnostics (Basel). 2023;13(8):1430. doi: 10.3390/diagnostics13081430
- Driver CN, Bowles BS, Bartholmai BJ, Greenberg-Worisek AJ. Artificial Intelligence in Radiology: A Call for Thoughtful Application. Clinical and Translational Science. 2020;13(2):216–218. doi: 10.1111/cts.12704
- Yoo H, Kim EY, Kim H, et al. Artificial intelligence-based identification of normal chest radiographs: a simulation study in a multicenter health screening cohort. Korean Journal of Radiology. 2022;23(10):1009–1018. doi: 10.3348/kjr.2022.0189
- Suganuma N, Yoshida S, Takeuchi Y, et al. Artificial intelligence in quantitative chest imaging analysis for occupational lung disease. Seminars in Respiratory and Critical Care Medicine. 2023;44(3):362–369. doi: 10.1055/s-0043-1767760
- Brown C, Nazeer R, Gibbs A, et al. Breaking Bias: The role of artificial intelligence in improving clinical decision-making. Cureus. 2023;15(3):e36415. doi: 10.7759/cureus.36415
- Kaviani P, Kalra MK, Digumarthy SR, et al. Frequency of missed findings on chest radiographs (CXRs) in an international, multicenter study: application of AI to reduce missed findings. Diagnostics (Basel). 2022;12(10):2382. doi: 10.3390/diagnostics12102382
- de Groot PM, Carter BW, Abbott GF, Wu CC. Pitfalls in chest radiographic interpretation: blind spots. Seminars in Roentgenology. 2015;50(3):197–209. doi: 10.1053/j.ro.2015.01.008
- Gefter WB, Post BA, Hatabu H. Commonly missed findings on chest radiographs: causes and consequences. Chest. 2023;163(3):650–661. doi: 10.1016/j.chest.2022.10.039
- Vasilev YuA, Vladzimirsky AV, editors. Komp’yuternoe zrenie v luchevoj diagnostike: pervyj etap Moskovskogo eksperimenta. Moscow: Izdatelskie resheniya; 2022. (In Russ.)
- Morozov SP, Burenchev DV, Vladzimirsky AV, et al. Principy i pravila opisanij rezul’tatov luchevyh issledovanij. The series “Best practices of radiation and instrumental diagnostics”. Issue 97. Moscow: State Budget-Funded Health Care Institution of the City of Moscow “Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department”; 2021. (In Russ.)
- Choi YR, Yoon SH, Kim J, et al. Chest Radiography of Tuberculosis: Determination of Activity Using Deep Learning Algorithm. Tuberculosis and Respiratory Diseases. 2023;86(3):226–233. doi: 10.4046/trd.2023.0020
- Sun Z, Zhou J, Zhao L. Application status and problems summary of artificial intelligence in chest imaging. Asian Journal of Surgery. 2023;46(10):4437–4438. doi: 10.1016/j.asjsur.2023.04.100
- Bernstein MH, Atalay MK, Dibble EH, et al. Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography. European Radiology. 2023;33(11):8263–8269. doi: 10.1007/s00330-023-09747-1
- Becker J, Decker JA, Römmele C, et al. Artificial Intelligence-based detection of pneumonia in chest radiographs. Diagnostics (Basel). 2022;12(6):1465. doi: 10.3390/diagnostics12061465
- Dasegowda G, Bizzo BC, Gupta RV, et al. Radiologist-trained AI model for identifying suboptimal chest-radiographs. Academic Radiology. 2023;30(12):2921–2930. doi: 10.1016/j.acra.2023.03.006
- Fanni SC, Greco G, Rossi S, et al. Role of artificial intelligence in oncologic emergencies: a narrative review. Exploration of Targeted Anti-tumor Therapy. 2023;4(2):344–354. doi: 10.37349/etat.2023.00138
- Hwang EJ, Goo JM, Nam JG, et al. Conventional versus artificial intelligence-assisted interpretation of chest radiographs in patients with acute respiratory symptoms in emergency department: a pragmatic randomized clinical trial. Korean Journal of Radiology. 2023;24(3):259–270. doi: 10.3348/kjr.2022.0651
- Tan H, Xu H, Yu N, et al. The value of deep learning-based computer aided diagnostic system in improving diagnostic performance of rib fractures in acute blunt trauma. BMC Medical Imaging. 2023;23(1):55. doi: 10.1186/s12880-023-01012-7
- Wu J, Liu N, Li X, et al. Convolutional neural network for detecting rib fractures on chest radiographs: a feasibility study. BMC Medical Imaging. 2023;23(1):18. doi: 10.1186/s12880-023-00975-x
- Lee HW, Jin KN, Oh S, et al. Artificial intelligence solution for chest radiographs in respiratory outpatient clinics: multicenter prospective randomized clinical trial. Annals of the American Thoracic Society. 2023;20(5):660–667. doi: 10.1513/AnnalsATS.202206-481OC
- Hillis JM, Bizzo BC, Mercaldo S, et al. Evaluation of an artificial intelligence model for detection of pneumothorax and tension pneumothorax in chest radiographs. JAMA Network Open. 2022;5(12):e2247172. doi: 10.1001/jamanetworkopen.2022.47172
Supplementary files
