“We are in a scientific reproducibility crisis!”, Luis Martí-Bonmatí stated in his keynote lecture at this year’s Annual Meeting of the European Society of Medical Imaging Informatics (EuSoMII). After introducing the concept of imaging biomarkers and outlining different ways for imaging biomarker development (including hand-crafted extraction as well as approaches driven by AI), Martí-Bonmatí eloquently explained the current reproducibility crisis of science in the field of imaging biomarkers, radiomics and AI. According to a study from 2016 published in Nature (Baker M. 1,500 scientists lift the lid on reproducibility), researchers were able to reproduce only a minority of published science – a problem which is expected to be even aggravated during our journey towards big data and AI.
As Martí-Bonmatí states, one of the main problems in imaging when facing the clinical translation of novel imaging biomarkers is, that “signal comes from voxels, which are complex structures in their components and properties, reconstructed from different machines, from different vendors, having different releases, and with different acquisition and reconstruction parameters.”, thereby addressing the concerning lack of standardization of the images used for big data, radiomic, and AI studies.
During the following part of his lecture, Martí-Bonmati then outlined potential solutions to this crisis: adequate sample sizes, a decreased level of significance (p values < .005), and randomized prospective studies. In addition, he stressed the importance of real-world data, which might be able to “address clinical and policy-relevant questions that cannot be answered with data from clinical trials.”
Martí-Bonmati concluded his lecture by the statement, that “harmonization (of data) by AI-based approaches could be a disruptive solution by making it possible for complex and deep CNN to learn from the lack of homogeneity in the collected data.”
In my opinion, Martí-Bonmati made a substantial contribution to the ongoing discussion on AI and Big Data in Medicine with his excellent lecture. He did not replicate the hyping comments, which can be found everywhere, but set out for a critical, but thoughtful and, nonetheless, optimistic look at the current challenges of imaging biomarkers. We should all be aware of these challenges, and some of us will have to address the “reproducibility crisis” as well as the standardization issue in order to allow for the valuable clinical translation of imaging biomarkers and radiomics in the future.
