Results of the first fastMRI image reconstruction challenge

Facebook AI and NYU Langone Health created the fastMRI project to accelerate MRI scans through AI. By using AI to create ground-truth-accurate images from significantly less raw data, fastMRI aims to make scans as much as ten times faster than they are today, thereby improving patient experience and making MRI scans less expensive and more accessible. As part of this project, we released the largest publicly available dataset of de-identified raw MRI knee measurements, and we’re pleased to now share the results of the first fastMRI image reconstruction challenge.

Through this challenge, researchers from across the AI community were able to explore new approaches and compare their results. Thirty-four teams participated, using u-nets, deformable convolutional networks, recurrent neural networks, and other model architectures. Three teams lead respectively by Patrick Putzky of the University of Amsterdam (single-coil 4x acceleration), Puyang Wang of United Imaging Intelligence/Johns Hopkins University (multi-coil 4x), and Nicola Pezzotti of Philips (multi-coil 4x and 8x) were judged to be the strongest performers. The teams have been invited to present at the Medical Imaging Meets NeurIPS workshop on Saturday, December 14, at the NeurIPS conference in Vancouver.

Submissions were first evaluated by structural similarity measure (SSIM), which quantifies changes in the structural information of an image. The top four results by SSIM score were then judged by a panel of radiologists on their visual quality. Though this evaluation is useful for gaining insight into their relative strengths, the determination of whether the approaches are clinically viable will require thorough clinical studies, which have not yet been performed. (Note that the dataset used in the challenge included two of the MRI sequences used in the clinical setting.)

Overall, the top models in the challenge all produced SSIM scores within a tenth of a decimal place of one another. This suggests that multiple approaches will be effective and adds to our optimism that AI will be able to improve MRI scans.

A challenge designed with radiologists’ needs in mind

Challenge participants trained their models using the open source fastMRI knee dataset and then used the challenge dataset to reconstruct knee MRIs for evaluation. They submitted their reconstructions for either a single-coil track with 4x acceleration or multi-coil tracks with 4x or 8x accelerations. The multi-coil track is more clinically relevant, while the single-coil track provided a less complex entry point to the challenge of creating accelerated scans. It is important to note that the challenge is not a clinical study, and the results should not be viewed as determining which approaches are most clinically viable.

To ensure reproducibility and enable the community to continue to build on this work, we encouraged challenge participants to share their code. Several have done so already. Facebook AI and NYU Langone Health intend to open-source many of the models they have developed together for the fastMRI project.

Once the entrants with the highest SSIM scores were determined, seven expert subspecialized musculoskeletal radiologists scored the top four on several measures: contrast to noise ratio, artifacts, sharpness, diagnostic confidence, and overall image quality. Submissions were then ranked and their scores averaged to determine the strongest performers. Note that the diagnostic interchangeability of the approaches with respect to the original ground truth MRIs was not tested.

The fastMRI challenge results also underscore the importance of radiologists’ review. When the radiologists assessed the entries with the highest SSIM marks, they sometimes preferred entries that did not have the absolute highest score.

“The radiologist review is an important part of the challenge, because appealing to radiologists will ensure the technology will be widely adopted when it is finally available. That’s why for the AI-reconstructed images from NYU Langone and Facebook AI, we’ve also conducted an interchangeability study with radiologists and plan to publish our results soon in a peer-reviewed journal,” said Michael P. Recht, M.D., the Louis Marx Professor and Chair of the Department of Radiology at NYU Langone Health. Recht will present additional details on December 5 at the Radiological Society of North America’s annual conference in Chicago.

Working toward using AI to improve MRIs in the real world

We’re encouraged by the challenge and happy to see the broader community successfully engaging on the project. We are confident that by working openly and collaboratively we will make progress toward our ultimate goal more quickly.

Computer vision research in other subfields often benefits from hosting these types of open, public challenges. But they are not yet common in the medical imaging space. We hope the fastMRI initiative will advance AI research, improve access to potentially life-saving technology, and inspire more open and reproducible research practices in the field.