Using multitask learning to improve image classification for histopathology

May 4

What the research is:

A gigapixel histopathology image classification pipeline based on convolutional neural networks (CNNs). Unlike most current approaches, which typically require experts to make pixel-level annotations for each histopathology classification task, we propose the use of an encoder network trained in a supervised way on multiple histopathology datasets and classification tasks.

Our approach is inspired by the workflow of pathologists: When analyzing a new image classification task or dataset, they transfer the knowledge about relevant classification patterns from previously studied diseases. We show that by exploiting the four histopathology datasets, the CNN encoder learns features that generalize to new datasets and new tasks. We exhibit state-of-the art results on the Tumor Proliferation Assessment Challenge and competitive performance on colorectal liver metastasis classification, and show potential to assist the prediction of patient outcome.

Image from Camelyon16 dataset shows lymph node tissue section with small breast cancer metastasis. Courtesy of Radboud University Medical Center, Nijmegen, Netherlands.

How it works:

Together with our collaborators at Radboud University Medical Center (Netherlands) and Erasmus Medical Center (Netherlands), we propose to leverage multitask and transfer learning approaches to mitigate the low signal-to-noise ratio problem in histopathology images. In our approach, we take advantage of four histopathology datasets built to identify different pathologies and learn a compressed image representation (an encoder), which suppresses low-level pixel noise and spurious correlations, while identifying and extracting transferable features that work well in a variety of downstream tasks. Once the encoder is learned, any histopathology dataset can be encoded, and a classifier can be trained using the image-level labels.

First, we train an encoder using multiple histopathology classification tasks and a multitask learning training objective (step 1). Once the embedding function is learned, we can encode any histopathology dataset and learn a classifier for a new classification task (step 2).

Why it matters:

Histopathology image classification is particularly challenging, since the image size is in the order of 10 billion pixels and the classification signal might be very weak (low signal-to-noise image classification scenario). For example, a specimen represented on the histopathology image can capture hundreds of thousands of cells, and the label might depend on the presence or absence of a single tumor cell. It’s important to understand and improve CNN generalization for gigapixel histopathology image classification scenarios where only one label per image is available. We previously developed an image classification toolbox, in collaboration with Radboud University Medical Center and Imperial College London (United Kingdom), which allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios on two simulated datasets.

Low signal-to-noise image classification is a very common and relevant problem in medical imaging, where the questions often include whether there is a lesion in the image, what the grade of the disease is, and whether the image shows healthy or abnormal tissue. Collecting a large number of pixel-level annotations is time-consuming and requires significant levels of domain-specific expertise. This often causes a bottleneck in medical imaging datasets, which can limit the dataset sizes.

A successful, data-efficient solution to low signal-to-noise classification has the potential to scale up the dataset sizes by reducing the time and resources needed for labeling. Such a solution could enable automatic analysis of images in applications for which pixel-level annotations are not available, such as prediction of patient outcome, cancer recurrence, and treatment response.