Blog

The latest from Google Research

Understanding Transfer Learning for Medical Imaging

Friday, December 6, 2019

Posted by Maithra Raghu and Chiyuan Zhang, Research Scientists, Google Researchtransfer learningpretrainingImageNetfine-tuningmedical imaginginterpreting chest x-raysidentifying eye diseasesearly detection of Alzheimer’s diseasethe effects on performance improvementcontribution of the underlying architecturepretraining dataset type and sizeNeurIPS 2019Transfusion: Understanding Transfer Learning for Medical Imagingfeature independentPerformance Evaluationdiabetic retinopathyfundus photographsfive different diseases from chest x-raysResNet50Inception-v3convolutional neural networksconvolution-batchnorm-ReLU

Surprisingly, transfer learning does not significantly affect performance on medical imaging tasks, with models trained from scratch performing nearly as well as standard ImageNet transferred models.

On the medical imaging tasks, the much smaller CBR models perform at a level comparable to the standard ImageNet architectures.

As the CBR models are much smaller and shallower than the standard ImageNet models, they perform much worse on ImageNet classification, highlighting that ImageNet performance is not indicative of performance on medical tasks.

The two medical tasks are much smaller in size than ImageNet (~200k vs ~1.2m training images), but in the very small data regime, there may only be a few thousand training examples. We evaluated transfer learning in this very small data regime, finding that while there was a larger gap in performance between transfer and training from scratch for large models (ResNet) this was not true for smaller models (CBRs), suggesting that the large models designed for ImageNet might be too overparameterized for the very small data regime.

Representation Analysishidden representationsrecentmethodsingular vector canonical correlation analysiscode and tutorialscanonical correlation analysissimilarity score

Representation similarity scores between networks trained from random initialization and networks trained from pretrained ImageNet weights (orange), and baseline similarity scores of representations trained from two different random initializations (blue). Higher values indicate greater similarity. For larger models, representations learned from random initialization are much more similar to each other than those learned through transfer. This is not the case for smaller models.

even

Effects on Convergence: Feature Independent Benefits and Hybrid Approachesfeature independentweight scalingMean Var Init

Filter visualization of weights initialized according to pretrained ImageNet weights, Random Init, and Mean Var Init. Only the ImageNet Init filters have pretrained (Gabor-like) structure, as Rand Init and Mean Var weights are iid.

weight transfusion

Learning curves comparing the convergence speed with AUC on the test set. Using only the scaling of the pretrained weights (Mean Var Init) helps with convergence speed. The figures compare the standard transfer learning and the Mean Var initialization scheme to training from random initialization.

hybrid approaches

Hybrid approaches to transfer learning on Resnet50 (left) and CBR models (right) — reusing a subset of the weights and slimming the remainder of the network (Slim), and using mathematically synthesized Gabors for conv1 (Synthetic Gabor).

mathematically synthesized Gabor filters

Synthetic Gabor filters used to initialize the first layer if neural networks in some of the experiments in this paper. The Gabor filters are generated as grayscale images and repeated across the RGB channels. Left: Low frequencies. Right: High frequencies.

Conclusion and Open QuestionsAcknowledgementsSpecial thanks to Samy Bengio and Jon Kleinberg, who are co-authors on this work. Thanks also to Geoffrey Hinton for helpful feedback.