Blog

The latest news from Google AI

Open-Sourcing BiT: Exploring Large-Scale Pre-training for Computer Vision

Thursday, May 21, 2020

Posted by Lucas Beyer and Alexander Kolesnikov, Research Engineers, Google Research, ZürichOpenImagesPlacesthis amount of labeled datapre-trainedImageNetBERTT5large-scaleBig Transfer (BiT): General Visual Representation LearningILSVRC-2012share the best BiT modelscode in TF2, Jax, and PyTorchPre-trainingImageNet-21kJFTResNet

Left: In order to make effective use of a larger dataset for pre-training, one needs to increase model capacity. The red arrows exemplify this: small architectures (smaller point) become worse when pre-trained on the larger ImageNet-21k, whereas the larger architectures (larger points) improve. Right: Pre-training on a larger dataset alone does not necessarily result in improved performance, e.g., when going from ILSVRC-2012 to the relatively larger ImageNet-21k. However, by also increasing the computational budget and training for longer, the performance improvement is pronounced.

batch normalizationstabilizes training by normalizing activationsgroup normalizationweight standardization

Summary of our pre-training strategy: take a standard ResNet, increase depth and width, replace BatchNorm (BN) with GroupNorm and Weight Standardization (GNWS), and train on a very large and generic dataset for many more iterations.

Transfer LearningBERT“BiT-HyperRule”

Once the BiT model is pre-trained, it can be fine-tuned on any task, even if only few labeled examples are available.

The curves depict median accuracy over 5 independent runs (light points) when transferring to CIFAR-10 with only 1 or 5 images per class (10 or 50 images total). It is evident that large architectures pre-trained on large datasets are significantly more data-efficient.

VTAB-1kthe previous state-of-the-artPetsFlowersCIFARRetinaNetMSCOCO-2017

Left: Accuracy of BiT-L compared to the previous state-of-the-art general model on various standard computer vision benchmarks. Right: Results in average precision (AP) of using BiT as backbone for RetinaNet on MSCOCO-2017.

BiT-HyperRuleEvaluation with ObjectNetObjectNetprevious state-of-the-art

Results of BiT on the ObjectNet evaluation dataset. Left: top-5 accuracy, right: top-1 accuracy.

ConclusionreleaseAcknowledgementsWe would like to thank Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby who have co-authored the BiT paper and been involved in all aspects of its development, as well as the Brain team in Zürich. We also would like to thank Andrei Giurgiu for his help in debugging input pipelines. We thank Tom Small for creating the animations used in this blogpost. Finally, we refer the interested reader to the related approaches in this direction by our colleagues in Google Research, Noisy Student, as well as Facebook Research’s highly relevant Exploring the Limits of Weakly Supervised Pretraining.