Google AI Blog

Open-Sourcing BiT: Exploring Large-Scale Pre-training for Computer Vision

Thursday, May 21, 2020

Posted by Lucas Beyer and Alexander Kolesnikov, Research Engineers, Google Research, ZürichOpenImagesPlacesthis amount of labeled datapre-trainedImageNetBERTT5large-scaleBig Transfer (BiT): General Visual Representation LearningILSVRC-2012share the best BiT modelscode in TF2, Jax, and PyTorchPre-trainingImageNet-21kJFTResNet

Left: In order to make effective use of a larger dataset for pre-training, one needs to increase model capacity. The red arrows exemplify this: small architectures (smaller point) become worse when pre-trained on the larger ImageNet-21k, whereas the larger architectures (larger points) improve. Right: Pre-training on a larger dataset alone does not necessarily result in improved performance, e.g., when going from ILSVRC-2012 to the relatively larger ImageNet-21k. However, by also increasing the computational budget and training for longer, the performance improvement is pronounced.

batch normalizationstabilizes training by normalizing activationsgroup normalizationweight standardization

Summary of our pre-training strategy: take a standard ResNet, increase depth and width, replace BatchNorm (BN) with GroupNorm and Weight Standardization (GNWS), and train on a very large and generic dataset for many more iterations.

Transfer LearningBERT“BiT-HyperRule”

Once the BiT model is pre-trained, it can be fine-tuned on any task, even if only few labeled examples are available.

The curves depict median accuracy over 5 independent runs (light points) when transferring to CIFAR-10 with only 1 or 5 images per class (10 or 50 images total). It is evident that large architectures pre-trained on large datasets are significantly more data-efficient.

VTAB-1kthe previous state-of-the-artPetsFlowersCIFARRetinaNetMSCOCO-2017

Left: Accuracy of BiT-L compared to the previous state-of-the-art general model on various standard computer vision benchmarks. Right: Results in average precision (AP) of using BiT as backbone for RetinaNet on MSCOCO-2017.

BiT-HyperRuleEvaluation with ObjectNetObjectNetprevious state-of-the-art

Results of BiT on the ObjectNet evaluation dataset. Left: top-5 accuracy, right: top-1 accuracy.

ConclusionreleaseAcknowledgementsWe would like to thank Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby who have co-authored the BiT paper and been involved in all aspects of its development, as well as the Brain team in Zürich. We also would like to thank Andrei Giurgiu for his help in debugging input pipelines. We thank Tom Small for creating the animations used in this blogpost. Finally, we refer the interested reader to the related approaches in this direction by our colleagues in Google Research, Noisy Student, as well as Facebook Research’s highly relevant Exploring the Limits of Weakly Supervised Pretraining.

Announcing the 7th Fine-Grained Visual Categorization Workshop

Wednesday, May 20, 2020

Posted by Christine Kaeser-Chen, Software Engineer and Serge Belongie, Visiting Faculty, Google ResearchDanaus plexippusLimenitis archippusFGVCCUB7th Workshop on Fine-Grained Visual CategorizationIEEEComputer Vision and Pattern Recognition

The FGVC workshop at CVPR 2020 focuses on subordinate categories, including (from left to right) wildlife camera traps, plant pathology, birds, herbarium sheets, apparel, and museum artifacts.

Real-World Impact of the FGVC ChallengesFungi challengeDanish Mycological Societya team from Czech Technical University and the University of West Bohemiaa new workflowconference paperapp for AndroidiOSopen access modelTensorFlow Hub

The Svampeatlas app for mushroom recognition is a result of a Danish-Czech collaboration spun out of the FGVC 2018 Fungi challenge. The underlying model is now published on TF Hub. Images used with permission of the Danish Mycological Society.

iCassava Disease ChallengeMakerere UniversityNational Crops Resources Research Institute

Examples of cassava leaf disease represented in the 2019 iCassava challenge.

This Year’s ChallengesiWildCammaking generalization difficultusing informationbeyondthe image itselfiNaturalist competition datasets

Side-by-side comparison of image quality from iWildcam, captured from wildlife camera traps, (left) and iNaturalist (right), captured by conventional cameras. Images are from the 2020 iWildCam Challenge, and the iNaturalist competition datasets from 2017 and 2018.

Herbarium ChallengeNew York Botanical Garden

Representative examples of specimens from the 2020 Herbarium challenge. Images used with permission of the New York Botanical Garden.

iMat Fashioninstance segmentationiMetMetropolitan Museum of ArtSemi-Supervised AvesiNaturalistsemi-supervised learningPlant PathologyiCassavaInvitation to ParticipateAcknowledgementsWe’d like to thank our colleagues and friends on the FGVC7 organizing committee for working together to advance this important area. At Google we would like to thank Hartwig Adam, Kiat Chuan Tan, Arvi Gjoka, Kimberly Wilber, Sara Beery, Mikhail Sirotenko, Denis Brulé, Timnit Gebru, Ernest Mwebaze, Wojciech Sirko, Maggie Demkin.

Enabling E-Textile Microinteractions: Gestures and Light through Helical Structures

Friday, May 15, 2020

Posted by Alex Olwal, Research Scientist, Google Research

A scalable interactive E-textile architecture with embedded touch sensing, gesture recognition and visual feedback.

E-textile MicrointeractionsACM CHI 2020previously introduced E-textile architectureACM UIST 2018video about E-textile microinteractionsvideo about the E-textile architecture

E-textile microinteractions combining continuous sensing with discrete motion and grasps.

The Helical Sensing Matrix (HSM)capacitive sensinghelical sensing matrixtransmitreceive

Left: A Helical Sensing Matrix based on a 4×4 braid (8 conductive threads spiraled around the core). Magenta/cyan are conductive yarns, used as receive/transmit lines. Grey are passive yarns (cotton). Center: Flattened matrix, that illustrates the infinite number of 4×4 matrices (colored circles 0-F), which repeat along the length of the cord. Right: Yellow are fiber optic lines, which provide visual feedback.

Rotation Detection

Rotation is deduced from horizontal finger motion across the columns. The plots below show the relative capacitive signal strengths, which change with finger proximity.

Interaction Techniques and Design Guidelines

Simple gestures. We design for short interactions where the user either makes a single discrete gesture or performs a continuous manipulation.

Closed-loop feedback. We want to help the user discover functionality and get continuous feedback on their actions. Where possible, we provide visual, tactile, and audio feedback integrated in the device.

Our e-textile enables interaction based on capacitive sensing of proximity, contact area, contact time, roll, and pressure.

Braided fiber optics strands create the illusion of directional motion.

Motion Gestures (Flicks and Slides) and Grasping Styles (Pinch, Grab, Pinch)

Gesture elicitation study with imagined touch sensing.

Twelve participants (horizontal axis) performed 9 repetitions (animation) for the eight gestures (vertical axis). Each sub-image shows 16 overlaid feature vectors, interpolated to 80 observations over time.

User-Independent, Continuous Twist: Quantifying Precision and Speeduser-independent

Left: Weighted average subjective feedback. We mapped the 7-point Likert scale to a score in the range [-3, 3] and multiplied by the number of times the technique received that rating, and computed an average for all the scores. Right: Mean completion times for target distances show that Buttons were consistently slower.

Gesture Prototypes: Headphones, Hoodie Drawstrings, and Speaker Cord

Left: Tap = Play/Pause; Center: Double-tap = Next track; Right: Roll = Volume +/-

Interactive speaker cord for simultaneous use of continuous (twisting/rolling) and discrete gestures (pinch/pat) to control music playback.

Conclusions and Future DirectionsAcknowledgementsThis work is a collaboration across multiple teams at Google. Key contributors to the project include Alex Olwal, Thad Starner, Jon Moeller, Greg Priest-Dorman, Ben Carroll, and Gowa Mainini. We thank the Google ATAP Jacquard team for our collaboration, especially Shiho Fukuhara, Munehiko Sato, and Ivan Poupyrev. We thank Google Wearables, and Kenneth Albanowski and Karissa Sawyer, in particular. Finally, we would like to thank Mark Zarich for illustrations, Bryan Allen for videography, Frank Li for data processing, Mathieu Le Goc for valuable discussions, and Carolyn Priest-Dorman for textile advice.

Announcing Meta-Dataset: A Dataset of Datasets for Few-Shot Learning

Wednesday, May 13, 2020

Posted by Eleni Triantafillou, Student Researcher, and Vincent Dumoulin, Research Scientist, Google Researchmanually annotated training datasparked interestMeta-Dataset: A Dataset of Datasets for Learning to Learn from Few ExamplesICLR 2020ImageNetCUB-200-2011FungipublicnotebookTensorFlowPyTorchBackground: Few-shot Classificationtest taskstest setsupport set query set Comparison of Meta-Dataset with Previous Benchmarksmini-ImageNetImageNetclass splitsRecentworks

Test tasks from mini-ImageNet. Each task is a classification problem between previously unseen (test) classes. The model can use the support set of a few labeled examples of the new classes to adapt to the task at hand and then predicts labels for the query examples of these new classes. The evaluation metric is the query set accuracy, averaged over examples within each task and across tasks.

recentpapers

Test tasks from Meta-Dataset. Contrary to the mini-ImageNet tasks shown above, different tasks here originate from (the test classes of) different datasets. Further, the number of classes and the support set sizes differ across tasks and the support sets might be class-imbalanced.

Initial Investigation and Findings on Meta-Datasetpre-trainingmeta-learningnearest-neighbor comparisonssurgeofattention1) Existing approaches have trouble leveraging heterogeneous training data sources.OmniglotQuickdraw

Comparison of test performance on each dataset after having trained on ImageNet (ILSVRC-2012) only or on all datasets.

2) Some models are more capable than others of exploiting additional data at test time.ProtoNetfo-Proto-MAML

Comparison of test performance averaged across different datasets to the number of examples available per class in test tasks (“shots”). Performance is measured in terms of class precision: the proportion of the examples of a class that are correctly labeled, averaged across classes.

3) The adaptation algorithm of a meta-learner is more heavily responsible for its performance than the fact that it is trained end-to-end (i.e. meta-trained).

Comparison of three different meta-learner variants to their corresponding inference-only baselines, when training on ImageNet (ILSVRC-1012) only or all datasets. Each bar represents the difference between meta-training and inference-only, so positive values indicate improved performance from meta-training.

Conclusiontaskconditioninghyperparameter tuningmeta-baselinefeature selectionAcknowledgementsMeta-Dataset was developed by Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol and Hugo Larochelle. We would like to thank Pablo Castro for his valuable guidance on this blog post, Chelsea Finn for fruitful discussions and ensuring the correctness of fo-MAML’s implementation, as well as Zack Nado and Dan Moldovan for the initial dataset code that was adapted, Cristina Vasconcelos for spotting an issue in the ranking of models and John Bronskill for suggesting that we experiment with a larger inner-loop learning rate for MAML which indeed significantly improved our fo-MAML results.

Speeding Up Neural Network Training with Data Echoing

Tuesday, May 12, 2020

Posted by Dami Choi, Student Researcher and George Dahl, Senior Research Scientist, Google ResearchMoore's lawGPUsTPUsWe know there are limits

An example training pipeline representative of many large-scale computer vision programs. The stages that come before applying the mini-batch stochastic gradient descent (SGD) update generally do not benefit from specialized hardware accelerators.

Faster Neural Network Training with Data Echoing

Left: Without data echoing, downstream computational capacity is idle 50% of the time. Right: Data echoing with echoing factor 2 reclaims downstream computational capacity.

Repeating Data to Train Fastershuffle bufferexample echoing,batch echoingData Echoing Across Workloadsimage classificationlanguage modelingobject detection

Data echoing, when each data item is repeated twice, either reduces or does not change the number of fresh examples needed to reach the target out-of-sample performance. Dashed lines indicate the values we would expect if repeated examples were as useful as fresh examples.

Reduction in Training TimeResNet-50ImageNet

Data echoing can reduce training time for ResNet-50 on ImageNet. In this experiment, reading a batch of training data from cloud storage took 6 times longer than the code that used each batch of data to perform a training step. The Echoing factor in the legend refers to the number of times each data item was repeated. Dashed lines indicate the expected values if repeated examples were as useful as fresh examples and there was no overhead from echoing.

Data Echoing Preserves Predictive Performance

Comparing the individual trials that achieved the best out-of-sample performance during training for both with and without data echoing shows that reusing data does not harm final model quality. Here validation cross entropy is equivalent to log perplexity.

AcknowledgementsThe Data Echoing project was conducted by Dami Choi, Alexandre Passos, Christopher J. Shallue, and George E. Dahl while Dami Choi was a Google AI Resident. We would also like to thank Roy Frostig, Luke Metz, Yiding Jiang, and Ting Chen for helpful discussions.

Agile and Intelligent Locomotion via Deep Reinforcement Learning

Wednesday, May 6, 2020

Posted by Yuxiang Yang and Deepali Jain, AI Residents, Robotics at Googleusing off-policy dataimitating animal behaviorsperforming meta learningData Efficient Reinforcement Learning for Legged RobotsHierarchical Reinforcement Learning for Quadruped LocomotionData Efficient Reinforcement Learning for Legged RobotsSoft Actor-Criticmore than an hour of datadifficult to collectwe presentmodel predictive control

Overview of the model-based learning pipeline. The system alternates between fitting the dynamics model and collecting trajectories using model predictive control (MPC).

We separate action planning and execution on different threads.

multi-step losstrajectory generatorgeneralizable

The robot learns to walk using only 4.5 minutes of data.

The robot learns to backflip and walk with rear legs using the same framework.

Combining low-level controller with high-level planningour second paper

Framework of Hierarchical Policy: The policy gets observations from the robot and sends motor commands to execute desired actions. It is split into two levels (high and low). The high-level policy gives a latent command to the low-level policy and also decides the duration for which low-level will run.

PPOSACaugmented random search

Successful trajectory of a robot on a curved path. Left: A plot of the trajectory traversed by the robot with dots along the trajectory marking the positions where the high-level policy sent a new latent command to the low-level policy. Middle: The robot walking along the path in the simulated environment. Right: The robot walking around the path in the real world.

Left: Visualization of a learned 2D latent command space. Vector directions correspond to the movement direction of the robot. Vector length is proportional to the distance covered. Right: Transfer of low level policy: An HRL policy was trained on a single path (right, top). The learned low-level policy was then reused when training the high-level policy on other paths (e.g., right, bottom).

ConclusionAcknowledgementsBoth Deepali Jain and Yuxiang Yang are residents in the AI Residency program, mentored by Ken Caluwaerts and Atil Iscen. We would also like to thank Jie Tan and Vikas Sindhwani for support of the research, and Noah Broestl for managing the New York AI Residency Program.

Understanding the Shape of Large-Scale Data

Tuesday, May 5, 2020

Posted by Anton Tsitsulin, Research Intern and Bryan Perozzi, Senior Research Scientist, Graph Mining Team, Google ResearchgraphInternet graphlink together friends

Graphs are discrete objects that can model the relationships between many different types of data, including web pages (left), social connections (center), or molecules (right).

Just SLaQ When You Approximate: Accurate Spectral Distances for Web-Scale GraphsWWW'20DDGK: Learning Graph Representations for Deep Divergence Graph KernelsWWW’19graph embeddingsFully Unsupervised Learning of Graph SimilarityIn our 2019 paper,deep divergence graph kernels

Here is a t-SNE visualization of the latent representations learned by DDGK to compare proteins. Blue points indicate proteins that encode enzymes and the red points are for those that do not. We can see that the encoding correlates with a structural property of the protein (whether or not it encodes enzymes), even though this context was not provided during training. (Note that this is a projection of the representations, and so the absolute axis values aren’t meaningful.)

The pairwise distance between different datasets encoded and aligned using DDGK. Color indicates distance in the latent space, and the scale of similarity ranges from 0 (identical) to 1.0 (very different). We see that the representations can be clustered to group similar datasets together — for example, the datasets nci1 and ptc are both datasets of chemical compounds.

Fast and Accurate Approximation of Spectral Descriptorsspectrumdrum3D shapesgraphsgeneral high-dimensional dataAutoMLanomaly detection in dynamic graphschemical molecule characterizationOur more recent paperVon Neumann Graph EntropyEstrada Indexgraph energyNetLSDWikipedia

Left: The well-known Karate graph represents the social interactions of two martial arts clubs. Right: The spectral descriptors (NetLSD, VNGE, and Estrada Index) computed for the original graph in blue and the version with removed edges in red.

ConclusionsSLaQDDGKrecommendation systemsAcknowledgementsWe thank Marina Munkhoeva, Rami Al-Rfou, and Dustin Zelle who contributed to these works. For more information on the Graph Mining team (part of Algorithm and Optimization Group) visit our pages.

Blog