Google AI Blog: June 2020

SpineNet: A Novel Architecture for Object Detection Discovered with Neural Architecture Search

Tuesday, June 30, 2020

Posted by Xianzhi Du, Software Engineer and Jaeyoun Kim, Technical Program Manager, Google ResearchConvolutional neural networks created for image tasks typically encode an input image into a sequence of intermediate features that capture the semantics of an image (from local to global), where each subsequent layer has a lower spatial dimension. However, this scale-decreased model may not be able to deliver strong features for multi-scale visual recognition tasks where recognition and localization are both important (e.g., object detection and segmentation). Several works including FPN and DeepLabv3+ propose multi-scale encoder-decoder architectures to address this issue, where a scale-decreased network (e.g., a ResNet) is taken as the encoder (commonly referred to as a backbone model). A decoder network is then applied to the backbone to recover the spatial information.
While this architecture has yielded improved success for image recognition and localization tasks, it still relies on a scale-decreased backbone that throws away spatial information by down-sampling, which the decoder then must attempt to recover. What if one were to design an alternate backbone model that avoids this loss of spatial information, and is thus inherently well-suited for simultaneous image recognition and localization?CVPR 2020SpineNet: Learning Scale-Permuted Backbone for Recognition and Localizationscale-permuted modelneural architecture searchTensorflow TPU GitHub repositoryTensorFlow Model Garden GitHub repository

A scale-decreased backbone is shown on the left and a scale-permuted backbone is shown on the right. Each rectangle represents a building block. Colors and shapes represent different spatial resolutions and feature dimensions. Arrows represent connections among building blocks.

Design of SpineNet ArchitectureCOCO dataset

Scale permutations: The orderings of network building blocks are important because each block can only be built from those that already exist (i.e., with a “lower ordering”). We define the search space of scale permutations by rearranging intermediate and output blocks, respectively.

Cross-scale connections: We define two input connections for each block in the search space. The parent blocks can be any block with a lower ordering or a block from the stem network.

Block adjustments (optional): We allow the block to adjust its scale level and type.

The architecture search process from a scale-decreased backbone to a scale-permuted backbone.

ResNet-50ResNet-50-FPNaverage precisionFLOPsresidual blockbottleneck blockResNet

The architecture comparison between a ResNet backbone (left) and the SpineNet backbone (right) derived from it using NAS.

PerformanceCOCOiNaturalistEdit 2020-12-12: Experiment results on COCO can be viewed and downloaded on TensorBoard.dev. Metric definitions and naming conventions correspond directly to those defined by COCO. Terminal values on TensorBoard may exceed those in the below table due to TF1 to TF2 conversion, and as a consequence of a series of random variants in the pipeline.

Performance comparisons of SpineNet models and ResNet-FPN models adopting the RetinaNet detection framework on COCO bounding box detection.

Performance comparisons of SpineNet models and ResNet models on ImageNet classification and iNaturalist fine-grained image classification.

ConclusionAcknowledgementsSpecial thanks to the co-authors of the paper: Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V. Le, and Xiaodan Song. We also would like to acknowledge Yeqing Li, Youlong Cheng, Jing Li, Jianwei Xie, Russell Power, Hongkun Yu, Chad Richards, Liang-Chieh Chen, Anelia Angelova, and the larger Google Brain Team for their help.

Leveraging Temporal Context for Object Detection

Friday, June 26, 2020

Posted by Sara Beery, Student Researcher and Jonathan Huang, Research Scientist, Google Researchmountain pass road conditionsecosystem phenologyleading to poor generalization to novel deploymentsLILA BCWildlife Insights

What’s in this image? Objects in images from static cameras can be very challenging to detect and categorize. Here, a foggy morning has made it very difficult to see a herd of wildebeest walking along the crest of a hill. [Image from Snapshot Serengeti]

Context R-CNN: Long Term Temporal Context for Per-Camera Object DetectionFaster R-CNNTF Object Detection API

Here, we can see how additional examples from the same scene help experts determine that the object is an animal and not background. Context such as the shape & size of the object, its attachment to a herd, and habitual grazing at certain times of day help determine that the species is a wildebeest. Useful examples occur throughout the month.

The Context R-CNN ModelFaster R-CNNmemory bankattention

High-level architecture diagram, showing how Context R-CNN incorporates long-term context within the Faster R-CNN model architecture.

similarity-based attentionmemory bankMM

Context R-CNN is able to leverage context (spanning up to 1 month) to correctly categorize the challenging wildebeest example we saw above. The green values are the corresponding attention weights for each boxed object.

Compared to a Faster R-CNN baseline (left), Context R-CNN (right) is able to capture challenging objects such as an elephant occluded by a tree, two poorly-lit impala, and a vervet monkey leaving the frame. [Images from Snapshot Serengeti]

ResultsSnapshot SerengetiCaltech Camera Trapsmean average precisionS3D

Comparison to a single frame Faster R-CNN baseline, showing both mean average precision (mAP) and average recall (AR) detection metrics.

Ongoing and Future WorkWildlife InsightsiWildCamCVPR Fine-Grained Visual Recognition WorkshopAcknowledgementsThis post reflects the work of the authors as well as the following group of core contributors: Vivek Rathod, Guanhang Wu, Ronny Votel. We are also grateful to Zhichao Lu, David Ross, Tanya Birch and the Wildlife Insights AI team, and Pietro Perona and the Caltech Computational Vision Lab.

Sensing Force-Based Gestures on the Pixel 4

Wednesday, June 24, 2020

Posted by Philip Quinn and Wenxin Feng, Research Scientists, Android UXsoftfirmrecent update to the Pixel 4Touch Sensor Technology and Finger Biomechanicsdrive electrodesense electrodedielectriccell

Left: A finger interacts with a touch sensor cell by ‘stealing’ charge from the projected field around two electrodes. Right: A capacitive touch sensor is constructed from rows and columns of electrodes, separated by a dielectric. The electrodes overlap at cells, where capacitance is measured.

Slowed touch sensor recordings of a user tapping (left), pressing (middle), and scrolling (right).

per se

Touch sensor signals are saturated around the centre of the finger’s contact, but fall off at the edges. This allows us to sense small deformations in the finger’s contact shape caused by changes in the finger’s force.

Machine Learning for Touch Interactionpress gestureconvolutionalrecurrentTensorFlow Lite

An overview of the classification model’s architecture.

User Experience Integrationpress

A user long pressing (left) and firmly pressing (right) on a launcher icon.

GestureDetectorViewMotionEvent classificationAcknowledgementsThis project is a collaborative effort between the Android UX, Pixel software, and Android framework teams.

Presenting a Challenge and Workshop in Efficient Open-Domain Question Answering

Tuesday, June 23, 2020

Posted by Eunsol Choi, Visiting Faculty Researcher and Tom Kwiatkowski, Research Scientist, Google Researchknowledge graphWikipediaT5EfficientQA competition and workshopNeurIPS 2020Princeton UniversityUniversity of Washingtonquestion answering

An illustration of how the memory budget changes as a neural network and retrieval corpus grow and shrink. It is possible that successful systems will also use other resources such as a knowledge graph.

Competition OverviewNatural Questions datasetcompetition track at NeurIPS 20202017 NeurIPS Human-Computer competitionParticipationcompetition siteAcknowledgementsAdam Roberts, Colin Raffel, Chris Alberti, Jordan Boyd-Graber, Jennimaria Palomaki, Kenton Lee, Kelvin Guu, Michael CollinsSewon Min Hannaneh HajishirzDanqi Chen

RepNet: Counting Repetitions in Videos

Monday, June 22, 2020

Posted by Debidatta Dwibedi, Research Scientist, Robotics at GoogleCounting Out Time: Class Agnostic Video Repetition Counting in the WildRepNetour previous workdatasetColab notebookRepNetIn the pastResNetResNetimagevideotemporal self-similarity matrix

Demonstration of how the TSM processes images of the Earth’s day-night cycle.

Transformers

Overview of the RepNet model.

Temporal Self-Similarity Matrix

Jumping Jacks (constant period; video from Kinetics), Bouncing ball (decreasing period; Kinetics), Mixing concrete (aperiodic segments present in video; PERTUBE dataset).

Data

Our synthetic data generation pipeline that produces videos with repetitions from any video.

camera motion augmentationaffine motion

Left: An example of a synthetic repeating video generated from a random video. Right: An example of a video with camera motion augmentation, which is tougher for the model, but results in better generalization to real repeating videos (both from Kinetics).

EvaluationKinetics datasetCountixApplications

RepNet can count repeated activities from a range of domains, such as slicing onions (left; video from Kinetics dataset), Earth’s diurnal cycle (middle; Himawari satellite data), or even a cheetah in motion (right; video from imgur.com).

Predicted heart rates: 45 bpm (left) and 75 bpm (right). True heart rates 46-50 bpm and 78-79 bpm, respectively. RepNet’s prediction of the heart rate across different devices is encouragingly close to the rate measured by the device. (Source for left and right)

In this video, we see RepNet counting accelerating cellular oscillations observed under a laser microscope even though it has never seen such a video during training, (from Nature article).

Left: Person performing a “mountain climber” exercise. Right: The 1D projection of the RepNet embeddings using principal component analysis, capturing the moment that the person changes their speed during the exercise. (Video from Kinetics)

ReleaseannotationsColab notebookAcknowledgementsThis is joint work with Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Special thanks to Tom Small for designing the visual explanation of TSM. The authors thank Anelia Angelova, Relja Arandjelović, Sourish Chaudhuri, Aishwarya Gomatam, Meghana Thotakuri, and Vincent Vanhoucke for their help with this project.

Improving Speech Representations and Personalized Models Using Self-Supervision

Thursday, June 18, 2020

Posted by Joel Shor, Software Engineer and Oran Lang, Software Engineer, Google Research, Israelautomatic speech recognitionBERTALBERTInception layersSimCLR

Bottom:A large speech dataset is used to train a model, which is then rolled out to other environments. Top Left: On-device personalization — personalized, on-device models combine security and privacy. Top Middle: Small model on embeddings — general-use representations transform high-dimensional, few-example datasets to a lower dimension without sacrificing accuracy; smaller models train faster and are regularized. Top Right: Full model fine-tuning — large datasets can use the embedding model as pre-training to improve performance

T5Visual Task Adaptation BenchmarkTowards Learning a Universal Non-Semantic Representation of SpeechThese datasetsTensorFlow DatasetsTRIpLet Loss networkopen-source the codeA New Benchmark for Speech Embeddings

Datasets for downstream benchmark tasks. *VoxCeleb results in our study were computed using a subset of the dataset that was filtered according to internal policy.

kkinternet of thingsadded the six datasets in our benchmark to TensorFlow Datasetsopen sourced the evaluation frameworkTRILL: A New State of the Art in Non-semantic Speech Classificationpersonalizing speech recognizersvoice imitation text-to-speech from few samplesAudioSetpreviousworkBERT

TRILL loss: Embeddings from the same audio are closer in embedding space than embeddings from different audio.

MobileNetResNet50Benchmark ResultsOpenSMILEstrong baselinesy

Effect of model on accuracy.

Interspeech 2020 Computational Paralinguistics Challenge (ComParE)baseline modelSummaryGitHubTensorFlow DatasetsAI HubAcknowledgementsThe core team behind this work includes Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Felix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, and Yinnon Haviv. We'd also like to thank Avinatan Hassidim and Yossi Matias for technical guidance.

Using Selective Attention in Reinforcement Learning Agents

Thursday, June 18, 2020

Posted by Yujin Tang, Research Software Engineer and David Ha, Staff Research Scientist, Google Research, TokyoInattentional blindnessselective attentiondeepreinforcement learningpredictingfuturesequencesreducingGECCO 2020Neuroevolution of Self-Interpretable Agentsbottleneck

AttentionAgent learned to attend to task critical regions in its visual inputs. In a car driving task (CarRacing, top row), the agent mostly attends to the road borders, but shifts its focus to the turns before it changes heading directions. In a fireball dodging game (DoomTakeCover, bottom row), the agent focuses on fireballs and enemy monsters. Left: Visual inputs to the agent. Center: Agent’s attention overlaid on the visual inputs, the white patches indicate where the agent focuses its attention. Right: Visual cues based on which the agent makes decisions.

Agent with Artificial Attentionseveralworksabilitiesself-attentionlong short-term memorybackpropagationdifferentiablederivative-free optimization

Overview of our method and illustration of data processing flow in AttentionAgent. Top: Input transformation — A sliding window segments an input image into smaller patches, and then “flattens” them for future processing. Middle: Patch election — The modified self-attention module holds votes between patches to generate a patch importance vector. Bottom: Action generation — AttentionAgent picks the patches of the highest importance, extracts corresponding features and makes decisions based on them.

Generalization to Unseen Modifications of the EnvironmentVizDoomTakeCover

DoomTakeCover Generalization: The AttentionAgent is trained in the environment with no modifications (left). It is able to adapt to changes in the environment, such as a higher wall (middle, left), a different floor texture (middle, right), or floating text (right).

CarRacing-v0

CarRacing Generalization: No modification (left); color perturbation (middle, left); vertical bars on left and right (middle, right); added red blob (right).

Limitations and Future Work

AttentionAgent fails to generalize to drastically modified environments. Left: The background suddenly becomes a cat (Creative Commons video). Middle: The background suddenly becomes an arcade game (Creative Commons video). Right: AttentionAgent learned to drive on pure noise background by avoiding noise patches.

CarRacingExtensionAcknowledgementsThis research was conducted by Yujin Tang, Duong Nguyen, and David Ha. We would like to thank Yingtao Tian, Lana Sinapayen, Shixin Luo, Krzysztof Choromanski, Sherjil Ozair, Ben Poole, Kai Arulkumaran, Eric Jang, Brian Cheung, Kory Mathewson, Ankur Handa, and Jeff Dean for valuable discussions.

Machine Learning-based Damage Assessment for Disaster Relief

Tuesday, June 16, 2020

Posted by Joseph Xu, Senior Software Engineer and Pranav Khaitan, Engineering Lead, Google ResearchBuilding Damage Detection in Satellite Imagery Using Convolutional Neural NetworksUnited Nations World Food Program (WFP) Innovation AcceleratorThe Approach

convolutional neural networkhistogram equalizationTraining DataUNOSATREACHGoogle Earth Engine

Examples of individual image patches that capture before and after images of damaged and undamaged buildings from different disasters.

ResultsROCtrue positive and false positive rates

Area under the

Event

Accuracy

ROC curve

2010 Haiti earthquake

77%

0.83

2017 Mexico City earthquake

71%

0.79

2018 Indonesia earthquake

78%

0.86

Evaluation of model predictions against human expert assessments (higher is better).

Example model predictions from the 2010 Haiti earthquake. Prediction values closer to 1.0 means the model is more confident that the building is damaged. Values closer to 0.0 means the building is not damaged. A threshold value of 0.5 is typically used to distinguish between damaged/undamaged predictions, but this can be tuned to make the predictions more or less sensitive.

Future WorkAcknowledgementsThis post reflects the work of our co-authors Wenhan Lu and Zebo Li. We would also like to thank Maolin Zuo for his contributions to the project. In tackling this problem, we have had a very productive partnership with the United Nations World Food Programme (WFP) Innovation Accelerator, an organization that identifies, funds, and supports startups and innovative projects to disrupt world hunger.

Blog