Google AI Blog: March 2019

Google Faculty Research Awards 2018

Friday, March 15, 2019

Posted by Maggie Johnson, VP, Education and Negar Saei, Program Manager, University RelationsGoogle Faculty Research Awardshuman computer interactionmachine learningmachine perceptionsystemsrecipientsour websitehere

Harnessing Organizational Knowledge for Machine Learning

Thursday, March 14, 2019

Posted by Alex Ratner, Stanford University and Cassandra Xia, Google AISnorkel Drybell: A Case Study in Deploying Weak Supervision at Industrial Scaleweak supervisionSnorkelorganizational knowledge resources—labeling functionsnamed-entity recognition

In our example of a labeling function, rather than hand-labeling a data point (1), one utilizes an existing knowledge resource—in this case, a NER model (2)—together with some simple logic expressed in code (3) to heuristically label data.

generative modeling techniquecovariance matrixusing a new matrix completion-style approachTensorFlowUsing Diverse Knowledge Sources as Weak Supervision

Heuristics and rules: e.g. existing human-authored rules about the target domain.

Topic models, taggers, and classifiers: e.g. machine learning models about the target domain or a related domain.

Aggregate statistics: e.g. tracked metrics about the target domain.

Knowledge or entity graphs: e.g. databases of facts about the target domain.

In Snorkel DryBell, the goal is to train a machine learning model (C), for example to do content or event classification over web data. Rather than hand-labeling training data to do this, in Snorkel DryBell users write labeling functions that express various organizational knowledge resources (A), which are then automatically reweighted and combined (B).

MapReduceModeling the Accuracies to Combine & Repurpose Existing Sourcesgenerative modeling techniquestatistically estimatedTransferring Non-Servable Knowledge to Servable Modelsservable non-servable

In many settings, users write labeling functions that leverage organizational knowledge resources that are not servable in production (a)—e.g. aggregate statistics, internal models, or knowledge graphs that are too slow or expensive to use in production—in order to train models that are only defined over production-servable features (b), e.g. cheap, real-time web signals.

servablecross-feature Next Stepsour papersnorkel.stanford.eduAcknowledgmentsThis research was done in collaboration between Google, Stanford, and Brown. We would like to thank all the people who were involved, including Stephen Bach (Brown), Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Souvik Sen, Braden Hancock (Stanford), Houman Alborzi, Rahul Kuchhal, Christopher Ré (Stanford), Rob Malkin.

An All-Neural On-Device Speech Recognizer

Tuesday, March 12, 2019

Posted by Johan Schalkwyk, Google Fellow, Speech Team significant accuracy improvements with deep learningVoice Searchdeep neural networksrecurrent neural networkslong short-term memory networksconvolutional networksStreaming End-to-End Speech Recognition for Mobile DevicesRNN transducer

This video compares the production, server-side speech recognizer (left panel) to the new on-device recognizer (right panel) when recognizing the same spoken sentence. Video credit: Akshay Kannan and Elnaz Sarbar

A Bit of Historysingle attention-basedlisten-attend-spellgreat promiseconnectionist temporal classification (CTC)halve the latencyRecurrent Neural Network Transducers

Representation of an RNN-T, with the input audio samples, x, and the predicted symbols y. The predicted symbols (outputs of the Softmax layer) are fed back into the model through the Prediction network, as y_u-1, ensuring that the predictions are conditioned both on the audio samples so far and on past outputs. The Prediction and Encoder Networks are LSTM RNNs, the Joint model is a feedforward network (paper). The Prediction Network comprises 2 layers of 2048 units, with a 640-dimensional projection layer. The Encoder Network comprises 8 such layers. Image credit: Chris Thornton

training techniqueparallel implementationTPU v2Offline Recognitionsearch graphFinite State Transducerbeam search through a single neural networkparameter quantization and hybrid kernel techniquesmodel optimization toolkitTensorFlow LiteAcknowledgements:Raziel Alvarez, Michiel Bacchiani, Tom Bagby, Françoise Beaufays, Deepti Bhatia, Shuo-yiin Chang, Zhifeng Chen, Chung-Chen Chiu, Yanzhang He, Alex Gruenstein, Anjuli Kannan, Bo Li, Wei Li, Qiao Liang, Ian McGraw, Patrick Nguyen, Ruoming Pang, Rohit Prabhavalkar, Golan Pundak, Kanishka Rao, David Rybach, Tara Sainath, Haşim Sak, June Yuan Shangguan, Matt Shannon, Mohammadinamul Sheik, Khe Chai Sim, Gabor Simko, Trevor Strohman, Mirkó Visontai, Ron Weiss, Yonghui Wu, Ding Zhao, Dan Zivkovic, and Yu Zhang.

Real-Time AR Self-Expression with Machine Learning

Friday, March 8, 2019

Posted by Artsiom Ablavatski and Ivan Grishchenko, Research Engineers, Google AIAugmented realityAR features coming to Google MapsPlayground - a creative mode in the Pixel cameraYouTube StoriesARCoreAugmented Faces API

Our 3D mesh and some of the effects it enables

TensorFlow Liteits new mobile GPU functionalityYouTube Stories' latest ARCore SDKML Kit Face Contour Detection APIAn ML Pipeline for Selfie AR

Our 3D mesh in action

transfer learningMLKitreasonable

Dataset expansion and improvement pipeline

Hardware-tailored InferenceTensorFlow Litenewly introduced GPU back-end

Inference time per frame: CPU vs. GPU

Comparison of the most complex (left) and the lightest models (right). Temporal consistency as well as lip and eye tracking is slightly degraded on light models.

Simulating light reflections via environmental mapping for realistic rendering of glasses

Natural lighting by casting virtual object shadows onto the face mesh

Modelling face occlusions to hide virtual object parts behind a face, e.g. virtual glasses, as shown below.

YouTube Stories includes Creator Effects like realistic virtual glasses, based on our 3D mesh

Modelling Specular reflections applied on lips and

Face painting by using luminance-aware material

Case study comparing real make-up against our AR make-up on 5 subjects under different lighting conditions.

latest ARCore SDKAcknowledgementsWe would like to thank Yury Kartynnik, Valentin Bazarevsky, Andrey Vakunov, Siargey Pisarchyk, Andrei Tkachenka, and Matthias Grundmann for collaboration on developing the current mesh technology; Nick Dufour, Avneesh Sud and Chris Bregler for an earlier version of the technology based on parametric models; Kanstantsin Sokal, Matsvei Zhdanovich, Gregory Karpiak, Alexander Kanaukou, Suril Shah, Buck Bourdon, Camillo Lugaresi, Siarhei Kazakou and Igor Kibalchich for building the ML pipeline to drive impressive effects; Aleksandra Volf and the annotation team for their diligence and dedication to perfection; Andrei Kulik, Juhyun Lee, Raman Sarokin, Ekaterina Ignasheva, Nikolay Chirkov, and Yury Pisarchyk for careful benchmarking and insights on mobile GPU-centric network architecture optimizations.

RNN-Based Handwriting Recognition in Gboard

Thursday, March 7, 2019

Posted by Sandro Feuz and Pedro Gonnet, Senior Software Engineers, Handwriting TeamGoogle Handwriting Inputinitial launchGboard for AndroidFast Multi-language LSTM-based Online Handwriting Recognition

Touch Points, Bézier Curves and Recurrent Neural Networks Bézier curvesrecurrent neural networkinputsprevious modelsxygogo

Character Decodingquasi-recurrent neural networksgoblankConnectionist Temporal Classificationweighted finite-state acceptor

blankMaking it Work, On-deviceTensorFlowTensorFlow LitequantizingMore to ComeAcknowledgementsWe would like to thank everybody who contributed to improving the handwriting experience in Gboard. In particular, Jatin Matani from the Gboard team, David Rybach from the Speech & Language Algorithms Team, Prabhu Kaliamoorthi‎ from the Expander Team, Pete Warden from the TensorFlow Lite team, as well as Henry Rowley‎, Li-Lun Wang‎, Mircea Trăichioiu‎, Philippe Gervais, and Thomas Deselaers from the Handwriting Team.

Exploring Neural Networks with Activation Atlases

Wednesday, March 6, 2019

Posted by Shan Carter, Software Engineer, Google AIde factoOpenAIExploring Neural Networks with Activation Atlasesjupyter notebooks

A detail view of an activation atlas from one of the layers of the InceptionV1 vision classification network. It reveals many of the visual detectors that the network uses to classify images, such as different types of fruit-like textures, honeycomb patterns and fabric-like textures.

Inceptionv1ImageNetcarbonarasnorkelfrying panUMAP

Left: A randomized set of one million images is fed through the network, collecting one random spatial activation per image. Center: The activations are fed through UMAP to reduce them to two dimensions. They are then plotted, with similar activations placed near each other. Right: We then draw a grid, average the activations that fall within a cell, and run feature inversion on the averaged activation.

one

An overview of an activation atlas for one of the many layers (mixed4c) within Inception v1. It is about halfway through the network.

In this detail, we can see detectors for different types of leaves and plants.

Here we can see different detectors for water, lakes and sandbars.

Here we see different types of buildings and bridges.

In an early layer, mixed4a, there is a vague "mammalian" area.

By the next layer in the network, mixed4b, animals and people have been disentangled, with some fruit and food emerging in the middle.

By layer mixed4c these concepts are further refined and differentiated into small "peninsulas".

Left: This early layer is very nonspecific in comparison to the others. Center: By the middle layer, the images definitely resemble leaves, but they could be any type of plant. Right: By the last layer the images are very specific to cabbage, leaves curved into rounded balls.

You can see how sand and water are distinct concepts in a middle layer, mixed4c (left and center), both with strong attributions to the classification of "sandbar". Contrast this with a later layer (right), mixed5b, where the two ideas seem to be fused into one activation.

Here we can more clearly see what the network is focusing on to classify a "red fox". There are pointy ears, white snouts surrounded by red fur, and wooded or snowy backgrounds.

Here we can see the many different scales and angles of detectors for "tile roof".

For "ibex", we see detectors for horns and brown fur, but also environments where we might find such animals, like rocky hillsides.

Like the detectors for tile roof, "artichoke" also has many different sizes of detectors for the texture of an artichoke, but we also get some purple flower detectors. These are presumably detecting the blossoms of an artichoke plant.

jupyter notebookscolabLucid

Introducing GPipe, an Open Source Library for Efficiently Training Large-scale Neural Network Models

Monday, March 4, 2019

Posted by Yanping Huang, Software Engineer, Google AIDeep neural networksBigGanBertGPT2.0ImageNet visual recognition challengeGoogleNetSqueeze-and-Excitation Networksstate-of-the-art image modelsTPUv2s

Strong correlation between ImageNet accuracy and model size for recently developed representative image classification models

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelismpipeline parallelismgradient descentAmoebaNetCIFARGPipe libraryLingvoFrom Mini- to Micro-Batchesmini-batchmicro-batches

Top: The naive model parallelism strategy leads to severe underutilization due to the sequential nature of the network. Only one accelerator is active at a time. Bottom: GPipe divides the input mini-batch into smaller micro-batches, enabling different accelerators to work on separate micro-batches at the same time.

Maximizing Memory and EfficiencyTransformer

Speedup of AmoebaNet-D using GPipe. This model could not fit into one accelerator. The baseline naive-2 is the performance of the native partition approach when the model is split into two partitions. Pipeline-k refers to the performance of GPipe that splits the model into k partitions with k accelerators.

Testing Accuracybetter ImageNet models transfer betterConclusionGPipeAcknowledgmentsSpecial thanks to the co-authors of the paper: Youlong Cheng, Dehao Che, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. We wish to thank Esteban Real, Alok Aggarwal, Xiaodan Song, Naveen Kumar, Mark Heffernan, Rajat Monga, Megan Kacholia, Samy Bengio, and Jeff Dean for their support and valuable input; Noam Shazeer, Patrick Nguyen, Xiaoqiang Zheng, Yonghui Wu, Barret Zoph, Ekin Cubuk, Jonathan Shen, Tianqi Chen, and Vijay Vasudevan for helpful discussions and inspirations; and the larger Google Brain team.

Blog