Google AI Blog

Simulated Policy Learning in Video Models

Monday, March 25, 2019

Posted by Łukasz Kaiser and Dumitru Erhan, Research Scientists, Google AIreinforcement learningclassic Atari 2600 gamesRecent workexploration regimesMontezuma's RevengeModel-Based Reinforcement Learning for Ataricodetensor2tensorLearning a SimPLe World Modelwell establishedrecentmodel-basedreinforcementlearningmethods

Main loop of SimPLe. 1) The agent starts interacting with the real environment. 2) The collected observations are used to update the current world model. 3) The agent updates the policy by learning inside the world model.

pixel spacetrajectoriesrealfeedforward convolutional networkPongpreviouswork

One example of an issue arising from stochasticity is seen when the SimPle model is applied to Kung Fu Master. In the animation, the left is the output of the model, the middle is the groundtruth, and the right panel is the pixel-wise difference between the two. Here the model's predictions deviate from the real game by spawning a different number of opponents.

Proximal Policy Optimization (PPO)FreewaySimPLe Efficiency Rainbow

The number of interactions needed by the respective model-free algorithms (left - Rainbow; right - PPO) to match the score achieved using our SimPLe training method. The red line indicates the number of interactions used by our method.

SimPLe SuccessFreeway

Breakout

Nearly pixel perfect predictions can be made by SimPLe, on Breakout (top) and Freeway (bottom). In each animation, the left is the output of the model, the middle is the groundtruth, and the right pane is the pixel-wise difference between the two.

SimPLe SurprisesAtlantisBattlezonePrivate Eye

In Battlezone, we find the model struggles with predicting small, relevant parts, such as the bullet.

ConclusionrepositorycolabAcknowledgementsThis work was done in collaboration with the University of Illinois at Urbana-Champaign, the University of Warsaw and deepsense.ai. We would like to give special recognition to paper co-authors Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker and Henryk Michalewski.

Reducing the Need for Labeled Data in Generative Adversarial Networks

Wednesday, March 20, 2019

Posted by Mario Lučić, Research Scientist and Marvin Ritter, Software Engineer, Google AI ZürichGenerative adversarial networksGANsthe generator,discriminator,high-fidelity natural image synthesisimproving learned image compression

Evolution of the generated samples as training progresses on ImageNet. The generator network is conditioned on the class (e.g., "great gray owl" or "golden retriever").

conditional GANsHigh-Fidelity Image Generation With Fewer Labels10x fewer labelsCompare GAN libraryImprovements via Semi-supervision and Self-supervisionself-supervisionrecently introduced

An unlabeled image is randomly rotated and the network is tasked with predicting the rotation angle. Successful models need to capture semantically meaningful image features which can then be used for other vision tasks.

previouslyFréchet Inception Distance

Given a latent vector the generator network produces an image. In each row, linear interpolation between the latent codes of the leftmost and the rightmost image results in a semantic interpolation in the image space.

Compare GAN: A Library for Training and Evaluating GANsCompare GAN

Training on GPUs and TPUs.

Lightweight configuration via Gin (examples).

A plethora of data sets via the TensorFlow datasets library.

Conclusions and Future Workincreasingly importantAcknowledgmentsWork conducted in collaboration with colleagues on the Google Brain team in Zürich, ETH Zürich and UCLA. We would like to thank our paper co-authors Michael Tschannen, Xiaohua Zhai, Olivier Bachem and Sylvain Gelly for their input and feedback. We would like to thank Alexander Kolesnikov, Lucas Beyer and Avital Oliver for helpful discussion on self-supervised learning and semi-supervised learning. We would like to thank Karol Kurach and Marcin Michalski for their major contributions to the Compare GAN library. We would also like to thank Andy Brock, Jeff Donahue and Karen Simonyan for their insights into training GANs on TPUs. The work described in this post also builds upon our work on “Self-Supervised Generative Adversarial Networks” with Ting Chen and Neil Houlsby.

Measuring the Limits of Data Parallel Training for Neural Networks

Tuesday, March 19, 2019

Posted by Chris Shallue, Senior Software Engineer and George Dahl, Senior Research Scientist, Google AIimage classificationmachine translationspeech recognitionCloud TPU Podsdistribute computationsmodel parallelismdata parallelismstochastic gradient descentbatch sizeMeasuring the Effects of Data Parallelism in Neural Network Trainingshare our raw data our paperUniversal Relationship Between Batch Size and Training Timestepsout-of-sample error

For all workloads we tested, we observed a universal relationship between batch size and training speed with three distinct regimes: perfect scaling (following the dashed line), diminishing returns (diverging from the dashed line), and maximal data parallelism (where the trend plateaus). The transition points between the regimes vary dramatically between different workloads.

Cloud TPU PodsResNet-8CIFAR-10ResNet-50ImageNetOptimizing Workloads

Left: A transformer neural network scales to much larger batch sizes than an LSTM neural network on the LM1B dataset. Right: The Common Crawl dataset does not benefit from larger batch sizes than the LM1B dataset, even though it is 1,000 times the size.

momentumFuture WorkAcknowledgementsThe authors of this study were Chris Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig and George Dahl (Chris and Jaehoon contributed equally). Many researchers have done work in this area that we have built on, so please see our paper for a full discussion of related work.

A Summary of the Google Flood Forecasting Meets Machine Learning Workshop

Monday, March 18, 2019

Posted by Sella Nevo, Senior Software Engineer and Rainier Aliment, Program ManagerGoogle Flood Forecasting Meets Machine Learningour belief

Panel on challenges and opportunities in flood forecasting, featuring (from left to right): Prof. Paolo Burlando (ETH Zürich), Dr. Tyler Erickson (Google Earth Engine), Dr. Peter Salamon (Joint Research Centre) and Prof. Dawei Han (University of Bristol).

Yossi Matiasrecentmachinelearningflood forecastingcrisis responseAI for Social GoodProf. Peter MolnarProf. Yishay MansourGooglefascinating talksposters

An overview of research areas in flood forecasting addressed in the workshop.

Dr. Dhanya C. T. of IIT Delhi gave a talk on satellite precipitation error characterization.

Adarsh M. S., Assistant Director of the Indian Ministry of Water Resources presented India's Central Water Commission's role and challenges.

Prof. Andras Bardossy of the University of Stuttgart discussed variation in discharge series and the challenges this presents.

Frederik Kratzert of Johannes Kepler University presented recent work on hydrologic modeling using LSTMs.

Prof. Paul Bates of the University of Bristol gave a keynote on the potential uses of machine learning in inundation modelling.

Prof. Emmanouil Anagnostou of the University of Connecticut spoke about hyper-resolution hydrologic simulations at global-scale.

Prof. Efrat Morin of the Hebrew University highlighted flood prediction challenges in dry climate regions.

Dr. Zachary Flamig of the University of Chicago presented NASA's new global flash flood prediction project.

Vova Anisimov presented our progress in hydraulic modeling.

Ami Weisel presented our research on remote discharge estimation.

Stephan Hoyer presented our work on data-driven discretization approach to solving partial differential equations.

Jason Hickey presented our efforts using machine learning for precipitation prediction.

Avinatan Hassidim presented lessons learned from previous projects in Google, and how they apply to our flood forecasting efforts.

Prof. Paolo BurlandoProf. Dawei HanDr. Peter SalamonDr. Tyler EricksonAI for Social Goodour continued engagementAcknowledgementsWe would like to thank Avinatan Hassidim, Carla Bromberg, Doron Kukliansky, Efrat Morin, Gal Elidan, Guy Shalev, Jennifer Ye, Nadav Rabani and Sasha Goldshtein for their contributions to making this workshop happen.

Google Faculty Research Awards 2018

Friday, March 15, 2019

Posted by Maggie Johnson, VP, Education and Negar Saei, Program Manager, University RelationsGoogle Faculty Research Awardshuman computer interactionmachine learningmachine perceptionsystemsrecipientsour websitehere

Harnessing Organizational Knowledge for Machine Learning

Thursday, March 14, 2019

Posted by Alex Ratner, Stanford University and Cassandra Xia, Google AISnorkel Drybell: A Case Study in Deploying Weak Supervision at Industrial Scaleweak supervisionSnorkelorganizational knowledge resources—labeling functionsnamed-entity recognition

In our example of a labeling function, rather than hand-labeling a data point (1), one utilizes an existing knowledge resource—in this case, a NER model (2)—together with some simple logic expressed in code (3) to heuristically label data.

generative modeling techniquecovariance matrixusing a new matrix completion-style approachTensorFlowUsing Diverse Knowledge Sources as Weak Supervision

Heuristics and rules: e.g. existing human-authored rules about the target domain.

Topic models, taggers, and classifiers: e.g. machine learning models about the target domain or a related domain.

Aggregate statistics: e.g. tracked metrics about the target domain.

Knowledge or entity graphs: e.g. databases of facts about the target domain.

In Snorkel DryBell, the goal is to train a machine learning model (C), for example to do content or event classification over web data. Rather than hand-labeling training data to do this, in Snorkel DryBell users write labeling functions that express various organizational knowledge resources (A), which are then automatically reweighted and combined (B).

MapReduceModeling the Accuracies to Combine & Repurpose Existing Sourcesgenerative modeling techniquestatistically estimatedTransferring Non-Servable Knowledge to Servable Modelsservable non-servable

In many settings, users write labeling functions that leverage organizational knowledge resources that are not servable in production (a)—e.g. aggregate statistics, internal models, or knowledge graphs that are too slow or expensive to use in production—in order to train models that are only defined over production-servable features (b), e.g. cheap, real-time web signals.

servablecross-feature Next Stepsour papersnorkel.stanford.eduAcknowledgmentsThis research was done in collaboration between Google, Stanford, and Brown. We would like to thank all the people who were involved, including Stephen Bach (Brown), Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Souvik Sen, Braden Hancock (Stanford), Houman Alborzi, Rahul Kuchhal, Christopher Ré (Stanford), Rob Malkin.

An All-Neural On-Device Speech Recognizer

Tuesday, March 12, 2019

Posted by Johan Schalkwyk, Google Fellow, Speech Team significant accuracy improvements with deep learningVoice Searchdeep neural networksrecurrent neural networkslong short-term memory networksconvolutional networksStreaming End-to-End Speech Recognition for Mobile DevicesRNN transducer

This video compares the production, server-side speech recognizer (left panel) to the new on-device recognizer (right panel) when recognizing the same spoken sentence. Video credit: Akshay Kannan and Elnaz Sarbar

A Bit of Historysingle attention-basedlisten-attend-spellgreat promiseconnectionist temporal classification (CTC)halve the latencyRecurrent Neural Network Transducers

Representation of an RNN-T, with the input audio samples, x, and the predicted symbols y. The predicted symbols (outputs of the Softmax layer) are fed back into the model through the Prediction network, as y_u-1, ensuring that the predictions are conditioned both on the audio samples so far and on past outputs. The Prediction and Encoder Networks are LSTM RNNs, the Joint model is a feedforward network (paper). The Prediction Network comprises 2 layers of 2048 units, with a 640-dimensional projection layer. The Encoder Network comprises 8 such layers. Image credit: Chris Thornton

training techniqueparallel implementationTPU v2Offline Recognitionsearch graphFinite State Transducerbeam search through a single neural networkparameter quantization and hybrid kernel techniquesmodel optimization toolkitTensorFlow LiteAcknowledgements:Raziel Alvarez, Michiel Bacchiani, Tom Bagby, Françoise Beaufays, Deepti Bhatia, Shuo-yiin Chang, Zhifeng Chen, Chung-Chen Chiu, Yanzhang He, Alex Gruenstein, Anjuli Kannan, Bo Li, Wei Li, Qiao Liang, Ian McGraw, Patrick Nguyen, Ruoming Pang, Rohit Prabhavalkar, Golan Pundak, Kanishka Rao, David Rybach, Tara Sainath, Haşim Sak, June Yuan Shangguan, Matt Shannon, Mohammadinamul Sheik, Khe Chai Sim, Gabor Simko, Trevor Strohman, Mirkó Visontai, Ron Weiss, Yonghui Wu, Ding Zhao, Dan Zivkovic, and Yu Zhang.

Real-Time AR Self-Expression with Machine Learning

Friday, March 8, 2019

Posted by Artsiom Ablavatski and Ivan Grishchenko, Research Engineers, Google AIAugmented realityAR features coming to Google MapsPlayground - a creative mode in the Pixel cameraYouTube StoriesARCoreAugmented Faces API

Our 3D mesh and some of the effects it enables

TensorFlow Liteits new mobile GPU functionalityYouTube Stories' latest ARCore SDKML Kit Face Contour Detection APIAn ML Pipeline for Selfie AR

Our 3D mesh in action

transfer learningMLKitreasonable

Dataset expansion and improvement pipeline

Hardware-tailored InferenceTensorFlow Litenewly introduced GPU back-end

Inference time per frame: CPU vs. GPU

Comparison of the most complex (left) and the lightest models (right). Temporal consistency as well as lip and eye tracking is slightly degraded on light models.

Simulating light reflections via environmental mapping for realistic rendering of glasses

Natural lighting by casting virtual object shadows onto the face mesh

Modelling face occlusions to hide virtual object parts behind a face, e.g. virtual glasses, as shown below.

YouTube Stories includes Creator Effects like realistic virtual glasses, based on our 3D mesh

Modelling Specular reflections applied on lips and

Face painting by using luminance-aware material

Case study comparing real make-up against our AR make-up on 5 subjects under different lighting conditions.

latest ARCore SDKAcknowledgementsWe would like to thank Yury Kartynnik, Valentin Bazarevsky, Andrey Vakunov, Siargey Pisarchyk, Andrei Tkachenka, and Matthias Grundmann for collaboration on developing the current mesh technology; Nick Dufour, Avneesh Sud and Chris Bregler for an earlier version of the technology based on parametric models; Kanstantsin Sokal, Matsvei Zhdanovich, Gregory Karpiak, Alexander Kanaukou, Suril Shah, Buck Bourdon, Camillo Lugaresi, Siarhei Kazakou and Igor Kibalchich for building the ML pipeline to drive impressive effects; Aleksandra Volf and the annotation team for their diligence and dedication to perfection; Andrei Kulik, Juhyun Lee, Raman Sarokin, Ekaterina Ignasheva, Nikolay Chirkov, and Yury Pisarchyk for careful benchmarking and insights on mobile GPU-centric network architecture optimizations.

RNN-Based Handwriting Recognition in Gboard

Thursday, March 7, 2019

Posted by Sandro Feuz and Pedro Gonnet, Senior Software Engineers, Handwriting TeamGoogle Handwriting Inputinitial launchGboard for AndroidFast Multi-language LSTM-based Online Handwriting Recognition

Touch Points, Bézier Curves and Recurrent Neural Networks Bézier curvesrecurrent neural networkinputsprevious modelsxygogo

Character Decodingquasi-recurrent neural networksgoblankConnectionist Temporal Classificationweighted finite-state acceptor

blankMaking it Work, On-deviceTensorFlowTensorFlow LitequantizingMore to ComeAcknowledgementsWe would like to thank everybody who contributed to improving the handwriting experience in Gboard. In particular, Jatin Matani from the Gboard team, David Rybach from the Speech & Language Algorithms Team, Prabhu Kaliamoorthi‎ from the Expander Team, Pete Warden from the TensorFlow Lite team, as well as Henry Rowley‎, Li-Lun Wang‎, Mircea Trăichioiu‎, Philippe Gervais, and Thomas Deselaers from the Handwriting Team.

Exploring Neural Networks with Activation Atlases

Wednesday, March 6, 2019

Posted by Shan Carter, Software Engineer, Google AIde factoOpenAIExploring Neural Networks with Activation Atlasesjupyter notebooks

A detail view of an activation atlas from one of the layers of the InceptionV1 vision classification network. It reveals many of the visual detectors that the network uses to classify images, such as different types of fruit-like textures, honeycomb patterns and fabric-like textures.

Inceptionv1ImageNetcarbonarasnorkelfrying panUMAP

Left: A randomized set of one million images is fed through the network, collecting one random spatial activation per image. Center: The activations are fed through UMAP to reduce them to two dimensions. They are then plotted, with similar activations placed near each other. Right: We then draw a grid, average the activations that fall within a cell, and run feature inversion on the averaged activation.

one

An overview of an activation atlas for one of the many layers (mixed4c) within Inception v1. It is about halfway through the network.

In this detail, we can see detectors for different types of leaves and plants.

Here we can see different detectors for water, lakes and sandbars.

Here we see different types of buildings and bridges.

In an early layer, mixed4a, there is a vague "mammalian" area.

By the next layer in the network, mixed4b, animals and people have been disentangled, with some fruit and food emerging in the middle.

By layer mixed4c these concepts are further refined and differentiated into small "peninsulas".

Left: This early layer is very nonspecific in comparison to the others. Center: By the middle layer, the images definitely resemble leaves, but they could be any type of plant. Right: By the last layer the images are very specific to cabbage, leaves curved into rounded balls.

You can see how sand and water are distinct concepts in a middle layer, mixed4c (left and center), both with strong attributions to the classification of "sandbar". Contrast this with a later layer (right), mixed5b, where the two ideas seem to be fused into one activation.

Here we can more clearly see what the network is focusing on to classify a "red fox". There are pointy ears, white snouts surrounded by red fur, and wooded or snowy backgrounds.

Here we can see the many different scales and angles of detectors for "tile roof".

For "ibex", we see detectors for horns and brown fur, but also environments where we might find such animals, like rocky hillsides.

Like the detectors for tile roof, "artichoke" also has many different sizes of detectors for the texture of an artichoke, but we also get some purple flower detectors. These are presumably detecting the blossoms of an artichoke plant.