Google AI Blog

Introducing PlaNet: A Deep Planning Network for Reinforcement Learning

Friday, February 15, 2019

Posted by Danijar Hafner, Student Researcher, Google AIartificial agentsreinforcement learningModel-free directlyDeepMind's DQNcontrolrobotsModel-based RLplan ahead, AlphaGounknowndynamicsDeepMindDeep Planning Network (PlaNet)source code

The PlaNet agent learning to solve a variety of continuous control tasks from images in 2000 attempts. Previous agents that do not learn a model of the environment often require 50 times as many attempts to reach comparable performance.

How PlaNet Worksplan over imageslatent dynamics model

Learned Latent Dynamics Model: In a latent dynamics model, the information of the input images is integrated into the hidden states (green) using the encoder network (grey trapezoids). The hidden state is then projected forward in time to predict future images (blue trapezoids) and rewards (blue rectangle).

A Recurrent State Space Model: A latent dynamics model with both deterministic and stochastic components, allowing to predict a variety of possible futures as needed for robust planning, while remembering information over many time steps. Our experiments indicate both components to be crucial for high planning performance.

A Latent Overshooting Objective: We generalize the standard training objective for latent dynamics models to train multi-step predictions, by enforcing consistency between one-step and multi-step predictions in latent space. This yields a fast and effective objective that improves long-term predictions and is compatible with any latent sequence model.

Planning in Latent Space: For planning, we encode past images (gray trapezoid) into the current hidden state (green). From there, we efficiently predict future rewards for multiple action sequences. Note how the expensive image decoder (blue trapezoid) from the previous figure is gone. We then execute the first action of the best sequence found (red box).

world modelsonline research paperPDF versionPlaNet vs. Model-Free Methods

A cartpole swing-up task, with a fixed camera, so the cart can move out of sight. The agent thus must absorb and remember information over multiple frames.

A finger spin task that requires predicting two separate objects, as well as the interactions between them.

A cheetah running task that includes contacts with the ground that are difficult to predict precisely, calling for a model that can predict multiple possible futures.

A cup task, which only provides a sparse reward signal once a ball is caught. This demands accurate predictions far into the future to plan a precise sequence of actions.

A walker task, in which a simulated robot starts off by lying on the ground, and must first learn to stand up and then walk.

PlaNet agents trained on a variety of image-based control tasks. The animation shows the input images as the agent is solving the tasks. The tasks pose different challenges: partial observability, contacts with the ground, sparse rewards for catching a ball, and controlling a challenging bipedal robot.

A3CD4PGDeepMind Control Suite

One Agent for All Tasks

Video predictions of the PlaNet agent trained on multiple tasks. Holdout episodes collected with the trained agent are shown above and open-loop agent hallucinations below. The agent observes the first 5 frames as context to infer the task and state and accurately predicts ahead for 50 steps given a sequence of actions.

Conclusion3D environmentsTPUsAcknowledgementsThis project is a collaboration with Timothy Lillicrap, Ian Fischer, Ruben Villegas, Honglak Lee, David Ha and James Davidson. We further thank everybody who commented on our paper draft and provided feedback at any point throughout the project.

Using Global Localization to Improve Navigation

Monday, February 11, 2019

Posted by Tilman Reinhardt‎, Software Engineer, Google MapsWhere exactly am I, andwhich way is north?" global localizationVisual Positioning ServiceStreet View

Due to limitations with accuracy and orientation, guidance via GPS alone is limited in urban environments. Using VPS, Street View and machine learning, Global Localization can provide better context on where you are relative to where you're going.

Where GPS Falls ShortlocalizationGPS

GPS signals bouncing off facades in an urban environment.

A New Approach to LocalizationexpectEnter Street ViewStreet View

Features matched from multiple images.

Combining Global Localization with Augmented RealityARCore

Local GuidesGoogle Lens

Announcing the Second Workshop and Challenge on Learned Image Compression

Wednesday, February 6, 2019

Posted by Nick Johnston, Software Engineer, Machine Perceptionwe announcedWorkshop and Challenge on Learned Image Compression CVPR 201823 accepted workshop papersthe competitionmean opinion scoreBetter Portable Graphics

An example image from the 2018 test set, comparing the original image to BPG, JPEG and the results from nine competing teams. All the methods are better than JPEG in color reproduction and many of them are comparable to BPG in their ability to create legible text on the sign.

Workshop and Challenge on Learned Image CompressionCVPR 2019PSNRMS-SSIMhuman evaluated rating taskcompression.ccAcknowledgementsThis workshop is being jointly hosted by researchers at Google, Twitter and ETH Zürich. We'd like to thank: George Toderici (Google), Michele Covell (Google), Johannes Ballé (Google), Nick Johnston (Google), Eirikur Agustsson (Google), Wenzhe Shi (Twitter), Lucas Theis (Twitter), Radu Timofte (ETH Zürich), Fabian Mentzer (ETH Zürich) for their contributions.

Real-time Continuous Transcription with Live Transcribe

Monday, February 4, 2019

Posted by Sagar Savla, Product Manager, Machine PerceptionWorld Health Organizationestimatesautomatic speech recognitionGoogle's ASRautomated captions in Youtubepresentations in Slidesmultiple improvementsCARTPalantypistSTTRLive TranscribeGoogle CloudBuilding Live Transcribeour previous workAudioSetpublished VGGish modelUser ExperienceGallaudet Universityeven small projectors

Displaying confidence level of the transcription. Yellow is high confidence, green is medium and blue is low confidence. White is fresh text awaiting context before finalizing. On the left, the coloring is at a per-phrase level while on the right is at a per-word level.¹ Research found them to be distracting to the user without providing conversational value.

UX research in this spacecocktail party problemrelative

The loudness and noise indicator is made of two concentric circles. The inner brighter circle, indicating the noise floor, tells a deaf user how audibly noisy the current environment is. The outer circle shows how well the speaker’s voice is received.Together, the circles visually show the relative difference intuitively.

Future Workon-device recognitionspeaker-separationspeech enhancementThe KeywordAcknowledgementsLive Transcribe was made by researchers Chet Gnegy, Dimitri Kanevsky, and Justin S. Paul in collaboration with Android Accessibility team members Brian Kemler, Thomas Lin, Alex Huang, Jacqueline Huang, Ben Chung, Richard Chang, I-ting Huang, Jessie Lin, Ausmus Chang, Weiwei Wei, Melissa Barnhart and Bingying Xia. We'd also like to thank our close partners from Gallaudet University, Christian Vogler, Norman Williams and Paula Tucker.
1 Eagle-eyed readers can see the phrase level confidence mode in use by Dr. Obeidat in the video above.^↩

Transformer-XL: Unleashing the Potential of Attention Models

Tuesday, January 29, 2019

Posted by Zhilin Yang and Quoc Le, Google AIlong-range dependenceGating-based RNNsthe gradient clipping techniquenot sufficientTransformers,a fixed-length context

Vanilla Transformer with a fixed-length context at training time.

The algorithm is not able to model dependencies that are longer than a fixed length.

The segments usually do not respect the sentence boundaries, resulting in context fragmentation which leads to inefficient optimization. This is particularly troublesome even for short sequences, where long range dependency isn't an issue.

Transformer-XLSegment-level Recurrence

Transformer-XL with segment-level recurrence at training time.

Relative Positional Encodingsother relative positional encoding schemes

Vanilla Transformer with a fixed-length context at evaluation time.

Transformer-XL with segment-level recurrence at evaluation time./td>

Results

Transformer-XL learns dependency that is about 80% longer than RNNs and 450% longer than vanilla Transformers, which generally have better performance than RNNs, but are not the best for long-range dependency modeling due to fixed-length contexts (please see our paper for details).

Transformer-XL is up to 1,800+ times faster than a vanilla Transformer during evaluation on language modeling tasks, because no re-computation is needed (see figures above).

Transformer-XL has better performance in perplexity (more accurate at predicting a sample) on long sequences because of long-term dependency modeling, and also on short sequences by resolving the context fragmentation problem.

enwiki8text8WikiText-103One Billion WordPenn Treebanklanguage model pretrainingBERTpaperGitHub

Natural Questions: a New Corpus and Challenge for Question Answering Research

Wednesday, January 23, 2019

Posted by Tom Kwiatkowski and Michael Collins, Research Scientists, Google AI LanguageOpen-domain question answeringnatural language understanding"Why is the sky blue?"this Wikipedia pagenaturally occurringNatural Questions¹quite easy for computers to solvechallengeThe Datalong answersshort answers

Natural Questions: a Benchmark for Question Answering ResearchTransactions of the Association for Computational LinguisticsNQ websiteThe Challengechallenge website
1 A reader just alerted us to DuReader, a dataset from Baidu that contains real queries and full documents. We will add this to our paper, in which we discuss NQ in relation to previous work.^↩

Expanding the Application of Deep Learning to Electronic Health Records

Tuesday, January 22, 2019

Posted by Alvin Rajkomar, MD and Eyal Oren, PhD, Google AI, Healthcarepublished a paperour work aims to further improve potential clinical benefitIntermountain Healthcare

TeleHealth critical care unit

Soft Actor-Critic: Deep Reinforcement Learning for Robotics

Friday, January 18, 2019

Posted by Tuomas Haarnoja, Student Researcher and Sergey Levine, Faculty Advisor, Robotics at Googlewe recently released Soft Actor-CriticRequirements for Real-World Robotic Learning

Good sample efficiency to lower the learning time

Minimal number of hyperparameters that require tuning

Reusing already collected data on different scenarios (known as off-policy learning)

Ensuring that learning and exploration does not damage the hardware

Soft Actor-Criticentropy BAIR blog post tutorialQ-functiontechnical reportPerformance of SACGhost RoboticsDynamixel Claw

Illustration of learned walking, using SAC implemented on the Minitaur robot. A full video of the learning process can be found at our project website.

prior workConclusionBAIR blog postearly preprint of the locomotion experimentmore complete description of the algorithmGitHubAcknowledgementsThis research was done in collaboration between Google and UC Berkeley. We would like to thank all the people who were involved, including Sehoon Ha, Kristian Hartikainen, Jie Tan, George Tucker, Vincent Vanhoucke and Aurick Zhou.

Looking Back at Google’s Research Efforts in 2018

Tuesday, January 15, 2019

Posted by Jeff Dean, Senior Fellow and Google AI Lead, on behalf of the entire Google Research Community
publications in 2018Ethical Principles and AIGoogle AI Principlesresponsible AI practicesevaluate our own development of AIAvoid creating or reinforcing unfair biasBe accountable to peopleML fairnessmodel interpretabilityreducing gender biasesexplorationreleaseFairness ModuleMachine Learning Crash CourseAI for Social Goodwork on flood prediction earthquake aftershock predictionTensorFlowusing convolutional neural networks to identify humpback whale callsdetecting new exoplanetsidentifying diseased cassava plants Google AI for Social Impact ChallengeGoogle.orgAssistive TechnologyGoogle DuplexCan you book me a haircut at 4 PM today?Smart ComposeSound SearchNow PlayingSmart Linkify

support more languagesbetter understanding of semantic similarityspeech synthesistext-to-speech for languages without much training data availableQuantum computingquantum supremacya number of exciting new resultsBristlecone

A Bristlecone chip being installed by Research Scientist Marissa Giustina at the Quantum AI Lab in Santa Barbara.

Cirqquantum computers could be used for neural networksunderstanding performance fluctuations in quantum processorsquantum computers might be useful as a computational substrate for neural networksNatural Language Understandingmix of basic research as well as product-focused collaborationsTransformer work from 2017Universal TransformerBERT

BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks.

handle multilingual use cases better, with the goal of makingallPerceptionpetslearn about the natural worldanswer questions in real-timedo more with Lens in Google Imagesnew capabilities in vision and video in Cloud ML APIsface-related on-device building blocks

Google Lens can help you learn more about the world around you. Here, Lens identifies the breed of this dog. Learn more in this blog post.

stereo magnificationa fast bottom-up model for joint pose estimation and person instance segmentationsystem for visualizing complex motionsystemdistillation3D convolutionsunsupervised learning of semantic audio representationsexpressive and human-like speech synthesisLooking to ListenMobileNetV2next-generation mobile computer vision modelMorphNetautomatic generation of mobile network architecturesComputational PhotographyHDR+Motion Photos in Pixel 2Augmented Reality mode in Motion Stills

Motion photos on the Pixel 2 in Google Photos. For more examples, check out this Google Photos album.

Augmented chicken family with Motion Stills AR mode.

Night Sightpressusersusing ML to provide better portrait mode shotsSuper Res ZoomTop ShotGoogle Clips

Left: iPhone XS (full resolution image here). Right: Pixel 3 Night Sight (full resolution image here).

Algorithms and Theoryrouting algorithms behind Google tripsconsistent hashing for Google cloudalgorithms and theorygraph miningour work on studying convergence of stochastic optimization algorithms for training neural networksICLR 2018ADAM

Performance comparison of ADAM and AMSGRAD on a synthetic example of a simple one dimensional convex problem inspired by our examples of non-convergence. The first two plots (left and center) are for the online setting and the the last one (right) is for the stochastic setting.

distributed optimizationvia round compressionvia core-setssubmodular maximizationk-core decompositionset cover at scale via sketchingbalanced partitioning and hierarchical clusteringonline delivery services WWW'18OR-tools platform2018 Minizinc new models reconstruction learning a mixture of multinomial logits learnable by neural networksimprove classiconline algorithmsby iterationby shufflingincentive-aware learning new researchmarket algorithmstest incentive compatibilityoptimizing ad refresh for in-app advertisinglack of prediction of futurenoisy forecastsheterogenous buyer behaviourdynamic double auctionsnew online allocation algorithms for stochastic input with traffic spikesnew bandit algorithms robust to corrupted dataSoftware Systemsdynamic control flow Mesh TensorFlowreleased

The TF-Ranking library supports multi-item scoring architecture, an extension of traditional single-item scoring.

JAXXLACleverHansTensorFlow Privacyplacement of computations onto deviceslearning memory access patternslearned indices

The Hierarchical Planner's placement of a NMT (4-layer) model. White denotes CPU and the four colors each represent one of the GPUs. Note that every step of every layer is allocated across multiple GPUs. This placement is 53.7% faster than that generated by a human expert.

SpectreMeltdownProject Zerotoolthe evolution of Google's Software Defined Networking WANstand-alone, federated query processing platforma report on our extensive use of code reviewdynamicconsistent hashing schemetight provable guaranteesdeployed itGoogle Cloud Pub/Subearlier version of our paperVimeohaproxyalmost 8AutoMLevolutionary algorithms can be used to automatically discover state-of-the-art neural network architecturesautomatically generate image transformation sequences find new symbolic optimization expressions that are more effective than the commonly used optimization update rulesAdaNet showed how to have a fast and flexible AutoML algorithm with learning guarantees

AdaNet adaptively growing an ensemble of neural networks. At each iteration, it measures the ensemble loss for each candidate, and selects the best one to move onto the next iteration.

combining the accuracy of a model with its inference computation time in the reward function for a reinforcement learning architecture searchexplored using ML to learn to automatically compress ML modelsTPUsTensor Processing UnitsBERTvia ColabTensorFlow Research CloudCloud TPUslarge-scale ML training

An individual TPU v3 device (left) and a portion of a TPU v3 Pod (right). TPU v3 is the latest generation of Google's Tensor Processing Unit (TPU) hardware. Available to external customers as Cloud TPU v3, these systems are liquid-cooled for maximum performance (computer chips + liquid = exciting!), and a full TPU v3 Pod can apply more than 100 petaflops of computational power to the world's largest ML problems.

Open Source Software and DatasetsTensorFlowreleased in November 2015TensorFlow LiteTensorFlow.jsTensorFlow ProbabilityContent Delivery Network10,000 stars on Githubintroduced a new framework for flexible and reproducible reinforcement learningnew visualization tools to rapidly understand the characteristics of a dataseta high-level library for expressing machine learning problems that involve learning-to-ranka framework for fast and flexible AutoML solutions with learning guaranteeslibrary for doing in-browser realtime t-SNE visualizationsFHIR tools and software for working with electronic healthcare data

Real-time evolution of the tSNE embedding for the complete MNIST dataset. The dataset contains images of 60,000 handwritten digits. You can find a live demo here.

Google Dataset Searchcurated and released many new, novel datasets

Pictures from India & Singapore added to Open Images Extended using the Crowdsource app.

Open Images V4extended this dataset to add more diversity of people and scenes from all over the worldcrowdsource.google.comAVAhuman actionsspeech in video.announced an updated YouTube-8M, and the 2nd YouTube-8M Large-Scale Video Understanding Challenge and WorkshopHDR+ Burst Photography DatasetGoogle-LandmarksFluid Annotation

Visualization of the fluid annotation interface in action on image from COCO dataset. Image credit: gamene, original image.

Inclusive Images ChallengeiNaturalist 2018 Challenge Kaggle "Quick, Draw!" Doodle Recognition ChallengeConceptual CaptionsRoboticsteach robots to grasp novel objectsto learn about objects without human supervisioncombining ML and sampling-based methodsICRA'18learning robot geometryperceive the structure of the worlddeep reinforcement learning models online on real robotslearn stable approaches

Applications of AI to Other FieldsNature Methodshigh-precision automated reconstruction of neuronsimproves the accuracy of automated interpretation of connectomics data by an order of magnitude

Our algorithm in action as it traces a single neurite in 3d in a songbird brain.

Finding new planets outside our solar system by data mining light curves of stars

Recognizing the origin or function of short DNA sequences

Automatically detecting out-of-focus microscope images

Digitally creating images of the same cells with multiple stains

Automatically mapping mass spectrometry output to peptide chains

A pre-trained TensorFlow model rates focus quality for a montage of microscope image patches of cells in Fiji (ImageJ). Hue and lightness of the borders denote predicted focus quality and prediction uncertainty, respectively.

Healthapplying ML to healthpublished workis on-par with retinal specialistspairing ophthalmologists and this ML model allow them to make more accurate decisionsVerilyAravind Eye HospitalsRajavithi Hospital affiliated with the Ministry of Health in Thailand

On the left is a retinal fundus image graded as having moderate DR ("Mo") by an adjudication panel of ophthalmologists (ground truth). On the top right is an illustration of the predicted scores ("N" = no DR, "Mi" = Mild DR, "Mo" = Moderate DR) from the model. On the bottom right is the set of scores given by physicians without assistance ("Unassisted") and those who saw the model's predictions ("Grades Only").

medical and eye specialists found quite remarkablecan assess cardiovascular risk from retinal imageshow to improve the grading of prostate cancer using MLdetect metastatic breast cancer with deep learningprototype for an augmented-reality microscopeUniversity of Chicago MedicineUCSFStanford Medicinepublished work in Nature Digital Medicineopen sourced softwareFast Healthcare Interoperability ResourcesGitHub repositoryimproved the accuracy, speed and utility of our deep learning-based variant caller, DeepVariantNature Biotechnologyto proactively advance health equityResearch Outreachproviding multi-year Ph.D. fellowshipsannual Google Ph.D. Fellowship Summit Google AI Residencyits third yearglobal officesmachine learningperceptionalgorithms and optimizationlanguage understandinghealthcaremuch moreGoogle Faculty Research Awards programworkshop on AI/ML Research and PracticeAlgorithms & Optimization Workshopmuch of our research is published openlyICLR 2018NAACL 2018ICML 2018CVPR 2018NeurIPS 2018ECCV 2018EMNLP 2018ASPLOSHPCAICSEIEEE Security & PrivacyOSDISIGCOMMNew Places, New Facesannounced our first AI research office in AfricaParisTokyoAmsterdamPrincetonhere