Google AI Blog: February 2019

Long-Range Robotic Navigation via Automated Reinforcement Learning

Thursday, February 28, 2019

Aleksandra Faust, Senior Research Scientist and Anthony Francis, Senior Software Engineer, Robotics at Googlegrasp objectsrobot locomotionagentsLearning Navigation Behaviors End-to-End with AutoRLPRM-RL: Long-Range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based PlanningLong-Range Indoor Navigation with PRM-RLlidar Automating Reinforcement Learning (AutoRL) first paperDDPGcatastrophic forgetfulnesslarge-scale hyperparameter optimization

Automating reinforcement learning with reward and neural network architecture search.

AutoRL (red) success over short distances (up to 10 meters) in several unseen buildings. Compared to hand-tuned DDPG (dark-red), artificial potential fields (light blue), dynamic window approach (blue), and behavior cloning (green).

AutoRL local planner policy transfer to robots in real, unstructured environments

Achieving Long Range Navigation with PRM-RLSampling-based plannersprobabilistic roadmapssecond paperBest Paper in Service RoboticsICRA 2018Monte Carlo

Roadmap being built with 3 Monte Carlo simulations per randomly selected node pair.

The largest map was 288 meters by 163 meters and contains almost 700,000 edges, collected over 4 days using 300 workers in a cluster requiring 1.1 billion collision checks.

third paperSimultaneous Localization and Mapping

Navigation over 100 meters success rates in several buildings. First paper -AutoRL local planner only (blue); original PRMs (red); path-guided artificial potential fields (yellow); second paper (green); third paper - PRMs with AutoRL (orange).

On-robot experiments

ConclusionAcknowledgementsThe research was done by, in alphabetical order, Hao-Tien Lewis Chiang, James Davidson, Aleksandra Faust, Marek Fiser, Anthony Francis, Jasmine Hsu, J. Chase Kew, Tsang-Wei Edward Lee, Ken Oslund, Oscar Ramirez from Robotics at Google and Lydia Tapia from University of New Mexico. We thank Alexander Toshev, Brian Ichter, Chris Harris, and Vincent Vanhoucke for helpful discussions.

Learning to Generalize from Sparse and Underspecified Rewards

Friday, February 22, 2019

Posted by Rishabh Agarwal, Google AI Resident and Mohammad Norouzi, Research ScientistReinforcement learningplaying video gamescontinuous controlrobotic learningsparseunderspecifiedlearning agent generalizei.e.,language understandinge.g.,10

In this instruction-following task, the action trajectories a₁, a₂ and a₃ reach the goal, but the sequences a₂ and a₃ do not follow the instructions. This illustrates the issue of underspecified rewards.

sparse underspecifiedreward hackingLearning to Generalize from Sparse and Underspecified RewardsunderspecifiedMeta Reward Learning MeRLMeRLexplorationsemantic parsinglogical formse.g.,SQLweakly-supervisedWhich nation won the most silver medals?a relevant Wikipedia tablei.e.,

WikiTableQuestionsWikiSQLprior work1.2%2.4%s,ground-truthpreviousrewardlearning

Overview of the proposed approach. We employ (1) mode covering exploration to collect a diverse set of successful trajectories in a memory buffer; (2) Meta-learning or Bayesian optimization to learn an auxiliary reward that provides more refined feedback for policy optimization.

Meta Reward Learning (MeRL) generalization performance validation setmeta learning

Schematic illustration of MeRL: The RL agent is trained via the reward signal obtained from the auxiliary reward model while the auxiliary rewards are trained using the generalization error of the agent.

Learning from Sparse RewardssparseOur paperthe two directionsKullback–Leiblerbimodalmode coveringmode seeking

Left: Optimizing mode covering KL. Right: Optimizing mode seeking KL

Conclusionmodelling reward functionscredit-assignment problemAcknowledgements This research was done in collaboration with Chen Liang and Dale Schuurmans. We thank Chelsea Finn and Kelvin Guu for their review of the paper.

On the Path to Cryogenic Control of Quantum Processors

Thursday, February 21, 2019

Posted by Joseph Bardin, Visiting Faculty Researcher and Erik Lucero, Staff Research Scientist and Hardware Lead, Google AI Quantum Teampractical problemsGoogle AI Quantum teamCurrent thresholds suggest Bristleconekelvinwe presentedInternational Solid State Circuits Conference

Google’s first generation cryogenic-CMOS single-qubit controller (center and zoomed on the right) packaged and ready to be deployed inside our cryostat. The controller measures 1mm by 1.6mm.

How to Control 72 Qubitsour lab in Santa Barbara

Research Scientist Amit Vainsencher checking the wiring on Bristlecone in one of Google's flagship cryostats. Blue coaxial cables are connected from custom analog control electronics (server rack on the right) to the quantum processor.

A Cool IdeaUniversity of Massachusetts Professor Joseph Bardinwithin

Schematic of the cryogenic-CMOS qubit controller mounted on the 3 kelvin stage of our dilution refrigerator and connected to a qubit. Our standard qubit control electronics were connected in parallel to enable control and measurement of the qubit as an in-situ check experiment.

Performance at Low TemperatureT1Rabi oscillationssingle qubit gates

Comparison of the qubit coherence time measured using the standard and cryogenic quantum controllers.

Measured Rabi amplitude oscillations using the cryogenic controller. The green and black traces are the probability of measuring the qubits in the 1 and 0 states, respectively.

Next StepsAcknowledgementsThis work was carried out with the support of the Google Visiting Researcher Program while Prof. Bardin, an Associate Professor with the University of Massachusetts Amherst, was on sabbatical with the Google AI Quantum Team. This work would not have been possible without the many contributions of members of the Google AI Quantum team, especially Evan Jeffrey for his integration of the cryo-CMOS controller into the qubit calibration software, Ted White for his on-demand qubit calibrations and Trent Huang for his tireless design rules checks.

Introducing PlaNet: A Deep Planning Network for Reinforcement Learning

Friday, February 15, 2019

Posted by Danijar Hafner, Student Researcher, Google AIartificial agentsreinforcement learningModel-free directlyDeepMind's DQNcontrolrobotsModel-based RLplan ahead, AlphaGounknowndynamicsDeepMindDeep Planning Network (PlaNet)source code

The PlaNet agent learning to solve a variety of continuous control tasks from images in 2000 attempts. Previous agents that do not learn a model of the environment often require 50 times as many attempts to reach comparable performance.

How PlaNet Worksplan over imageslatent dynamics model

Learned Latent Dynamics Model: In a latent dynamics model, the information of the input images is integrated into the hidden states (green) using the encoder network (grey trapezoids). The hidden state is then projected forward in time to predict future images (blue trapezoids) and rewards (blue rectangle).

A Recurrent State Space Model: A latent dynamics model with both deterministic and stochastic components, allowing to predict a variety of possible futures as needed for robust planning, while remembering information over many time steps. Our experiments indicate both components to be crucial for high planning performance.

A Latent Overshooting Objective: We generalize the standard training objective for latent dynamics models to train multi-step predictions, by enforcing consistency between one-step and multi-step predictions in latent space. This yields a fast and effective objective that improves long-term predictions and is compatible with any latent sequence model.

Planning in Latent Space: For planning, we encode past images (gray trapezoid) into the current hidden state (green). From there, we efficiently predict future rewards for multiple action sequences. Note how the expensive image decoder (blue trapezoid) from the previous figure is gone. We then execute the first action of the best sequence found (red box).

world modelsonline research paperPDF versionPlaNet vs. Model-Free Methods

A cartpole swing-up task, with a fixed camera, so the cart can move out of sight. The agent thus must absorb and remember information over multiple frames.

A finger spin task that requires predicting two separate objects, as well as the interactions between them.

A cheetah running task that includes contacts with the ground that are difficult to predict precisely, calling for a model that can predict multiple possible futures.

A cup task, which only provides a sparse reward signal once a ball is caught. This demands accurate predictions far into the future to plan a precise sequence of actions.

A walker task, in which a simulated robot starts off by lying on the ground, and must first learn to stand up and then walk.

PlaNet agents trained on a variety of image-based control tasks. The animation shows the input images as the agent is solving the tasks. The tasks pose different challenges: partial observability, contacts with the ground, sparse rewards for catching a ball, and controlling a challenging bipedal robot.

A3CD4PGDeepMind Control Suite

One Agent for All Tasks

Video predictions of the PlaNet agent trained on multiple tasks. Holdout episodes collected with the trained agent are shown above and open-loop agent hallucinations below. The agent observes the first 5 frames as context to infer the task and state and accurately predicts ahead for 50 steps given a sequence of actions.

Conclusion3D environmentsTPUsAcknowledgementsThis project is a collaboration with Timothy Lillicrap, Ian Fischer, Ruben Villegas, Honglak Lee, David Ha and James Davidson. We further thank everybody who commented on our paper draft and provided feedback at any point throughout the project.

Using Global Localization to Improve Navigation

Monday, February 11, 2019

Posted by Tilman Reinhardt‎, Software Engineer, Google MapsWhere exactly am I, andwhich way is north?" global localizationVisual Positioning ServiceStreet View

Due to limitations with accuracy and orientation, guidance via GPS alone is limited in urban environments. Using VPS, Street View and machine learning, Global Localization can provide better context on where you are relative to where you're going.

Where GPS Falls ShortlocalizationGPS

GPS signals bouncing off facades in an urban environment.

A New Approach to LocalizationexpectEnter Street ViewStreet View

Features matched from multiple images.

Combining Global Localization with Augmented RealityARCore

Local GuidesGoogle Lens

Announcing the Second Workshop and Challenge on Learned Image Compression

Wednesday, February 6, 2019

Posted by Nick Johnston, Software Engineer, Machine Perceptionwe announcedWorkshop and Challenge on Learned Image Compression CVPR 201823 accepted workshop papersthe competitionmean opinion scoreBetter Portable Graphics

An example image from the 2018 test set, comparing the original image to BPG, JPEG and the results from nine competing teams. All the methods are better than JPEG in color reproduction and many of them are comparable to BPG in their ability to create legible text on the sign.

Workshop and Challenge on Learned Image CompressionCVPR 2019PSNRMS-SSIMhuman evaluated rating taskcompression.ccAcknowledgementsThis workshop is being jointly hosted by researchers at Google, Twitter and ETH Zürich. We'd like to thank: George Toderici (Google), Michele Covell (Google), Johannes Ballé (Google), Nick Johnston (Google), Eirikur Agustsson (Google), Wenzhe Shi (Twitter), Lucas Theis (Twitter), Radu Timofte (ETH Zürich), Fabian Mentzer (ETH Zürich) for their contributions.

Real-time Continuous Transcription with Live Transcribe

Monday, February 4, 2019

Posted by Sagar Savla, Product Manager, Machine PerceptionWorld Health Organizationestimatesautomatic speech recognitionGoogle's ASRautomated captions in Youtubepresentations in Slidesmultiple improvementsCARTPalantypistSTTRLive TranscribeGoogle CloudBuilding Live Transcribeour previous workAudioSetpublished VGGish modelUser ExperienceGallaudet Universityeven small projectors

Displaying confidence level of the transcription. Yellow is high confidence, green is medium and blue is low confidence. White is fresh text awaiting context before finalizing. On the left, the coloring is at a per-phrase level while on the right is at a per-word level.¹ Research found them to be distracting to the user without providing conversational value.

UX research in this spacecocktail party problemrelative

The loudness and noise indicator is made of two concentric circles. The inner brighter circle, indicating the noise floor, tells a deaf user how audibly noisy the current environment is. The outer circle shows how well the speaker’s voice is received.Together, the circles visually show the relative difference intuitively.

Future Workon-device recognitionspeaker-separationspeech enhancementThe KeywordAcknowledgementsLive Transcribe was made by researchers Chet Gnegy, Dimitri Kanevsky, and Justin S. Paul in collaboration with Android Accessibility team members Brian Kemler, Thomas Lin, Alex Huang, Jacqueline Huang, Ben Chung, Richard Chang, I-ting Huang, Jessie Lin, Ausmus Chang, Weiwei Wei, Melissa Barnhart and Bingying Xia. We'd also like to thank our close partners from Gallaudet University, Christian Vogler, Norman Williams and Paula Tucker.
1 Eagle-eyed readers can see the phrase level confidence mode in use by Dr. Obeidat in the video above.^↩

Blog