Blog

The latest from Google Research

Agile and Intelligent Locomotion via Deep Reinforcement Learning

Wednesday, May 6, 2020

Posted by Yuxiang Yang and Deepali Jain, AI Residents, Robotics at Googleusing off-policy dataimitating animal behaviorsperforming meta learningData Efficient Reinforcement Learning for Legged RobotsHierarchical Reinforcement Learning for Quadruped LocomotionData Efficient Reinforcement Learning for Legged RobotsSoft Actor-Criticmore than an hour of datadifficult to collectwe presentmodel predictive control

Overview of the model-based learning pipeline. The system alternates between fitting the dynamics model and collecting trajectories using model predictive control (MPC).

We separate action planning and execution on different threads.

multi-step losstrajectory generatorgeneralizable

The robot learns to walk using only 4.5 minutes of data.

The robot learns to backflip and walk with rear legs using the same framework.

Combining low-level controller with high-level planningour second paper

Framework of Hierarchical Policy: The policy gets observations from the robot and sends motor commands to execute desired actions. It is split into two levels (high and low). The high-level policy gives a latent command to the low-level policy and also decides the duration for which low-level will run.

PPOSACaugmented random search

Successful trajectory of a robot on a curved path. Left: A plot of the trajectory traversed by the robot with dots along the trajectory marking the positions where the high-level policy sent a new latent command to the low-level policy. Middle: The robot walking along the path in the simulated environment. Right: The robot walking around the path in the real world.

Left: Visualization of a learned 2D latent command space. Vector directions correspond to the movement direction of the robot. Vector length is proportional to the distance covered. Right: Transfer of low level policy: An HRL policy was trained on a single path (right, top). The learned low-level policy was then reused when training the high-level policy on other paths (e.g., right, bottom).

ConclusionAcknowledgementsBoth Deepali Jain and Yuxiang Yang are residents in the AI Residency program, mentored by Ken Caluwaerts and Atil Iscen. We would also like to thank Jie Tan and Vikas Sindhwani for support of the research, and Noah Broestl for managing the New York AI Residency Program.