Blog

The latest from Google Research

DADS: Unsupervised Reinforcement Learning for Skill Discovery

Friday, May 29, 2020

Posted by Archit Sharma, AI Resident, Google Researchsupervisedreinforcement learninggrasping arbitrary objectslearning agile locomotionsensorsmanual-labelling of “goal” statesunsupervised learningintrinsiccuriosityresearchhas recentlyfocusedDynamics-Aware Unsupervised Discovery of SkillsEmergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

The behavior on the left is random and unpredictable, while the behavior on the right demonstrates systematic motion with predictable changes in the environment. Our goal is to learn potentially useful behaviors such as those on the right, without engineered reward functions.

Overview of DADSencouraging diversity) (predictability)skill-dynamics network

An overview of DADS.

Humanoid

Ant discovers galloping (top left) and skipping (bottom left), Humanoid discovers different locomotive gaits (middle, sped up 2x), and D’Claw from ROBEL (right) discovers different ways to rotate an object, all using DADS. More sample videos are available here.

Model-Based Control Using Skill-Dynamicswithout executing it in the environmentmodel-based planning

Using the skills discovered by the agents, we can traverse an arbitrary sequence of checkpoints without any additional training. The plot on the right follows the agent’s traversal from one checkpoint to another.

Real-World ResultsEmergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement LearningD’KittyROBELemergence of complex behaviors

Using off-DADS, we train D’Kitty from ROBEL to acquire diverse locomotion behaviors, which can then be used for goal-navigation through model-based control.

Future WorkAcknowledgementsWe would like to thank our coauthors, Michael Ahn, Sergey Levine, Vikash Kumar, Shixiang Gu and Karol Hausman. We would also like to acknowledge the support and feedback provided by various members of the Google Brain team and the Robotics at Google team.