Blog

The latest from Google Research

Exploring Evolutionary Meta-Learning in Robotics

Tuesday, April 21, 2020

Posted by Xingyou (Richard) Song, Software Engineer and Yuxiang Yang, AI Resident, Robotics at Googlesimulator engines imitation learningoffline reinforcement learningdomain randomizationspecificsimulationreal worldRapidly Adaptable Legged Robots via Evolutionary Meta-Learningevolutionary strategiespolicy gradients

Our algorithm quickly adapts a legged robot’s policy to dynamics changes. In this example, the battery voltage dropped from 16.8V to 10V which reduced motor power, and a 500g mass was also placed on the robot's side, causing it to turn rather than walk straight. The policy is able to adapt in only 50 episodes (or 150s of real-world data).

Meta-Learningmeta-learningmostfine-tunemodel-agnostic meta-learningpolicy gradientstochastichighly stochastic

Decreasing the policy’s stochasticity may be crucial, as otherwise the high-noise problem might be exacerbated by the additional randomness from the policy’s actions.

However, increasing the policy’s stochasticity may also benefit exploration, as the policy needs to use random actions to probe the type of environment to which it adapts.

beforeEvolutionary Strategies in RoboticsES-MAMLblack-box,deterministicparameterefficient compact policiesIt also worksguided ESAdaptation in Simulation

The meta-policy’s gait, which experiences issues when facing a difficult dynamics task. Left: The meta-policy lets the robot fall down. Center: The adapted policy ensures the robot continues to walk correctly. Right: Comparative measurement of the robot’s height.

The meta-policy’s gait, under changes to the robot’s leg settings. Left: The meta-policy allows the robot veer to the right. Center: The adapted policy ensures the robot continues to walk in a straight line. Right: Comparative measurement of the robot’s walking direction.

Adaptation in the Real Worldbatch hill-climbingprevious workhill-climbingdeterministicbatch

In the mass-voltage task (left), a 500g weight is placed on the robot’s side and the voltage is dropped to 10.0V from 16.8V. In the friction task (right), we replaced the rubber feet with tennis balls, to significantly reduce friction and hinder walking.

only simulation

Qualitative changes during the adaptation phase under the mass-voltage task.

PG-MAML

Comparisons between Domain Randomization and PG-MAML, and metric differences between our method’s meta-policy and adapted policy. Top: Comparison for the mass-voltage task. Our method stabilizes the robot’s roll angle. Bottom: Comparison for the friction task. Our method results in longer trajectories.

Future WorkAcknowledgementsThis research was conducted by the core ES-MAML team: Xingyou Song, Yuxiang Yang, Krzysztof Choromanski, Ken Caluwaerts, Wenbo Gao, Chelsea Finn, and Jie Tan. We would like to give special thanks to Vikas Sindhwani for his support on ES methods, and Daniel Seita for feedback on our paper.