Blog

The latest from Google Research

Teaching Uncalibrated Robots to Visually Self-Adapt

Friday, June 22, 2018

Posted by Fereshteh Sadeghi, Student Researcher, Google Brain Teamvisual motor integrationSim2Real Viewpoint Invariant Visual Servoing by Recurrent ControlCVPR 2018fully convolutional networkslong short-term memory

Viewpoint invariant manipulation for visually indicated goal reaching with a physical robotic arm. We learn a single policy that can reach diverse goals from sensory input captured from drastically different camera viewpoints. First row shows the visually indicated goals.
The Challengedegrees of freedom

How can we make it feasible to provide the right amount of experience for the robot to learn the self-adaptation behavior based on pure visual observations that simulate a lifelong learning paradigm?

How can we design a model that integrates robust perception and self-adaptive control such that it can quickly transfer to unseen environments?

Visually indicated goal reaching task with a physical robotic arm and diverse camera viewpoints.

Harnessing Simulation to Learn Complex Behaviorsdistributing the data collection and trials to multiple robotsvisual self-calibration

We use domain randomization technique to learn generalizable policies in simulation.

Sadeghi & Levineindoor navigationobject localizationpick and placingreinforcement learning

Viewpoint invariant manipulation for visually indicated goal reaching with a simulated seven-DoF robotic arm. We learn a single policy that can reach diverse goals from sensory input captured from dramatically different camera viewpoints.
Disentangling Perception from Control

Real-world robot and moving camera setup. First row shows the scene arrangements and the second row shows the visual sensory input to the robot.

Early Results

After adapting the visual features with the small amount of real images, performance was boosted by more than 10%. All used real objects are drastically different from the objects seen in simulation.

AcknowledgementsThis research was conducted by Fereshteh Sadeghi, Alexander Toshev, Eric Jang and Sergey Levine. We would also like to thank Erwin Coumans and Yunfei Bai for providing pybullet, and Vincent Vanhoucke for insightful discussions.