Blog

The latest from Google Research

Grasp2Vec: Learning Object Representations from Self-Supervised Grasping

Tuesday, December 11, 2018

Posted by Eric Jang, Software Engineer, Robotics at Google and Coline Devin, Berkeley PhD Student and former Research Interncognitive developmental researchself-supervision:object permanenceGrasp2Vec

our prior collaboration with X RoboticsConstructing a Perceptual Reward Functionreinforcement learningdiverse grasping skills from scratchbutton that supplies rewards directly to a robot when it is pushedperceptual understandingDo these objects match?

On the left, the gripper is holding the brush and there are some objects (yellow cup, blue plastic block) in the background. On the right, the gripper is holding the yellow cup and the brush is in the background. If the left image was the desired outcome, a good reward function should “understand” that the two images above correspond to different objects.

can be compressed into a low-dimensional spaceframes in a video can be predicted from previous framesphysically

Left: Objects before the grasp. Center: Objects after the grasp. Right: The Grasped object.

objects_before_grasp - objects_after_grasp = grasped_object

spatial feature mapN-Pairs objective

Add caption

1. Object Similaritycosine distance

2. Localizing Target Objects

Using Grasp2Vec embeddings to localize objects in a scene. The image on the top left shows the objects in the bin. On the bottom left is the query object we wish to grasp. By taking the dot product of the query object vector with the spatial features of the scene image, we get a per-pixel “activation map” (top right image) of how similar that region of the image is to the query. This response map can be used to approach the object for grasping.

Conclusionour papergraspingpushingmanipulatingAcknowledgementsThis research was conducted by Eric Jang, Coline Devin, Vincent Vanhoucke, and Sergey Levine. We’d like to thank Adrian Li, Alex Irpan, Anthony Brohan, Chelsea Finn, Christian Howard, Corey Lynch, Dmitry Kalashnikov, Ian Wilkes, Ivonne Fajardo, Julian Ibarz, Ming Zhao, Peter Pastor, Pierre Sermanet, Stephen James, Tsung-Yi Lin, Yunfei Bai, and many others at Google, X, and the broader robotics community who contributed to improving this work.