Blog

The latest news from Google AI

Learning to Assemble and to Generalize from Self-Supervised Disassembly

Thursday, October 31, 2019

Posted by Kevin Zakka, Research Intern and Andy Zeng, Research Scientist, Robotics at GoogleForm2Fit

Form2Fit learns to assemble a wide variety of kits by finding geometric correspondences between object surfaces and their target placement locations. By leveraging geometric information learned from multiple kits during training, the system generalizes to new objects and kits.

Data-Driven Shape Descriptors For Generalizable Assemblyorientation-sensitive geometric

Overview of Form2Fit. The suction and place networks infer candidate picking and placing locations in the scene respectively. The matching network generates pixel-wise orientation-sensitive descriptors to match picking locations to their corresponding placing locations. The planner then integrates it all to control the robot to execute the next best pick and place action.

Learning Assembly from Disassembly

An example of self-supervision through time-reversal: rewinding a disassembly sequence of a deodorant kit over time generates a valid assembly sequence.

time-reversed disassemblyGeneralization Results

Form2Fit policies are robust to a wide range of rotations and translations of the kits.

Form2Fit policies can generalize to novel kit configurations such as multiple versions of the same kit and mixtures of different kits.

Form2Fit policies can generalize to never-before-seen single and multi-object kits.

What Have the Descriptors Learned?t-SNE

The t-SNE embedding of the learned object descriptors. Similarly oriented objects of the same category display identical colors (e.g. A, B or F, G) while different objects (e.g. C, H) and same objects but different orientation (e.g. A, C, D or H, F) exhibit different colors.

Limitations & Future WorkGitHub repositoryAcknowledgmentsThis research was done by Kevin Zakka, Andy Zeng, Johnny Lee, and Shuran Song (faculty at Columbia University), with special thanks to Nick Hynes, Alex Nichol, and Ivan Krasin for fruitful technical discussions; Adrian Wong, Brandon Hurd, Julian Salazar, and Sean Snyder for hardware support; Ryan Hickman for valuable managerial support; and Chad Richards for helpful feedback on writing.