Blog
The latest news from Google AI
Improving Connectomics by an Order of Magnitude
Monday, July 16, 2018
Posted by Viren Jain, Research Scientist and Technical Lead and Michal Januszewski, Software Engineer, Connectomics at Google
The field of
connectomics
aims to comprehensively map the structure of the neuronal networks that are found in the nervous system, in order to better understand how the brain works. This process requires imaging brain tissue in 3D at nanometer resolution (typically using
electron microscopy
), and then analyzing the resulting image data to trace the brain’s
neurites
and identify individual
synaptic
connections. Due to the high resolution of the imaging, even a cubic millimeter of brain tissue can generate over 1,000 terabytes of data! When combined with the fact that the structures in these images can be extraordinarily subtle and complex, the primary bottleneck in brain mapping has been automating the interpretation of these data, rather than acquisition of the data itself.
Today, in collaboration with
colleagues at the Max Planck Institute of Neurobiology
, we published “
High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks
” in
Nature Methods
, which shows how a new type of
recurrent neural network
can improve the accuracy of automated interpretation of connectomics data by an order of magnitude over previous deep learning techniques. An open-access version of this work is also available from
biorXiv
(2017).
3D Image Segmentation with Flood-Filling Networks
Tracing neurites in large-scale electron microscopy data is an example of an
image segmentation
problem. Traditional algorithms have divided the process into at least two steps: finding boundaries between neurites using an
edge detector
or a machine-learning
classifier
, and then grouping together image pixels that are not separated by a boundary using an algorithm like
watershed
or
graph cut
. In 2015, we began experimenting with an alternative approach based on
recurrent neural networks
that unifies these two steps. The algorithm is seeded at a specific pixel location and then iteratively “fills” a region using a recurrent
convolutional neural network
that predicts which pixels are part of the same object as the seed. Since 2015, we have been working to apply this new approach to large-scale connectomics datasets and rigorously quantify its accuracy.
A flood-filling network segmenting an object in 2d. The yellow dot is the center of the current area of focus; the algorithm expands the segmented region (blue) as it iteratively examines more of the overall image.
Measuring Accuracy via Expected Run Length
Working with our partners at the Max Planck Institute, we devised a metric we call “expected run length” (ERL) that measures the following: given a random point within a random neuron in a 3d image of a brain, how far can we trace the neuron before making some kind of mistake? This is an example of a
mean-time-between-failure
metric, except that in this case we measure the amount of space between failures rather than the amount of time. For engineers, the appeal of ERL is that it relates a linear, physical path length to the frequency of individual mistakes that are made by an algorithm, and that it can be computed in a straightforward way. For biologists, the appeal is that a particular numerical value of ERL can be related to
biologically relevant quantities
, such as the average path length of neurons in different parts of the nervous system.
Progress in expected run length (blue line) leading up to the results shared today in Nature Methods. The red line shows progress in the “merge rate,” which measures the frequency with which two separate neurites were erroneously traced as a single object; achieving a very low merge rate is important for enabling efficient strategies for manual identification and correction of the remaining errors in the reconstruction.
Songbird Connectomics
We used ERL to measure our progress on a ground-truth set of neurons within a 1-million cubic micron
zebra finch
song-bird brain imaged by our collaborators using
serial block-face scanning electron microscopy
and found that our approach performed much better than
previous deep learning pipelines
applied to the same dataset.
Our algorithm in action as it traces a single neurite in 3d in a songbird brain.
We segmented every neuron in a small portion of a zebra finch song-bird brain using the new flood-filling network approach, as depicted here:
Reconstruction of a portion of
zebra finch
brain. Colors denote distinct objects in the segmentation that was automatically generated using a flood-filling network. Gold spheres represent synaptic locations automatically identified using a
previously published approach
.
By combining these automated results with a small amount of additional human effort required to fix the remaining errors, our collaborators at the Max Planck Institute are now able to study the songbird connectome to derive new insights into how zebra finch birds sing their song and
test theories related to how they learn their song
.
Next Steps
We will continue to improve connectomics reconstruction technology, with the aim of fully automating synapse-resolution connectomics and contributing to ongoing connectomics projects at the Max Planck Institute and elsewhere. In order to help support the larger research community in developing connectomics techniques, we have also
open-sourced the TensorFlow code
for the flood-filling network approach, along with
WebGL visualization software for 3d datasets
that we developed to help us understand and improve our reconstruction results.
Acknowledgements
We would like to acknowledge core contributions from Tim Blakely, Peter Li, Larry Lindsey, Jeremy Maitin-Shepard, Art Pope and Mike Tyka (Google), as well as Joergen Kornfeld and Winfried Denk (Max Planck Institute).
Accelerated Training and Inference with the Tensorflow Object Detection API
Friday, July 13, 2018
Posted by Jonathan Huang, Research Scientist and Vivek Rathod, Software Engineer, Google AI Perception
Last year
we announced
the
TensorFlow Object Detection API
, and since then we’ve released a number of new features, such as models learned via
Neural Architecture Search
,
instance segmentation support
and models trained on new datasets such as
Open Images
. We have been amazed at how it is being used – from
finding scofflaws on the streets of NYC
to
diagnosing diseases on cassava plants in Tanzania
.
Today, as part of Google’s commitment to democratizing computer vision, and using feedback from the research community on how to make this codebase even more useful, we’re excited to announce a number of additions to our API. Highlights of this release include:
Support for accelerated training of object detection models via
Cloud TPUs
Improving the mobile deployment process by accelerating inference and making it easy to export a model to mobile with the
TensorFlow Lite
format
Several new model architecture definitions including:
RetinaNet (
Lin et al., 2017
)
A
MobileNet
adaptation of RetinaNet
A novel SSD-based architecture called the
Pooling Pyramid Network
(PPN) whose model size is >3x smaller than that of SSD MobileNet v1 with minimal loss in accuracy.
Additionally, we are releasing pre-trained weights for each of the above models based on the
COCO dataset
.
Accelerated Training via Cloud TPUs
Users spend a great deal of time on optimizing hyperparameters and retraining object detection models, therefore having fast turnaround times on experiments is critical. The models released today belong to the
single shot detector
(SSD) class of architectures that are optimized for training on Cloud TPUs. For example, we can now train a ResNet-50 based
RetinaNet
model to achieve 35%
mean Average Precision
(mAP) on the COCO dataset in < 3.5 hrs.
Accelerated Inference via Quantization and TensorFlow Lite
To better support low-latency requirements on mobile and embedded devices, the models we are providing are now natively compatible with
TensorFlow Lite
, which enables on-device machine learning inference with low latency and a small binary size. As part of this, we have implemented: (1)
model quantization
and (2) detection-specific operations natively in TensorFlow Lite. Our
model quantization
follows the strategy outlined in
Jacob et al.
(2018) and the
whitepaper by Krishnamoorthi
(2018) which applies quantization to both model weights and activations at training and inference time, yielding smaller models that run faster.
Quantized detection models are faster and smaller (e.g., a quantized 75% depth-reduced SSD Mobilenet model runs at >15 fps on a Pixel 2 CPU with a 4.2 Mb footprint) with minimal loss in detection accuracy compared to the full floating point model.
Try it Yourself with a New Tutorial!
To get started training your own model on Cloud TPUs,
check out our new tutorial
! This walkthrough will take you through the process of training a quantized pet face detector on Cloud TPU then exporting it to an Android phone for inference via TensorFlow Lite conversion.
We hope that these new additions will help make high-quality computer vision models accessible to anyone wishing to solve an object detection problem, and provide a more seamless user experience, from training a model with quantization to exporting to a TensorFlow Lite model ready for on-device deployment. We would like to thank everyone in the community who have contributed features and bug fixes. As always, contributions to the codebase are welcome, and please stay tuned for more updates!
Acknowledgements
This post reflects the work of the following group of core contributors: Derek Chow, Aakanksha Chowdhery, Jonathan Huang, Pengchong Jin, Zhichao Lu, Vivek Rathod, Ronny Votel and Xiangxin Zhu. We would also like to thank the following colleagues: Vasu Agrawal, Sourabh Bajaj, Chiachen Chou, Tom Jablin, Wenzhe Li, Tsung-Yi Lin, Hernan Moraldo, Kevin Murphy, Sara Robinson, Andrew Selle, Shashi Shekhar, Yash Sonthalia, Zak Stone, Pete Warden and Menglong Zhu.
Automating Drug Discoveries Using Computer Vision
Thursday, July 12, 2018
Vincent Vanhoucke, Principal Scientist, Google Brain Team
“Every time you miss a protein crystal, because they are so rare, you risk missing on an important biomedical discovery.”
-
Patrick Charbonneau
, Duke University Dept. of Chemistry and Lead Researcher,
MARCO initiative
.
Protein crystallization
is a key step to biomedical research concerned with discovering the structure of complex biomolecules. Because that structure determines the molecule’s function, it helps scientists design new drugs that are specifically targeted to that function. However, protein crystals are rare and difficult to find. Hundreds of experiments are typically run for each protein, and while the setup and imaging are mostly automated, finding individual protein crystals remains largely performed through visual inspection and thus prone to human error. Critically, missing these structures can result in lost opportunity for important biomedical discoveries for advancing the state of medicine.
In collaboration with researchers from the
MAchine Recognition of Crystallization Outcomes
(MARCO) initiative, we have published “
Classification of Crystallization Outcomes using Deep Convolutional Neural Networks
” in
PLOS One
(
ArXiv preprint
), in which we discuss how we used some of the most recent architectures of deep convolutional networks and customized them to achieve an accuracy of more than 94% on the visual recognition task of identifying protein crystals. In order to spur further research in this area, we have made the data freely accessible, and
open-sourced our model
as part of the TensorFlow research model repository, and available to researchers as a
Cloud ML Engine
endpoint.
Image of protein crystal, courtesy of the MARCO repository (CC-BY-4.0 license)
The MARCO initiative is a joint project between several pharmaceutical companies and academic research centers to pool and host
a large repository of curated crystallography images
, and make them available to the community to help develop better image analysis tools. When a member of the initiative reached out to Google with a well-defined problem, and half a million labelled images, we embraced the challenge of trying to apply the recent advances in deep learning to the problem.
Due to the large variability between imaging technologies and data acquisition approaches, coming up with a single approach to the visual recognition problem may appear daunting. Crystals can be very small, which makes them rare structures in a large image containing otherwise undifferentiated visual clutter.
Samples from the MARCO repository, illustrating the degree of variability between data sources.
Fortunately, given sufficient training data, modern deep convolutional networks are well suited to handle extreme variability in visual appearance. We modified the basic
Inception V3
model to handle larger images while still being able to be trained quickly. The model achieves a level of precision and recall that makes its use practical in automated assessment pipelines.
This work is a great example of the effectiveness of multi-institutional collaborations aimed at solving problems that require data in amounts and level of diversity that no single collaborator has access to. We invite researchers to take advantage of these resources that are the result of this work and share what they learn. This research was conducted as a personal 20% project by the author. To learn more about this work, please see our paper
here
and read the recent
Duke Research Blog post
.
Google at ICML 2018
Monday, July 9, 2018
Posted by Christian Howard, Editor-in-Chief, Google AI Communications
Machine learning is a key strategic focus at Google, with highly active groups pursuing research in virtually all aspects of the field, including deep learning and more classical algorithms, exploring theory as well as application. We utilize scalable tools and architectures to build machine learning systems that enable us to solve deep scientific and engineering challenges in areas of language, speech, translation, music, visual processing and more.
As a leader in machine learning research, Google is proud to be a Platinum Sponsor of the thirty-fifth
International Conference on Machine Learning
(ICML 2018), a premier annual event supported by the
International Machine Learning Society
taking place this week in Stockholm, Sweden. With over 130 Googlers attending the conference to present publications and host workshops, we look forward to our continued collaboration with the larger ML research community.
If you're attending ICML 2018, we hope you'll visit the Google booth and talk with our researchers to learn more about the exciting work, creativity and fun that goes into solving some of the field's most interesting challenges. Our researchers will also be available to talk about
TensorFlow Hub
, the latest work from the
Magenta project
, a Q&A session on the
Google AI Residenc
y program and much more. You can also learn more about our research being presented at ICML 2018 in the list below (Googlers highlighted in blue).
ICML 2018 Committees
Board Members include:
Andrew McCallum
,
Corinna Cortes
,
Hugo Larochelle
,
William Cohen
Sponsorship Co-Chair:
Ryan Adams
Accepted Publications
Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
Nataly Brukhim,
Amir Globerson
Quickshift++: Provably Good Initializations for Sample-Based Mean Shift
Heinrich Jiang
, Jennifer Jang, Samory Kpotufe
Learning a Mixture of Two Multinomial Logits
Flavio Chierichetti,
Ravi Kumar
,
Andrew Tomkins
Structured Evolution with Compact Architectures for Scalable Policy Optimization
Krzysztof Choromanski
, Mark Rowland,
Vikas Sindhwani
, Richard E Turner, Adrian Weller
Fixing a Broken ELBO
Alexander Alemi
, Ben Poole,
Ian Fischer
,
Joshua Dillon
,
Rif Saurous
,
Kevin Murphy
Hierarchical Long-term Video Prediction without Supervision
Nevan Wichers
, Ruben Villegas,
Dumitru Erhan
,
Honglak Lee
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
John Co-Reyes, Yu Xuan Liu, Abhishek Gupta,
Benjamin Eysenbach
, Pieter Abbeel, Sergey Levine
Well Tempered Lasso
Yuanzhi Li,
Yoram Singer
Programmatically Interpretable Reinforcement Learning
Abhinav Verma, Vijayaraghavan Murali,
Rishabh Singh
, Pushmeet Kohli, Swarat Chaudhuri
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Lechao Xiao
,
Yasaman Bahri
,
Jascha Sohl-Dickstein
,
Samuel Schoenholz
,
Jeffrey Pennington
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
Sanjeev Arora, Nadav Cohen,
Elad Hazan
Scalable Deletion-Robust Submodular Maximization: Data Summarization with Privacy and Fairness Constraints
Ehsan Kazemi,
Morteza Zadimoghaddam
, Amin Karbasi
Data Summarization at Scale: A Two-Stage Submodular Approach
Marko Mitrovic, Ehsan Kazemi,
Morteza Zadimoghaddam
, Amin Karbasi
Machine Theory of Mind
Neil Rabinowitz, Frank Perbet, Francis Song,
Chiyuan Zhang
, S. M. Ali Eslami, Matthew Botvinick
Learning to Optimize Combinatorial Functions
Nir Rosenfeld, Eric Balkanski,
Amir Globerson
, Yaron Singer
Proportional Allocation: Simple, Distributed, and Diverse Matching with High Entropy
Shipra Agarwal,
Morteza Zadimoghaddam
,
Vahab Mirrokni
Path Consistency Learning in Tsallis Entropy Regularized MDPs
Yinlam Chow,
Ofir Nachum
,
Mohammad Ghavamzadeh
Efficient Neural Architecture Search via Parameters Sharing
Hieu Pham, Melody Guan,
Barret Zoph
,
Quoc Le
,
Jeff Dean
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Noam Shazeer
, Mitchell Stern
Learning Memory Access Patterns
Milad Hashemi
,
Kevin Swersky
,
Jamie Smith
, Grant Ayers, Heiner Litz,
Jichuan Chang
, Christos Kozyrakis,
Parthasarathy Ranganathan
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
Bo Dai, Albert Shaw,
Lihong Li
, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le Song
Scalable Bilinear Pi Learning Using State and Action Features
Yichen Chen,
Lihong Li
, Mengdi Wang
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go?
Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Peter Glynn, Yinyu Ye,
Li-Jia Li
, Li Fei-Fei
Shampoo: Preconditioned Stochastic Tensor Optimization
Vineet Gupta
,
Tomer Koren, Yoram Singer
Parallel and Streaming Algorithms for K-Core Decomposition
Hossein Esfandiari,
Silvio Lattanzi
,
Vahab Mirrokni
Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?
Maithra Raghu
,
Alexander Irpan
, Jacob Andreas, Bobby Kleinberg,
Quoc Le
, Jon Kleinberg
Is Generator Conditioning Causally Related to GAN Performance?
Augustus Odena
,
Jacob Buckman
,
Catherine Olsson
,
Tom Brown
,
Christopher Olah
,
Colin Raffel
,
Ian Goodfellow
The Mirage of Action-Dependent Baselines in Reinforcement Learning
George Tucker
,
Surya Bhupatiraju
, Shixiang Gu, Richard E Turner, Zoubin Ghahramani,
Sergey Levine
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Lu Jiang
, Zhengyuan Zhou, Thomas Leung,
Li-Jia Li
,
Li Fei-Fei
Loss Decomposition for Fast Learning in Large Output Spaces
En-Hsu Yen,
Satyen Kale
,
Felix Xinnan Yu
,
Daniel Holtmann-Rice
,
Sanjiv Kumar
, Pradeep Ravikumar
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
Adam Roberts
,
Jesse Engel
,
Colin Raffel
,
Curtis Hawthorne
,
Douglas Eck
Smoothed Action Value Functions for Learning Gaussian Policies
Ofir Nachum
,
Mohammad Norouzi
,
George Tucker
,
Dale Schuurmans
Fast Decoding in Sequence Models Using Discrete Latent Variables
Lukasz Kaiser
,
Samy Bengio
,
Aurko Roy
,
Ashish Vaswani
,
Niki Parmar
,
Jakob Uszkoreit
,
Noam Shazeer
Accelerating Greedy Coordinate Descent Methods
Haihao Lu, Robert Freund,
Vahab Mirrokni
Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions
Shuaiwen Wang, Wenda Zhou, Haihao Lu, Arian Maleki,
Vahab Mirrokni
Image Transformer
Niki Parmar
,
Ashish Vaswani
,
Jakob Uszkoreit
,
Lukasz Kaiser
,
Noam Shazeer
, Alexander Ku,
Dustin Tran
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
RJ Skerry-Ryan
,
Eric Battenberg
,
Ying Xiao
,
Yuxuan Wang
,
Daisy Stanton
,
Joel Shor
,
Ron Weiss
,
Robert Clark
,
Rif Saurous
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Minmin Chen
,
Jeffrey Pennington
,
, Samuel Schoenholz
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Yuxuan Wang
,
Daisy Stanton
,
Yu Zhang
,
RJ Skerry-Ryan
,
Eric Battenberg
,
Joel Shor
,
Ying Xiao
,
Ye Jia
,
Fei Ren
,
Rif Saurous
Constrained Interacting Submodular Groupings
Andrew Cotter
,
Mahdi Milani Fard
,
Seungil You
,
Maya Gupta
, Jeff Bilmes
Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training
Xi Wu
, Uyeong Jang, Jiefeng Chen, Lingjiao Chen, Somesh Jha
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
Been Kim
,
Martin Wattenberg
,
Justin Gilmer
,
Carrie Cai
,
James Wexler
,
Fernanda Viégas
,
Rory Sayres
Online Learning with Abstention
Corinna Cortes
,
Giulia DeSalvo
,
Claudio Gentile
,
Mehryar Mohri
, Scott Yang
Online Linear Quadratic Control
Alon Cohen
,
Avinatan Hasidim
,
Tomer Koren
,
Nevena Lazic
,
Yishay Mansour
,
Kunal Talwar
Competitive Caching with Machine Learned Advice
Thodoris Lykouris,
Sergei Vassilvitskii
Efficient Neural Audio Synthesis
Nal Kalchbrenner,
Erich Elsen
, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, Koray Kavukcuoglu
Gradient Descent with Identity Initialization Efficiently Learns Positive Definite Linear Transformations by Deep Residual Networks
Peter Bartlett, Dave Helmbold,
Phil Long
Understanding and Simplifying One-Shot Architecture Search
Gabriel Bender
,
Pieter-Jan Kindermans
,
Barret Zoph
,
Vijay Vasudevan
,
Quoc Le
Approximation Algorithms for Cascading Prediction Models
Matthew Streeter
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Trieu Trinh
,
Andrew Dai
,
Thang Luong
,
Quoc Le
Self-Imitation Learning
Junhyuk Oh, Yijie Guo, Satinder Singh,
Honglak Lee
Adaptive Sampled Softmax with Kernel Based Sampling
Guy Blanc,
Steffen Rendle
Workshops
2018 Workshop on Human Interpretability in Machine Learning (WHI)
Organizers:
Been Kim
, Kush Varshney, Adrian Weller
Invited Speakers include:
Fernanda Viégas
,
Martin Wattenberg
Exploration in Reinforcement Learning
Organizers:
Ben Eysenbach
,
Surya Bhupatiraju
,
Shane Gu
, Junhyuk Oh,
Vincent Vanhoucke
, Oriol Vinyals, Doina Precup
Theoretical Foundations and Applications of Deep Generative Models
Invited speakers include:
Honglak Lee
Scalable Deep Reinforcement Learning for Robotic Manipulation
Thursday, June 28, 2018
Posted Alex Irpan, Software Engineer, Google Brain Team and Peter Pastor, Senior Roboticist, X
How can robots acquire skills that generalize effectively to diverse, real-world objects and situations? While designing robotic systems that effectively perform repetitive tasks in controlled environments, like building products on an assembly line, is fairly routine, designing robots that can observe their surroundings and decide the best course of action while reacting to unexpected outcomes is exceptionally difficult. However, there are two tools that can help robots acquire such skills from experience:
deep learning
, which is excellent at handling unstructured real-world scenarios, and
reinforcement learning
, which enables longer-term reasoning while exhibiting more complex and robust sequential decision making. Combining these two techniques has the potential to enable robots to learn continuously from their experience, allowing them to master basic sensorimotor skills using data rather than manual engineering.
Designing reinforcement learning algorithms for robot learning introduces its own set of challenges: real-world objects span a wide variety of visual and physical properties, subtle differences in contact forces can make predicting object motion difficult and objects of interest can be obstructed from view. Furthermore, robotic sensors are inherently noisy, adding to the complexity. All of these factors makes it incredibly difficult to learn a general solution, unless there is enough variety in the training data, which takes time to collect. This motivates exploring learning algorithms that can effectively reuse past experience, similar to our
previous work
on grasping which benefited from large datasets. However, this previous work could not reason about the long-term consequences of its actions, which is important for learning how to grasp. For example, if multiple objects are clumped together, pushing one of them apart (called “singulation”) will make the grasp easier, even if doing so does not directly result in a successful grasp.
Examples of singulation.
To be more efficient, we need to use off-policy reinforcement learning, which can learn from data that was collected hours, days, or weeks ago. To design such an off-policy reinforcement learning algorithm that can benefit from large amounts of diverse experience from past interactions, we combined
large-scale distributed optimization
with a new fitted deep
Q-learning
algorithm that we call QT-Opt. A preprint is available on
arXiv
.
QT-Opt is a distributed Q-learning algorithm that supports continuous action spaces, making it well-suited to robotics problems. To use QT-Opt, we first train a model entirely offline, using whatever data we’ve already collected. This doesn’t require running the real robot, making it easier to scale. We then deploy and finetune that model on the real robot, further training it on newly collected data. As we run QT-Opt, we accumulate more offline data, letting us train better models, which lets us collect better data, and so on.
To apply this approach to robotic grasping, we used 7 real-world robots, which ran for 800 total robot hours over the course of 4 months. To bootstrap collection, we started with a hand-designed policy that succeeded 15-30% of the time. Data collection switched to the learned model when it started performing better. The policy takes a camera image and returns how the arm and gripper should move. The offline data contained grasps on over 1000 different objects.
Some of the training objects used.
In the past, we’ve seen
that sharing experience across robots can accelerate learning
. We scaled this training and data gathering process to ten GPUs, seven robots, and many CPUs, allowing us to collect and process a large dataset of over 580,000 grasp attempts. At the end of this process, we successfully trained a grasping policy that runs on a real world robot and generalizes to a diverse set of challenging objects that were not seen at training time.
Seven robots collecting grasp data.
Quantitatively, the QT-Opt approach succeeded in 96% of the grasp attempts across 700 trial grasps on previously unseen objects. Compared to our previous supervised-learning based grasping approach, which had a 78% success rate, our method reduced the error rate by more than a factor of five.
The objects used at evaluation time. To make the task challenging, we aimed for a large variety of object sizes, textures, and shapes.
Notably, the policy exhibits a variety of closed-loop, reactive behaviors that are often not found in standard robotic grasping systems:
When presented with a set of interlocking blocks that cannot be picked up together, the policy separates one of the blocks from the rest before picking it up.
When presented with a difficult-to-grasp object, the policy figures out it should reposition the gripper and regrasp it until it has a firm hold.
When grasping in clutter, the policy probes different objects until the fingers hold one of them firmly, before lifting.
When we perturbed the robot by intentionally swatting the object out of the gripper -- something it had not seen during training -- it automatically repositioned the gripper for another attempt.
Crucially, none of these behaviors were engineered manually. They emerged automatically from self-supervised training with QT-Opt, because they improve the model’s long-term grasp success.
Examples of the learned behaviors. In the left GIF, the policy corrects for the moved ball. In the right GIF, the policy tries several grasps until it succeeds at picking up the tricky object.
Additionally, we’ve found that QT-Opt reaches this higher success rate using less training data, albeit with taking longer to converge. This is especially exciting for robotics, where the bottleneck is usually collecting real robot data, rather than training time. Combining this with other data efficiency techniques (such as our prior work on
domain adaptation
for grasping) could open several interesting avenues in robotics. We’re also interested in combining QT-Opt with
recent work on learning how to self-calibrate
, which could further improve the generality.
Overall, the QT-Opt algorithm is a general reinforcement learning approach that’s giving us good results on real world robots. Besides the reward definition, nothing about QT-Opt is specific to robot grasping. We see this as a strong step towards more general robot learning algorithms, and are excited to see what other robotics tasks we can apply it to. You can learn more about this work in the short video below.
Acknowledgements
This research was conducted by Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. We’d also like to give special thanks to Iñaki Gonzalo and John-Michael Burke for overseeing the robot operations, Chelsea Finn, Timothy Lillicrap, and Arun Nair for valuable discussions, and other people at Google and
X
who’ve contributed their expertise and time towards this research. A preprint is available on
arXiv
.
Self-Supervised Tracking via Video Colorization
Wednesday, June 27, 2018
Posted by Carl Vondrick, Research Scientist, Machine Perception
Tracking objects in video is a fundamental problem in computer vision, essential to applications such as
activity recognition
,
object interaction
, or
video stylization
. However, teaching a machine to visually track objects is challenging partly because it requires large, labeled tracking datasets for training, which are impractical to annotate at scale.
In “
Tracking Emerges by Colorizing Videos
”, we introduce a convolutional network that colorizes grayscale videos, but is constrained to copy colors from a single reference frame. In doing so, the network learns to visually track objects automatically without supervision. Importantly, although the model was never trained explicitly for tracking, it can follow multiple objects, track through occlusions, and remain robust over deformations without requiring
any
labeled training data.
Example tracking predictions on the publicly-available, academic dataset
DAVIS 2017
. After learning to colorize videos, a mechanism for tracking automatically emerges without supervision. We specify regions of interest (indicated by different colors) in the first frame, and our model propagates it forward without any additional learning or supervision.
Learning to Recolorize Video
Our hypothesis is that the temporal coherency of color provides excellent large-scale training data for teaching machines to track regions in video. Clearly, there are exceptions when color is not temporally coherent (such as lights turning on suddenly), but in general color is stable over time. Furthermore, most videos contain color, providing a scalable self-supervised learning signal. We decolor videos, and then add the colorization step because there may be multiple objects with the same color, but by colorizing we can teach machines to track specific objects or regions.
In order to train our system, we use videos from the
Kinetics dataset
, which is a large public collection of videos depicting everyday activities. We convert all video frames except the first frame into gray-scale, and train a convolutional network to predict the original colors in the subsequent frames. We expect the model to learn to follow regions in order to accurately recover the original colors. Our main observation is the need to follow objects for colorization will cause a model for object tracking to be automatically learned.
We illustrate the video recolorization task using video from
the DAVIS 2017 dataset
. The model receives as input one color frame and a gray-scale video, and predicts the colors for the rest of the video. The model learns to copy colors from the reference frame, which enables a mechanism for tracking to be learned without human supervision.
Learning to copy colors from the single reference frame requires the model to learn to internally point to the right region in order to copy the right colors. This forces the model to learn an explicit mechanism that we can use for tracking. To see how the video colorization model works, we show some predicted colorizations from videos in the Kinetics dataset below.
Examples of predicted colors from colorized reference frame applied to input video using the publicly-available
Kinetics dataset
.
Although the network is trained without ground-truth identities, our model learns to track any visual region specified in the first frame of a video. We can track outlined objects or a single point in the video. The only change we make is that, instead of propagating colors throughout the video, we now propagate labels representing the regions of interest.
Analyzing the Tracker
Since the model is trained on large amounts of unlabeled video, we want to gain insight into what the model learns. The videos below show a standard trick to visualize the embeddings learned by our model by projecting them down to three dimensions using
Principal Component Analysis
(PCA) and plotting it as an RGB movie. The results show that nearest neighbors in the learned embedding space tend to correspond to object identity, even over deformations and viewpoint changes.
Top Row: We show videos from the
DAVIS 2017 dataset
. Bottom Row: We visualize the internal embeddings from the colorization model. Similar embeddings will have a similar color in this visualization. This suggests the learned embedding is grouping pixels by object identity.
Tracking Pose
We found the model can also track human poses given key-points in an initial frame. We show results on the publicly-available, academic dataset
JHMDB
where we track a human joint skeleton.
Examples of using the model to track movements of the human skeleton. In this case the input was a human pose for the first frame and subsequent movement is automatically tracked. The model can track human poses even though it was never explicitly trained for this task.
While we do not yet outperform heavily supervised models, the colorization model learns to track video segments and human pose well enough to outperform the
latest methods
based on
optical flow
. Breaking down performance by motion type suggests that our model is a more robust tracker than optical flow for many natural complexities, such as dynamic backgrounds, fast motion, and occlusions. Please see
the paper
for details.
Future Work
Our results show that video colorization provides a signal that can be used for learning to track objects in videos without supervision. Moreover, we found that the failures from our system are correlated with failures to colorize the video, which suggests that further improving the video colorization model can advance progress in self-supervised tracking.
Acknowledgements
This project was only possible thanks to several collaborations at Google. The core team includes Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama and Kevin Murphy. We also thank David Ross, Bryan Seybold, Chen Sun and Rahul Sukthankar.
Teaching Uncalibrated Robots to Visually Self-Adapt
Friday, June 22, 2018
Posted by Fereshteh Sadeghi, Student Researcher, Google Brain Team
People are remarkably proficient at manipulating objects without needing to adjust their viewpoint to a fixed or specific pose. This capability (referred to as
visual motor integration
) is learned during childhood from manipulating objects in various situations, and governed by a self-adaptation and mistake correction mechanism that uses rich sensory cues and vision as feedback. However, this capability is quite difficult for vision-based controllers in robotics, which until now have been built on a rigid setup for reading visual input data from a fixed mounted camera which should not be moved or repositioned at train and test time. The ability to quickly acquire visual motor control skills under large viewpoint variation would have substantial implications for autonomous robotic systems — for example, this capability would be particularly desirable for robots that can help rescue efforts in emergency or disaster zones.
In “
Sim2Real Viewpoint Invariant Visual Servoing by Recurrent Control
” presented at
CVPR 2018
this week, we study a novel deep network architecture (consisting of two
fully convolutional networks
and a
long short-term memory
unit) that learns from a past history of actions and observations to self-calibrate. Using diverse simulated data consisting of demonstrated trajectories and reinforcement learning objectives, our visually-adaptive network is able to control a robotic arm to reach a diverse set of visually-indicated goals, from various viewpoints and independent of camera calibration.
Viewpoint invariant manipulation for visually indicated goal reaching with a physical robotic arm. We learn a single policy that can reach diverse goals from sensory input captured from drastically different camera viewpoints. First row shows the visually indicated goals.
The Challenge
Discovering how the controllable
degrees of freedom
(DoF) affect visual motion can be ambiguous and underspecified from a single image captured from an unknown viewpoint. Identifying the effect of actions on image-space motion and successfully performing the desired task requires a robust perception system augmented with the ability to maintain a memory of past actions. To be able to tackle this challenging problem, we had to address the following essential questions:
How can we make it feasible to provide the right amount of experience for the robot to learn the self-adaptation behavior based on pure visual observations that simulate a lifelong learning paradigm?
How can we design a model that integrates robust perception and self-adaptive control such that it can quickly transfer to unseen environments?
To do so, we devised a new manipulation task where a seven-DoF robot arm is provided with an image of an object and is directed to reach that particular goal amongst a set of distractor objects, while viewpoints change drastically from one trial to another. In doing so, we were able to simulate both the learning of complex behaviors and the transfer to unseen environments.
Visually indicated goal reaching task with a physical robotic arm and diverse camera viewpoints.
Harnessing Simulation to Learn Complex Behaviors
Collecting robot experience data is difficult and time-consuming. In a previous post, we showed how to scale up learning skills by
distributing the data collection and trials to multiple robots
. Although this approach expedited learning, it is still not feasibly extendable to learning complex behaviors such as
visual self-calibration
, where we need to expose robots to a huge space of various viewpoints. Instead, we opt to learn such complex behavior in simulation where we can collect unlimited robot trials and easily move the camera to various random viewpoints. In addition to fast data collection in simulation, we can also surpass hardware limitations requiring the installation of multiple cameras around a robot.
We use domain randomization technique to learn generalizable policies in simulation.
To learn visually robust features to transfer to unseen environments, we used a technique known as domain randomization (a.k.a. simulation randomization) introduced by
Sadeghi & Levine
(2017), that enables robots to learn vision-based policies entirely in simulation such that they can generalize to the real world. This technique was shown to work well for various robotic tasks such as
indoor navigation
,
object localization
,
pick and placing
, etc. In addition, to learn complex behaviors like self-calibration, we harnessed the simulation capabilities to generate synthetic demonstrations and combined
reinforcement learning
objectives to learn a robust controller for the robotic arm.
Viewpoint invariant manipulation for visually indicated goal reaching with a simulated seven-DoF robotic arm. We learn a single policy that can reach diverse goals from sensory input captured from dramatically different camera viewpoints.
Disentangling Perception from Control
To enable fast transfer to unseen environments, we devised a deep neural network that combines perception and control trained end-to-end simultaneously, while also allowing each to be learned independently if needed. This disentanglement between perception and control eases transfer to unseen environments, and makes the model both flexible and efficient in that each of its parts (i.e. 'perception' or 'control') can be independently adapted to new environments with small amounts of data. Additionally, while the control portion of the network was entirely trained by the simulated data, the perception part of our network was complemented by collecting a small amount of static images with object bounding boxes without needing to collect the whole action sequence trajectory with a physical robot. In practice, we fine-tuned the perception part of our network with only 76 object bounding boxes coming from 22 images.
Real-world robot and moving camera setup. First row shows the scene arrangements and the second row shows the visual sensory input to the robot.
Early Results
We tested the visually-adapted version of our network on a physical robot and on real objects with drastically different appearances than the ones used in simulation. Experiments were performed with both one or two objects on a table — “seen objects” (as labeled in the figure below) were used for visual adaptation using small collection of real static images, while “unseen objects” had not been seen during visual adaptation. During the test, the robot arm was directed to reach a visually indicated object from various viewpoints. For the two object experiments the second object was to "fool" the robotic arm. While the simulation-only network has good generalization capability (due to being trained with domain randomization technique), the very small amount of static visual data to visually adapt the controller boosted the performance, due to the flexible architecture of our network.
After adapting the visual features with the small amount of real images, performance was boosted by more than 10%. All used real objects are drastically different from the objects seen in simulation.
We believe that learning online visual self-adaptation is an important and yet challenging problem with the goal of learning generalizable policies for robots that can act in diverse and unstructured real world setup. Our approach can be extended to any sort of automatic self-calibration. See the video below for more information on this work.
Acknowledgements
This research was conducted by Fereshteh Sadeghi, Alexander Toshev, Eric Jang and Sergey Levine. We would also like to thank Erwin Coumans and Yunfei Bai for providing
pybullet
, and Vincent Vanhoucke for insightful discussions.
How Can Neural Network Similarity Help Us Understand Training and Generalization?
Thursday, June 21, 2018
Posted by Maithra Raghu, Google Brain Team and Ari S. Morcos, DeepMind
In order to solve tasks, deep neural networks (DNNs) progressively transform input data into a sequence of complex representations (i.e., patterns of activations across individual neurons). Understanding these representations is critically important, not only for interpretability, but also so that we can more intelligently design machine learning systems. However, understanding these representations has proven quite difficult, especially when comparing representations across networks. In a
previous post
, we outlined the benefits of
Canonical Correlation Analysis
(CCA) as a tool for understanding and comparing the representations of
convolutional neural networks
(CNNs), showing that they converge in a bottom-up pattern, with early layers converging to their final representations before later layers over the course of training.
In “
Insights on Representational Similarity in Neural Networks with Canonical Correlation
” we develop this work further to provide new insights into the representational similarity of CNNs, including differences between networks which memorize (e.g., networks which can only classify images they have seen before) from those which generalize (e.g., networks which can correctly classify previously unseen images). Importantly, we also extend this method to provide insights into the dynamics of
recurrent neural networks
(RNNs), a class of models that are particularly useful for sequential data, such as language. Comparing RNNs is difficult in many of the same ways as CNNs, but RNNs present the additional challenge that their representations change over the course of a sequence. This makes CCA, with its helpful invariances, an ideal tool for studying RNNs in addition to CNNs. As such, we have additionally open sourced the
code used for applying CCA on neural networks
with the hope that will help the research community better understand network dynamics.
Representational Similarity of Memorizing and Generalizing CNNs
Ultimately, a machine learning system is only useful if it can generalize to new situations it has never seen before. Understanding the factors which differentiate between networks that generalize and those that don’t is therefore essential, and may lead to new methods to improve generalization performance. To investigate whether representational similarity is predictive of generalization, we studied two types of CNNs:
generalizing networks
: CNNs trained on data with unmodified, accurate labels and which learn solutions which generalize to novel data.
memorizing networks
: CNNs trained on datasets with randomized labels such that they must memorize the training data and cannot, by definition, generalize (as in
Zhang et al., 2017
).
We trained multiple instances of each network, differing only in the initial randomized values of the network weights and the order of the training data, and used a new weighted approach to calculate the CCA distance measure (see
our paper
for details) to compare the representations within each group of networks and between memorizing and generalizing networks.
We found that groups of
different
generalizing networks consistently converged to more similar representations (especially in later layers) than groups of memorizing networks (see figure below). At the
softmax
, which denotes the network’s ultimate prediction, the CCA distance for each group of generalizing and memorizing networks decreases substantially, as the networks in each separate group make similar predictions.
Groups of generalizing networks (blue) converge to more similar solutions than groups of memorizing networks (red). CCA distance was calculated between groups of networks trained on real CIFAR-10 labels (“Generalizing”) or randomized CIFAR-10 labels (“Memorizing”) and between pairs of memorizing and generalizing networks (“Inter”).
Perhaps most surprisingly, in later hidden layers, the representational distance between any given pair of memorizing networks was about the same as the representational distance between a memorizing and generalizing network (“Inter” in the plot above), despite the fact that these networks were trained on data with entirely different labels. Intuitively, this result suggests that
while there are many different ways to memorize the training data (resulting in greater CCA distances), there are fewer ways to learn generalizable solutions
. In future work, we plan to explore whether this insight can be used to regularize networks to learn more generalizable solutions.
Understanding the Training Dynamics of Recurrent Neural Networks
So far, we have only applied CCA to CNNs trained on image data. However, CCA can also be applied to calculate representational similarity in RNNs, both over the course of training and over the course of a sequence. Applying CCA to RNNs, we first asked whether the RNNs exhibit the same
bottom-up
convergence pattern we observed in our
previous work
for CNNs. To test this, we measured the CCA distance between the representation at each layer of the RNN over the course of training with its final representation at the end of training. We found that the CCA distance for layers closer to the input dropped earlier in training than for deeper layers, demonstrating that, like CNNs, RNNs also converge in a bottom-up pattern (see figure below).
Convergence dynamics for RNNs over the course of training exhibit bottom up convergence, as layers closer to the input converge to their final representations earlier in training than later layers. For example, layer 1 converges to its final representation earlier in training than layer 2 than layer 3 and so on. Epoch designates the number of times the model has seen the entire training set while different colors represent the convergence dynamics of different layers.
Additional findings in our paper show that wider networks (e.g., networks with more neurons at each layer) converge to more similar solutions than narrow networks. We also found that trained networks with identical structures but different learning rates converge to distinct clusters with similar performance, but highly dissimilar representations. We also apply CCA to RNN dynamics over the course of a single sequence, rather than simply over the course of training, providing some initial insights into the various factors which influence RNN representations over time.
Conclusions
These findings reinforce the utility of analyzing and comparing DNN representations in order to provide insights into network function, generalization, and convergence. However, there are still many open questions: in future work, we hope to uncover which aspects of the representation are conserved across networks, both in CNNs and RNNs, and whether these insights can be used to improve network performance. We encourage others to try out the
code
used for the paper to investigate what CCA can tell us about other neural networks!
Acknowledgements
Special thanks to Samy Bengio, who is a co-author on this work. We also thank Martin Wattenberg, Jascha Sohl-Dickstein and Jon Kleinberg for helpful comments.
Labels
accessibility
ACL
ACM
Acoustic Modeling
Adaptive Data Analysis
ads
adsense
adwords
Africa
AI
Algorithms
Android
Android Wear
API
App Engine
App Inventor
April Fools
Art
Audio
Augmented Reality
Australia
Automatic Speech Recognition
Awards
Cantonese
Chemistry
China
Chrome
Cloud Computing
Collaboration
Computational Imaging
Computational Photography
Computer Science
Computer Vision
conference
conferences
Conservation
correlate
Course Builder
crowd-sourcing
CVPR
Data Center
Data Discovery
data science
datasets
Deep Learning
DeepDream
DeepMind
distributed systems
Diversity
Earth Engine
economics
Education
Electronic Commerce and Algorithms
electronics
EMEA
EMNLP
Encryption
entities
Entity Salience
Environment
Europe
Exacycle
Expander
Faculty Institute
Faculty Summit
Flu Trends
Fusion Tables
gamification
Gboard
Gmail
Google Accelerated Science
Google Books
Google Brain
Google Cloud Platform
Google Docs
Google Drive
Google Genomics
Google Maps
Google Photos
Google Play Apps
Google Science Fair
Google Sheets
Google Translate
Google Trips
Google Voice Search
Google+
Government
grants
Graph
Graph Mining
Hardware
HCI
Health
High Dynamic Range Imaging
ICLR
ICML
ICSE
Image Annotation
Image Classification
Image Processing
Inbox
India
Information Retrieval
internationalization
Internet of Things
Interspeech
IPython
Journalism
jsm
jsm2011
K-12
KDD
Keyboard Input
Klingon
Korean
Labs
Linear Optimization
localization
Low-Light Photography
Machine Hearing
Machine Intelligence
Machine Learning
Machine Perception
Machine Translation
Magenta
MapReduce
market algorithms
Market Research
Mixed Reality
ML
MOOC
Moore's Law
Multimodal Learning
NAACL
Natural Language Processing
Natural Language Understanding
Network Management
Networks
Neural Networks
Nexus
Ngram
NIPS
NLP
On-device Learning
open source
operating systems
Optical Character Recognition
optimization
osdi
osdi10
patents
Peer Review
ph.d. fellowship
PhD Fellowship
PhotoScan
Physics
PiLab
Pixel
Policy
Professional Development
Proposals
Public Data Explorer
publication
Publications
Quantum AI
Quantum Computing
renewable energy
Research
Research Awards
resource optimization
Robotics
schema.org
Search
search ads
Security and Privacy
Semantic Models
Semi-supervised Learning
SIGCOMM
SIGMOD
Site Reliability Engineering
Social Networks
Software
Speech
Speech Recognition
statistics
Structured Data
Style Transfer
Supervised Learning
Systems
TensorBoard
TensorFlow
TPU
Translate
trends
TTS
TV
UI
University Relations
UNIX
User Experience
video
Video Analysis
Virtual Reality
Vision Research
Visiting Faculty
Visualization
VLDB
Voice Search
Wiki
wikipedia
WWW
YouTube
Archive
2018
Jul
Jun
May
Apr
Mar
Feb
Jan
2017
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Jul
May
Apr
Mar
Feb
2007
Oct
Sep
Aug
Jul
Jun
Feb
2006
Dec
Nov
Sep
Aug
Jul
Jun
Apr
Mar
Feb
Feed
Google
on
Follow @googleai
Give us feedback in our
Product Forums
.
