Blog

The latest from Google Research

A Structured Approach to Unsupervised Depth Learning from Monocular Videos

Tuesday, November 27, 2018

Posted by Anelia Angelova, Research Scientist, Robotics at GoogleLIDARego-motionZhou et al.prior researchDepth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular VideosAAAI 2019open sourced the code in TensorFlow

Previous work (middle row) has not been able to correctly estimate depth of moving objects mapping them to infinity (dark blue regions in the heatmap). Our approach (right) provides much better depth estimates.

Structureneural network to learn depth directlyKITTICityscapessolve

Previous work with monocular inputs were not able to extract moving objects and incorrectly map them to infinity.

Example depth results for a dynamic scene together with estimates of the motion vectors of the individual objects (rotation angles are estimated too, but for simplicity are not shown).

Depth prediction from monocular video input on the KITTI dataset, middle row, compared to ground truth depth from a Lidar sensor; the latter does not cover the full scene and has missing and noisy values. Ground truth depth is not used during training.

Depth prediction on the Cityscapes dataset. Left to right: image, baseline, our method and ground truth provided by stereo. Note the missing values in the stereo ground truth. Also note that our algorithm is able to achieve these results without any ground truth depth supervision.

Ego-motion

Depth and ego-motion prediction. Follow the speed and the turning angle indicator to see the estimates when the car is taking a turn or stopping for a red light.

Transfer Across Domains

Online refinement when training on the Cityscapes Data and testing on KITTI. The images show depth prediction of the trained model, and of the trained model with online refinement. Depth prediction with online refinement better outlines the objects in the scene.

Fetch robot

Results of online adaptation when transferring the learning model from Cityscapes (an outdoors dataset collected from a moving car) to a dataset collected indoors by the Fetch robot. The bottom row shows improved depth after applying online refinement.

AcknowledgementsThis research was conducted by Vincent Casser, Soeren Pirk, Reza Mahjourian and Anelia Angelova. We would like to thank Ayzaan Wahid for his help with data collection and Martin Wicke and Vincent Vanhoucke for their support and encouragement.