Blog

The latest news from Google AI

Learning to Predict Depth on the Pixel 3 Phones

Thursday, November 29, 2018

Posted by Rahul Garg, Research Scientist and Neal Wadhwa, Software Engineerdual-pixel autofocustraditional non-learnedstereo algorithm

Left: The original HDR+ image. Right: A comparison of Portrait Mode results using depth from traditional stereo and depth from machine learning. The learned depth result has fewer errors. Notably, in the traditional stereo result, many of the horizontal lines behind the man are incorrectly estimated to be at the same depth as the man and are kept sharp.
(Mike Milne)

A Short Recaplast year’s blog postparallax

The two PDAF images on the left and center look very similar, but in the crop on the right you can see the parallax between them. It is most noticeable on the circular structure in the middle of the crop.

the aperture problemImproving Depth Estimationdefocussemanticconvolutional neural networkTensorFlow

Our convolutional neural network takes as input the PDAF images and outputs a depth map. The network uses an encoder-decoder style architecture with skip connections and residual blocks.

Training the Neural Networkstructure from motionmulti-view stereo

Left: Custom rig used to collect training data. Middle: An example capture flipping between the five images. Synchronization between the cameras ensures that we can calculate depth for dynamic scenes, such as this one. Right: Ground truth depth. Low confidence points, i.e., points where stereo matches are not reliable due to weak texture, are colored in black and are not used during training. (Sam Ansari and Mike Milne)

Five viewpoints ensure that there is parallax in multiple directions and hence no aperture problem.

The arrangement of the cameras ensures that a point in an image is usually visible in at least one other image resulting in fewer points with no correspondences.

The baseline, i.e., the distance between the cameras is much larger than our PDAF baseline resulting in more accurate depth estimation.

Synchronization between the cameras ensure that we can calculate depth for dynamic scenes like the one above.

Portability of the rig ensures that we can capture photos in the wild simulating the photos users take with their smartphones.

Putting it All TogetherTensorFlow LiteTry it YourselfGoogle Photos depth editorthirdpartydepthextractorsalbumAcknowledgmentsThis work wouldn’t have been possible without Sam Ansari, Yael Pritch Knaan, David Jacobs, Jiawen Chen, Juhyun Lee and Andrei Kulik. Special thanks to Mike Milne and Andy Radin who captured data with the five-camera rig.