Google AI Blog: November 2018

Highlights from the 2018 Google PhD Fellowship Summit

Friday, November 30, 2018

Posted by Susie Kim, Program Manager, University RelationsPhD Fellowship ProgramAustraliaChina and East AsiaIndiaNorth America, Europe, the Middle EastAfricaSummit HighlightsMaggie JohnsonMaya GuptaAndrew TomkinsRahul SukthankarSai Teja PeddintiAmin VahdatMartin StumpeEd ChiCiera JaspanJeff Dean

Starting clockwise from top left: Researchers Rahul Sukthankar and Ed Chi talking with Fellow attendees; Jeff Dean delivering the closing talk; Poster session in full swing.

Google Fellows attending the 2018 PhD Fellowship Summit.

The Complete List of 2018 Google PhD FellowsAlgorithms, Optimizations and MarketsMassachusetts Institute of TechnologyETH ZurichChennai Mathematical InstituteCornell UniversityDuke UniversityComputational NeuroscienceColumbia UniversityUniversity of California - BerkeleyNew York UniversityHuman Computer InteractionCarnegie Mellon UniversityUniversity of WaterlooGeorgia Institute of TechnologyThe University of MelbourneAustralian National UniversityThe University of New South WalesMachine LearningIndian Institute of Technology - BombayStanford UniversityUniversity of TokyoYale UniversityUniversity of Southern CaliforniaHebrew UniversityChinese University of Hong KongUniversity of California - San DiegoUniversity of California - BerkeleyUniversity of the WitwatersrandUniversity of MichiganMassachusetts Institute of TechnologyMachine Perception, Speech Technology and Computer VisionINRIAUniversity of OxfordIndian Institute of Technology - MadrasUniversity of WashingtonUniversity of Illinois - Urbana-ChampaignGeorgia Institute of TechnologyMobile ComputingHong Kong University of Science & TechnologyEgypt-Japan University of Science and TechnologyUniversity of California - San DiegoNatural Language ProcessingUniversity of PennsylvaniaShanghai Jiao Tong UniversityTel Aviv UniversityTechnion - Israel Institute of TechnologyIndian Institute of Science - BangaloreUniversity of EdinburghHarvard UniversityShanghai Jiao Tong UniversityUniversité Internationale de RabatPrivacy and SecurityUniversity of California - Los AngelesProgramming Technology and Software EngineeringPurdue UniversityUniversity of Dar es SalaamUniversity of California - IrvineFondation Sciences Mathématiques de ParisQuantum ComputingCalifornia Institute of TechnologyStructured Data and Database ManagementUniversity of Wisconsin - MadisonSystems and NetworkingETH ZurichTsinghua UniversityEPFLUniversity of CambridgeUniversity of Massachusetts - AmherstThe University of Sydney

Learning to Predict Depth on the Pixel 3 Phones

Thursday, November 29, 2018

Posted by Rahul Garg, Research Scientist and Neal Wadhwa, Software Engineerdual-pixel autofocustraditional non-learnedstereo algorithm

Left: The original HDR+ image. Right: A comparison of Portrait Mode results using depth from traditional stereo and depth from machine learning. The learned depth result has fewer errors. Notably, in the traditional stereo result, many of the horizontal lines behind the man are incorrectly estimated to be at the same depth as the man and are kept sharp.
(Mike Milne)

A Short Recaplast year’s blog postparallax

The two PDAF images on the left and center look very similar, but in the crop on the right you can see the parallax between them. It is most noticeable on the circular structure in the middle of the crop.

the aperture problemImproving Depth Estimationdefocussemanticconvolutional neural networkTensorFlow

Our convolutional neural network takes as input the PDAF images and outputs a depth map. The network uses an encoder-decoder style architecture with skip connections and residual blocks.

Training the Neural Networkstructure from motionmulti-view stereo

Left: Custom rig used to collect training data. Middle: An example capture flipping between the five images. Synchronization between the cameras ensures that we can calculate depth for dynamic scenes, such as this one. Right: Ground truth depth. Low confidence points, i.e., points where stereo matches are not reliable due to weak texture, are colored in black and are not used during training. (Sam Ansari and Mike Milne)

Five viewpoints ensure that there is parallax in multiple directions and hence no aperture problem.

The arrangement of the cameras ensures that a point in an image is usually visible in at least one other image resulting in fewer points with no correspondences.

The baseline, i.e., the distance between the cameras is much larger than our PDAF baseline resulting in more accurate depth estimation.

Synchronization between the cameras ensure that we can calculate depth for dynamic scenes like the one above.

Portability of the rig ensures that we can capture photos in the wild simulating the photos users take with their smartphones.

Putting it All TogetherTensorFlow LiteTry it YourselfGoogle Photos depth editorthirdpartydepthextractorsalbumAcknowledgmentsThis work wouldn’t have been possible without Sam Ansari, Yael Pritch Knaan, David Jacobs, Jiawen Chen, Juhyun Lee and Andrei Kulik. Special thanks to Mike Milne and Andy Radin who captured data with the five-camera rig.

A Structured Approach to Unsupervised Depth Learning from Monocular Videos

Tuesday, November 27, 2018

Posted by Anelia Angelova, Research Scientist, Robotics at GoogleLIDARego-motionZhou et al.prior researchDepth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular VideosAAAI 2019open sourced the code in TensorFlow

Previous work (middle row) has not been able to correctly estimate depth of moving objects mapping them to infinity (dark blue regions in the heatmap). Our approach (right) provides much better depth estimates.

Structureneural network to learn depth directlyKITTICityscapessolve

Previous work with monocular inputs were not able to extract moving objects and incorrectly map them to infinity.

Example depth results for a dynamic scene together with estimates of the motion vectors of the individual objects (rotation angles are estimated too, but for simplicity are not shown).

Depth prediction from monocular video input on the KITTI dataset, middle row, compared to ground truth depth from a Lidar sensor; the latter does not cover the full scene and has missing and noisy values. Ground truth depth is not used during training.

Depth prediction on the Cityscapes dataset. Left to right: image, baseline, our method and ground truth provided by stereo. Note the missing values in the stereo ground truth. Also note that our algorithm is able to achieve these results without any ground truth depth supervision.

Ego-motion

Depth and ego-motion prediction. Follow the speed and the turning angle indicator to see the estimates when the car is taking a turn or stopping for a red light.

Transfer Across Domains

Online refinement when training on the Cityscapes Data and testing on KITTI. The images show depth prediction of the trained model, and of the trained model with online refinement. Depth prediction with online refinement better outlines the objects in the scene.

Fetch robot

Results of online adaptation when transferring the learning model from Cityscapes (an outdoors dataset collected from a moving car) to a dataset collected indoors by the Fetch robot. The bottom row shows improved depth after applying online refinement.

AcknowledgementsThis research was conducted by Vincent Casser, Soeren Pirk, Reza Mahjourian and Anelia Angelova. We would like to thank Ayzaan Wahid for his help with data collection and Martin Wicke and Vincent Vanhoucke for their support and encouragement.

Improved Grading of Prostate Cancer Using Deep Learning

Friday, November 16, 2018

Posted by Martin Stumpe, Technical Lead and Craig Mermel, Product Manager, Healthcare, Google AI(Updated Jun 7, 2019: The work described in this blogpost has been published in Nature Partner Journal (npj) Digital Medicine.)1 in 9 menmost common cancer in malesprostatectomyrisk stratificationGleason gradedeep learningGoogleothersDevelopment and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Canceroverall Gleason grade group

Visual examples of Gleason patterns, which are used in the Gleason system for grading prostate cancer. Individual cancer patches are assigned a Gleason pattern based on how closely the cancer resembles normal prostate tissue, with lower numbers corresponding to more well differentiated tumors. Image Source: National Institutes of Health.

needle core biopsiesgenitourinary

Comparison of scoring performance of the DLS with pathologists. a: Accuracy of the DLS (in red) compared with the mean accuracy among a cohort-of-29 pathologists (in green). Error bars indicate 95% confidence intervals. b: Comparison of risk stratification provided by the DLS, the cohort-of-29 pathologists, and the genitourinary specialist pathologists. Patients are divided into low and high risk groups based on their Gleason grade group, where a larger separation between the Kaplan-Meier curves of these risk groups indicates better stratification.

Assessing the region-level classification of the DLS. a: Annotations from 3 pathologists compared to DLS predictions. The pathologists show general concordance on the location and the extent of tumor areas, but poor agreement in classifying Gleason patterns. The DLS’s precision Gleason pattern for each region is represented by interpolating between the DLS’s prediction patterns for Gleason patterns 3 (green), 4 (yellow), and 5 (red). b: DLS prediction patterns compared to the distribution of pathologists’ Gleason pattern classifications on 41 million annotated image patches from the test dataset. On patches where pathologists are discordant, where the tissue is more likely to be on the cusp of two patterns, the DLS reflects this ambiguity in its prediction scores.

AcknowledgementsThis work involved the efforts of a multidisciplinary team of software engineers, researchers, clinicians and logistics support staff. Key contributors to this project include Kunal Nagpal, Davis Foote, Yun Liu, Po-Hsuan (Cameron) Chen, Ellery Wulczyn, Fraser Tan, Niels Olson, Jenny L. Smith, Arash Mohtashamian, James H. Wren, Greg S. Corrado, Robert MacDonald, Lily H. Peng, Mahul B. Amin, Andrew J. Evans, Ankur R. Sangoi, Craig H. Mermel, Jason D. Hipp and Martin C. Stumpe. We would also like to thank Tim Hesterberg, Michael Howell, David Miller, Alvin Rajkomar, Benny Ayalew, Robert Nagle, Melissa Moran, Krishna Gadepalli, Aleksey Boyko, and Christopher Gammage. Lastly, this work would not have been possible without the aid of the pathologists who annotated data for this study.References

Interobserver Variability in Histologic Evaluation of Radical Prostatectomy Between Central and Local Pathologists: Findings of TAX 3501 Multinational Clinical Trial, Netto, G. J., Eisenberger, M., Epstein, J. I. & TAX 3501 Trial Investigators, Urology 77, 1155–1160 (2011).

Phase 3 Study of Adjuvant Radiotherapy Versus Wait and See in pT3 Prostate Cancer: Impact of Pathology Review on Analysis, Bottke, D., Golz, R., Störkel, S., Hinke, A., Siegmann, A., Hertle, L., Miller, K., Hinkelbein, W., Wiegel, T., Eur. Urol. 64, 193–198 (2013).

Utility of Quantitative Gleason Grading in Prostate Biopsies and Prostatectomy Specimens, Sauter, G. Steurer, S., Clauditz, T. S., Krech, T., Wittmer, C., Lutz, F., Lennartz, M., Janssen, T., Hakimi, N., Simon, R., von Petersdorff-Campen, M., Jacobsen, F., von Loga, K., Wilczak, W., Minner, S., Tsourlakis, M. C., Chirico, V., Haese, A., Heinzer, H., Beyer, B., Graefen, M., Michl, U., Salomon, G., Steuber, T., Budäus, L. H., Hekeler, E., Malsy-Mink, J., Kutzera, S., Fraune, C., Göbel, C., Huland, H., Schlomm, T., Clinical Eur. Urol. 69, 592–598 (2016).

Night Sight: Seeing in the Dark on Pixel Phones

Wednesday, November 14, 2018

Posted by Marc Levoy, Distinguished Engineer and Yael Pritch, Staff Software EngineerHDR+

Left: iPhone XS (full resolution image here). Right: Pixel 3 Night Sight (full resolution image here).

Why is Low-light Photography Hard?image noiseshot noiseread noisesignal-to-noise ratioHDR+How Dark is Dark?

Capturing the Datazero-shutter-lagHDR+Super Res Zoompositive-shutter-lagoptical flow¹

Left: 15-frame burst captured by one of two side-by-side Pixel 3 phones. Center: Night Sight shot with motion metering disabled, causing this phone to use 73ms exposures. The dog’s head is motion blurred in this crop. Right: Night Sight shot with motion metering enabled, causing this phone to notice the motion and use shorter 48ms exposures. This shot has less motion blur. (Mike Milne)

Left: Crop from a handheld Night Sight shot of the sky (full resolution image here). There was slight handshake, so Night Sight chose 333ms x 15 frames = 5.0 seconds of capture. Right: Tripod shot (full resolution image here). No handshake was detected, so Night Sight used 1.0 second x 6 frames = 6.0 seconds. The sky is cleaner (less noise), and you can see more stars. (Florian Kainz)

Alignment and Mergingexposure stackingSynthcamsteadily became cleaner as you watchedHDR+'s merging algorithmSuper Res ZoomOther Challenges1. Auto white balancing (AWB) fails in low light.color constancycolor temperatureauto white balancingill-posed problemlearning-based AWB algorithm

Left: The white balancer in the Pixel’s default camera mode doesn't know how yellow the illumination was on this shack on the Vancouver waterfront (full resolution image here). Right: Our learning-based AWB algorithm does a better job (full resolution image here). (Marc Levoy)

2. Tone mapping of scenes that are too dark to see.cone cells

Yosemite valley at nighttime, Canon DSLR, 28mm f/4 lens, 3-minute exposure, ISO 100. It's nighttime, since you can see stars, but it looks like daytime (full resolution image here). (Jesse Levinson)

A Philosopher Lecturing on the Orrery, by Joseph Wright of Derby, 1766 (image source: Wikidata). The artist uses pigments from black to white, but the scene depicted is evidently dark. How does he accomplish this? He increases contrast, surrounds the scene with darkness, and drops shadows to black, because we cannot see detail there.

S-curve

Pixel 3, 6-second Night Sight shot, with tripod (full resolution image here). (Alex Savu)

How Dark can Night Sight Go?hyperfocal distanceblog postHow to Get the Most out of Night SightexamplesA/B comparisons- Night Sight can't operate in complete darkness, so pick a scene with some light falling on it.- Soft, uniform lighting works better than harsh lighting, which creates dark shadows.- To avoid lens flare artifacts, try to keep very bright light sources out of the field of view.- To increase exposure, tap on various objects, then move the exposure slider. Tap again to disable.- To decrease exposure, take the shot and darken later in Google’s Photos editor; it will be less noisy.- If it’s so dark the camera can’t focus, tap on a high-contrast edge, or the edge of a light source.- If this won’t work for your scene, use the Near (4 feet) or Far (12 feet) focus buttons (see below).- To maximize image sharpness, brace your phone against a wall or tree, or prop it on a table or rock.- Night Sight works for selfies too, as in the A/B album, with optional illumination from the screen itself.

Manual focus buttons (Pixel 3 only).

AcknowledgementsNight Sight was a collaboration of several teams at Google. Key contributors to the project include: from the Gcam team Charles He, Nikhil Karnad, Orly Liba, David Jacobs, Tim Brooks, Michael Milne, Andrew Radin, Navin Sarma, Jon Barron, Yun-Ta Tsai, Jiawen Chen, Kiran Murthy, Tianfan Xue, Dillon Sharlet, Ryan Geiss, Sam Hasinoff and Alex Schiffhauer; from the Super Res Zoom team Bart Wronski, Peyman Milanfar and Ignacio Garcia Dorado; from the Google camera app team Gabriel Nava, Sushil Nath, Tim Smith , Justin Harrison, Isaac Reynolds and Michelle Chen.
1 By the way, the exposure time shown in Google Photos (if you press "i") is per-frame, not total time, which depends on the number of frames captured. You can get some idea of the number of frames by watching the animation while the camera is collecting light. Each tick around the circle is one captured frame.^↩

2 For a wonderful analysis of these techniques, look at von Helmholtz, "On the relation of optics to painting" (1876).^↩

Accurate Online Speaker Diarization with Supervised Learning

Monday, November 12, 2018

Posted by Chong Wang, Research Scientist, Google AISpeaker diarizationunderstanding medical conversationsvideo captioning

Online speaker diarization on streaming audio input. Different colors in the bottom axis indicate different speakers.

Fully Supervised Speaker DiarizationNIST SRE 2000 CALLHOMEprevious clustering-based methoddeep neural network embedding methodsopen sourcing the core algorithmsClustering versus Interleaved-state RNNk-meansspectral clusteringrecurrent neural networkinterleavedblueyellowpinkgreenChinese restaurant processblueyellowbluey₇green

The generative process of our model. Colors indicate labels for speaker segments.

stochastic gradient descentFuture Workour paperGithub pageAcknowledgmentsThis work was done as a close collaboration between Google AI and Speech & Assistant teams. Contributors include Aonan Zhang (intern), Quan Wang, Zhengyao Zhu and Chong Wang.

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing

Friday, November 2, 2018

Posted by Jacob Devlin and Ming-Wei Chang, Research Scientists, Google AI Languagenatural language processingbillionspre-trainingfine-tunedquestion answeringsentiment analysisopen sourcedBERTransformersBERTCloud TPUTensorFlowIn our associated paperStanford Question Answering DatasetWhat Makes BERT Different? Semi-supervised Sequence LearningGenerative Pre-TrainingELMoULMFitdeeply bidirectionalunsupervisedWikipediacontext-freecontextualcontextualunidirectionalbidirectionalword2vecGloVeword embeddingbankbank accountbank of the river.I accessed the bank accountbankI accessed theaccountbankI accessed theaccount

BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.

The Strength of Bidirectionalityand

very long timeABBA

Training with Cloud TPUsCloud TPUsTransformer model architectureopen source releasetensor2tensor libraryResults with BERTSQuAD v1.1

GLUE benchmarkimproves upon the state-of-the-art

Making BERT Work for Youhttp://goo.gl/language/bertColabBERT FineTuning with Cloud TPUsBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Blog