Google AI Blog

Providing Gender-Specific Translations in Google Translate

Monday, December 10, 2018

Posted by Melvin Johnson, Senior Software Engineer, Google TranslateGoogle Translateend-to-end neural network-based systemHe/she is a doctorHe/she is a nursewe announced

Gender-specific translations on the Google Translate website.

Detecting Gender-Neutral Queriesmorphologically complexO, OnaBiliyor mu?Does she know?Does he know?Cloud Natural Language APIGenerating Gender-Specific TranslationsNeural Machine Translation

Identifying and dividing our parallel training data into those with feminine words, those with masculine and those with ungendered words.

Adding an additional input token to the beginning of the sentence to specify the required gender to translate to, similar to how we build multilingual NMT systems:

<2MALE> O bir doktor → He is a doctor
<2FEMALE> O bir doktor → She is a doctor

Training our enhanced NMT model on the feminine, masculine and ungendered data sources. We experimented with various mixing ratios for these sources to enable the model to perform equally well on the three tasks.

Checking for Accuracy

If the requested feminine translation is feminine.

If the requested masculine translation is masculine.

If the feminine and masculine translations are exactly equivalent with the exception of gender-related changes. Even minor changes in the wording between the translations will result in being filtered.

Top: The masculine and feminine translations differ only with respect to gender i.e. “he” and “his” vs “she” and “her”. Hence, we will show gender-specific translations. Bottom: The masculine and feminine translations differ correctly with respect to gender i.e. “he” vs “she”. However, the change from “really” to “actually” is not related to gender. Hence, we will filter gender-specific translations and display the default translation.

fairness in machine learningAcknowledgements:This effort has been successful thanks to the hard work of a lot of people including, but not limited to, the following (in alphabetical order of last name): Lindsey Boran, HyunJeong Choe, Héctor Fernández Alcalde, Orhan Firat, Qin Gao, Rick Genter, Macduff Hughes, Tolga Kayadelen, James Kuczmarski, Tatiana Lando, Liu Liu, Michael Mandl, Nihal Meriç Atilla, Mengmeng Niu, Adnan Ozturel, Emily Pitler, Kathy Ray, John Richardson, Larissa Rinaldi, Alex Rudnick, Apu Shah, Jason Smith, Antonio Stella, Romina Stella, Jana Strnadova, Katrin Tomanek, Barak Turovsky, Dan Schwarz, Shilp Vaishnav, Clayton Watts, Kellie Webster, Colin Young, Pendar Yousefi, Candice Zhang and Min Zhao.

Adding Diversity to Images with Open Images Extended

Friday, December 7, 2018

Posted by Anurag Batra and Parker Barnes, Product Managers, Google AIwe introducedInclusive Images Kaggle competitionNeurIPS 2018 Competition TrackOpen Images ExtendedCrowdsource appAbout the Crowdsourced Extension of Open Images Extendedhere

Pictures from India and Singapore contributed using the Crowdsource app.

Get InvolvedCrowdsource Android appcontact usAcknowledgementsThe release of Open Images Extended has been possible thanks to the hard work of a lot of people including, but not limited to the following (in alphabetical order of last name): James Atwood, Pallavi Baljekar, Peggy Chi, Tulsee Doshi, Tom Duerig, Vittorio Ferrari, Akshay Gaur, Victor Gomes, Yoni Halpern, Gursheesh Kaur, Mahima Pushkarna, Jigyasa Saxena, D. Sculley, Richa Singh, Rachelle Summers.

TF-Ranking: A Scalable TensorFlow Library for Learning-to-Rank

Wednesday, December 5, 2018

Posted by Xuanhui Wang and Michael Bendersky, Software Engineers, Google AImachine translationdialogue systemscomputational biology.learning-to-rank—TensorFlowTF-Rankingpaperpairwise or listwise loss functionsmulti-item scoringranking metric optimizationunbiased learning-to-rankExisting Algorithms and Metrics Supportloss functionprior workembeddingsMean Reciprocal RankNormalized Discounted Cumulative GainTensorBoard

An example of the NDCG metric (Y-axis) along the training steps (X-axis) displayed in the TensorBoard. It shows the overall progress of the metrics during training. Different methods can be compared directly on the dashboard. Best models can be selected based on the metric.

Multi-Item Scoringscored jointly

The TF-Ranking library supports multi-item scoring architecture, an extension of traditional single-item scoring.

recent workLETORRanking Metric OptimizationArea Under the Curvediscontinuous or flatrecent work,expectation-maximizationUnbiased Learning-to-RankPrior researchunbiased learning-to-rankGetting Started with TF-RankingTensorFlow EstimatorTensorboardTensorFlow ServingGitHub repotutorial examplesAcknowledgementsThis project was only possible thanks to the members of the core TF-Ranking team: Rama Pasumarthi, Cheng Li, Sebastian Bruch, Nadav Golbandi, Stephan Wolf, Jan Pfeifer, Rohan Anil, Marc Najork, Patrick McGregor and Clemens Mewald‎. We thank the members of the TensorFlow team for their advice and support: Alexandre Passos, Mustafa Ispir, Karmel Allison, Martin Wicke, and others. Finally, we extend our special thanks to our collaborators, interns and early adopters: Suming Chen, Zhen Qin, Chirag Sethi, Maryam Karimzadehgan, Makoto Uchida, Yan Zhu, Qingyao Ai, Brandon Tran, Donald Metzler, Mike Colagrosso, and many others at Google who helped in evaluating and testing the early versions of TF-Ranking.

The NeurIPS 2018 Test of Time Award: The Trade-Offs of Large Scale Learning

Tuesday, December 4, 2018

Posted by Anna Ukhanova, Program Manager, Google AI ZürichThe Trade-Offs of Large Scale LearningNEC LabsFacebook AI ResearchGoogle AI, Zürich)stochastic gradient descentOptimization and the Challenge of Scale

A model, which is a set of possible functions that will be used to fit the data.

An optimization algorithm which specifies how to find the best function in that set.

Kernel MachinesSVMsquadratic programmingLarge Scale Kernel MachinesStochastic Algorithms Scale BetterMysteries Remain

Google at NeurIPS 2018

Monday, December 3, 2018

Posted by Slav Petrov, Principal Scientist, GooglendConference on Neural Information Processing Systemsdefine a new frameworkblueNeurIPS Foundation BoardCorinna Cortes, John C. Platt, Fernando PereiraNeurIPS Organizing CommitteeSamy BengioHugo LarochelleDouglas EckKatherine A. HellerNeurIPS Program CommitteeAngela Yu, Claudio Gentile, Cordelia Schmid, Corinna Cortes, Csaba Szepesvari, Dale Schuurmans, Elad Hazan, Mehryar Mohri, Raia Hadsell, Satyen Kale, Yishay Mansour, Afshin Rostamizadeh, Alex Kulesza Amin Karbasi, Amir Globerson, Amit Daniely, Andras Gyorgy, Andriy Mnih, Been Kim, Branislav Kveton, Ce Liu, D Sculley, Danilo Rezende, Danny Tarlow, David Balduzzi, Denny Zhou, Dilan Gorur, Dumitru Erhan, George Dahl, Graham Taylor, Ian Goodfellow, Jasper Snoek, Jean-Philippe Vert, Jia Deng, Jon Shlens, Karen Simonyan, Kevin Swersky, Kun Zhang, Lihong Li, Marc G. Bellemare, Marco Cuturi, Maya Gupta, Michael Bowling, Michalis Titsias, Mohammad Norouzi, Mouhamadou Moustapha Cisse, Nicolas Le Roux, Remi Munos, Sanjiv Kumar, Sanmi Koyejo, Sergey Levine, Silvia Chiappa, Slav Petrov, Surya Ganguli, Timnit Gebru, Timothy Lillicrap, Viren Jain, Vitaly Feldman, Vitaly Kuznetsov Mehryar Mohri, Sergey LevineAccepted Papers3D-Aware Scene Manipulation via Inverse GraphicsShunyu Yao, Tzu Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. TenenbaumA Retrieve-and-Edit Framework for Predicting Structured OutputsTatsunori Hashimoto, Kelvin Guu, Yonatan Oren, Percy LiangAdversarial Attacks on Stochastic BanditsKwang-Sung Jun, Lihong Li, Yuzhe Ma, Xiaojin ZhuAdversarial Examples that Fool both Computer Vision and Time-Limited HumansGamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, Jascha Sohl-DicksteinAdversarially Robust Generalization Requires More DataLudwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, Aleksander MadryAre GANs Created Equal? A Large-Scale StudyMario Lucic, Karol Kurach, Marcin Michalski, Olivier Bousquet, Sylvain GellyCollaborative Learning for Deep Neural NetworksGuocong Song, Wei ChaiCompleting State Representations using Spectral LearningNan Jiang, Alex Kulesza, Santinder SinghContent Preserving Text Generation with Attribute ControlsLajanugen Logeswaran, Honglak Lee, Samy BengioContext-aware Synthesis and Placement of Object InstancesDonghoon Lee, Sifei Liu, Jinwei Gu, Ming-Yu Liu, Ming-Hsuan Yang, Jan KautzCo-regularized Alignment for Unsupervised Domain AdaptationAbhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerlo Feris, William T. Freeman, Gregory WornellcpSGD: Communication-efficient and differentially-private distributed SGDNaman Agarwal, Ananda Theertha Suresh, Felix Yu, Sanjiv Kumar, H. Brendan McmahanData Center Cooling Using Model-Predictive ControlNevena Lazic, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, MK Ryu, Greg ImwalleData-Efficient Hierarchical Reinforcement LearningOfir Nachum, Shixiang Gu, Honglak Lee, Sergey LevineDeep Attentive Tracking via Reciprocative LearningShi Pu, Yibing Song, Chao Ma, Honggang Zhang, Ming-Hsuan YangGeneralizing Point Embeddings Using the Wasserstein Space of Elliptical DistributionsBoris Muzellec, Marco CuturiGLoMo: Unsupervised Learning of Transferable Relational GraphsZhilin Yang, Jake (Junbo) Zhao, Bhuwan Dhingra, Kaiming He, William W. Cohen, Ruslan Salakhutdinov, Yann LeCunGroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model ShrinkingPatrick Chen, Si Si, Yang Li, Ciprian Chelba, Cho-Jui HsiehInterpreting Neural Network Judgments via Minimal, Stable, and Symbolic CorrectionsXin Zhang, Armando Solar-Lezama, Rishabh SinghLearning Hierarchical Semantic Image Manipulation through Structured RepresentationsSeunghoon Hong, Xinchen Yan, Thomas Huang, Honglak LeeLearning Temporal Point Processes via Reinforcement LearningShuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, Le SongLearning Towards Minimum Hyperspherical EnergyWeiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, Le SongMesh-TensorFlow: Deep Learning for SupercomputersNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake HechtmanMiME: Multilevel Medical Embedding of Electronic Health Records for Predictive HealthcareEdward Choi, Cao Xiao, Walter F. Stewart, Jimeng SunSearching for Efficient Multi-Scale Architectures for Dense Image PredictionLiang-Chieh Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, Jonathon ShlensSplineNets: Continuous Neural Decision GraphsCem Keskin, Shahram IzadiTask-Driven Convolutional Recurrent Models of the Visual SystemAran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, Daniel L. K. YaminsTo Trust or Not to Trust a ClassifierHeinrich Jiang, Been Kim, Melody Guan, Maya GuptaTransfer Learning from Speaker Verification to Multispeaker Text-To-Speech SynthesisYe Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui WuAlgorithms and Theory for Multiple-Source AdaptationJudy Hoffman, Mehryar Mohri, Ningshan ZhangA Lyapunov-based Approach to Safe Reinforcement LearningYinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, Mohammad GhavamzadehAdaptive Methods for Nonconvex OptimizationManzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, Sanjiv KumarAssessing Generative Models via Precision and RecallMehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain GellyA Loss Framework for Calibrated Anomaly DetectionAditya Menon, Robert WilliamsonBlockwise Parallel Decoding for Deep Autoregressive ModelsMitchell Stern, Noam Shazeer, Jakob UszkoreitBreaking the Curse of Horizon: Infinite-Horizon Off-Policy EstimationQiang Liu, Lihong Li, Ziyang Tang, Dengyong ZhouContextual Pricing for Lipschitz BuyersJieming Mao, Renato Leme, Jon SchneiderCoupled Variational Bayes via Optimization EmbeddingBo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, Le SongData Amplification: A Unified and Competitive Approach to Property EstimationYi HAO, Alon Orlitsky, Ananda Theertha Suresh, Yihong WuDeep Network for the Integrated 3D Sensing of Multiple People in Natural ImagesElisabeta Marinoiu, Mihai Zanfir, Alin-Ionut Popa, Cristian SminchisescuDeep Non-Blind Deconvolution via Generalized Low-Rank ApproximationWenqi Ren, Jiawei Zhang, Lin Ma, Jinshan Pan, Xiaochun Cao, Wei Liu, Ming-Hsuan YangDiminishing Returns Shape Constraints for Interpretability and RegularizationMaya Gupta, Dara Bahri, Andrew Cotter, Kevin CaniniDropBlock: A Regularization Method for Convolutional NetworksGolnaz Ghiasi, Tsung-Yi Lin, Quoc V. LeGeneralization Bounds for Uniformly Stable AlgorithmsVitaly Feldman, Jan VondrakGeometrically Coupled Monte Carlo SamplingMark Rowland, Krzysztof Choromanski, Francois Chalus, Aldo Pacchiano, Tamas Sarlos, Richard E. Turner, Adrian WellerGILBO: One Metric to Measure Them AllAlexander A. Alemi, Ian FischerInsights on Representational Similarity in Neural Networks with Canonical CorrelationAri S. Morcos, Maithra Raghu, Samy BengioImproving Online Algorithms via ML PredictionsManish Purohit, Zoya Svitkina, Ravi KumarLearning to Exploit Stability for 3D Scene ParsingYilun Du, Zhijan Liu, Hector Basevi, Ales Leonardis, William T. Freeman, Josh Tenembaum, Jiajun WuMaximizing Induced Cardinality Under a Determinantal Point ProcessJennifer Gillenwater, Alex Kulesza, Sergei Vassilvitskii, Zelda MarietMemory Augmented Policy Optimization for Program Synthesis and Semantic ParsingChen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V. Le, Ni LaoPCA of High Dimensional Random Walks with Comparison to Neural Network TrainingJoseph M. Antognini, Jascha Sohl-DicksteinPredictive Approximate Bayesian Computation via Saddle PointsYingxiang Yang, Bo Dai, Negar Kiyavash, Niao HeRecurrent World Models Facilitate Policy EvolutionDavid Ha, Jürgen SchmidhuberSanity Checks for Saliency MapsJulius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been KimSimple, Distributed, and Accelerated Probabilistic ProgrammingDustin Tran, Matthew Hoffman, Dave Moore, Christopher Suter, Srinivas Vasudevan, Alexey Radul, Matthew Johnson, Rif A. SaurousTangent: Automatic Differentiation Using Source-Code Transformation for Dynamically Typed Array ProgrammingBart van Merriënboer, Dan Moldovan, Alex WiltschkoThe Emergence of Multiple Retinal Cell Types Through Efficient Coding of Natural MoviesSamuel A. Ocko, Jack Lindsey, Surya Ganguli, Stephane DenyThe Everlasting Database: Statistical Validity at a Fair PriceBlake Woodworth, Vitaly Feldman, Saharon Rosset, Nathan SrebroThe Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural NetworkJeffrey Pennington, Pratik WorahA Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial AttacksKimin Lee, Kibok Lee, Honglak Lee, Jinwoo ShinAutoconj: Recognizing and Exploiting Conjugacy Without a Domain-Specific LanguageMatthew D. Hoffman, Matthew Johnson, Dustin TranA Bayesian Nonparametric View on Count-Min SketchDiana Cai, Michael Mitzenmacher, Ryan Adams (no longer at Google)Automatic Differentiation in ML: Where We are and Where We Should be GoingBart van Merriënboer, Olivier Breuleux, Arnaud Bergeron, Pascal LamblinAssessing the Scalability of Biologically-Motivated Deep Learning Algorithms and ArchitecturesSergey Bartunov, Adam Santoro, Blake A. Richards, Geoffrey E. Hinton, Timothy P. LillicrapDeep Generative Models for Distribution-Preserving Lossy CompressionMichael Tschannen, Eirikur Agustsson, Mario LucicDeep Structured Prediction with Nonlinear Output TransformationsColin Graber, Ofer Meshi, Alexander SchwingDiscovery of Latent 3D Keypoints via End-to-end Geometric ReasoningSupasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, Mohammad NorouziTransfer Learning with Neural AutoMLCatherine Wong, Neil Houlsby, Yifeng Lu, Andrea GesmundoEfficient Gradient Computation for Structured Output Learning with Rational and Tropical LossesCorinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, Dmitry Storcheus, Scott YangCooperative neural networks (CoNN): Exploiting prior independence structure for improved classificationHarsh Shrivastava, Eugene Bart, Bob Price, Hanjun Dai, Bo Dai, Srinivas AluruGraph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic OptimizationBlake Woodworth, Jialei Wang, Brendan McMahan, Nathan SrebroHierarchical Reinforcement Learning for Zero-shot Generalization with Subtask DependenciesSungryull Sohn, Junhyuk Oh, Honglak LeeHuman-in-the-Loop Interpretability PriorIsaac Lage, Andrew Slavin Ross, Been Kim, Samuel J. Gershman, Finale Doshi-VelezJoint Autoregressive and Hierarchical Priors for Learned Image CompressionDavid Minnen, Johannes Ballé, George D TodericiLarge-Scale Computation of Means and Clusters for Persistence Diagrams Using Optimal TransportThéo Lacombe, Steve Oudot, Marco CuturiLearning to Reconstruct Shapes from Unseen ClassesXiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, William T. Freeman, Jiajun WuLarge Margin Deep Networks for ClassificationGamaleldin Fathy Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, Samy BengioMallows Models for Top-k ListsFlavio Chierichetti, Anirban Dasgupta, Shahrzad Haddadan, Ravi Kumar, Silvio LattanziMeta-Learning MCMC ProposalsTongzhou Wang, YI WU, Dave Moore, Stuart RussellNon-delusional Q-Learning and Value-IterationTyler Lu, Dale Schuurmans, Craig BoutilierOnline Learning of Quantum StatesScott Aaronson, Xinyi Chen, Elad Hazan, Satyen Kale, Ashwin NayakOnline Reciprocal Recommendation with Theoretical Performance GuaranteesFabio Vitale, Nikos Parotsidis, Claudio GentileOptimal Algorithms for Continuous Non-monotone Submodular and DR-Submodular MaximizationRad Niazadeh, Tim Roughgarden, Joshua R. WangPolicy Regret in Repeated GamesRaman Arora, Michael Dinitz, Teodor Vanislavov Marinov, Mehryar MohriProvable Variational Inference for Constrained Log-Submodular ModelsJosip Djolonga, Stefanie Jegelka, Andreas KrauseRealistic Evaluation of Deep Semi-Supervised Learning AlgorithmsAvital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, Ian J. GoodfellowSample-Efficient Reinforcement Learning with Stochastic Ensemble Value ExpansionJacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak LeeVisual Object Networks: Image Generation with Disentangled 3D RepresentationsJunYan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, William T. FreemanWatch Your Step: Learning Node Embeddings via Graph AttentionSami Abu-El-Haija, Bryan Perozzi, Rami AlRfou, Alexander AlemiWorkshops2nd Workshop on Machine Learning on the Phone and Other Consumer DevicesSujith Ravi, Wei Chai, Hrishikesh AradhyeBayesian Deep Learning Kevin MurphyContinual Learning Marc PickettThe Second Conversational AI Workshop – Today's Practice and Tomorrow's Potential Dilek Hakkani-TurVisually Grounded Interaction and Language Olivier PietquinWorkshop on Ethical, Social and Governance Issues in AI D. SculleyAI for Social GoodSamuel Greydanus Black in AIMouhamadou Moustapha Cisse, Timnit GebruIrwan Bello, Samy Bengio, Ian Goodfellow, Hugo Larochelle, Margaret MitchellInterpretability and Robustness in Audio, Speech, and Language Ehsan Variani, Bhuvana RamabhadranLatinX in AIPablo Samuel Castro Sergio Guadarrama Machine Learning for Systems Anna Goldie, Azalia Mirhoseini, Kevin Swersky, Milad Hashemi Simon Kornblith, Nicholas Frosst, Amir Yazdanbakhsh, Azade Nazi, James Bradbury, Sharan Narang, Martin Maas, Carlos VillaviejaQueer in AIRaphael Gontijo LopesSecond Workshop on Machine Learning for Creativity and Design Jesse Engel, Adam RobertsWorkshop on Security in Machine LearningNicolas PapernotTutorialVisualization for Machine LearningFernanda Viégas, Martin Wattenberg

Highlights from the 2018 Google PhD Fellowship Summit

Friday, November 30, 2018

Posted by Susie Kim, Program Manager, University RelationsPhD Fellowship ProgramAustraliaChina and East AsiaIndiaNorth America, Europe, the Middle EastAfricaSummit HighlightsMaggie JohnsonMaya GuptaAndrew TomkinsRahul SukthankarSai Teja PeddintiAmin VahdatMartin StumpeEd ChiCiera JaspanJeff Dean

Starting clockwise from top left: Researchers Rahul Sukthankar and Ed Chi talking with Fellow attendees; Jeff Dean delivering the closing talk; Poster session in full swing.

Google Fellows attending the 2018 PhD Fellowship Summit.

The Complete List of 2018 Google PhD FellowsAlgorithms, Optimizations and MarketsMassachusetts Institute of TechnologyETH ZurichChennai Mathematical InstituteCornell UniversityDuke UniversityComputational NeuroscienceColumbia UniversityUniversity of California - BerkeleyNew York UniversityHuman Computer InteractionCarnegie Mellon UniversityUniversity of WaterlooGeorgia Institute of TechnologyThe University of MelbourneAustralian National UniversityThe University of New South WalesMachine LearningIndian Institute of Technology - BombayStanford UniversityUniversity of TokyoYale UniversityUniversity of Southern CaliforniaHebrew UniversityChinese University of Hong KongUniversity of California - San DiegoUniversity of California - BerkeleyUniversity of the WitwatersrandUniversity of MichiganMassachusetts Institute of TechnologyMachine Perception, Speech Technology and Computer VisionINRIAUniversity of OxfordIndian Institute of Technology - MadrasUniversity of WashingtonUniversity of Illinois - Urbana-ChampaignGeorgia Institute of TechnologyMobile ComputingHong Kong University of Science & TechnologyEgypt-Japan University of Science and TechnologyUniversity of California - San DiegoNatural Language ProcessingUniversity of PennsylvaniaShanghai Jiao Tong UniversityTel Aviv UniversityTechnion - Israel Institute of TechnologyIndian Institute of Science - BangaloreUniversity of EdinburghHarvard UniversityShanghai Jiao Tong UniversityUniversité Internationale de RabatPrivacy and SecurityUniversity of California - Los AngelesProgramming Technology and Software EngineeringPurdue UniversityUniversity of Dar es SalaamUniversity of California - IrvineFondation Sciences Mathématiques de ParisQuantum ComputingCalifornia Institute of TechnologyStructured Data and Database ManagementUniversity of Wisconsin - MadisonSystems and NetworkingETH ZurichTsinghua UniversityEPFLUniversity of CambridgeUniversity of Massachusetts - AmherstThe University of Sydney

Learning to Predict Depth on the Pixel 3 Phones

Thursday, November 29, 2018

Posted by Rahul Garg, Research Scientist and Neal Wadhwa, Software Engineerdual-pixel autofocustraditional non-learnedstereo algorithm

Left: The original HDR+ image. Right: A comparison of Portrait Mode results using depth from traditional stereo and depth from machine learning. The learned depth result has fewer errors. Notably, in the traditional stereo result, many of the horizontal lines behind the man are incorrectly estimated to be at the same depth as the man and are kept sharp.
(Mike Milne)

A Short Recaplast year’s blog postparallax

The two PDAF images on the left and center look very similar, but in the crop on the right you can see the parallax between them. It is most noticeable on the circular structure in the middle of the crop.

the aperture problemImproving Depth Estimationdefocussemanticconvolutional neural networkTensorFlow

Our convolutional neural network takes as input the PDAF images and outputs a depth map. The network uses an encoder-decoder style architecture with skip connections and residual blocks.

Training the Neural Networkstructure from motionmulti-view stereo

Left: Custom rig used to collect training data. Middle: An example capture flipping between the five images. Synchronization between the cameras ensures that we can calculate depth for dynamic scenes, such as this one. Right: Ground truth depth. Low confidence points, i.e., points where stereo matches are not reliable due to weak texture, are colored in black and are not used during training. (Sam Ansari and Mike Milne)

Five viewpoints ensure that there is parallax in multiple directions and hence no aperture problem.

The arrangement of the cameras ensures that a point in an image is usually visible in at least one other image resulting in fewer points with no correspondences.

The baseline, i.e., the distance between the cameras is much larger than our PDAF baseline resulting in more accurate depth estimation.

Synchronization between the cameras ensure that we can calculate depth for dynamic scenes like the one above.

Portability of the rig ensures that we can capture photos in the wild simulating the photos users take with their smartphones.

Putting it All TogetherTensorFlow LiteTry it YourselfGoogle Photos depth editorthirdpartydepthextractorsalbumAcknowledgmentsThis work wouldn’t have been possible without Sam Ansari, Yael Pritch Knaan, David Jacobs, Jiawen Chen, Juhyun Lee and Andrei Kulik. Special thanks to Mike Milne and Andy Radin who captured data with the five-camera rig.

A Structured Approach to Unsupervised Depth Learning from Monocular Videos

Tuesday, November 27, 2018

Posted by Anelia Angelova, Research Scientist, Robotics at GoogleLIDARego-motionZhou et al.prior researchDepth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular VideosAAAI 2019open sourced the code in TensorFlow

Previous work (middle row) has not been able to correctly estimate depth of moving objects mapping them to infinity (dark blue regions in the heatmap). Our approach (right) provides much better depth estimates.

Structureneural network to learn depth directlyKITTICityscapessolve

Previous work with monocular inputs were not able to extract moving objects and incorrectly map them to infinity.

Example depth results for a dynamic scene together with estimates of the motion vectors of the individual objects (rotation angles are estimated too, but for simplicity are not shown).

Depth prediction from monocular video input on the KITTI dataset, middle row, compared to ground truth depth from a Lidar sensor; the latter does not cover the full scene and has missing and noisy values. Ground truth depth is not used during training.

Depth prediction on the Cityscapes dataset. Left to right: image, baseline, our method and ground truth provided by stereo. Note the missing values in the stereo ground truth. Also note that our algorithm is able to achieve these results without any ground truth depth supervision.

Ego-motion

Depth and ego-motion prediction. Follow the speed and the turning angle indicator to see the estimates when the car is taking a turn or stopping for a red light.

Transfer Across Domains

Online refinement when training on the Cityscapes Data and testing on KITTI. The images show depth prediction of the trained model, and of the trained model with online refinement. Depth prediction with online refinement better outlines the objects in the scene.

Fetch robot

Results of online adaptation when transferring the learning model from Cityscapes (an outdoors dataset collected from a moving car) to a dataset collected indoors by the Fetch robot. The bottom row shows improved depth after applying online refinement.

AcknowledgementsThis research was conducted by Vincent Casser, Soeren Pirk, Reza Mahjourian and Anelia Angelova. We would like to thank Ayzaan Wahid for his help with data collection and Martin Wicke and Vincent Vanhoucke for their support and encouragement.