Google AI Blog: May 2019

Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction

Thursday, May 23, 2019

Posted by Tali Dekel, Research Scientist and Forrester Cole, Software Engineer, Machine Perceptioncomputer visiontriangulationGoogle’s Jump

Left: The traditional stereo setup assumes that at least two viewpoints capture the scene at the same time. Right: We consider the setup where both camera and subject are moving.

Learning the Depths of Moving People by Watching Frozen Peopledeep learning-baseddepth mapsaugmented reality

Our model predicts the depth map (right; brighter=closer to the camera) from a regular video (left), where both the people in the scene and the camera are freely moving.

Sourcing the Training Datapeople imitate mannequinsmulti-view-stereo

Videos of people imitating mannequins while a camera tours the scene, which we used for training. We use traditional MVS algorithms to estimate depth, which serves as supervision during training of our depth-prediction model.

Inferring the Depth of Moving Peopleand moving peoplemotion parallaxstatic objects2D optical flowhuman-segmentation network

Depth prediction network: The input to the model includes an RGB image (Frame t), a mask of the human region, and an initial depth for the non-human regions, computed from motion parallax (optical flow) between the input frame and another frame in the video. The model outputs a full depth map for Frame t. Supervision for training is provided by the depth map, computed by MVS.

Comparison of depth prediction models to a video clip with moving cameras and people. Top: Learning based monocular depth prediction methods (DORN; Chen et al.). Bottom: Learning based stereo method (DeMoN), and our result.

3D Video Effects Using Our Depth Maps

Bokeh video effect produced using our estimated depth maps. Video courtesy of Wind Walk Travel Videos.

AcknowledgementsThe research described in this post was done by Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu and Bill Freeman. We would like to thank Miki Rubinstein for his valuable feedback.

Introducing Translatotron: An End-to-End Speech-to-Speech Translation Model

Wednesday, May 15, 2019

Posted by Ye Jia and Ron Weiss, Software Engineers, Google AIpast several decadesautomatic speech recognitionmachine translationtext-to-speech synthesiscascadeGoogle TranslateDirect speech-to-speech translation with a sequence-to-sequence modelattentive sequence-to-sequence modelTranslatotronTranslatotronthe feasibilityoutperform cascade modelsleveraging weakly supervised dataspectrogramsvocodermultitask objective

Model architecture of Translatotron.

PerformanceBLEU score

Input (Spanish)

Reference translation (English)

Baseline cascade translation

Translatotron translation

herePreserving Vocal Characteristicsspeaker verificationspeaker adaptation for TTS

Input (Spanish)

Reference translation (English)

Baseline cascade translation

Translatotron translation (canonical voice)

Translatotron translation (original speaker’s voice)

hereConclusionAcknowledgmentsThis research was a joint work between the Google Brain, Google Translate, and Google Speech teams. Contributors include Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, Mengmeng Niu, Quan Wang, Jason Pelecanos, Ignacio Lopez Moreno, Tom Walters, Heiga Zen, Patrick Nguyen, Yu Zhang, Jonathan Shen, Orhan Firat, and Yonghui Wu. We also thank Jorge Pereira and Stella Laurenzo for verifying the quality of the translation from Translatotron.

An End-to-End AutoML Solution for Tabular Data at KaggleDays

Thursday, May 9, 2019

Posted by Yifeng Lu, Software Engineer, Google AIfeature engineeringhyper-parameter tuningneural architecture searchNasNetAmoebaNetMNasNet

Full automation: Data and computation resources are the only inputs, while a servable TensorFlow model is the output. The whole process requires no human intervention.

Extensive coverage: The solution is applicable to the majority of arbitrary tasks in the tabular data domain.

High quality: Models generated by AutoML has comparable quality to models manually crafted by top ML experts.

KaggleDays SF HackathonKaggleDays event Kaggle progression systemfinishing second place by a narrow marginfinal leaderboardarchitecture searchbootstrap aggregating

public kernel

Potential model quality improvement on final leaderboard if AutoML models were merged with other Kagglers’ models. “Erkut & Mark, Google AutoML”, includes the top winner “Erkut & Mark” and the second place “Google AutoML” models. Erkut Aykutlug and Mark Peng used XGBoost with creative feature engineering whereas AutoML uses both neural network and gradient boosting tree (TFBT) with automatic feature engineering and hyperparameter tuning.

Google Cloud AutoML TablesGoogle Cloud AutoML TablesGoogle Cloud Next ‘19.

Third party benchmark of AutoML Tables on multiple Kaggle competitions

AcknowledgementsThis project was only possible thanks to Google Brain team members Ming Chen, Da Huang, Yifeng Lu, Quoc V. Le and Vishy Tirumalashetty. We also thank Dawei Jia, Chenyu Zhao and Tin-yun Ho from the Cloud AutoML Tables team for great infrastructure and product landing collaboration. Thanks to Walter Reade, Julia Elliott and Kaggle for organizing such an engaging competition.

Announcing Open Images V5 and the ICCV 2019 Open Images Challenge

Wednesday, May 8, 2019

Posted by Vittorio Ferrari, Research Scientist, Machine PerceptionOpen ImagesupdatesOpen Images V4bounding-boxesOpen Images V5Open Images ChallengeOpen Images V5

Example masks on the training set of Open Images V5. These have been produced by our interactive segmentation process. The first example also shows a bounding box, for comparison. From left to right, top to bottom: Tea and cake at the Fitzwilliam Museum by Tim Regan, Pilota II by Euskal kultur erakundea Institut culturel basque, Rheas by Dag Peak, Wuxi science park, 1995 by Gary Stevens, Cat Cafe Shinjuku calico by Ari Helminen, and Untitled by Todd Huffman. All images used under CC BY 2.0 license.

segmentation masks on the training setinteractive segmentation processintersection-over-unionmasks on the validation and test sets

Example masks on the validation and test sets of Open Images V5, drawn completely manually. From left to right: thistle flowers by sophie, still life with ax by liz west, Fischkutter KOŁ-180 in Kolobrzeg (PL) by zeesenboot. All images used under CC BY 2.0 license.

Open Images Challenge 2019Open Images Challenge2019 International Conference on Computer Vision2018 editionavailable now

Google at ICLR 2019

Monday, May 6, 2019

Posted by Andrew Helton, Editor, Google AI CommunicationsICLR 2019machine learningdeep learningneural networksblueOfficers and Board MembersHugo LarochelleSamy BengioTara SainathGeneral ChairTara SainathWorkshop ChairsBeen Kim, Graham TaylorProgram Committee includes:Chelsea Finn, Dale Schuurmans, Dumitru Erhan, Katherine Heller, Lihong Li, Samy Bengio, Rohit Prabhavalkar, Alex Wiltschko, Slav Petrov, George DahlOral ContributionsGenerating High Fidelity Images with Subscale Pixel Networks and Multidimensional UpscalingJacob Menick, Nal KalchbrennerEnabling Factorized Piano Music Modeling and Generation with the MAESTRO DatasetCurtis Hawthorne, Andrew Stasyuk, Adam Roberts, Ian Simon, Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, Douglas EckMeta-Learning Update Rules for Unsupervised Representation LearningLuke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Sohl-DicksteinPostersA Data-Driven and Distributed Approach to Sparse Signal Representation and RecoveryAli Mousavi, Gautam Dasarathy, Richard G. BaraniukBayesian Deep Convolutional Networks with Many Channels are Gaussian ProcessesRoman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-DicksteinDiversity-Sensitive Conditional Generative Adversarial NetworksDingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak LeeDiversity and Depth in Per-Example Routing ModelsPrajit Ramachandran, Quoc V. LeEidetic 3D LSTM: A Model for Video Prediction and BeyondYunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-FeiGANSynth: Adversarial Neural Audio SynthesisJesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, Adam RobertsK for the Price of 1: Parameter-efficient Multi-task and Transfer LearningPramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, Andrew HowardLearning to Describe Scenes with ProgramsYunchao Liu, Zheng Wu, Daniel Ritchie, William Freeman, Joshua B Tenenbaum, Jiajun WuLearning to Infer and Execute 3D Shape ProgramsYonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William Freeman, Joshua B Tenenbaum, Jiajun WuThe Singular Values of Convolutional LayersHanie Sedghi, Vineet Gupta, Philip M. LongUnsupervised Discovery of Parts, Structure, and DynamicsZhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William Freeman, Joshua B Tenenbaum, Jiajun WuAdversarial Reprogramming of Neural NetworksGamaleldin Elsayed, Ian Goodfellow (no longer at Google), Jascha Sohl-DicksteinDiscriminator Rejection SamplingIan Goodfellow (no longer at Google), Jascha Sohl-DicksteinOn Self Modulation for Generative Adversarial NetworksTing Chen, Mario Lucic, Neil Houlsby, Sylvain GellyTowards GAN Benchmarks Which Require GeneralizationIshaan Gulrajani, Colin Raffel, Luke MetzUnderstanding and Improving Interpolation in Autoencoders via an Adversarial RegularizerDavid Berthelot, Colin Raffel, Aurko Roy, Ian Goodfellow (no longer at Google)A new dog learns old tricks: RL finds classic optimization algorithmsWeiwei Kong, Christopher Liaw, Aranyak Mehta, D. SivakumarContingency-Aware Exploration in Reinforcement LearningJongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, Honglak LeeDiscriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation LearningIlya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan TompsonDiversity is All You Need: Learning Skills without a Reward FunctionBenjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey LevineEpisodic Curiosity through ReachabilityNikolay Savinov, Anton Raichuk, Raphael Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain GellyLearning to Navigate the WebIzzeddin Gur, Ulrich Rueckert, Aleksandra Faust, Dilek Hakkani-TurMeta-Learning Probabilistic Inference for PredictionJonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. TurnerMulti-step Retriever-Reader Interaction for Scalable Open-domain Question AnsweringRajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Andrew McCallumNear-Optimal Representation Learning for Hierarchical Reinforcement LearningOfir Nachum, Shixiang Gu, Honglak Lee, Sergey LevineNeural Logic MachinesHonghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, Dengyong ZhouNeural Program Repair by Jointly Learning to Localize and RepairMarko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, Rishabh SinghOptimal Completion Distillation for Sequence LearningSara Sabour, William Chan, Mohammad NorouziRecall Traces: Backtracking Models for Efficient Reinforcement LearningAnirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua BengioSample Efficient Adaptive Text-to-Speech Yutian Chen, Yannis M Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Caglar Gulcehre, Aaron van den Oord, Oriol Vinyals, Nando de FreitasSynthetic Datasets for Neural Program SynthesisRichard Shin, Neel Kant, Kavi Gupta, Chris Bender, Brandon Trabucco, Rishabh Singh, Dawn SongThe Laplacian in RL: Learning Representations with Efficient ApproximationsYifan Wu, George Tucker, Ofir NachumA Mean Field Theory of Batch NormalizationGreg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S SchoenholzEfficient Training on Very Large Corpora via Gramian EstimationWalid Krichene, Nicolas Mayoraz, Steffen Rendle, Li Zhang, Xinyang Yi, Lichan Hong, Ed Chi, John AndersonPredicting the Generalization Gap in Deep Networks with Margin DistributionsYiding Jiang, Dilip Krishnan, Hossein Mobahi, Samy BengioInfoBot: Transfer and Exploration via the Information BottleneckAnirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, Yoshua BengioAntisymmetricRNN: A Dynamical System View on Recurrent Neural NetworksBo Chang, Minmin Chen, Eldad Haber, Ed H. ChiComplement Objective TrainingHao-Yun Chen, Pei-Hsin Wang, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, Da-Cheng JuanDOM-Q-NET: Grounded RL on Structured LanguageSheng Jia, Jamie Kiros, Jimmy BaFrom Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction FollowingJustin Fu, Anoop Korattikara Balan, Sergey Levine, Sergio GuadarramaHarmonic Unpaired Image-to-image TranslationRui Zhang, Tomas Pfister, Li-Jia LiHierarchical Generative Modeling for Controllable Speech SynthesisWei-Ning Hsu, Yu Zhang, Ron Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, Patrick Nguyen, Ruoming PangLearning Finite State Representations of Recurrent Policy NetworksAnurag Koul, Alan Fern, Samuel GreydanusLearning to Screen for Fast Softmax Inference on Large Vocabulary Neural NetworksPatrick Chen, Si Si, Sanjiv Kumar, Yang Li, Cho-Jui HsiehMusic Transformer: Generating Music with Long-Term StructureChen-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew Dai, Matthew D Hoffman, Monica Dinculescu, Douglas EckUniversal TransformersMostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz KaiserWhat do you learn from context? Probing for sentence structure in contextualized word representationsIan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Tom McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, Ellie PavlickDoubly Reparameterized Gradient Estimators for Monte Carlo ObjectivesGeorge Tucker, Dieterich Lawson, Shixiang Gu, Chris J. MaddisonHow Important Is a Neuron?Kedar Dhamdhere, Mukund Sundararajan, Qiqi YanInteger Networks for Data Compression with Latent-Variable ModelsJohannes Ballé, Nick Johnston, David MinnenModeling Uncertainty with Hedged Instance EmbeddingsSeong Joon Oh, Andrew Gallagher, Kevin Murphy, Florian Schroff, Jiyan Pan, Joseph RothPreventing Posterior Collapse with delta-VAEsAli Razavi, Aaron van den Oord, Ben Poole, Oriol VinyalsSpectral Inference Networks: Unifying Deep and Spectral LearningDavid Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, Kimberly L StachenfeldStochastic Prediction of Multi-Agent Interactions from Partial ObservationsChen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, Kevin MurphyWorkshopsLearning from Limited Labeled DataDeep Reinforcement Learning Meets Structured PredictionChen LiangMohammad NorouziDebugging Machine Learning ModelsD. SculleyDan MoldovanStructure & Priors in Reinforcement Learning (SPiRL)Chelsea FinnTask-Agnostic Reinforcement Learning (TARL)Danijar Hafner, Marc G. BellemareChelsea FinnAI for Social GoodErnest MwebazeSafe Machine Learning Specification, Robustness and AssuranceNicholas CarliniRepresentation Learning on Graphs and ManifoldsBryan Perozzi

Announcing Google-Landmarks-v2: An Improved Dataset for Landmark Recognition & Retrieval

Friday, May 3, 2019

Posted by Bingyi Cao and Tobias Weyand, Software Engineers, Google AIGoogle-Landmarksinstance-level recognitionimage retrievalKaggle challengesLandmark Recognition 2018Landmark Retrieval 2018machine learningGoogle-Landmarks-v2Landmark Recognition 2019Landmark Retrieval 2019source code and modelDetect-to-Retrieve

Heatmap of the landmark locations in Google-Landmarks-v2, which demonstrates the increase in the scale of the dataset and the improved geographic coverage compared to last year’s dataset.

Creating the Dataset

Selection of images from Google-Landmarks-v2. Landmarks include (left to right, top to bottom) Neuschwanstein Castle, Golden Gate Bridge, Kiyomizu-dera, Burj khalifa, Great Sphinx of Giza, and Machu Picchu.

Wikimedia CommonsWiki Loves Monumentsinstance recognitionThe Kaggle ChallengesLandmark Recognition 2019Landmark Retrieval 2019Second Landmark Recognition WorkshopCVPR 2019Open Sourcing our Modelopen-source codeDetect-to-RetrieveCVPR 2019bounding boxeshereLandmark Recognition 2019Landmark Retrieval 2019Second Landmark Recognition WorkshopCVPR 2019instance recognitionimage retrievalCommon Visual Data FoundationAcknowledgmentsThe core contributors to this project are Andre Araujo, Bingyi Cao, Jack Sim and Tobias Weyand. We would like to thank our team members Daniel Kim, Emily Manoogian, Nicole Maffeo, and Hartwig Adam for their kind help. Thanks also to Marvin Teichmann and Menglong Zhu for their contribution to collecting the landmark bounding boxes and developing the Detect-to-Retrieve technique. We would like to thank Will Cukierski and Maggie Demkin for their help organizing the Kaggle challenge, Elan Hourticolon-Retzler, Yuan Gao, Qin Guo, Gang Huang, Yan Wang, Zhicheng Zheng for their help with data collection, Tsung-Yi Lin for his support with CVDF hosting, as well as our CVPR workshop co-organizers Bohyung Han, Shih-Fu Chang, Ondrej Chum, Torsten Sattler, Giorgos Tolias, and Xu Zhang. We have great appreciation for the Wikimedia Commons Community and their volunteer contributions to an invaluable photographic archive of the world’s cultural heritage. And finally, we’d like to thank the Common Visual Data Foundation for hosting the dataset.

Blog

Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction

Introducing Translatotron: An End-to-End Speech-to-Speech Translation Model

An End-to-End AutoML Solution for Tabular Data at KaggleDays

Announcing Open Images V5 and the ICCV 2019 Open Images Challenge

Google at ICLR 2019

Announcing Google-Landmarks-v2: An Improved Dataset for Landmark Recognition & Retrieval

Labels

Archive

Feed