Google AI Blog

Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction

Thursday, May 23, 2019

Posted by Tali Dekel, Research Scientist and Forrester Cole, Software Engineer, Machine Perceptioncomputer visiontriangulationGoogle’s Jump

Left: The traditional stereo setup assumes that at least two viewpoints capture the scene at the same time. Right: We consider the setup where both camera and subject are moving.

Learning the Depths of Moving People by Watching Frozen Peopledeep learning-baseddepth mapsaugmented reality

Our model predicts the depth map (right; brighter=closer to the camera) from a regular video (left), where both the people in the scene and the camera are freely moving.

Sourcing the Training Datapeople imitate mannequinsmulti-view-stereo

Videos of people imitating mannequins while a camera tours the scene, which we used for training. We use traditional MVS algorithms to estimate depth, which serves as supervision during training of our depth-prediction model.

Inferring the Depth of Moving Peopleand moving peoplemotion parallaxstatic objects2D optical flowhuman-segmentation network

Depth prediction network: The input to the model includes an RGB image (Frame t), a mask of the human region, and an initial depth for the non-human regions, computed from motion parallax (optical flow) between the input frame and another frame in the video. The model outputs a full depth map for Frame t. Supervision for training is provided by the depth map, computed by MVS.

Comparison of depth prediction models to a video clip with moving cameras and people. Top: Learning based monocular depth prediction methods (DORN; Chen et al.). Bottom: Learning based stereo method (DeMoN), and our result.

3D Video Effects Using Our Depth Maps

Bokeh video effect produced using our estimated depth maps. Video courtesy of Wind Walk Travel Videos.

AcknowledgementsThe research described in this post was done by Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu and Bill Freeman. We would like to thank Miki Rubinstein for his valuable feedback.

Introducing Translatotron: An End-to-End Speech-to-Speech Translation Model

Wednesday, May 15, 2019

Posted by Ye Jia and Ron Weiss, Software Engineers, Google AIpast several decadesautomatic speech recognitionmachine translationtext-to-speech synthesiscascadeGoogle TranslateDirect speech-to-speech translation with a sequence-to-sequence modelattentive sequence-to-sequence modelTranslatotronTranslatotronthe feasibilityoutperform cascade modelsleveraging weakly supervised dataspectrogramsvocodermultitask objective

Model architecture of Translatotron.

PerformanceBLEU score

Input (Spanish)

Reference translation (English)

Baseline cascade translation

Translatotron translation

herePreserving Vocal Characteristicsspeaker verificationspeaker adaptation for TTS

Input (Spanish)

Reference translation (English)

Baseline cascade translation

Translatotron translation (canonical voice)

Translatotron translation (original speaker’s voice)

hereConclusionAcknowledgmentsThis research was a joint work between the Google Brain, Google Translate, and Google Speech teams. Contributors include Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, Mengmeng Niu, Quan Wang, Jason Pelecanos, Ignacio Lopez Moreno, Tom Walters, Heiga Zen, Patrick Nguyen, Yu Zhang, Jonathan Shen, Orhan Firat, and Yonghui Wu. We also thank Jorge Pereira and Stella Laurenzo for verifying the quality of the translation from Translatotron.

An End-to-End AutoML Solution for Tabular Data at KaggleDays

Thursday, May 9, 2019

Posted by Yifeng Lu, Software Engineer, Google AIfeature engineeringhyper-parameter tuningneural architecture searchNasNetAmoebaNetMNasNet

Full automation: Data and computation resources are the only inputs, while a servable TensorFlow model is the output. The whole process requires no human intervention.

Extensive coverage: The solution is applicable to the majority of arbitrary tasks in the tabular data domain.

High quality: Models generated by AutoML has comparable quality to models manually crafted by top ML experts.

KaggleDays SF HackathonKaggleDays event Kaggle progression systemfinishing second place by a narrow marginfinal leaderboardarchitecture searchbootstrap aggregating

public kernel

Potential model quality improvement on final leaderboard if AutoML models were merged with other Kagglers’ models. “Erkut & Mark, Google AutoML”, includes the top winner “Erkut & Mark” and the second place “Google AutoML” models. Erkut Aykutlug and Mark Peng used XGBoost with creative feature engineering whereas AutoML uses both neural network and gradient boosting tree (TFBT) with automatic feature engineering and hyperparameter tuning.

Google Cloud AutoML TablesGoogle Cloud AutoML TablesGoogle Cloud Next ‘19.

Third party benchmark of AutoML Tables on multiple Kaggle competitions

AcknowledgementsThis project was only possible thanks to Google Brain team members Ming Chen, Da Huang, Yifeng Lu, Quoc V. Le and Vishy Tirumalashetty. We also thank Dawei Jia, Chenyu Zhao and Tin-yun Ho from the Cloud AutoML Tables team for great infrastructure and product landing collaboration. Thanks to Walter Reade, Julia Elliott and Kaggle for organizing such an engaging competition.

Announcing Open Images V5 and the ICCV 2019 Open Images Challenge

Wednesday, May 8, 2019

Posted by Vittorio Ferrari, Research Scientist, Machine PerceptionOpen ImagesupdatesOpen Images V4bounding-boxesOpen Images V5Open Images ChallengeOpen Images V5

Example masks on the training set of Open Images V5. These have been produced by our interactive segmentation process. The first example also shows a bounding box, for comparison. From left to right, top to bottom: Tea and cake at the Fitzwilliam Museum by Tim Regan, Pilota II by Euskal kultur erakundea Institut culturel basque, Rheas by Dag Peak, Wuxi science park, 1995 by Gary Stevens, Cat Cafe Shinjuku calico by Ari Helminen, and Untitled by Todd Huffman. All images used under CC BY 2.0 license.

segmentation masks on the training setinteractive segmentation processintersection-over-unionmasks on the validation and test sets

Example masks on the validation and test sets of Open Images V5, drawn completely manually. From left to right: thistle flowers by sophie, still life with ax by liz west, Fischkutter KOŁ-180 in Kolobrzeg (PL) by zeesenboot. All images used under CC BY 2.0 license.

Open Images Challenge 2019Open Images Challenge2019 International Conference on Computer Vision2018 editionavailable now

Google at ICLR 2019

Monday, May 6, 2019

Posted by Andrew Helton, Editor, Google AI CommunicationsICLR 2019machine learningdeep learningneural networksblueOfficers and Board MembersHugo LarochelleSamy BengioTara SainathGeneral ChairTara SainathWorkshop ChairsBeen Kim, Graham TaylorProgram Committee includes:Chelsea Finn, Dale Schuurmans, Dumitru Erhan, Katherine Heller, Lihong Li, Samy Bengio, Rohit Prabhavalkar, Alex Wiltschko, Slav Petrov, George DahlOral ContributionsGenerating High Fidelity Images with Subscale Pixel Networks and Multidimensional UpscalingJacob Menick, Nal KalchbrennerEnabling Factorized Piano Music Modeling and Generation with the MAESTRO DatasetCurtis Hawthorne, Andrew Stasyuk, Adam Roberts, Ian Simon, Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, Douglas EckMeta-Learning Update Rules for Unsupervised Representation LearningLuke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Sohl-DicksteinPostersA Data-Driven and Distributed Approach to Sparse Signal Representation and RecoveryAli Mousavi, Gautam Dasarathy, Richard G. BaraniukBayesian Deep Convolutional Networks with Many Channels are Gaussian ProcessesRoman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-DicksteinDiversity-Sensitive Conditional Generative Adversarial NetworksDingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak LeeDiversity and Depth in Per-Example Routing ModelsPrajit Ramachandran, Quoc V. LeEidetic 3D LSTM: A Model for Video Prediction and BeyondYunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-FeiGANSynth: Adversarial Neural Audio SynthesisJesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, Adam RobertsK for the Price of 1: Parameter-efficient Multi-task and Transfer LearningPramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, Andrew HowardLearning to Describe Scenes with ProgramsYunchao Liu, Zheng Wu, Daniel Ritchie, William Freeman, Joshua B Tenenbaum, Jiajun WuLearning to Infer and Execute 3D Shape ProgramsYonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William Freeman, Joshua B Tenenbaum, Jiajun WuThe Singular Values of Convolutional LayersHanie Sedghi, Vineet Gupta, Philip M. LongUnsupervised Discovery of Parts, Structure, and DynamicsZhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William Freeman, Joshua B Tenenbaum, Jiajun WuAdversarial Reprogramming of Neural NetworksGamaleldin Elsayed, Ian Goodfellow (no longer at Google), Jascha Sohl-DicksteinDiscriminator Rejection SamplingIan Goodfellow (no longer at Google), Jascha Sohl-DicksteinOn Self Modulation for Generative Adversarial NetworksTing Chen, Mario Lucic, Neil Houlsby, Sylvain GellyTowards GAN Benchmarks Which Require GeneralizationIshaan Gulrajani, Colin Raffel, Luke MetzUnderstanding and Improving Interpolation in Autoencoders via an Adversarial RegularizerDavid Berthelot, Colin Raffel, Aurko Roy, Ian Goodfellow (no longer at Google)A new dog learns old tricks: RL finds classic optimization algorithmsWeiwei Kong, Christopher Liaw, Aranyak Mehta, D. SivakumarContingency-Aware Exploration in Reinforcement LearningJongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, Honglak LeeDiscriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation LearningIlya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan TompsonDiversity is All You Need: Learning Skills without a Reward FunctionBenjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey LevineEpisodic Curiosity through ReachabilityNikolay Savinov, Anton Raichuk, Raphael Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain GellyLearning to Navigate the WebIzzeddin Gur, Ulrich Rueckert, Aleksandra Faust, Dilek Hakkani-TurMeta-Learning Probabilistic Inference for PredictionJonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. TurnerMulti-step Retriever-Reader Interaction for Scalable Open-domain Question AnsweringRajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Andrew McCallumNear-Optimal Representation Learning for Hierarchical Reinforcement LearningOfir Nachum, Shixiang Gu, Honglak Lee, Sergey LevineNeural Logic MachinesHonghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, Dengyong ZhouNeural Program Repair by Jointly Learning to Localize and RepairMarko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, Rishabh SinghOptimal Completion Distillation for Sequence LearningSara Sabour, William Chan, Mohammad NorouziRecall Traces: Backtracking Models for Efficient Reinforcement LearningAnirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua BengioSample Efficient Adaptive Text-to-Speech Yutian Chen, Yannis M Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Caglar Gulcehre, Aaron van den Oord, Oriol Vinyals, Nando de FreitasSynthetic Datasets for Neural Program SynthesisRichard Shin, Neel Kant, Kavi Gupta, Chris Bender, Brandon Trabucco, Rishabh Singh, Dawn SongThe Laplacian in RL: Learning Representations with Efficient ApproximationsYifan Wu, George Tucker, Ofir NachumA Mean Field Theory of Batch NormalizationGreg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S SchoenholzEfficient Training on Very Large Corpora via Gramian EstimationWalid Krichene, Nicolas Mayoraz, Steffen Rendle, Li Zhang, Xinyang Yi, Lichan Hong, Ed Chi, John AndersonPredicting the Generalization Gap in Deep Networks with Margin DistributionsYiding Jiang, Dilip Krishnan, Hossein Mobahi, Samy BengioInfoBot: Transfer and Exploration via the Information BottleneckAnirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, Yoshua BengioAntisymmetricRNN: A Dynamical System View on Recurrent Neural NetworksBo Chang, Minmin Chen, Eldad Haber, Ed H. ChiComplement Objective TrainingHao-Yun Chen, Pei-Hsin Wang, Chun-Hao Liu, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, Da-Cheng JuanDOM-Q-NET: Grounded RL on Structured LanguageSheng Jia, Jamie Kiros, Jimmy BaFrom Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction FollowingJustin Fu, Anoop Korattikara Balan, Sergey Levine, Sergio GuadarramaHarmonic Unpaired Image-to-image TranslationRui Zhang, Tomas Pfister, Li-Jia LiHierarchical Generative Modeling for Controllable Speech SynthesisWei-Ning Hsu, Yu Zhang, Ron Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, Patrick Nguyen, Ruoming PangLearning Finite State Representations of Recurrent Policy NetworksAnurag Koul, Alan Fern, Samuel GreydanusLearning to Screen for Fast Softmax Inference on Large Vocabulary Neural NetworksPatrick Chen, Si Si, Sanjiv Kumar, Yang Li, Cho-Jui HsiehMusic Transformer: Generating Music with Long-Term StructureChen-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew Dai, Matthew D Hoffman, Monica Dinculescu, Douglas EckUniversal TransformersMostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz KaiserWhat do you learn from context? Probing for sentence structure in contextualized word representationsIan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Tom McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, Ellie PavlickDoubly Reparameterized Gradient Estimators for Monte Carlo ObjectivesGeorge Tucker, Dieterich Lawson, Shixiang Gu, Chris J. MaddisonHow Important Is a Neuron?Kedar Dhamdhere, Mukund Sundararajan, Qiqi YanInteger Networks for Data Compression with Latent-Variable ModelsJohannes Ballé, Nick Johnston, David MinnenModeling Uncertainty with Hedged Instance EmbeddingsSeong Joon Oh, Andrew Gallagher, Kevin Murphy, Florian Schroff, Jiyan Pan, Joseph RothPreventing Posterior Collapse with delta-VAEsAli Razavi, Aaron van den Oord, Ben Poole, Oriol VinyalsSpectral Inference Networks: Unifying Deep and Spectral LearningDavid Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, Kimberly L StachenfeldStochastic Prediction of Multi-Agent Interactions from Partial ObservationsChen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, Kevin MurphyWorkshopsLearning from Limited Labeled DataDeep Reinforcement Learning Meets Structured PredictionChen LiangMohammad NorouziDebugging Machine Learning ModelsD. SculleyDan MoldovanStructure & Priors in Reinforcement Learning (SPiRL)Chelsea FinnTask-Agnostic Reinforcement Learning (TARL)Danijar Hafner, Marc G. BellemareChelsea FinnAI for Social GoodErnest MwebazeSafe Machine Learning Specification, Robustness and AssuranceNicholas CarliniRepresentation Learning on Graphs and ManifoldsBryan Perozzi

Announcing Google-Landmarks-v2: An Improved Dataset for Landmark Recognition & Retrieval

Friday, May 3, 2019

Posted by Bingyi Cao and Tobias Weyand, Software Engineers, Google AIGoogle-Landmarksinstance-level recognitionimage retrievalKaggle challengesLandmark Recognition 2018Landmark Retrieval 2018machine learningGoogle-Landmarks-v2Landmark Recognition 2019Landmark Retrieval 2019source code and modelDetect-to-Retrieve

Heatmap of the landmark locations in Google-Landmarks-v2, which demonstrates the increase in the scale of the dataset and the improved geographic coverage compared to last year’s dataset.

Creating the Dataset

Selection of images from Google-Landmarks-v2. Landmarks include (left to right, top to bottom) Neuschwanstein Castle, Golden Gate Bridge, Kiyomizu-dera, Burj khalifa, Great Sphinx of Giza, and Machu Picchu.

Wikimedia CommonsWiki Loves Monumentsinstance recognitionThe Kaggle ChallengesLandmark Recognition 2019Landmark Retrieval 2019Second Landmark Recognition WorkshopCVPR 2019Open Sourcing our Modelopen-source codeDetect-to-RetrieveCVPR 2019bounding boxeshereLandmark Recognition 2019Landmark Retrieval 2019Second Landmark Recognition WorkshopCVPR 2019instance recognitionimage retrievalCommon Visual Data FoundationAcknowledgmentsThe core contributors to this project are Andre Araujo, Bingyi Cao, Jack Sim and Tobias Weyand. We would like to thank our team members Daniel Kim, Emily Manoogian, Nicole Maffeo, and Hartwig Adam for their kind help. Thanks also to Marvin Teichmann and Menglong Zhu for their contribution to collecting the landmark bounding boxes and developing the Detect-to-Retrieve technique. We would like to thank Will Cukierski and Maggie Demkin for their help organizing the Kaggle challenge, Elan Hourticolon-Retzler, Yuan Gao, Qin Guo, Gang Huang, Yan Wang, Zhicheng Zheng for their help with data collection, Tsung-Yi Lin for his support with CVDF hosting, as well as our CVPR workshop co-organizers Bohyung Han, Shih-Fu Chang, Ondrej Chum, Torsten Sattler, Giorgos Tolias, and Xu Zhang. We have great appreciation for the Wikimedia Commons Community and their volunteer contributions to an invaluable photographic archive of the world’s cultural heritage. And finally, we’d like to thank the Common Visual Data Foundation for hosting the dataset.

Announcing the 6th Fine-Grained Visual Categorization Workshop

Monday, April 29, 2019

Posted by Christine Kaeser-Chen, Software Engineer and Serge Belongie, Visiting Faculty, Google AIiNaturalistiMaterialistindigo buntinglazuli buntingmorethan500FGVC5 at CVPR 2018domain-specific transfer learningestimating test-time priors6th annual workshop on Fine-Grained Visual CategorizationCVPR 2019This Year’s Challengesupdated iNaturalist challengefashionproductswildlife camera trapsfoodbutterflies & mothsfashion designcassava leaf diseaseThe Metropolitan Museum of ArtiMet Collection challengeNew York Botanical GardenHerbarium challenge

The FGVC workshop at CVPR focuses on subordinate categories, including (from left to right, top to bottom) animal species from wildlife camera traps, retail products, fashion attributes, cassava leaf disease, Melastomataceae species from herbarium sheets, animal species from citizen science photos, butterfly and moth species, cuisine of dishes, and fine-grained attributes for museum art objects.

iMet Collection challenge

A diverse sample of images included in the iMet Collection challenge dataset. Images were taken from the Metropolitan Museum of Art’s public domain dataset.

Kernels-only competition Herbarium challengeSteere Herbarium

In the Herbarium challenge, participants will identify species from the flowering plant family Melastomataceae. The New York Botanical Garden (NYBG) provided a dataset of over 46,000 herbarium specimens including over 680 species. Images used with permission of the NYBG.

Kaggle leaderboardsSeek by iNaturalist Merlin Bird IDInvitation to ParticipateAcknowledgementsWe’d like to thank our colleagues and friends on the FGVC6 organizing committee for working together to advance this important area. At Google we would like to thank Hartwig Adam, Chenyang Zhang, Yulong Liu, Kiat Chuan Tan, Mikhail Sirotenko, Denis Brulé, Cédric Deltheil, Timnit Gebru, Ernest Mwebaze, Weijun Wang, Grace Chu, Jack Sim, Andrew Howard, R.V. Guha, Srikanth Belwadi, Tanya Birch, Katherine Chou, Maggie Demkin, Elizabeth Park, and Will Cukierski.

Evaluating the Unsupervised Learning of Disentangled Representations

Wednesday, April 24, 2019

Posted by Olivier Bachem, Research Scientist, Google AI Zürichunsuperviseddeep learningdisentangled representationscuriosity driven explorationabstract reasoningvisual concept learningdomain adaptation for reinforcement learningChallenging Common Assumptions in the Unsupervised Learning of Disentangled RepresentationsICML 2019disentanglement_libUnderstanding DisentanglementShapes3D

Visualization of the ground-truth factors of the Shapes3D data set: Floor color (upper left), wall color (upper middle), object color (upper right), object size (bottom left), object shape (bottom middle), and camera angle (bottom right).

Visualization of the latent dimensions learned by a FactorVAE model (see below). The ground-truth factors wall and floor color as well as rotation of the camera are disentangled (see top right, top center and bottom center panels), while the ground-truth factors object shape, size and color are entangled (see top left and the two bottom left images).

Key Results of this Reproducible Large-scale Studyvariational autoencodersBetaVAEAnnealedVAEFactorVAEDIP-VAE I/IIBeta-TCVAEBetaVAE scoreFactorVAE scoreMIGSAPModularityDCI Disentanglement

We do not find any empirical evidence that the considered models can be used to reliably learn disentangled representations in an unsupervised way, since random seeds and hyperparameters seem to matter more than the model choice. In other words, even if one trains a large number of models and some of them are disentangled, these disentangled representations seemingly cannot be identified without access to ground-truth labels. Furthermore, good hyperparameter values do not appear to consistently transfer across the data sets in our study. These results are consistent with the theorem we present in the paper, which states that the unsupervised learning of disentangled representations is impossible without inductive biases on both the data set and the models (i.e., one has to make assumptions about the data set and incorporate those assumptions into the model).

For the considered models and data sets, we cannot validate the assumption that disentanglement is useful for downstream tasks, e.g., that with disentangled representations it is possible to learn with fewer labeled observations.

The violin plots show the distribution of FactorVAE scores attained by different models on the Cars3D data set. The left plot shows how the distribution changes as different disentanglement models are considered while the right plot displays the different distributions as the regularization strength in a FactorVAE model is varied. The key observation is that the violin plots substantially overlap which indicates that all methods strongly depend on the random seed.

Given the theoretical result that the unsupervised learning of disentangled representations without inductive biases is impossible, future work should clearly describe the imposed inductive biases and the role of both implicit and explicit supervision.

Finding good inductive biases for unsupervised model selection that work across multiple data sets persists as a key open problem.

The concrete practical benefits of enforcing a specific notion of disentanglement of the learned representations should be demonstrated. Promising directions include robotics, abstract reasoning and fairness.

Experiments should be conducted in a reproducible experimental setup on a diverse selection of data sets.

Open Sourcing disentanglement_libdisentanglement_libdisentanglement_libdisentanglement_lib>10,000 pretrained disentanglement_lib modelsdisentanglement_libAcknowledgmentsThis research was done in collaboration with Francesco Locatello, Mario Lucic, Stefan Bauer, Gunnar Rätsch, Sylvain Gelly and Bernhard Schölkopf at Google AI Zürich, ETH Zürich and the Max-Planck Institute for Intelligent Systems. We also wish to thank Josip Djolonga, Ilya Tolstikhin, Michael Tschannen, Sjoerd van Steenkiste, Joan Puigcerver, Marcin Michalski, Marvin Ritter, Irina Higgins and the rest of the Google Brain team for helpful discussions, comments, technical help and code contributions.

SpecAugment: A New Data Augmentation Method for Automatic Speech Recognition

Monday, April 22, 2019

Posted by Daniel S. Park, AI Resident and William Chan, Research Scientistthe ongoing development of deep neural networksdata augmentationimage classificationSpecAugment: A Simple Data Augmentation Method for Automatic Speech RecognitionLibriSpeech 960hSwitchboard 300hSpecAugmentspectrogram

A waveform is typically converted into a visual representation (in our case, a log mel spectrogram; steps 1 through 3 of this article) before being fed into a network.

warping

The log mel spectrogram is augmented by warping in the time direction, and masking (multiple) blocks of consecutive time steps (vertical masks) and mel frequency channels (horizontal masks). The masked portion of the spectrogram is displayed in purple for emphasis.

Listen Attend and Spell (LAS) networksWord Error Rate

Performance of networks on the test sets of LibriSpeech with and without augmentation. The LibriSpeech test set is divided into two portions, test-clean and test-other, the latter of which contains noisier audio data.

Training, clean (dev-clean) and noisy (dev-other) development set performance with and without augmentation.

State-of-the-Art ResultsLibriSpeech 960hSwitchboard 300h

Word error rates (%) for state-of-the-art results for the tasks LibriSpeech 960h and Switchboard 300h. The test set for both tasks have a clean (clean/Switchboard) and a noisy (other/CallHome) subset. Previous SOTA results taken from Li et. al (2019), Yang et. al (2018) and Zeyer et. al (2018).

Performance of various classes of networks on LibriSpeech and Switchboard tasks. The performance of LAS models is compared to classical (e.g., HMM) and other end-to-end models (e.g., CTC/ASG) over time.

Language ModelsLanguage models

Word error rates for LibriSpeech and Switchboard tasks with and without LMs. SpecAugment outperforms previous state-of-the-art even before the inclusion of a language model.

AcknowledgementsWe would like to thank the co-authors of our paper Chung-Cheng Chiu, Ekin Dogus Cubuk, Quoc Le, Yu Zhang and Barret Zoph. We also thank Yuan Cao, Ciprian Chelba, Kazuki Irie, Ye Jia, Anjuli Kannan, Patrick Nguyen, Vijay Peddinti, Rohit Prabhavalkar, Yonghui Wu and Shuyuan Zhang for useful discussions.