Google AI Blog

Multilingual Universal Sentence Encoder for Semantic Retrieval

Friday, July 12, 2019

Posted by Yinfei Yang and Amin Ahmad, Software Engineers, Google Researchintroduced last yearUniversal Sentence Encoder (USE) for EnglishTensorflow Hubsentiment classifierthree new USE multilingual modulesquestion-answer retrievalmulti-task dual-encoder frameworkdual-encoder with additive margin softmax approach

Multi-task training structure of the Universal Sentence Encoder. A variety of tasks and task structures are joined by shared encoder layers/parameters (pink boxes).

Semantic Retrieval Applicationsnearest neighbor problemprecision and recallapproximate

A prototypical semantic retrieval pipeline, used for textual similarity.

Semantic Similarity Modules

Multilingual Semantic Textual Similarity Retrieval
Most existing approaches for finding semantically similar text require being given a pair of texts to compare. However, using the Universal Sentence Encoder, semantically similar text can be extracted directly from a very large database. For example, in an application like FAQ search, a system can first index all possible questions with associated answers. Then, given a user’s question, the system can search for known questions that are semantically similar enough to provide an answer. A similar approach was used to find comparable sentences from 50 million sentences in wikipedia. With the new multilingual USE models, this can be done in any of supported non-English languages.

Multilingual Translation Pair Retrieval
The newly released modules can also be used to mine translation pairs to train neural machine translation systems. Given a source sentence in one language (“How do I get to the restroom?”), they can find the potential translation target in any other supported language (“¿Cómo llego al baño?”).

Chidambaram et al. (2018)USE for Question-Answer Retrievalquestion-answer retrieval applications

Visualizing the action of a neural answer retrieval system. The blue point at the north pole represents the question vector. The other points represent the embeddings of various answers. The correct answer, highlighted here in red, is “closest” to the question, in that it minimizes the angular distance. The points in this diagram are produced by an actual USE-QA model, however, they have been projected downwards from ℝ500 to ℝ3 to assist the reader’s visualization.

Google Talk to BooksWhat fragrance brings back memories?And for me, the smell of jasmine along with the pan bagnat, it brings back my entire carefree childhood.fragrancesmellFor Researchers and Developerssemantic similarity for natural languageAcknowledgementsMandy Guo, Daniel Cer, Noah Constant, Jax Law, Muthuraman Chidambaram for core modeling, Gustavo Hernandez Abrego, Chen Chen, Mario Guajardo-Cespedes for infrastructure and colabs, Steve Yuan, Chris Tar, Yunhsuan Sung, Brian Strope, Ray Kurzweil for discussion of the model architecture.

Advancing Semi-supervised Learning with Unsupervised Data Augmentation

Wednesday, July 10, 2019

Posted by Qizhe Xie, Student Researcher and Thang Luong, Senior Research Scientist, Google Research, Brain TeamGPUTPUImageNetnatural language processingvisionspeechsupervised learning

Example augmentation operations for text-based (top) or image-based (bottom) training data.

Unsupervised Data Augmentation (UDA) for Consistency Trainingsemi-supervised learningrevival of semi-supervised learningBERTgithubUnsupervised Data Augmentation Explainedloss functionconsistency training

An overview of Unsupervised Data Augmentation (UDA). Left: Standard supervised loss is computed when labeled data is available. Right: With unlabeled data, a consistency loss is computed between an example and its augmented version.

Gaussian noiseadversarial noiseback translationAutoAugmentTF-IDFBenchmarks in NLP and Computer VisionIMDb

Benchmark on IMDb, a sentiment analysis task. UDA surpasses state-of-the-art results in supervised learning across different training sizes.

CIFAR-10VATICTPyramidNet+ShakeDropSVHN

SSL benchmark on CIFAR-10, an image classification task. UDA surpases existing semi-supervised learning methods, all of which use the Wide-ResNet-28-2 architecture. UDA applies the augmentation policy found by AutoAugment using 4,000 examples. At this sample size, it matches the performance of the fully supervised setting with 50,000 examples. The same augmentation policy is applied in cases with fewer labeled examples.

ImageNetReleasecodebaseAcknowledgementsSpecial thanks to the co-authors of the paper Zihang Dai, Eduard Hovy, and Quoc V. Le. We’d also like to thank Hieu Pham, Adams Wei Yu, Zhilin Yang, Colin Raffel, Olga Wichrowska, Ekin Dogus Cubuk, Guokun Lai, Jiateng Xie, Yulun Du, Trieu Trinh, Ran Zhao, Ola Spyra, Brandon Yang, Daiyi Peng, Andrew Dai, Samy Bengio and Jeff Dean for their help with this project. A preprint is available online.

Predicting the Generalization Gap in Deep Neural Networks

Tuesday, July 9, 2019

Posted by Yiding Jiang, Google AI ResidentDeepneural networksimage recognitionimage segmentationmachine translationVC-dimensionRademacher complexitygeneralize poorlyrecent workmassivelygeneralization gapmargindecision boundarysupport-vector machinesmargintheoretical upper bounds

An example of a support-vector machine decision boundary. The hyperplane defined by w∙x-b=0 is the "decision boundary" of this linear classifier, i.e., every point x lying on the hyperplane is equally likely to be in either class under this classifier.

ICLR 2019Predicting the Generalization Gap in Deep Networks with Margin Distributionsnormalized margingeneralization gapGithub repository

Each plot corresponds to a convolutional neural network trained on CIFAR-10 with different classification accuracies. The probability density (y-axis) of normalized margin distributions (x-axis) at 4 layers of a network is shown for three different models with increasingly better generalization (left to right). The normalized margin distributions are strongly correlated with test accuracy, which suggests they can be used as a proxy for predicting a network's generalization gap. Please see our paper for more details on these networks.

Margin Distributions as a Predictor of Generalizationlinear regression

Predicted generalization gap (x-axis) vs. true generalization gap (y-axis) on CIFAR-100 + ResNet-32. The points lie close to the diagonal line, which indicates that the predicted values of the log linear model fit the true generalization gap very well.

The Deep Model Generalization DatasetDeep Model GeneralizationNetwork-in-NetworkregularizationGithub repository

Announcing the YouTube-8M Segments Dataset

Friday, June 28, 2019

Posted by Joonseok Lee and Joe Yue-Hei Ng, Software Engineers, Google ResearchFirstSecond YouTube-8M Large-Scale Video Understanding Challenge and WorkshopYouTube-8M SegmentsKaggle video understanding challenge3rd Workshop on YouTube-8M Large-Scale Video Understanding2019 International Conference on Computer Vision

YouTube-8M Segmentscapturing special video momentsYouTube-8M releaseThe 3rd YouTube-8M Video Understanding ChallengeKaggle competition pageThe 3rd Workshop on YouTube-8M Large-Scale Video Understandingworkshop pageAcknowledgementsThis post reflects the work of many machine perception researchers including Ke Chen, Nisarg Kothari, Joonseok Lee, Hanhan Li, Paul Natsev, Joe Yue-Hei Ng, Naderi Parizi, David Ross, Cordelia Schmid, Javier Snaider, Rahul Sukthankar, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan, Yexin Wang, Zheng Xu, as well as Julia Elliott and Walter Reade from Kaggle. We are also grateful for the support and advice from our partners at YouTube.

Predicting Bus Delays with Machine Learning

Thursday, June 27, 2019

Posted by Alex Fabrikant, Research Scientist, Google Researchintroduced live traffic delays for busesfirst launched in India three weeks agoThe Beginnings of a Model

speech processingmachine translation

To model a bus trip (a) starting at the blue stop, the model (b) adds up the delay predictions from timeline units for the blue stop, the three road segments, the white stop, etc.

Modeling the "Where"

popular times at businessesparking difficultyLearning the Local Rhythms

The model's time representation (3 out of 4 dimensions) forms a loop, reimagined here as the circumference of a watch. The more location-dependent time windows like 4pm-9pm and 7am-9am get more complex "crumpling", while big featureless windows like 2am-5am get bent away with flat bends for simpler rules. (Artist's conception by Will Cassella, using textures from textures.com and HDRIs from hdrihaven.)

Putting it All Together

The Trip AheadAcknowledgementsThis work was the joint effort of James Cook, Alex Fabrikant, Ivan Kuznetsov, and Fangzhou Xu, on Google Research, and Anthony Bertuca, Julian Gibbons, Thierry Le Boulengé, Cayden Meyer, Anatoli Plotnikov, and Ivan Volosyuk on Google Maps. We thank Senaka Buthpitiya, Da-Cheng Juan, Reuben Kan, Ramesh Nagarajan, Andrew Tomkins, and the greater Transit team for support and helpful discussions; as well as Will Cassella for the inspired reimagining of the model's time embedding. We are also indebted to our partner agencies for providing the transit data feeds the system is trained on.

Innovations in Graph Representation Learning

Tuesday, June 25, 2019

Posted by Alessandro Epasto, Senior Research Scientist and Bryan Perozzi, Senior Research Scientist, Graph Mining Teamgraph nodesedgesfriendship relationsgraph analysisclusteringlink predictionprivacyotherslearn. graph embeddingDeepWalk

Left: The well-known Karate graph representing a social network. Right: A continuous space embedding of the nodes in the graph using DeepWalk.

Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social ContextsWWW’19Watch Your Step: Learning Node Embeddings via Graph AttentionNeurIPS’18graph embeddingsLearning Node Representations that Capture Multiple Social Contextssinglesingleoverlappingmultiple WWW’19 paperSplitterego-network analysispersona graph conceptGPpersona graphGpersona

Ego-net of node U

personas

The ego-splitting method separating the U nodes in 2 personas.

up to 90% reduction

Top Left: A typical graphs with highly overlapping communities. Top Right: A traditional embedding of the graph on the left using node2vec. Bottom Left: A persona graph of the graph above. Bottom Right: The Splitter embedding of the persona graph. Notice how the persona graph clearly disentangles the overlapping communities of the original graph and Splitter outputs well-separated embeddings.

Automatic hyper-parameter tuning via graph attention.link prediction and node classificationsecond paper.DeepWalkbackpropagation

Our new method for automatic hyper-parameter tuning, Watch Your Step, uses an attention model to learn different graph context distributions. Shown above are two example local neighborhoods about a center node (in yellow) and the context distributions (red gradient) that was learned by the model. The left-side graph shows a more diffused attention model, while the distribution on the right shows one concentrated on direct neighbors.

AutoMLneural architecture searchcodeAcknowledgementsWe thank Sami Abu-el-Haija who contributed to this work and is now a Ph.D. student at USC. For more information on the Graph Mining team (part of Algorithm and Optimization), visit our pages.

Off-Policy Classification - A New Reinforcement Learning Model Selection Method

Wednesday, June 19, 2019

Reinforcement learningoff-policy RLwalkinggraspingfully off-policy RLentirelytrainingevaluationAutoMLff-policy evaluation

A diagram for real-world model development. Assuming we can evaluate 10 models per day, without off-policy evaluation, we would need 100x as many days to evaluate our models.

Off-Policy Evaluation via Off-Policy Classificationoff-policy classificationimportance samplingHow OPC WorkseffectivecatastrophicQ-functionQ-learningpositive-unlabeled learningOff-Policy Evaluation for Sim-to-Real Learningsimulated datatransfer learning techniquessample complexity

An example of how simulated experience can differ from real-world experience. Here, simulated images (left) have much less visual complexity than real-world images (right).

Results

An experiment in the simulated grasping task. The red curve is the dimensionless SoftOPC score over the course of training, evaluated from old data. The blue curve is the grasp success rate in simulation. We see the SoftOPC on old data correlates well with grasp success of the model within our simulator.

SoftOPC score and true performance for 3 different sim-to-real methods: a baseline simulation, a simulation with random textures and lighting, and a model trained with RCAN. All three models are trained with no real data, then evaluated with off-policy evaluation on a validation set of real data. The ordering of the SoftOPC score matches the order of real grasp success.

temporal-difference error

Results from our sim-to-real evaluation experiment. On the left is a baseline, the temporal difference error of the model. On the right is one of our proposed methods, the SoftOPC. The shaded region is a 95% confidence interval. The correlation is significantly better with SoftOPC.

Future WorkAcknowledgementsThis research was conducted by Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz and Sergey Levine. We’d like to thank Razvan Pascanu, Dale Schuurmans, George Tucker and Paul Wohlhart for valuable discussions. A preprint is available on arXiv.

Google at CVPR 2019

Monday, June 17, 2019

Posted by Andrew Helton, Editor, Google AI Communications2019 Conference on Computer Vision and Pattern Recognitionworkshopstutorialsmachine perceptionpredicting pedestrian motionOpen Images V5 datasetblueArea Chairs Jonathan T. Barron, William T. Freeman, Ce Liu, Michael Ryoo, Noah SnavelyOral PresentationsRelational Action ForecastingChen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy, Cordelia SchmidPushing the Boundaries of View Extrapolation With Multiplane ImagesPratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, Noah SnavelyAuto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image SegmentationChenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, Li Fei-FeiAutoAugment: Learning Augmentation Strategies From DataEkin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, Quoc V. LeDeepView: View Synthesis With Learned Gradient DescentJohn Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, Richard TuckerNormalized Object Coordinate Space for Category-Level 6D Object Pose and Size EstimationHe Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, Leonidas J. GuibasDo Better ImageNet Models Transfer Better?Simon Kornblith, Jonathon Shlens, Quoc V. LeTextureNet: Consistent Local Parametrizations for Learning From High-Resolution Signals on MeshesJingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Niessner, Leonidas J. GuibasDiverse Generation for Multi-Agent Sports GamesRaymond A. Yeh, Alexander G. Schwing, Jonathan Huang, Kevin MurphyOccupancy Networks: Learning 3D Reconstruction in Function SpaceLars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas GeigerA General and Adaptive Robust Loss FunctionJonathan T. BarronLearning the Depths of Moving People by Watching Frozen PeopleZhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, William T. Freeman(CVPR 2019 Best Paper Honorable Mention)Composing Text and Image for Image Retrieval - an Empirical OdysseyNam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James HaysLearning to Synthesize Motion BlurTim Brooks, Jonathan T. BarronNeural Rerendering in the WildMoustafa Meshry, Dan B. Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-BruallaNeural Illumination: Lighting Prediction for Indoor EnvironmentsShuran Song, Thomas FunkhouserUnprocessing Images for Learned Raw DenoisingTim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. BarronPostersCo-Occurrent Features in Semantic SegmentationHang Zhang, Han Zhang, Chenguang Wang, Junyuan XieCrDoCo: Pixel-Level Domain Transfer With Cross-Domain ConsistencyYun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, Jia-Bin HuangIm2Pencil: Controllable Pencil Illustration From PhotographsYijun Li, Chen Fang, Aaron Hertzmann, Eli Shechtman, Ming-Hsuan YangMode Seeking Generative Adversarial Networks for Diverse Image SynthesisQi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, Ming-Hsuan YangRevisiting Self-Supervised Visual Representation LearningAlexander Kolesnikov, Xiaohua Zhai, Lucas BeyerScene Graph Generation With External Knowledge and Image ReconstructionJiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, Mingyang LingScene Memory Transformer for Embodied Agents in Long-Horizon TasksKuan Fang, Alexander Toshev, Li Fei-Fei, Silvio SavareseSpatially Variant Linear Representation Models for Joint FilteringJinshan Pan, Jiangxin Dong, Jimmy S. Ren, Liang Lin, Jinhui Tang, Ming-Hsuan YangTarget-Aware Deep TrackingXin Li, Chao Ma, Baoyuan Wu, Zhenyu He, Ming-Hsuan YangTemporal Cycle-Consistency LearningDebidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew ZissermanDepth-Aware Video Frame InterpolationWenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan YangMnasNet: Platform-Aware Neural Architecture Search for MobileMingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. LeA Compact Embedding for Facial Expression SimilarityRaviteja Vemulapalli, Aseem AgarwalaContrastive Adaptation Network for Unsupervised Domain AdaptationGuoliang Kang, Lu Jiang, Yi Yang, Alexander G. HauptmannDeepLight: Learning Illumination for Unconstrained Mobile Mixed RealityChloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, Paul DebevecDetect-To-Retrieve: Efficient Regional Aggregation for Image SearchMarvin Teichmann, Andre Araujo, Menglong Zhu, Jack SimFast Object Class Labelling via SpeechMichael Gygli, Vittorio FerrariLearning Independent Object Motion From Unlabelled Stereoscopic VideosZhe Cao, Abhishek Kar, Christian Hane, Jitendra MalikPeeking Into the Future: Predicting Future Person Activities and Locations in VideosJunwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G. Hauptmann, Li Fei-FeiSpotTune: Transfer Learning Through Adaptive Fine-TuningYunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, Rogerio FerisNAS-FPN: Learning Scalable Feature Pyramid Architecture for Object DetectionGolnaz Ghiasi, Tsung-Yi Lin, Quoc V. LeClass-Balanced Loss Based on Effective Number of SamplesYin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge BelongieFEELVOS: Fast End-To-End Embedding Learning for Video Object SegmentationPaul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, Liang-Chieh ChenInserting Videos Into VideosDonghoon Lee, Tomas Pfister, Ming-Hsuan YangVolumetric Capture of Humans With a Single RGBD Camera via Semi-Parametric LearningRohit Pandey, Anastasia Tkach, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Ricardo Martin-Brualla, Andrea Tagliasacchi, George Papandreou, Philip Davidson, Cem Keskin, Shahram Izadi, Sean FanelloYou Look Twice: GaterNet for Dynamic Filter Selection in CNNsZhourong Chen, Yang Li, Samy Bengio, Si SiInteractive Full Image Segmentation by Considering All Regions JointlyEirikur Agustsson, Jasper R. R. Uijlings, Vittorio FerrariLarge-Scale Interactive Object Segmentation With Human AnnotatorsRodrigo Benenson, Stefan Popov, Vittorio FerrariSelf-Supervised GANs via Auxiliary Rotation LossTing Chen, Xiaohua Zhai, Marvin Ritter, Mario Lučić, Neil HoulsbySim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation NetworksStephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, Konstantinos BousmalisUsing Unknown Occluders to Recover Hidden ScenesAdam B. Yedidia, Manel Baradad, Christos Thrampoulidis, William T. Freeman, Gregory W. WornellWorkshopsComputer Vision for Global ChallengesTimnit Gebru, Ernest Mwebaze, John QuinnDeep Vision 2019Pierre Sermanet, Chris BreglerLandmark RecognitionAndre Araujo, Bingyi Cao, Jack Sim, Tobias WeyandImage Matching: Local Features and BeyondEduard Trulls3D-WiDGET: Deep GEneraTive Models for 3D UnderstandingJulien ValentinFine-Grained Visual CategorizationChristine Kaeser-ChenHartwig AdamLow-Power Image Recognition Challenge (LPIRC)Aakanksha Chowdhery, Achille Brighton, Alec Go, Andrew Howard, Bo Chen, Jaeyoun Kim, Jeff GilbertNew Trends in Image Restoration and Enhancement Workshop and Associated ChallengesVivek Kwatra, Peyman Milanfar, Sebastian Nowozin, George Toderici, Ming-Hsuan YangSpatio-temporal Action Recognition (AVA) @ ActivityNet ChallengeDavid Ross, Sourish Chaudhuri, Radhika Marvin, Arkadiusz Stopczynski, Joseph Roth, Caroline Pantofaru, Chen Sun, Cordelia SchmidThird Workshop on Computer Vision for AR/VRSofien Bouaziz, Serge BelongieDAVIS Challenge on Video Object SegmentationJordi Pont-Tuset, Alberto MontesEfficient Deep Learning for Computer VisionAndrew HowardFairness Accountability Transparency and Ethics in Computer VisionTimnit Gebru, Margaret MitchellPrecognition Seeing through the FutureUtsav PrabhuWorkshop and Challenge on Learned Image CompressionGeorge Toderici, Michele Covell, Johannes Ballé, Eirikur Agustsson, Nick JohnstonWhen Blockchain Meets Computer Vision & AIChris BreglerApplications of Computer Vision and Pattern Recognition to Media ForensicsPaul Natsev, Christoph BreglerTutorialsTowards Relightable Volumetric Performance Capture of HumansSean Fanello, Christoph Rhemann, Graham Fyffe, Jonathan Taylor, Sofien Bouaziz, Paul Debevec, Shahram IzadiLearning Representations via Graph-structured NetworksMing-Hsuan Yang

Blog