Google AI Blog

Google at NAACL

Friday, June 8, 2018

Posted by Kenton Lee, Research Scientist and Slav Petrov, Principal Scientist, Language Team, Google AINorth American Association of Computational LinguisticsWidening Natural Language Processing Workshop

Googler Margaret Mitchell setting up our telepresence robots for remote presenters Diana Gonazelez and Gibran Fuentes Pineda to remotely present their first-place work on visual storytelling from Universidad Nacional Autonoma de Mexico.

Test of Time AwardblueBLEU: a Method for Automatic Evaluation of Machine TranslationKishore Papineni, Salim Roukos, Todd Ward, Wei-Jing ZhuDiscriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron AlgorithmsMichael CollinsThumbs up?: Sentiment Classification using Machine Learning TechniquesBo Pang, Lillian Lee, Shivakumar VaithyanathanblueGoogle AI Language Team pageArea Chairs include:Dan Bikel, Dilek Hakkani-Tur, Zornitsa Kozareva, Marius Pasca, Emily Pitler, Idan Szpektor, Taro WatanabePublications Co-ChairMargaret MitchellKeynoteGoogle Assistant or My Assistant? Towards Personalized Situated Conversational AgentsDilek Hakkani-TürPublicationsBootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement LearningPararth Shah, Dilek Hakkani-Tür, Bing Liu, Gokhan TürSHAPED: Shared-Private Encoder-Decoder for Text Style AdaptationYe Zhang, Nan Ding, Radu SoricutOlive Oil is Made of Olives, Baby Oil is Made for Babies: Interpreting Noun Compounds Using Paraphrases in a Neural ModelVered Schwartz, Chris WatersonAre All Languages Equally Hard to Language-Model?Ryan Cotterell, Sebastian J. Mielke, Jason Eisner, Brian RoarkSelf-Attention with Relative Position RepresentationsPeter Shaw, Jakob Uszkoreit, Ashish VaswaniDialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue SystemsBing Liu, Gokhan Tür, Dilek Hakkani-Tür, Parath Shah, Larry HeckWorkshopsSubword & Character Level Models in NLPManaal Faruqui, Hinrich Schütze, Isabel Trancoso, Yulia Tsvetkov, Yadollah YaghoobzadehStorytelling WorkshopMargaret Mitchell, Ishan Misra, Ting-Hao 'Kenneth' Huang, Frank FerraroEthics in NLPMichael Strube, Dirk Hovy, Margaret Mitchell, Mark AlfanoNAACL HLT PanelsCareers in IndustryPhilip Resnik (moderator), Jason Baldridge, Laura Chiticariu, Marie Mateer, Dan RothEthics in NLPDirk Hovy (moderator), Margaret Mitchell, Vinodkumar Prabhakaran, Mark Yatskar, Barbara Plank

Realtime tSNE Visualizations with TensorFlow.js

Thursday, June 7, 2018

Posted by Nicola Pezzotti, Software Engineering Intern, Google Züricht-distributed Stochastic Neighbor EmbeddingTensorFlow Embedding ProjectorTensorBoardLinear tSNE Optimization for the WebWebGLreleasing this work as an open source libraryTensorFlow.js

Real-time evolution of the tSNE embedding for the complete MNIST dataset with our technique. The dataset contains images of 60,000 handwritten digits. You can find a live demo here.

objective functionN-body simulationBarnes-HutMNIST

Rendering of the three functions used to approximate the repulsive effect created by a single point. In the above figure the repulsive forces show a point in a blue area is pushed to the left/bottom, while a point in the red area is pushed to the right/top while a point in the white region will not move.

This animation shows the evolution of the tSNE embedding (upper left) and of the scalar fields used to approximate its gradient with normalization term (upper right), horizontal shift (bottom left) and vertical shift (bottom right).

TensorFlow.jsan open source libraryFuture WorkN-body simulationsAcknowledgementsWe would like to thank Alexander Mordvintsev, Yannick Assogba, Matt Sharifi, Anna Vilanova, Elmar Eisemann, Nikhil Thorat, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Alessio Bazzica, Boudewijn Lelieveldt, Thomas Höllt, Baldur van Lew, Julian Thijssen and Marvin Ritter.

Announcing an updated YouTube-8M, and the 2nd YouTube-8M Large-Scale Video Understanding Challenge and Workshop

Tuesday, June 5, 2018

Posted by Joonseok Lee, Software Engineer, Google AIYouTube-8M Large-Scale Video Understanding ChallengeKaggleYouTube-8M datasetworkshop at CVPR’17update to the YouTube-8M dataseta new Kaggle video understanding challenge2nd Workshop on YouTube-8M Large-Scale Video Understanding2018 European Conference on Computer Vision

An Updated YouTube-8M Dataset (2018 Edition)ground truthstarter codeTensorFlowThe 2nd YouTube-8M Video Understanding Challenge2nd YouTube-8M Video Understanding ChallengeKaggle competition pageThe 2nd Workshop on YouTube-8M Large-Scale Video UnderstandingECCV’18workshop pageAcknowledgementsThis post reflects the work of many machine perception researchers including Sami Abu-El-Haija, Ke Chen, Nisarg Kothari, Joonseok Lee, Hanhan Li, Paul Natsev, Sobhan Naderi Parizi, Rahul Sukthankar, George Toderici, Balakrishnan Varadarajan, as well as Sohier Dane, Julia Elliott, Wendy Kan and Walter Reade from Kaggle. We are also grateful for the support and advice from our partners at YouTube.

Improving Deep Learning Performance with AutoAugment

Monday, June 4, 2018

Posted by Ekin Dogus Cubuk, Google AI Resident and Barret Zoph, Research Scientist, Google Brain Teamdesign neural network architecturesoptimizersAutoAugment: Learning Augmentation Policies from Datareinforcement learningAugmenting Training Datamixup

Left: An original image from the ImageNet dataset. Right: The same image transformed by a commonly used data augmentation transformation, a horizontal flip about the center.

32street view of house numbers

Left: An original image from the SVHN dataset. Right: The same image transformed by AutoAugment. In this case, the optimal transformation was a result of shearing the image and inverting the colors of the pixels.

CIFAR-10ImageNet

Left: An original image from the ImageNet dataset. Right: The same image transformed by the AutoAugment policy. First, the image contrast is maximized, after which the image is rotated.

ResultsStanford CarsFGVC-Aircraftappendix of the paperAcknowledgementsSpecial thanks to the co-authors of the paper Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. We’d also like to thank Alok Aggarwal, Gabriel Bender, Yanping Huang, Pieter-Jan Kindermans, Simon Kornblith, Augustus Odena, Avital Oliver, and Colin Raffel for their help with this project.

Advances in Semantic Textual Similarity

Thursday, May 17, 2018

Posted by Yinfei Yang, Software Engineer and Chris Tar, Engineering Manager, Google AISmart ComposeTalk to BooksTensorFlow HubSemantic Textual SimilarityLearning Semantic Textual Similarity from Conversations

Sentences are semantically similar if they can be answered by the same responses. Otherwise, they are semantically different.

SNLIentailmentSTSBenchmarkCQA task B

For a given input, classification is considered a ranking problem against potential candidates.

Universal Sentence EncoderUniversal Sentence Encoderskip-thought

Pairwise semantic similarity comparison via outputs from TensorFlow Hub Universal Sentence Encoder.

deep average networkTransformer

Multi-task training as described in “Universal Sentence Encoder”. A variety of tasks and task structures are joined by shared encoder layers/parameters (grey boxes).

New Models modelnewTensorFlow HubUniversal Sentence Encoder - LargeUniversal Sentence Encoder - Lite

The Large model is trained with the Transformer encoder described in our second paper. It targets scenarios requiring high precision semantic representations and the best model performance at the cost of speed & size.

The Lite model is trained on a Sentence Piece vocabulary instead of words in order to significantly reduce the vocabulary size, which is a major contributor of model size. It targets scenarios where resources like memory and CPU are limited, such as on-device or browser based implementations.

AcknowledgementsDaniel Cer, Mario Guajardo-Cespedes, Sheng-Yi Kong, Noah Constant for training the models, Nan Hua, Nicole Limtiaco, Rhomni St. John for transferring tasks, Steve Yuan, Yunhsuan Sung, Brian Strope, Ray Kurzweil for discussion of the model architecture. Special thanks to Sheng-Yi Kong and Noah Constant for training the Lite model.

Smart Compose: Using Neural Networks to Help Write Emails

Wednesday, May 16, 2018

Posted by Yonghui Wu, Principal Engineer, Google Brain TeamGoogle I/OSmart ComposeSmart Reply

Latency: Since Smart Compose provides predictions on a per-keystroke basis, it must respond ideally within 100ms for the user not to notice any delays. Balancing model complexity and inference speed was a critical issue.

Scale: Gmail is used by more than 1.4 billion diverse users. In order to provide auto completions that are useful for all Gmail users, the model has to have enough modeling capacity so that it is able to make tailored suggestions in subtly different contexts.

Fairness and Privacy: In developing Smart Compose, we needed to address sources of potential bias in the training process, and had to adhere to the same rigorous user privacy standards as Smart Reply, making sure that our models never expose user’s private information. Furthermore, researchers had no access to emails, which meant they had to develop and train a machine learning system to work on a dataset that they themselves cannot read.

Finding the Right Model ngramneural bag-of-wordsRNN languagesequence-to-sequenceword embeddings

Smart Compose RNN-LM model architecture. Subject and previous email message are encoded by averaging the word embeddings in each field. The averaged embeddings are then fed to the RNN-LM at each decoding step.

Accelerated Model Training & ServingTPUv2 PodFairness and PrivacyFairness in machine learningSemantics derived automatically from language corpora contain human-like biasesthis paperFuture workTransformer,RNMT+AcknowledgementsSmart Compose language generation model was developed by Benjamin Lee, Mia Chen, Gagan Bansal, Justin Lu, Jackie Tsay, Kaushik Roy, Tobias Bosch, Yinan Wang, Matthew Dierker, Katherine Evans, Thomas Jablin, Dehao Chen, Vinu Rajashekhar, Akshay Agrawal, Yuan Cao, Shuyuan Zhang, Xiaobing Liu, Noam Shazeer, Andrew Dai, Zhifeng Chen, Rami Al-Rfou, DK Choe, Yunhsuan Sung, Brian Strope, Timothy Sohn, Yonghui Wu, and many others.

Automatic Photography with Google Clips

Friday, May 11, 2018

Posted by Aseem Agarwala, Research Scientist, Clips Content Team LeadTo me, photography is the simultaneous recognition, in a fraction of a second, of the significance of an event as well as of a precise organization of forms which give that event its proper expression.Henri Cartier-BressonGoogle Clips

We wanted all computations to be performed on-device. In addition to extending battery life and reducing latency, on-device processing means that none of your clips leave the device unless you decide to save or share them, which is a key privacy control.

We wanted the device to capture short videos, rather than single photographs. Moments with motion can be more poignant and true-to-memory, and it is often easier to shoot a video around a compelling moment than it is to capture a perfect, single instant in time.

We wanted to focus on capturing candid moments of people and pets, rather than the more abstract and subjective problem of capturing artistic images. That is, we did not attempt to teach Clips to think about composition, color balance, light, etc.; instead, Clips focuses on selecting ranges of time containing people and animals doing interesting activities.

Learning to Recognize Great MomentsWe then hired

Training a Clips Quality Modelinrecognize over 27,000 different labelsMobileNetpiecewise linear regression model

Diagram of the training process for generating frame quality scores. Piecewise linear regression maps from an ICM embedding to a score which, when averaged across a video segment, yields a moment score. The moment score of the preferred segment should be higher.

most recent releaseShot Control

Respect Power & Thermals: We want the Clips battery to last roughly three hours, and we don’t want the device to overheat — the device can’t run at full throttle all the time. Clips spends much of its time in a low-power mode that captures one frame per second. If the quality of that frame exceeds a threshold set by how much Clips has recently shot, it moves into a high-power mode, capturing at 15 fps. Clips then saves a clip at the first quality peak encountered.

Avoid Redundancy: We don’t want Clips to capture all of its moments at once, and ignore the rest of a session. Our algorithms therefore cluster moments into visually similar groups, and limit the number of clips in each cluster.

The Benefit of Hindsight: It’s much easier to determine which clips are the best when you can examine the totality of clips captured. Clips therefore captures more moments than it intends to show to the user. When clips are ready to be transferred to the phone, the Clips device takes a second look at what it has shot, and only transfers the best and least redundant content.

Machine Learning Fairnessfairness in ML algorithmsConclusionAcknowledgementsThe algorithms described here were conceived and implemented by a large group of Google engineers, research scientists, and others. Figures were made by Lior Shapira. Thanks to Lior and Juston Payne for video content.

Custom On-Device ML Models with Learn2Compress

Wednesday, May 9, 2018

Posted by Sujith Ravi, Senior Staff Research Scientist, Google Expander TeamOn-device machine learningMobileNetsProjectionNetsownML KitTensorFlow LitehereHow it WorksProjectionNet

Learn2Compress for automatically generating on-device ML models.

Pruning reduces model size by removing weights or operations that are least useful for predictions (e.g.low-scoring weights). This can be very effective especially for on-device models involving sparse inputs or outputs, which can be reduced up to 2x in size while retaining 97% of the original prediction quality.

Quantization techniques are particularly effective when applied during training and can improve inference speed by reducing the number of bits used for model weights and activations. For example, using 8-bit fixed point representation instead of floats can speed up the model inference, reduce power and further reduce size by 4x.

Joint training and distillation approaches follow a teacher-student learning strategy — we use a larger teacher network (in this case, user-provided TensorFlow model) to train a compact student network (on-device model) with minimal loss in accuracy.

Joint training and distillation approach to learn compact student models.

The teacher network can be fixed (as in distillation) or jointly optimized, and even train multiple student models of different sizes simultaneously. So instead of a single model, Learn2Compress generates multiple on-device models in a single shot, at different sizes and inference speeds, and lets the developer pick one best suited for their application needs.

transfer learningHow well does it work?MobileNetsNASNetInceptionProjectionNet

Accuracy at various sizes for Learn2Compress models and full-sized baseline networks on CIFAR-10 (left) and ImageNet (right) image classification tasks. Student networks used to produce the compressed variants for CIFAR-10 and ImageNet are modeled using NASNet and MobileNet-inspired architectures, respectively.

ImageNetCIFAR-10

Computation cost and average prediction latency (on Pixel phone) for baseline and Learn2Compress models on CIFAR-10 image classification task. Learn2Compress-optimized models use NASNet-style network architecture.

FishbrainAcknowledgmentsI would like to acknowledge our core contributors Gaurav Menghani, Prabhu Kaliamoorthi and Yicheng Fan along with Wei Chai, Kang Lee, Sheng Xu and Pannag Sanketi. Special thanks to Dave Burke, Brahim Elbouchikhi, Hrishikesh Aradhye, Hugues Vincent, and Arun Venkatesan from the Android team; Sachin Kotwani, Wesley Tarle, Pavel Jbanov and from the Firebase team; Andrei Broder, Andrew Tomkins, Robin Dua, Patrick McGregor, Gaurav Nemade, the Google Expander team and TensorFlow team.

Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone

Tuesday, May 8, 2018

Posted by Yaniv Leviathan, Principal Engineer and Yossi Matias, Vice President, Engineering, GoogleGoogle voice searchWaveNet

Duplex scheduling a hair salon appointment:

Duplex calling a restaurant:

Conducting Natural Conversations“So umm Tuesday through Thursday we are open 11 to 2, and then reopen 4 to 9, and then Friday, Saturday, Sunday we... or Friday, Saturday we're open 11 to 9 and then Sunday we're open 1 to 9.”

Example of complex statement:

elaborations syncsinterruptionspausesEnter Duplexunderstandinginteractingtimingspeakingrecurrent neural networkTensorFlow Extended

Incoming sound is processed through an ASR system. This produces text that is analyzed with context data and other inputs to produce a response text that is read aloud through the TTS system.

Duplex handling interruptions:

Duplex elaborating:

Duplex responding to a sync:

Sounding NaturalTacotronWaveNetlatencymoreSystem Operationfully autonomouslyreal-time supervised trainingBenefits for Businesses and Users

Duplex calling a restaurant:

Duplex asking for holiday hours:

A user asks the Google Assistant for an appointment, which the Assistant then schedules by having Duplex call the business.

Google Assistant

Yaniv Leviathan, Google Duplex lead, and Matan Kalman, engineering manager on the project, enjoying a meal booked through a call from Duplex.

Duplex calling to book the above meal:

Deep Learning for Electronic Health Records

Tuesday, May 8, 2018

Posted by Alvin Rajkomar MD, Research Scientist and Eyal Oren PhD, Product Manager, Google AI

Scalable: Predictions should be straightforward to create for any important outcome and for different hospital systems. Since healthcare data is very complicated and requires much data wrangling, this requirement is not straightforward to satisfy.

Accurate: Predictions should alert clinicians to problems but not distract them with false alarms. With the widespread adoption of electronic health records, we set out to use that data to create more accurate prediction models.

UC San FranciscoStanford MedicineThe University of Chicago MedicineScalable and Accurate Deep Learning with Electronic Health RecordsNature Partner Journals: Digital Medicine

ScalabilityFast Healthcare Interoperability Resourcesin an earlier blog postrecurrent neural networksfeedforward

Data in a patient's record is represented as a timeline. For illustrative purposes, we display various types of clinical data (e.g. encounters, lab tests) by row. Each piece of data, indicated as a little grey dot, is stored in FHIR, an open data standard that can be used by any healthcare institution. A deep learning model analyzed a patient's chart by reading the timeline from left to right, from the beginning of a chart to the current hospitalization, and used this data to make different types of predictions.

Prediction Accuracyarea-under-the-receiver-operator curvenot

A deep learning model was used to render a prediction 24 hours after a patient was admitted to the hospital. The timeline (top of figure) contains months of historical data and the most recent data is shown enlarged in the middle. The model "attended" to information highlighted in red that was in the patient's chart to "explain" its prediction. In this case-study, the model highlighted pieces of information that make sense clinically. Figure from our paper.

What does this mean for patients and clinicians?

Blog