Google AI Blog: May 2018

Advances in Semantic Textual Similarity

Thursday, May 17, 2018

Posted by Yinfei Yang, Software Engineer and Chris Tar, Engineering Manager, Google AISmart ComposeTalk to BooksTensorFlow HubSemantic Textual SimilarityLearning Semantic Textual Similarity from Conversations

Sentences are semantically similar if they can be answered by the same responses. Otherwise, they are semantically different.

SNLIentailmentSTSBenchmarkCQA task B

For a given input, classification is considered a ranking problem against potential candidates.

Universal Sentence EncoderUniversal Sentence Encoderskip-thought

Pairwise semantic similarity comparison via outputs from TensorFlow Hub Universal Sentence Encoder.

deep average networkTransformer

Multi-task training as described in “Universal Sentence Encoder”. A variety of tasks and task structures are joined by shared encoder layers/parameters (grey boxes).

New Models modelnewTensorFlow HubUniversal Sentence Encoder - LargeUniversal Sentence Encoder - Lite

The Large model is trained with the Transformer encoder described in our second paper. It targets scenarios requiring high precision semantic representations and the best model performance at the cost of speed & size.

The Lite model is trained on a Sentence Piece vocabulary instead of words in order to significantly reduce the vocabulary size, which is a major contributor of model size. It targets scenarios where resources like memory and CPU are limited, such as on-device or browser based implementations.

AcknowledgementsDaniel Cer, Mario Guajardo-Cespedes, Sheng-Yi Kong, Noah Constant for training the models, Nan Hua, Nicole Limtiaco, Rhomni St. John for transferring tasks, Steve Yuan, Yunhsuan Sung, Brian Strope, Ray Kurzweil for discussion of the model architecture. Special thanks to Sheng-Yi Kong and Noah Constant for training the Lite model.

Smart Compose: Using Neural Networks to Help Write Emails

Wednesday, May 16, 2018

Posted by Yonghui Wu, Principal Engineer, Google Brain TeamGoogle I/OSmart ComposeSmart Reply

Latency: Since Smart Compose provides predictions on a per-keystroke basis, it must respond ideally within 100ms for the user not to notice any delays. Balancing model complexity and inference speed was a critical issue.

Scale: Gmail is used by more than 1.4 billion diverse users. In order to provide auto completions that are useful for all Gmail users, the model has to have enough modeling capacity so that it is able to make tailored suggestions in subtly different contexts.

Fairness and Privacy: In developing Smart Compose, we needed to address sources of potential bias in the training process, and had to adhere to the same rigorous user privacy standards as Smart Reply, making sure that our models never expose user’s private information. Furthermore, researchers had no access to emails, which meant they had to develop and train a machine learning system to work on a dataset that they themselves cannot read.

Finding the Right Model ngramneural bag-of-wordsRNN languagesequence-to-sequenceword embeddings

Smart Compose RNN-LM model architecture. Subject and previous email message are encoded by averaging the word embeddings in each field. The averaged embeddings are then fed to the RNN-LM at each decoding step.

Accelerated Model Training & ServingTPUv2 PodFairness and PrivacyFairness in machine learningSemantics derived automatically from language corpora contain human-like biasesthis paperFuture workTransformer,RNMT+AcknowledgementsSmart Compose language generation model was developed by Benjamin Lee, Mia Chen, Gagan Bansal, Justin Lu, Jackie Tsay, Kaushik Roy, Tobias Bosch, Yinan Wang, Matthew Dierker, Katherine Evans, Thomas Jablin, Dehao Chen, Vinu Rajashekhar, Akshay Agrawal, Yuan Cao, Shuyuan Zhang, Xiaobing Liu, Noam Shazeer, Andrew Dai, Zhifeng Chen, Rami Al-Rfou, DK Choe, Yunhsuan Sung, Brian Strope, Timothy Sohn, Yonghui Wu, and many others.

Automatic Photography with Google Clips

Friday, May 11, 2018

Posted by Aseem Agarwala, Research Scientist, Clips Content Team LeadTo me, photography is the simultaneous recognition, in a fraction of a second, of the significance of an event as well as of a precise organization of forms which give that event its proper expression.Henri Cartier-BressonGoogle Clips

We wanted all computations to be performed on-device. In addition to extending battery life and reducing latency, on-device processing means that none of your clips leave the device unless you decide to save or share them, which is a key privacy control.

We wanted the device to capture short videos, rather than single photographs. Moments with motion can be more poignant and true-to-memory, and it is often easier to shoot a video around a compelling moment than it is to capture a perfect, single instant in time.

We wanted to focus on capturing candid moments of people and pets, rather than the more abstract and subjective problem of capturing artistic images. That is, we did not attempt to teach Clips to think about composition, color balance, light, etc.; instead, Clips focuses on selecting ranges of time containing people and animals doing interesting activities.

Learning to Recognize Great MomentsWe then hired

Training a Clips Quality Modelinrecognize over 27,000 different labelsMobileNetpiecewise linear regression model

Diagram of the training process for generating frame quality scores. Piecewise linear regression maps from an ICM embedding to a score which, when averaged across a video segment, yields a moment score. The moment score of the preferred segment should be higher.

most recent releaseShot Control

Respect Power & Thermals: We want the Clips battery to last roughly three hours, and we don’t want the device to overheat — the device can’t run at full throttle all the time. Clips spends much of its time in a low-power mode that captures one frame per second. If the quality of that frame exceeds a threshold set by how much Clips has recently shot, it moves into a high-power mode, capturing at 15 fps. Clips then saves a clip at the first quality peak encountered.

Avoid Redundancy: We don’t want Clips to capture all of its moments at once, and ignore the rest of a session. Our algorithms therefore cluster moments into visually similar groups, and limit the number of clips in each cluster.

The Benefit of Hindsight: It’s much easier to determine which clips are the best when you can examine the totality of clips captured. Clips therefore captures more moments than it intends to show to the user. When clips are ready to be transferred to the phone, the Clips device takes a second look at what it has shot, and only transfers the best and least redundant content.

Machine Learning Fairnessfairness in ML algorithmsConclusionAcknowledgementsThe algorithms described here were conceived and implemented by a large group of Google engineers, research scientists, and others. Figures were made by Lior Shapira. Thanks to Lior and Juston Payne for video content.

Custom On-Device ML Models with Learn2Compress

Wednesday, May 9, 2018

Posted by Sujith Ravi, Senior Staff Research Scientist, Google Expander TeamOn-device machine learningMobileNetsProjectionNetsownML KitTensorFlow LitehereHow it WorksProjectionNet

Learn2Compress for automatically generating on-device ML models.

Pruning reduces model size by removing weights or operations that are least useful for predictions (e.g.low-scoring weights). This can be very effective especially for on-device models involving sparse inputs or outputs, which can be reduced up to 2x in size while retaining 97% of the original prediction quality.

Quantization techniques are particularly effective when applied during training and can improve inference speed by reducing the number of bits used for model weights and activations. For example, using 8-bit fixed point representation instead of floats can speed up the model inference, reduce power and further reduce size by 4x.

Joint training and distillation approaches follow a teacher-student learning strategy — we use a larger teacher network (in this case, user-provided TensorFlow model) to train a compact student network (on-device model) with minimal loss in accuracy.

Joint training and distillation approach to learn compact student models.

The teacher network can be fixed (as in distillation) or jointly optimized, and even train multiple student models of different sizes simultaneously. So instead of a single model, Learn2Compress generates multiple on-device models in a single shot, at different sizes and inference speeds, and lets the developer pick one best suited for their application needs.

transfer learningHow well does it work?MobileNetsNASNetInceptionProjectionNet

Accuracy at various sizes for Learn2Compress models and full-sized baseline networks on CIFAR-10 (left) and ImageNet (right) image classification tasks. Student networks used to produce the compressed variants for CIFAR-10 and ImageNet are modeled using NASNet and MobileNet-inspired architectures, respectively.

ImageNetCIFAR-10

Computation cost and average prediction latency (on Pixel phone) for baseline and Learn2Compress models on CIFAR-10 image classification task. Learn2Compress-optimized models use NASNet-style network architecture.

FishbrainAcknowledgmentsI would like to acknowledge our core contributors Gaurav Menghani, Prabhu Kaliamoorthi and Yicheng Fan along with Wei Chai, Kang Lee, Sheng Xu and Pannag Sanketi. Special thanks to Dave Burke, Brahim Elbouchikhi, Hrishikesh Aradhye, Hugues Vincent, and Arun Venkatesan from the Android team; Sachin Kotwani, Wesley Tarle, Pavel Jbanov and from the Firebase team; Andrei Broder, Andrew Tomkins, Robin Dua, Patrick McGregor, Gaurav Nemade, the Google Expander team and TensorFlow team.

Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone

Tuesday, May 8, 2018

Posted by Yaniv Leviathan, Principal Engineer and Yossi Matias, Vice President, Engineering, GoogleGoogle voice searchWaveNet

Duplex scheduling a hair salon appointment:

Duplex calling a restaurant:

Conducting Natural Conversations“So umm Tuesday through Thursday we are open 11 to 2, and then reopen 4 to 9, and then Friday, Saturday, Sunday we... or Friday, Saturday we're open 11 to 9 and then Sunday we're open 1 to 9.”

Example of complex statement:

elaborations syncsinterruptionspausesEnter Duplexunderstandinginteractingtimingspeakingrecurrent neural networkTensorFlow Extended

Incoming sound is processed through an ASR system. This produces text that is analyzed with context data and other inputs to produce a response text that is read aloud through the TTS system.

Duplex handling interruptions:

Duplex elaborating:

Duplex responding to a sync:

Sounding NaturalTacotronWaveNetlatencymoreSystem Operationfully autonomouslyreal-time supervised trainingBenefits for Businesses and Users

Duplex calling a restaurant:

Duplex asking for holiday hours:

A user asks the Google Assistant for an appointment, which the Assistant then schedules by having Duplex call the business.

Google Assistant

Yaniv Leviathan, Google Duplex lead, and Matan Kalman, engineering manager on the project, enjoying a meal booked through a call from Duplex.

Duplex calling to book the above meal:

Deep Learning for Electronic Health Records

Tuesday, May 8, 2018

Posted by Alvin Rajkomar MD, Research Scientist and Eyal Oren PhD, Product Manager, Google AI

Scalable: Predictions should be straightforward to create for any important outcome and for different hospital systems. Since healthcare data is very complicated and requires much data wrangling, this requirement is not straightforward to satisfy.

Accurate: Predictions should alert clinicians to problems but not distract them with false alarms. With the widespread adoption of electronic health records, we set out to use that data to create more accurate prediction models.

UC San FranciscoStanford MedicineThe University of Chicago MedicineScalable and Accurate Deep Learning with Electronic Health RecordsNature Partner Journals: Digital Medicine

ScalabilityFast Healthcare Interoperability Resourcesin an earlier blog postrecurrent neural networksfeedforward

Data in a patient's record is represented as a timeline. For illustrative purposes, we display various types of clinical data (e.g. encounters, lab tests) by row. Each piece of data, indicated as a little grey dot, is stored in FHIR, an open data standard that can be used by any healthcare institution. A deep learning model analyzed a patient's chart by reading the timeline from left to right, from the beginning of a chart to the current hospitalization, and used this data to make different types of predictions.

Prediction Accuracyarea-under-the-receiver-operator curvenot

A deep learning model was used to render a prediction 24 hours after a patient was admitted to the hospital. The timeline (top of figure) contains months of historical data and the most recent data is shown enlarged in the middle. The model "attended" to information highlighted in red that was in the patient's chart to "explain" its prediction. In this case-study, the model highlighted pieces of information that make sense clinically. Figure from our paper.

What does this mean for patients and clinicians?

Introducing Google AI

Monday, May 7, 2018

Posted by Christian Howard, Editor-in-Chief, Google AI CommunicationsHmm, have I made a wrong turn? I was looking for Google Research…computer visionhealthcare researchAutoMLproductsplatforms

Google AI websiteTwitterGoogle+

The Question of Quantum Supremacy

Friday, May 4, 2018

Posted by Sergio Boixo, Research Scientist and Theory Team Lead, and Charles Neill, Quantum Electronics Engineer, Quantum A.I. LabQuantum computinginformation technologyquantum mechanicsintractablecomputational tasksCharacterizing quantum supremacy in near-term devicesherebutterfly effectqubitArguablyexponentialseparationcomputationalimprovementsclassicalalgorithmssimulate quantum circuitsincrease the sizesimulation costherecross-entropy benchmarkingfault-toleranthighly complex

Space-time volume of a quantum circuit computation. The computational cost for quantum simulation increases with the volume of the quantum circuit, and in general grows exponentially with the number of qubits and the circuit depth. For asymmetric grids of qubits, the computational space-time volume grows slower with depth than for symmetric grids, and can result in circuits exponentially easier to simulate.

A blueprint for demonstrating quantum supremacy with superconducting qubitsherecross-entropy benchmarkingdigital quantum processorsurface code error correction

Photograph of two gmon superconducting qubits and their tunable coupler developed by Charles Neill and Pedram Roushan.

quantum processorsquantum algorithmsapplications

Blog