Google AI Blog

Moving Beyond Translation with the Universal Transformer

Wednesday, August 15, 2018

Posted by Stephan Gouws, Research Scientist, Google Brain Team and Mostafa Dehghani, University of Amsterdam PhD student and Google Research Internthe Transformerrecurrent neural networkssequentiallyrecurrenceabcabcabcNeural GPUNeural Turing MachineUniversal TransformersTuring completeparallel-in-time recurrence

The Universal Transformer repeatedly refines a series of vector representations (shown as h₁ to h_m) for each position of the sequence in parallel, by combining information from different positions using self-attention and applying a recurrent transition function. Arrows denote dependencies between operations.

adaptive computation mechanismI arrived at the bank after crossing the riverbankIriverbanksinglesequencedifferent transformationsbAbI linguistic reasoning taskLAMBADA language modeling taskBLEU¹hereAcknowledgementsThis research was conducted by Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Additional thanks go to Ashish Vaswani, Douglas Eck, and David Dohan for their fruitful comments and inspiration.
1 A translation quality benchmark widely used in the machine translation community, computed on the standard WMT newstest2014 English to German translation test data set.^↩

The Machine Learning Behind Android Smart Linkify

Thursday, August 9, 2018

Posted by Lukas Zilka, Software Engineer, Google AI, Zürichwe launched Android 9 PieSmart Linkify

Smart Text SelectionTextClassifier APITensorFlowTensorFlow LiteFlatBuffershereFinding EntitiesConfirmation number: 857-555-3556feedforward neural networks

For the given text string, the first network assigns low scores to non-entities and a high score for the candidate that correctly selects the whole phone number.

And call 857 555 3556tomorrow.857 555 3556And call 857 555 3556 tomorrowAnd

And call 857 555 3556 tomorrow.857 555 3556And call 857 555 3556 tomorrowAndTextual Features

Given a candidate entity span, we extract: Left context: five words before the entity, Entity start: first three words of the entity, Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many), Right context: five words after the entity, Entity content: bag of words inside the entity and Entity length: size of the entity in number of words. They are then concatenated together and fed as an input to the neural network.

Character N-grams. Instead of using the standard word embedding technique for representing words, which keeps a separate vector for each word in the model and thus would be infeasible for mobile devices because of their large storage size, we use the hashed charactergram embedding. This technique represents the word as a set of all character subsequences of certain length. We use lengths 1 to 5. These strings are additionally hashed and mapped to a fixed number of buckets (see here for more details on the technique). As a result, the final model only stores vectors for each of the hash buckets, not each word/character subsequence, and can be kept small. The embedding matrix for the hashed charactergrams that we use has 20,000 buckets and 12 dimensions.

A binary feature that indicates whether the word starts with a capital letter. This is important for the network to know because the capitalization in postal addresses is quite distinct, and helps the networks to discriminate.

A Training DatasetSchema.orgConfirmation number:ID:Making it Work

Quantizing the embedding matrix to 8 bits. We found that we could reduce the size of the model almost 4x without compromising the performance, by quantizing the embedding matrix values to 8-bit integers.

Sharing embedding matrices between the selection and classification networks. This brings almost no loss and makes the model 2x smaller.

Varying the size of the context before/after the entities. On mobile screens text is often short, with not enough context, so the network needs to be exposed to this during training as well.

Creating artificial negative examples out of the positive ones for the classification network. For example for the positive example: “call me 857 555-3556 today” with a label “phone” we generate “call me 857 555-3556 today” as a negative example with a label “other”. This teaches the classification network to be more precise about the entity span. Without doing this, the network would be merely a detector whether there is a phone number somewhere in the input, regardless of the span.

Internationalization is ImportantNext Stepsnext Thursdayin 3 weeksopen-source as part of Android framework

MnasNet: Towards Automating the Design of Mobile Machine Learning Models

Tuesday, August 7, 2018

Posted by Mingxing Tan, Software Engineer, Google Brain TeamConvolutional neural networksMobileNetMobileNetV2AutoML neural architecture searchMnasNet: Platform-Aware Neural Architecture Search for Mobilereinforcement learningMobileNetV2NASNetFLOPSRNNTensorFlow Litemulti-objective optimizationPareto optimal

Overall flow of our automated neural architecture search approach for Mobile.

Our MnasNet network, sampled from the novel factorized hierarchical search space,illustrating the layer diversity throughout the network architecture.

ImageNetCOCO

ImageNet Accuracy and Inference Latency comparison. MnasNets are our models.

MobileNetV2NASNetsqueeze-and-excitationResNet-50SSD300AcknowledgementsSpecial thanks to the co-authors of the paper Bo Chen, Quoc V. Le, Ruoming Pang and Vijay Vasudevan. We’d also like to thank Andrew Howard, Barret Zoph, Dmitry Kalenichenko, Guiheng Zhou, Jeff Dean, Mark Sandler, Megan Kacholia, Sheng Li, Vishy Tirumalashetty, Wen Wang, Xiaoqiang Zheng and Yifeng Lu for their help, and the TensorFlow Lite and Google Brain teams.

Machine Learning in Google BigQuery

Wednesday, July 25, 2018

Posted by Umar Syed and Sergei Vassilvitskii, Research Scientists, Google AI, NYCGoogle BigQuerySQLBigQuery ML

gradient descent^*stochasticperform poorly when the data is suboptimally orderedfartrickierregularizationpreconditioningpaperBigQuery consoleuser guideCREATE MODEL dataset.model_name OPTIONS(model_type=’linear_reg’, input_label_cols=[‘input_label’]) AS SELECT * FROM input_table; Acknowledgements BigQuery ML is the result of a large collaboration across many teams at Google. Key contributors and sponsors include Hossein Ahmadi, Corinna Cortes, Grzegorz Czajkowski, JD Degenaar, Dan Delorey, Mingge Deng, Danielle Hanks, Amir Hormati, Abhishek Kashyap, Jing Jing Long, Dan McClary, Chris Meyers, Ross Popoff-Walker, Girishkumar Sabhnani, Vivek Sharma, Jordan Tigani, Chad Verbowski, Jiaxun Wu and Lisa Yin.
* For example, a gradient vector can be computed using the SUM and GROUP BY operators, and the weights of a model can be updated using an INNER JOIN.^↩

Announcing Cirq: An Open Source Framework for NISQ Algorithms

Wednesday, July 18, 2018

Posted by Alan Ho, Product Lead and Dave Bacon, Software Lead, Google AI Quantum Teamquantum computingquantum algorithmsNoisy Intermediate Scale Quantum^*First International Workshop on Quantum Software and Quantum Machine LearningGoogle AI Quantum teampublic alpha of Cirqcomputational problems of practical importanceApache 2

installed

OpenFermion-CirqOpenFermionlatest advances

Zapata Computing: simulation of a quantum autoencoder (example code, video tutorial)

QC Ware: QAOA implementation and integration into QC Ware’s AQUA platform (example code, video tutorial)

Quantum Benchmark: integration of True-Q software tools for assessing and extending hardware capabilities (video tutorial)

Heisenberg Quantum Simulations: simulating the Anderson Model

Cambridge Quantum Computing: integration of proprietary quantum compiler t|ket> (video tutorial)

NASA: architecture-aware compiler based on temporal-planning for QAOA (slides) and simulator of quantum computers (slides)

Bristlecone processorCirqOpenFermion-CirqAcknowledgementsWe would like to thank Craig Gidney for leading the development of Cirq, Ryan Babbush and Kevin Sung for building OpenFermion-Cirq and a whole host of code contributors to both frameworks.
* An analogous situation is how early classical programmers needed to run complex programs in very small memory spaces by paying careful attention to the lowest level details of the hardware.^↩

Improving Connectomics by an Order of Magnitude

Monday, July 16, 2018

Posted by Viren Jain, Research Scientist and Technical Lead and Michal Januszewski, Software Engineer, Connectomics at Googleconnectomicselectron microscopyneuritessynapticcolleagues at the Max Planck Institute of NeurobiologyHigh-Precision Automated Reconstruction of Neurons with Flood-Filling NetworksNature Methodsrecurrent neural networkbiorXiv3D Image Segmentation with Flood-Filling Networksimage segmentationedge detectorclassifierwatershedgraph cutrecurrent neural networksconvolutional neural network

A flood-filling network segmenting an object in 2d. The yellow dot is the center of the current area of focus; the algorithm expands the segmented region (blue) as it iteratively examines more of the overall image.

Measuring Accuracy via Expected Run Lengthmean-time-between-failurebiologically relevant quantities

Progress in expected run length (blue line) leading up to the results shared today in Nature Methods. The red line shows progress in the “merge rate,” which measures the frequency with which two separate neurites were erroneously traced as a single object; achieving a very low merge rate is important for enabling efficient strategies for manual identification and correction of the remaining errors in the reconstruction.

Songbird Connectomicszebra finchserial block-face scanning electron microscopyprevious deep learning pipelines

Our algorithm in action as it traces a single neurite in 3d in a songbird brain.

Reconstruction of a portion of zebra finch brain. Colors denote distinct objects in the segmentation that was automatically generated using a flood-filling network. Gold spheres represent synaptic locations automatically identified using a previously published approach.

test theories related to how they learn their songNext Stepsopen-sourced the TensorFlow codeWebGL visualization software for 3d datasetsAcknowledgementsWe would like to acknowledge core contributions from Tim Blakely, Peter Li, Larry Lindsey, Jeremy Maitin-Shepard, Art Pope and Mike Tyka (Google), as well as Joergen Kornfeld and Winfried Denk (Max Planck Institute).

Accelerated Training and Inference with the Tensorflow Object Detection API

Friday, July 13, 2018

Posted by Jonathan Huang, Research Scientist and Vivek Rathod, Software Engineer, Google AI Perceptionwe announcedTensorFlow Object Detection APINeural Architecture Searchinstance segmentation supportOpen Imagesfinding scofflaws on the streets of NYCdiagnosing diseases on cassava plants in Tanzania

Support for accelerated training of object detection models via Cloud TPUs

Improving the mobile deployment process by accelerating inference and making it easy to export a model to mobile with the TensorFlow Lite format

Several new model architecture definitions including:

RetinaNet (Lin et al., 2017)
A MobileNet adaptation of RetinaNet
A novel SSD-based architecture called the Pooling Pyramid Network (PPN) whose model size is >3x smaller than that of SSD MobileNet v1 with minimal loss in accuracy.

COCO datasetAccelerated Training via Cloud TPUssingle shot detectorRetinaNetmean Average Precision

Accelerated Inference via Quantization and TensorFlow Lite TensorFlow Litemodel quantizationmodel quantizationJacob et al.whitepaper by Krishnamoorthi

Quantized detection models are faster and smaller (e.g., a quantized 75% depth-reduced SSD Mobilenet model runs at >15 fps on a Pixel 2 CPU with a 4.2 Mb footprint) with minimal loss in detection accuracy compared to the full floating point model.

Try it Yourself with a New Tutorial!check out our new tutorialAcknowledgementsThis post reflects the work of the following group of core contributors: Derek Chow, Aakanksha Chowdhery, Jonathan Huang, Pengchong Jin, Zhichao Lu, Vivek Rathod, Ronny Votel and Xiangxin Zhu. We would also like to thank the following colleagues: Vasu Agrawal, Sourabh Bajaj, Chiachen Chou, Tom Jablin, Wenzhe Li, Tsung-Yi Lin, Hernan Moraldo, Kevin Murphy, Sara Robinson, Andrew Selle, Shashi Shekhar, Yash Sonthalia, Zak Stone, Pete Warden and Menglong Zhu.

Automating Drug Discoveries Using Computer Vision

Thursday, July 12, 2018

Vincent Vanhoucke, Principal Scientist, Google Brain Team“Every time you miss a protein crystal, because they are so rare, you risk missing on an important biomedical discovery.”Patrick CharbonneauMARCO initiativeProtein crystallizationMAchine Recognition of Crystallization OutcomesClassification of Crystallization Outcomes using Deep Convolutional Neural NetworksPLOS OneArXiv preprintopen-sourced our modelCloud ML Engine

Image of protein crystal, courtesy of the MARCO repository (CC-BY-4.0 license)

a large repository of curated crystallography images

Samples from the MARCO repository, illustrating the degree of variability between data sources.

Inception V3hereDuke Research Blog post

Google at ICML 2018

Monday, July 9, 2018

Posted by Christian Howard, Editor-in-Chief, Google AI CommunicationsInternational Conference on Machine LearningInternational Machine Learning SocietyTensorFlow HubMagenta projectGoogle AI ResidencICML 2018 CommitteesAndrew McCallum, Corinna Cortes, Hugo Larochelle, William CohenRyan AdamsAccepted PublicationsPredict and Constrain: Modeling Cardinality in Deep Structured PredictionNataly Brukhim, Amir GlobersonQuickshift++: Provably Good Initializations for Sample-Based Mean ShiftHeinrich Jiang, Jennifer Jang, Samory KpotufeLearning a Mixture of Two Multinomial LogitsFlavio Chierichetti, Ravi Kumar, Andrew TomkinsStructured Evolution with Compact Architectures for Scalable Policy OptimizationKrzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard E Turner, Adrian WellerFixing a Broken ELBOAlexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif Saurous, Kevin MurphyHierarchical Long-term Video Prediction without SupervisionNevan Wichers, Ruben Villegas, Dumitru Erhan, Honglak LeeSelf-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory EmbeddingsJohn Co-Reyes, Yu Xuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, Sergey LevineWell Tempered LassoYuanzhi Li, Yoram SingerProgrammatically Interpretable Reinforcement LearningAbhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat ChaudhuriDynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural NetworksLechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, Jeffrey PenningtonOn the Optimization of Deep Networks: Implicit Acceleration by OverparameterizationSanjeev Arora, Nadav Cohen, Elad HazanScalable Deletion-Robust Submodular Maximization: Data Summarization with Privacy and Fairness ConstraintsEhsan Kazemi, Morteza Zadimoghaddam, Amin KarbasiData Summarization at Scale: A Two-Stage Submodular ApproachMarko Mitrovic, Ehsan Kazemi, Morteza Zadimoghaddam, Amin KarbasiMachine Theory of MindNeil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, Matthew BotvinickLearning to Optimize Combinatorial FunctionsNir Rosenfeld, Eric Balkanski, Amir Globerson, Yaron SingerProportional Allocation: Simple, Distributed, and Diverse Matching with High EntropyShipra Agarwal, Morteza Zadimoghaddam, Vahab MirrokniPath Consistency Learning in Tsallis Entropy Regularized MDPsYinlam Chow, Ofir Nachum, Mohammad GhavamzadehEfficient Neural Architecture Search via Parameters SharingHieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff DeanAdafactor: Adaptive Learning Rates with Sublinear Memory CostNoam Shazeer, Mitchell SternLearning Memory Access PatternsMilad Hashemi, Kevin Swersky, Jamie Smith, Grant Ayers, Heiner Litz, Jichuan Chang, Christos Kozyrakis, Parthasarathy RanganathanSBEED: Convergent Reinforcement Learning with Nonlinear Function ApproximationBo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le SongScalable Bilinear Pi Learning Using State and Action FeaturesYichen Chen, Lihong Li, Mengdi WangDistributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go?Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Peter Glynn, Yinyu Ye, Li-Jia Li, Li Fei-FeiShampoo: Preconditioned Stochastic Tensor OptimizationVineet Gupta, Tomer Koren, Yoram SingerParallel and Streaming Algorithms for K-Core DecompositionHossein Esfandiari, Silvio Lattanzi, Vahab MirrokniCan Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?Maithra Raghu, Alexander Irpan, Jacob Andreas, Bobby Kleinberg, Quoc Le, Jon KleinbergIs Generator Conditioning Causally Related to GAN Performance?Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, Ian GoodfellowThe Mirage of Action-Dependent Baselines in Reinforcement LearningGeorge Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, Sergey LevineMentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted LabelsLu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-FeiLoss Decomposition for Fast Learning in Large Output SpacesEn-Hsu Yen, Satyen Kale, Felix Xinnan Yu, Daniel Holtmann-Rice, Sanjiv Kumar, Pradeep RavikumarA Hierarchical Latent Vector Model for Learning Long-Term Structure in MusicAdam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas EckSmoothed Action Value Functions for Learning Gaussian PoliciesOfir Nachum, Mohammad Norouzi, George Tucker, Dale SchuurmansFast Decoding in Sequence Models Using Discrete Latent VariablesLukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, Noam ShazeerAccelerating Greedy Coordinate Descent MethodsHaihao Lu, Robert Freund, Vahab MirrokniApproximate Leave-One-Out for Fast Parameter Tuning in High DimensionsShuaiwen Wang, Wenda Zhou, Haihao Lu, Arian Maleki, Vahab MirrokniImage TransformerNiki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin TranTowards End-to-End Prosody Transfer for Expressive Speech Synthesis with TacotronRJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Robert Clark, Rif SaurousDynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural NetworksMinmin Chen, Jeffrey Pennington,, Samuel SchoenholzStyle Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech SynthesisYuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, Rif SaurousConstrained Interacting Submodular GroupingsAndrew Cotter, Mahdi Milani Fard, Seungil You, Maya Gupta, Jeff BilmesReinforcing Adversarial Robustness using Model Confidence Induced by Adversarial TrainingXi Wu, Uyeong Jang, Jiefeng Chen, Lingjiao Chen, Somesh JhaInterpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viégas, Rory SayresOnline Learning with AbstentionCorinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, Scott YangOnline Linear Quadratic ControlAlon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, Kunal TalwarCompetitive Caching with Machine Learned AdviceThodoris Lykouris, Sergei VassilvitskiiEfficient Neural Audio SynthesisNal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, Koray KavukcuogluGradient Descent with Identity Initialization Efficiently Learns Positive Definite Linear Transformations by Deep Residual NetworksPeter Bartlett, Dave Helmbold, Phil LongUnderstanding and Simplifying One-Shot Architecture SearchGabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc LeApproximation Algorithms for Cascading Prediction ModelsMatthew StreeterLearning Longer-term Dependencies in RNNs with Auxiliary LossesTrieu Trinh, Andrew Dai, Thang Luong, Quoc LeSelf-Imitation LearningJunhyuk Oh, Yijie Guo, Satinder Singh, Honglak LeeAdaptive Sampled Softmax with Kernel Based SamplingGuy Blanc, Steffen RendleWorkshops2018 Workshop on Human Interpretability in Machine Learning (WHI)Been Kim, Kush Varshney, Adrian WellerFernanda Viégas, Martin WattenbergExploration in Reinforcement LearningBen Eysenbach, Surya Bhupatiraju, Shane Gu, Junhyuk Oh, Vincent Vanhoucke, Oriol Vinyals, Doina PrecupTheoretical Foundations and Applications of Deep Generative ModelsHonglak Lee

Scalable Deep Reinforcement Learning for Robotic Manipulation

Thursday, June 28, 2018

Posted Alex Irpan, Software Engineer, Google Brain Team and Peter Pastor, Senior Roboticist, Xdeep learningreinforcement learningprevious work

Examples of singulation.large-scale distributed optimizationQ-learningarXiv

Some of the training objects used.

that sharing experience across robots can accelerate learning

Seven robots collecting grasp data.

The objects used at evaluation time. To make the task challenging, we aimed for a large variety of object sizes, textures, and shapes.

When presented with a set of interlocking blocks that cannot be picked up together, the policy separates one of the blocks from the rest before picking it up.

When presented with a difficult-to-grasp object, the policy figures out it should reposition the gripper and regrasp it until it has a firm hold.

When grasping in clutter, the policy probes different objects until the fingers hold one of them firmly, before lifting.

When we perturbed the robot by intentionally swatting the object out of the gripper -- something it had not seen during training -- it automatically repositioned the gripper for another attempt.

Examples of the learned behaviors. In the left GIF, the policy corrects for the moved ball. In the right GIF, the policy tries several grasps until it succeeds at picking up the tricky object.domain adaptationrecent work on learning how to self-calibrateAcknowledgementsThis research was conducted by Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. We’d also like to give special thanks to Iñaki Gonzalo and John-Michael Burke for overseeing the robot operations, Chelsea Finn, Timothy Lillicrap, and Arun Nair for valuable discussions, and other people at Google and X who’ve contributed their expertise and time towards this research. A preprint is available on arXiv.