Google AI Blog: January 2019

Transformer-XL: Unleashing the Potential of Attention Models

Tuesday, January 29, 2019

Posted by Zhilin Yang and Quoc Le, Google AIlong-range dependenceGating-based RNNsthe gradient clipping techniquenot sufficientTransformers,a fixed-length context

Vanilla Transformer with a fixed-length context at training time.

The algorithm is not able to model dependencies that are longer than a fixed length.

The segments usually do not respect the sentence boundaries, resulting in context fragmentation which leads to inefficient optimization. This is particularly troublesome even for short sequences, where long range dependency isn't an issue.

Transformer-XLSegment-level Recurrence

Transformer-XL with segment-level recurrence at training time.

Relative Positional Encodingsother relative positional encoding schemes

Vanilla Transformer with a fixed-length context at evaluation time.

Transformer-XL with segment-level recurrence at evaluation time./td>

Results

Transformer-XL learns dependency that is about 80% longer than RNNs and 450% longer than vanilla Transformers, which generally have better performance than RNNs, but are not the best for long-range dependency modeling due to fixed-length contexts (please see our paper for details).

Transformer-XL is up to 1,800+ times faster than a vanilla Transformer during evaluation on language modeling tasks, because no re-computation is needed (see figures above).

Transformer-XL has better performance in perplexity (more accurate at predicting a sample) on long sequences because of long-term dependency modeling, and also on short sequences by resolving the context fragmentation problem.

enwiki8text8WikiText-103One Billion WordPenn Treebanklanguage model pretrainingBERTpaperGitHub

Natural Questions: a New Corpus and Challenge for Question Answering Research

Wednesday, January 23, 2019

Posted by Tom Kwiatkowski and Michael Collins, Research Scientists, Google AI LanguageOpen-domain question answeringnatural language understanding"Why is the sky blue?"this Wikipedia pagenaturally occurringNatural Questions¹quite easy for computers to solvechallengeThe Datalong answersshort answers

Natural Questions: a Benchmark for Question Answering ResearchTransactions of the Association for Computational LinguisticsNQ websiteThe Challengechallenge website
1 A reader just alerted us to DuReader, a dataset from Baidu that contains real queries and full documents. We will add this to our paper, in which we discuss NQ in relation to previous work.^↩

Expanding the Application of Deep Learning to Electronic Health Records

Tuesday, January 22, 2019

Posted by Alvin Rajkomar, MD and Eyal Oren, PhD, Google AI, Healthcarepublished a paperour work aims to further improve potential clinical benefitIntermountain Healthcare

TeleHealth critical care unit

Soft Actor-Critic: Deep Reinforcement Learning for Robotics

Friday, January 18, 2019

Posted by Tuomas Haarnoja, Student Researcher and Sergey Levine, Faculty Advisor, Robotics at Googlewe recently released Soft Actor-CriticRequirements for Real-World Robotic Learning

Good sample efficiency to lower the learning time

Minimal number of hyperparameters that require tuning

Reusing already collected data on different scenarios (known as off-policy learning)

Ensuring that learning and exploration does not damage the hardware

Soft Actor-Criticentropy BAIR blog post tutorialQ-functiontechnical reportPerformance of SACGhost RoboticsDynamixel Claw

Illustration of learned walking, using SAC implemented on the Minitaur robot. A full video of the learning process can be found at our project website.

prior workConclusionBAIR blog postearly preprint of the locomotion experimentmore complete description of the algorithmGitHubAcknowledgementsThis research was done in collaboration between Google and UC Berkeley. We would like to thank all the people who were involved, including Sehoon Ha, Kristian Hartikainen, Jie Tan, George Tucker, Vincent Vanhoucke and Aurick Zhou.

Looking Back at Google’s Research Efforts in 2018

Tuesday, January 15, 2019

Posted by Jeff Dean, Senior Fellow and Google AI Lead, on behalf of the entire Google Research Community
publications in 2018Ethical Principles and AIGoogle AI Principlesresponsible AI practicesevaluate our own development of AIAvoid creating or reinforcing unfair biasBe accountable to peopleML fairnessmodel interpretabilityreducing gender biasesexplorationreleaseFairness ModuleMachine Learning Crash CourseAI for Social Goodwork on flood prediction earthquake aftershock predictionTensorFlowusing convolutional neural networks to identify humpback whale callsdetecting new exoplanetsidentifying diseased cassava plants Google AI for Social Impact ChallengeGoogle.orgAssistive TechnologyGoogle DuplexCan you book me a haircut at 4 PM today?Smart ComposeSound SearchNow PlayingSmart Linkify

support more languagesbetter understanding of semantic similarityspeech synthesistext-to-speech for languages without much training data availableQuantum computingquantum supremacya number of exciting new resultsBristlecone

A Bristlecone chip being installed by Research Scientist Marissa Giustina at the Quantum AI Lab in Santa Barbara.

Cirqquantum computers could be used for neural networksunderstanding performance fluctuations in quantum processorsquantum computers might be useful as a computational substrate for neural networksNatural Language Understandingmix of basic research as well as product-focused collaborationsTransformer work from 2017Universal TransformerBERT

BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks.

handle multilingual use cases better, with the goal of makingallPerceptionpetslearn about the natural worldanswer questions in real-timedo more with Lens in Google Imagesnew capabilities in vision and video in Cloud ML APIsface-related on-device building blocks

Google Lens can help you learn more about the world around you. Here, Lens identifies the breed of this dog. Learn more in this blog post.

stereo magnificationa fast bottom-up model for joint pose estimation and person instance segmentationsystem for visualizing complex motionsystemdistillation3D convolutionsunsupervised learning of semantic audio representationsexpressive and human-like speech synthesisLooking to ListenMobileNetV2next-generation mobile computer vision modelMorphNetautomatic generation of mobile network architecturesComputational PhotographyHDR+Motion Photos in Pixel 2Augmented Reality mode in Motion Stills

Motion photos on the Pixel 2 in Google Photos. For more examples, check out this Google Photos album.

Augmented chicken family with Motion Stills AR mode.

Night Sightpressusersusing ML to provide better portrait mode shotsSuper Res ZoomTop ShotGoogle Clips

Left: iPhone XS (full resolution image here). Right: Pixel 3 Night Sight (full resolution image here).

Algorithms and Theoryrouting algorithms behind Google tripsconsistent hashing for Google cloudalgorithms and theorygraph miningour work on studying convergence of stochastic optimization algorithms for training neural networksICLR 2018ADAM

Performance comparison of ADAM and AMSGRAD on a synthetic example of a simple one dimensional convex problem inspired by our examples of non-convergence. The first two plots (left and center) are for the online setting and the the last one (right) is for the stochastic setting.

distributed optimizationvia round compressionvia core-setssubmodular maximizationk-core decompositionset cover at scale via sketchingbalanced partitioning and hierarchical clusteringonline delivery services WWW'18OR-tools platform2018 Minizinc new models reconstruction learning a mixture of multinomial logits learnable by neural networksimprove classiconline algorithmsby iterationby shufflingincentive-aware learning new researchmarket algorithmstest incentive compatibilityoptimizing ad refresh for in-app advertisinglack of prediction of futurenoisy forecastsheterogenous buyer behaviourdynamic double auctionsnew online allocation algorithms for stochastic input with traffic spikesnew bandit algorithms robust to corrupted dataSoftware Systemsdynamic control flow Mesh TensorFlowreleased

The TF-Ranking library supports multi-item scoring architecture, an extension of traditional single-item scoring.

JAXXLACleverHansTensorFlow Privacyplacement of computations onto deviceslearning memory access patternslearned indices

The Hierarchical Planner's placement of a NMT (4-layer) model. White denotes CPU and the four colors each represent one of the GPUs. Note that every step of every layer is allocated across multiple GPUs. This placement is 53.7% faster than that generated by a human expert.

SpectreMeltdownProject Zerotoolthe evolution of Google's Software Defined Networking WANstand-alone, federated query processing platforma report on our extensive use of code reviewdynamicconsistent hashing schemetight provable guaranteesdeployed itGoogle Cloud Pub/Subearlier version of our paperVimeohaproxyalmost 8AutoMLevolutionary algorithms can be used to automatically discover state-of-the-art neural network architecturesautomatically generate image transformation sequences find new symbolic optimization expressions that are more effective than the commonly used optimization update rulesAdaNet showed how to have a fast and flexible AutoML algorithm with learning guarantees

AdaNet adaptively growing an ensemble of neural networks. At each iteration, it measures the ensemble loss for each candidate, and selects the best one to move onto the next iteration.

combining the accuracy of a model with its inference computation time in the reward function for a reinforcement learning architecture searchexplored using ML to learn to automatically compress ML modelsTPUsTensor Processing UnitsBERTvia ColabTensorFlow Research CloudCloud TPUslarge-scale ML training

An individual TPU v3 device (left) and a portion of a TPU v3 Pod (right). TPU v3 is the latest generation of Google's Tensor Processing Unit (TPU) hardware. Available to external customers as Cloud TPU v3, these systems are liquid-cooled for maximum performance (computer chips + liquid = exciting!), and a full TPU v3 Pod can apply more than 100 petaflops of computational power to the world's largest ML problems.

Open Source Software and DatasetsTensorFlowreleased in November 2015TensorFlow LiteTensorFlow.jsTensorFlow ProbabilityContent Delivery Network10,000 stars on Githubintroduced a new framework for flexible and reproducible reinforcement learningnew visualization tools to rapidly understand the characteristics of a dataseta high-level library for expressing machine learning problems that involve learning-to-ranka framework for fast and flexible AutoML solutions with learning guaranteeslibrary for doing in-browser realtime t-SNE visualizationsFHIR tools and software for working with electronic healthcare data

Real-time evolution of the tSNE embedding for the complete MNIST dataset. The dataset contains images of 60,000 handwritten digits. You can find a live demo here.

Google Dataset Searchcurated and released many new, novel datasets

Pictures from India & Singapore added to Open Images Extended using the Crowdsource app.

Open Images V4extended this dataset to add more diversity of people and scenes from all over the worldcrowdsource.google.comAVAhuman actionsspeech in video.announced an updated YouTube-8M, and the 2nd YouTube-8M Large-Scale Video Understanding Challenge and WorkshopHDR+ Burst Photography DatasetGoogle-LandmarksFluid Annotation

Visualization of the fluid annotation interface in action on image from COCO dataset. Image credit: gamene, original image.

Inclusive Images ChallengeiNaturalist 2018 Challenge Kaggle "Quick, Draw!" Doodle Recognition ChallengeConceptual CaptionsRoboticsteach robots to grasp novel objectsto learn about objects without human supervisioncombining ML and sampling-based methodsICRA'18learning robot geometryperceive the structure of the worlddeep reinforcement learning models online on real robotslearn stable approaches

Applications of AI to Other FieldsNature Methodshigh-precision automated reconstruction of neuronsimproves the accuracy of automated interpretation of connectomics data by an order of magnitude

Our algorithm in action as it traces a single neurite in 3d in a songbird brain.

Finding new planets outside our solar system by data mining light curves of stars

Recognizing the origin or function of short DNA sequences

Automatically detecting out-of-focus microscope images

Digitally creating images of the same cells with multiple stains

Automatically mapping mass spectrometry output to peptide chains

A pre-trained TensorFlow model rates focus quality for a montage of microscope image patches of cells in Fiji (ImageJ). Hue and lightness of the borders denote predicted focus quality and prediction uncertainty, respectively.

Healthapplying ML to healthpublished workis on-par with retinal specialistspairing ophthalmologists and this ML model allow them to make more accurate decisionsVerilyAravind Eye HospitalsRajavithi Hospital affiliated with the Ministry of Health in Thailand

On the left is a retinal fundus image graded as having moderate DR ("Mo") by an adjudication panel of ophthalmologists (ground truth). On the top right is an illustration of the predicted scores ("N" = no DR, "Mi" = Mild DR, "Mo" = Moderate DR) from the model. On the bottom right is the set of scores given by physicians without assistance ("Unassisted") and those who saw the model's predictions ("Grades Only").

medical and eye specialists found quite remarkablecan assess cardiovascular risk from retinal imageshow to improve the grading of prostate cancer using MLdetect metastatic breast cancer with deep learningprototype for an augmented-reality microscopeUniversity of Chicago MedicineUCSFStanford Medicinepublished work in Nature Digital Medicineopen sourced softwareFast Healthcare Interoperability ResourcesGitHub repositoryimproved the accuracy, speed and utility of our deep learning-based variant caller, DeepVariantNature Biotechnologyto proactively advance health equityResearch Outreachproviding multi-year Ph.D. fellowshipsannual Google Ph.D. Fellowship Summit Google AI Residencyits third yearglobal officesmachine learningperceptionalgorithms and optimizationlanguage understandinghealthcaremuch moreGoogle Faculty Research Awards programworkshop on AI/ML Research and PracticeAlgorithms & Optimization Workshopmuch of our research is published openlyICLR 2018NAACL 2018ICML 2018CVPR 2018NeurIPS 2018ECCV 2018EMNLP 2018ASPLOSHPCAICSEIEEE Security & PrivacyOSDISIGCOMMNew Places, New Facesannounced our first AI research office in AfricaParisTokyoAmsterdamPrincetonhere