Google AI Blog

Grounding Natural Language Instructions to Mobile UI Actions

Friday, July 10, 2020

Posted by Yang Li, Research Scientist, Google ResearchMapping Natural Language Instructions to Mobile UI Action SequencesACL 2020automatic action sequence mappingGitHub repositoryConstructing Language Grounding Modelslanguage groundingaction phrase-extractiongrounding

The workflow of grounding language instructions to executable actions.

Transformerarea attention

The action phrase extraction model takes a word sequence of a natural language instruction and outputs a sequence of spans (denoted in red boxes) that indicate the phrases describing the operation, the object and the argument of each action in the task.

The grounding model takes the extracted spans as input and grounds them to executable actions, including the object an action is applied to, given the UI screen at each step during execution.

Resultspublic android UI corpusgraph convolutional networkfeedforward networkConclusionAcknowledgements Many thanks to my collaborators on this work at Google Research. Xin Zhou and Jiacong He contributed substantially to the data pipelines and the creation of the datasets. Yuan Zhang and Jason Baldridge provided much valuable advice for the project and contributed to the presentation of the work. Gang Li provided generous help for creating open-source datasets. Many thanks to Ashwin Kakarla, Muqthar Mohammad and Mohd Majeed for their help with the annotations.

AutoML-Zero: Evolving Code that Learns

Thursday, July 9, 2020

Posted by Esteban Real, Staff Software Engineer, and Chen Liang, Software Engineer, Google Research, Brain Teamdeep neural networksAutoMLneural architecture searchtopicmuchresearchlearning rulesince the early 90sour researchrecent paperAutoML-Zerobackpropagationlinear regressionEvolving Learning Algorithms from Scratchvariantevolutionary methodsproved usefulsince the 80smakes them especially suitable for the discovery of learning algorithmsparentmutated

A population is initialized with empty programs. Many generations later, we see a more evolved population and two of its algorithms compete. The most accurate wins to produce a child. After many such events, the final population contains highly accurate classifiers.

Exploring a Difficult Search Space12 migrationcodestochastic gradient descent

Progress of an evolution experiment. As time passes, from left to right, we see the algorithms becoming more complex and more accurate.

The Evolved Algorithmnoise injection as data augmentationbilinear modelgradient normalizationweight averagingnoisy ReLUConclusion^*AcknowledgementsWe want to thank our co-authors, David R. So and Quoc V. Le, and the many who helped us through discussions during the project and paper writing, including Samy Bengio, Vincent Vanhoucke, Doug Eck, Charles Sutton, Yanping Huang, Jacques Pienaar, Jeff Dean, and particularly Gabriel Bender, Hanxiao Liu, Rishabh Singh, Chiyuan Zhang, and Hieu Pham. We also want to especially thank Tom Small for contributing the animations in this post.* The electricity consumption for the experiments (run in 2019) was matched with the purchase of renewable energy. ↩

Duality — A New Approach to Reinforcement Learning

Wednesday, July 8, 2020

Posted by Ofir Nachum and Bo Dai, Research Scientists, Google ResearchReinforcement learningQ-learningactor-criticconstraint-satisfaction problemmathematical issuesReinforcement Learning via Fenchel-Rockafellar Dualityconvex dualityA Duality-Based Solutionconvex regularizer f-divergenceunconstrainedoff-policyoffline

do notExperimental Evaluation

A plot of the average reward achieved by an agent using the duality-based approach (blue) compared to an agent using standard actor-critic (orange). In addition to being more mathematically principled, our approach also yields better practical results.

Conclusionwithout constraintspolicy optimizationpolicy evaluationimitation learning

Google at ACL 2020

Monday, July 6, 2020

Posted by Cat Armato and Emily Knapp, Program ManagersACL 2020Diamond Level sponsor of ACL 2020Google virtual boothboldedCommitteesVinodkumar Prabhakaran Sushant Kafle Kristina ToutanovaYi Luan Anders Søgaard, Ankur Parikh, Annie Louis, Bhuvana Ramabhadran, Christo Kirov, Daniel Cer, Dipanjan Das, Diyi Yang, Emily Pitler, Eunsol Choi, George Foster, Idan Szpektor, Jacob Eisenstein, Jason Baldridge, Jun Suzuki, Kenton Lee, Luheng He, Marius Pasca, Ming-Wei Chang, Sebastian Gehrmann, Shashi Narayan, Slav Petrov, Vinodkumar Prabhakaran, Waleed Ammar, William CohenLong PapersCross-modal Language Generation using Pivot Stabilization for Web-scale Language CoverageAshish V. Thapliyal, Radu Soricut Automatic Detection of Generated Text is Easiest when Humans are FooledDaphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas Eck On Faithfulness and Factuality in Abstractive SummarizationJoshua Maynez, Shashi Narayan, Bernd Bohnet, Ryan McDonald MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby StepsWang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, Fei ShaDynamic Programming Encoding for Subword Segmentation in Neural Machine TranslationXuanli He, Gholamreza Haffari, Mohammad Norouzi GoEmotions: A Dataset of Fine-Grained EmotionsDorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, Sujith Ravi TaPas: Weakly Supervised Table Parsing via Pre-trainingsee blog post)Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, Julian EisenschlosToxicity Detection: Does Context Really Matter?John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, Ion Androutsopoulos (Re)construing Meaning in NLPSean Trott, Tiago Timponi Torrent, Nancy Chang, Nathan SchneiderPretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language ModelsDan Iter, Kelvin Guu, Larry Lansing, Dan Jurafsky Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question AnsweringHao Cheng, Ming-Wei Chang, Kenton Lee, Kristina Toutanova AdvAug: Robust Adversarial Augmentation for Neural Machine TranslationYong Cheng, Lu Jiang, Wolfgang Macherey, Jacob Eisenstein Named Entity Recognition as Dependency ParsingJuntao Yu, Bernd Bohnet, Massimo Poesio Cross-modal Coherence Modeling for Caption GenerationMalihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, Matthew Stone Representation Learning for Information Extraction from Form-like Documents(see blog post)Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, Marc NajorkLow-Dimensional Hyperbolic Knowledge Graph EmbeddingsInes Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, Christopher Ré What Question Answering can Learn from Trivia NerdsJordan Boyd-Graber, Benjamin Börschinger Learning a Multi-Domain Curriculum for Neural Machine Translation(see blog post)Wei Wang, Ye Tian, Jiquan Ngiam, Yinfei Yang, Isaac Caswell, Zarana Parekh Translationese as a Language in "Multilingual" NMTParker Riley, Isaac Caswell, Markus Freitag, David Grangier Mapping Natural Language Instructions to Mobile UI Action Sequences(see blog post)Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge BLEURT: Learning Robust Metrics for Text Generation(see blog post)Thibault Sellam, Dipanjan Das, Ankur Parikh Exploring Unexplored Generalization Challenges for Cross-Database Semantic ParsingAlane Suhr, Ming-Wei Chang, Peter Shaw, Kenton Lee Frugal Paradigm CompletionAlexander Erdmann, Tom Kenter, Markus Becker, Christian Schallhart Short PapersReverse Engineering Configurations of Neural Text Generation ModelsYi Tay, Dara Bahri, Che Zheng, Clifford Brunk, Donald Metzler, Andrew TomkinsSyntactic Data Augmentation Increases Robustness to Inference HeuristicsJunghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine TranslationAditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, Yonghui WuSocial Biases in NLP Models as Barriers for Persons with DisabilitiesBen Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, Stephen Denuyl Toward Better Storylines with Sentence-Level Language ModelsDaphne Ippolito, David Grangier, Douglas Eck, Chris Callison-Burch TACL PapersTYDI QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages(see blog post)Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria PalomakiPhonotactic Complexity and Its Trade-offsTiago Pimentel, Brian Roark, Ryan CotterellDemosMultilingual Universal Sentence Encoder for Semantic Retrievalsee blog postYinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, Ray KurzweilWorkshopsIWPT - The 16th International Conference on Parsing TechnologiesYuji Matsumoto, Stephan Oepen, Kenji Sagae, Anders Søgaard, Weiwei Sun and Reut TsarfatyALVR - Workshop on Advances in Language and Vision ResearchXin Wang, Jesse Thomason, Ronghang Hu, Xinlei Chen, Peter Anderson, Qi Wu, Asli Celikyilmaz, Jason Baldridge and William Yang WangWNGT - The 4th Workshop on Neural Generation and Translation Alexandra Birch, Graham Neubig, Andrew Finch, Hiroaki Hayashi, Kenneth Heafield, Ioannis Konstas, Yusuke Oda and Xian LiNLPMC - NLP for Medical ConversationsParminder Bhatia, Chaitanya Shivade, Mona Diab, Byron Wallace, Rashmi Gangadharaiah, Nan Du, Izhak Shafran and Steven LinAutoSimTrans - The 1st Workshop on Automatic Simultaneous TranslationHua Wu, Colin Cherry, James Cross, Liang Huang, Zhongjun He, Mark Liberman and Yang LiuTutorialsInterpretability and Analysis in Neural NLP (cutting-edge)Yonatan Belinkov, Sebastian Gehrmann, Ellie PavlickCommonsense Reasoning for Natural Language Processing (Introductory)Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, Dan Roth

SmartReply for YouTube Creators

Wednesday, July 1, 2020

Posted by Rami Al-Rfou, Research Scientist, Google ResearchSmartReplyGmail launchAndroid MessagesAndroid WearPlay Developer ConsoleMLKitTFLite

Deep RetrievalSmartReply for Inboxrecurrent neural networkwe foundmaximum inner product searchexpand SmartReply to GmailBeyond Wordsour recent workresearchTransformerself-attention layersWaveNetcontrastive objectiveaverage pooling

A dual encoder network that maximizes the mutual information between the comments and their replies through a contrastive objective. Each encoder is fed a sequence of bytes and is implemented as a computationally efficient dilated transformer network.

A Model to Learn Them AllThis is a great video,Thanks!

A 2D projection of the model encodings when presented with a hypothetical comment and a small list of potential replies. The neighborhood surrounding English comments (black color) consists of appropriate replies in English and their counterparts in Spanish and Arabic. Note that the network learned to align English replies with their translations without access to any parallel corpus.

When to Suggest?ConclusionAcknowledgementsSmartReply for YouTube creators was developed by Golnaz Farhadi, Ezequiel Baril, Cheng Lee, Claire Yuan, Coty Morrison‎, Joe Simunic‎, Rachel Bransom‎, Rajvi Mehta, Jorge Gonzalez‎, Mark Williams, Uma Roy and many more. We are grateful for the leadership support from Nikhil Dandekar, Eileen Long, Siobhan Quinn, Yun-hsuan Sung, Rachel Bernstein, and Ray Kurzweil.

SpineNet: A Novel Architecture for Object Detection Discovered with Neural Architecture Search

Tuesday, June 30, 2020

Posted by Xianzhi Du, Software Engineer and Jaeyoun Kim, Technical Program Manager, Google ResearchConvolutional neural networks created for image tasks typically encode an input image into a sequence of intermediate features that capture the semantics of an image (from local to global), where each subsequent layer has a lower spatial dimension. However, this scale-decreased model may not be able to deliver strong features for multi-scale visual recognition tasks where recognition and localization are both important (e.g., object detection and segmentation). Several works including FPN and DeepLabv3+ propose multi-scale encoder-decoder architectures to address this issue, where a scale-decreased network (e.g., a ResNet) is taken as the encoder (commonly referred to as a backbone model). A decoder network is then applied to the backbone to recover the spatial information.
While this architecture has yielded improved success for image recognition and localization tasks, it still relies on a scale-decreased backbone that throws away spatial information by down-sampling, which the decoder then must attempt to recover. What if one were to design an alternate backbone model that avoids this loss of spatial information, and is thus inherently well-suited for simultaneous image recognition and localization?CVPR 2020SpineNet: Learning Scale-Permuted Backbone for Recognition and Localizationscale-permuted modelneural architecture searchTensorflow TPU GitHub repositoryTensorFlow Model Garden GitHub repository

A scale-decreased backbone is shown on the left and a scale-permuted backbone is shown on the right. Each rectangle represents a building block. Colors and shapes represent different spatial resolutions and feature dimensions. Arrows represent connections among building blocks.

Design of SpineNet ArchitectureCOCO dataset

Scale permutations: The orderings of network building blocks are important because each block can only be built from those that already exist (i.e., with a “lower ordering”). We define the search space of scale permutations by rearranging intermediate and output blocks, respectively.

Cross-scale connections: We define two input connections for each block in the search space. The parent blocks can be any block with a lower ordering or a block from the stem network.

Block adjustments (optional): We allow the block to adjust its scale level and type.

The architecture search process from a scale-decreased backbone to a scale-permuted backbone.

ResNet-50ResNet-50-FPNaverage precisionFLOPsresidual blockbottleneck blockResNet

The architecture comparison between a ResNet backbone (left) and the SpineNet backbone (right) derived from it using NAS.

PerformanceCOCOiNaturalist

Performance comparisons of SpineNet models and ResNet-FPN models adopting the RetinaNet detection framework on COCO bounding box detection.

Performance comparisons of SpineNet models and ResNet models on ImageNet classification and iNaturalist fine-grained image classification.

ConclusionAcknowledgementsSpecial thanks to the co-authors of the paper: Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V. Le, and Xiaodan Song. We also would like to acknowledge Yeqing Li, Youlong Cheng, Jing Li, Jianwei Xie, Russell Power, Hongkun Yu, Chad Richards, Liang-Chieh Chen, Anelia Angelova, and the larger Google Brain Team for their help.

Leveraging Temporal Context for Object Detection

Friday, June 26, 2020

Posted by Sara Beery, Student Researcher, and Jonathan Huang, Research Scientist, Google Researchmountain pass road conditionsecosystem phenologyleading to poor generalization to novel deploymentsLILA BCWildlife Insights

What’s in this image? Objects in images from static cameras can be very challenging to detect and categorize. Here, a foggy morning has made it very difficult to see a herd of wildebeest walking along the crest of a hill. [Image from Snapshot Serengeti]

Context R-CNN: Long Term Temporal Context for Per-Camera Object DetectionFaster R-CNNTF Object Detection API

Here, we can see how additional examples from the same scene help experts determine that the object is an animal and not background. Context such as the shape & size of the object, its attachment to a herd, and habitual grazing at certain times of day help determine that the species is a wildebeest. Useful examples occur throughout the month.

The Context R-CNN ModelFaster R-CNNmemory bankattention

High-level architecture diagram, showing how Context R-CNN incorporates long-term context within the Faster R-CNN model architecture.

similarity-based attentionmemory bankMM

Context R-CNN is able to leverage context (spanning up to 1 month) to correctly categorize the challenging wildebeest example we saw above. The green values are the corresponding attention weights for each boxed object.

Compared to a Faster R-CNN baseline (left), Context R-CNN (right) is able to capture challenging objects such as an elephant occluded by a tree, two poorly-lit impala, and a vervet monkey leaving the frame. [Images from Snapshot Serengeti]

ResultsSnapshot SerengetiCaltech Camera Trapsmean average precisionS3D

Comparison to a single frame Faster R-CNN baseline, showing both mean average precision (mAP) and average recall (AR) detection metrics.

Ongoing and Future WorkWildlife InsightsiWildCamCVPR Fine-Grained Visual Recognition WorkshopAcknowledgementsThis post reflects the work of the authors as well as the following group of core contributors: Vivek Rathod, Guanhang Wu, Ronny Votel. We are also grateful to Zhichao Lu, David Ross, Tanya Birch and the Wildlife Insights AI team, and Pietro Perona and the Caltech Computational Vision Lab.

Sensing Force-Based Gestures on the Pixel 4

Wednesday, June 24, 2020

Posted by Philip Quinn and Wenxin Feng, Research Scientists, Android UXsoftfirmrecent update to the Pixel 4Touch Sensor Technology and Finger Biomechanicsdrive electrodesense electrodedielectriccell

Left: A finger interacts with a touch sensor cell by ‘stealing’ charge from the projected field around two electrodes. Right: A capacitive touch sensor is constructed from rows and columns of electrodes, separated by a dielectric. The electrodes overlap at cells, where capacitance is measured.

Slowed touch sensor recordings of a user tapping (left), pressing (middle), and scrolling (right).

per se

Touch sensor signals are saturated around the centre of the finger’s contact, but fall off at the edges. This allows us to sense small deformations in the finger’s contact shape caused by changes in the finger’s force.

Machine Learning for Touch Interactionpress gestureconvolutionalrecurrentTensorFlow Lite

An overview of the classification model’s architecture.

User Experience Integrationpress

A user long pressing (left) and firmly pressing (right) on a launcher icon.

GestureDetectorViewMotionEvent classificationAcknowledgementsThis project is a collaborative effort between the Android UX, Pixel software, and Android framework teams.

Presenting a Challenge and Workshop in Efficient Open-Domain Question Answering

Tuesday, June 23, 2020

Posted by Eunsol Choi, Visiting Faculty Researcher and Tom Kwiatkowski, Research Scientist, Google Researchknowledge graphWikipediaT5EfficientQA competition and workshopNeurIPS 2020Princeton UniversityUniversity of Washingtonquestion answering

An illustration of how the memory budget changes as a neural network and retrieval corpus grow and shrink. It is possible that successful systems will also use other resources such as a knowledge graph.

Competition OverviewNatural Questions datasetcompetition track at NeurIPS 20202017 NeurIPS Human-Computer competitionParticipationcompetition siteAcknowledgementsAdam Roberts, Colin Raffel, Chris Alberti, Jordan Boyd-Graber, Jennimaria Palomaki, Kenton Lee, Kelvin Guu, Michael CollinsSewon Min Hannaneh HajishirzDanqi Chen

RepNet: Counting Repetitions in Videos

Monday, June 22, 2020

Posted by Debidatta Dwibedi, Research Scientist, Robotics at GoogleCounting Out Time: Class Agnostic Video Repetition Counting in the WildRepNetour previous work, whichdatasetColab notebookRepNetIn the pastResNetResNetimagevideotemporal self-similarity matrix

Demonstration of how the TSM processes images of the Earth’s day-night cycle.

Transformers

Overview of the RepNet model.

Temporal Self-Similarity Matrix

Jumping Jacks (constant period; video from Kinetics), Bouncing ball (decreasing period; Kinetics), Mixing concrete (aperiodic segments present in video; PERTUBE dataset).

Data

Our synthetic data generation pipeline that produces videos with repetitions from any video.

camera motion augmentationaffine motion

Left: An example of a synthetic repeating video generated from a random video. Right: An example of a video with camera motion augmentation, which is tougher for the model, but results in better generalization to real repeating videos (both from Kinetics).

EvaluationKinetics datasetCountixApplications

RepNet can count repeated activities from a range of domains, such as slicing onions (left; video from Kinetics dataset), Earth’s diurnal cycle (middle; Himawari satellite data), or even a cheetah in motion (right; video from imgur.com).

Predicted heart rates: 45 bpm (left) and 75 bpm (right). True heart rates 46-50 bpm and 78-79 bpm, respectively. RepNet’s prediction of the heart rate across different devices is encouragingly close to the rate measured by the device. (Source for left and right)

In this video, we see RepNet counting accelerating cellular oscillations observed under a laser microscope even though it has never seen such a video during training, (from Nature article).

Left: Person performing a “mountain climber” exercise. Right: The 1D projection of the RepNet embeddings using principal component analysis, capturing the moment that the person changes their speed during the exercise. (Video from Kinetics)

ReleaseannotationsColab notebookAcknowledgementsThis is joint work with Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Special thanks to Tom Small for designing the visual explanation of TSM. The authors thank Anelia Angelova, Relja Arandjelović, Sourish Chaudhuri, Aishwarya Gomatam, Meghana Thotakuri, and Vincent Vanhoucke for their help with this project.