Google AI Blog: October 2019

Learning to Assemble and to Generalize from Self-Supervised Disassembly

Thursday, October 31, 2019

Posted by Kevin Zakka, Research Intern and Andy Zeng, Research Scientist, Robotics at GoogleForm2Fit

Form2Fit learns to assemble a wide variety of kits by finding geometric correspondences between object surfaces and their target placement locations. By leveraging geometric information learned from multiple kits during training, the system generalizes to new objects and kits.

Data-Driven Shape Descriptors For Generalizable Assemblyorientation-sensitive geometric

Overview of Form2Fit. The suction and place networks infer candidate picking and placing locations in the scene respectively. The matching network generates pixel-wise orientation-sensitive descriptors to match picking locations to their corresponding placing locations. The planner then integrates it all to control the robot to execute the next best pick and place action.

Learning Assembly from Disassembly

An example of self-supervision through time-reversal: rewinding a disassembly sequence of a deodorant kit over time generates a valid assembly sequence.

time-reversed disassemblyGeneralization Results

Form2Fit policies are robust to a wide range of rotations and translations of the kits.

Form2Fit policies can generalize to novel kit configurations such as multiple versions of the same kit and mixtures of different kits.

Form2Fit policies can generalize to never-before-seen single and multi-object kits.

What Have the Descriptors Learned?t-SNE

The t-SNE embedding of the learned object descriptors. Similarly oriented objects of the same category display identical colors (e.g. A, B or F, G) while different objects (e.g. C, H) and same objects but different orientation (e.g. A, C, D or H, F) exhibit different colors.

Limitations & Future WorkGitHub repositoryAcknowledgmentsThis research was done by Kevin Zakka, Andy Zeng, Johnny Lee, and Shuran Song (faculty at Columbia University), with special thanks to Nick Hynes, Alex Nichol, and Ivan Krasin for fruitful technical discussions; Adrian Wong, Brandon Hurd, Julian Salazar, and Sean Snyder for hardware support; Ryan Hickman for valuable managerial support; and Chad Richards for helpful feedback on writing.

On-Device Captioning with Live Caption

Tuesday, October 29, 2019

Posted by Michelle Tadmor-Ramanovich and Nadav Bar, Senior Software Engineers, Google Research, Tel-Aviv

Recently we introduced Live Caption, a new Android feature that automatically captions media playing on your phone. The captioning happens in real time, completely on-device, without using network resources, thus preserving privacy and lowering latency. The feature is currently available on Pixel 4 and Pixel 4 XL, will roll out to Pixel 3 models later this year, and will be more widely available on other Android devices soon.

When media is playing, Live Caption can be launched with a single tap from the volume control to display a caption box on the screen.

Building Live Caption for Accuracy and Efficiencyrecurrent neural networksequence transductionRNN-Tunspoken punctuationconvolutional neural networksound events classification[APPLAUSE][MUSIC]

Incoming sound is processed through a Sound Recognition and ASR feedback loop. The produced text or sound label is formatted and added to the caption.

sound events detectionAudioSetautomatic speech recognition[MUSIC]optimized for edge-devicesperformswellTensorFlow LiteLooking forwardmulti-speaker contentAcknowledgementsThe core team includes Robert Berry, Anthony Tripaldi, Danielle Cohen, Anna Belozovsky, Yoni Tsafir, Elliott Burford, Justin Lee, Kelsie Van Deman, Nicole Bleuel, Brian Kemler, and Benny Schlesinger. We would like to thank the Google Speech team, especially Qiao Liang, Arun Narayanan, and Rohit Prabhavalkar for their insightful work on the ASR model as well as Chung-Cheng Chiu from Google Brain Team; Dan Ellis and Justin Paul for their help with integrating the Sound Recognition model; Tal Remez for his help in developing the punctuation model; Kevin Rocard and Eric Laurent‎ for their help with the Android audio capture API; and Eugenio Marchiori, Shivanker Goel, Ye Wen, Jay Yoo, Asela Gunawardana, and Tom Hume for their help with the Android infrastructure work.

Introducing the Schema-Guided Dialogue Dataset for Conversational Assistants

Monday, October 28, 2019

Posted by Abhinav Rastogi, Software Engineer and Pranav Khaitan, Engineering Lead, Google Researchvirtual assistantsGoogle AssistantTowards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue DatasetSchema-Guided Dialogue datasetintent predictionslot fillinglanguage generationschema-guided approach BERTopen sourceAPIsThe Datasetbankseventsmedia, calendartravelweather. Wizard-of-Ozprobabilistic automaton

Steps for obtaining dialogues, with assistant turns marked in red and user turns in blue. Left: The simulator generates a dialogue skeleton using a finite set of actions. Center: Actions are converted into utterances using templates (~50 per service) and slot values are replaced with natural variations. Right: Paraphrasing via crowdsourcing to make the flow cohesive.

The Schema-Guided Approachopen-sourced dialogue state tracking modelEighth Dialogue System Technology ChallengeDialog System Technology ChallengesSchema-Guided Dialogue State Tracking8th DSTCDSTC8 workshopAAAI-20AcknowledgementsThis post reflects the work of our co-authors Xiaoxue Zang, Srinivas Sunkara and Raghav Gupta. We also thank Amir Fayazi and Maria Wang for help with data collection and Guan-Lin Chao for insights on model design and implementation.

Google at ICCV 2019

Monday, October 28, 2019

Posted by Andrew Helton, Editor, Google Research CommunicationsInternational Conference on Computer Vision 2019workshopstutorialsMediaPipeOpen ImagesGoogle Lens

Distinguished Researcher Award — Bill Freeman, Research Scientist, Google Research

Helmholtz Prize (Test of Time Award) — ICCV 2009 paper, "Building Rome in a Day", by Sameer Agarwal, Noah Snavely, Ian Simon, Steve Seitz and Rick Szeliski

Marr Prize (Best Paper Award) — ICCV 2019 paper, "SinGAN: Learning a Generative Model from a Single Natural Image", by Tamar Rott Shaham, Tali Dekel and Tomer Michaeli

blueOrganizing Committee Oral PresentationsLearning Single Camera Depth Estimation using Dual-PixelsRahul Garg, Neal Wadhwa, Sameer Ansari, Jonathan Barron RIO: 3D Object Instance Re-Localization in Changing Indoor EnvironmentsJohanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, Matthias Niessner ShapeMask: Learning to Segment Novel Objects by Refining Shape PriorsWeicheng Kuo, Anelia Angelova, Jitendra Malik, Tsung-Yi Lin PuppetGAN: Cross-Domain Image Manipulation by DemonstrationBen Usman, Nick Dufour, Kate Saenko, Chris Bregler
COCO-GAN: Generation by Parts via Conditional CoordinatingChieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, Hwann-Tzong ChenTowards Unconstrained End-to-End Text SpottingSiyang Qin, Alessandro Bissaco, Michalis Raptis, Yasuhisa Fujii, Ying XiaoSinGAN: Learning a Generative Model from a Single Natural ImageTamar Rott Shaham, Tali Dekel, Tomer Michaeli Generative Modeling for Small-Data Object DetectionLanlan Liu, Michael Muelly, Jia Deng, Tomas Pfister, Li-Jia Li Searching for MobileNetV3Andrew Howard, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, Yukun Zhu, Ruoming Pang, Hartwig Adam, Quoc Le S⁴L: Self-Supervised Semi-supervised LearningLucas Beyer, Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance PropagationJanis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, Federico TombariLinearized Multi-sampling for Differentiable Image TransformationWei Jiang, Weiwei Sun, Andrea Tagliasacchi, Eduard Trulls, Kwang Moo Yi
Poster PresentationsELF: Embedded Localisation of Features in Pre-trained CNNAssia Benbihi, Matthieu Geist, Cedric Pradalier Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown CamerasAriel Gordon, Hanhan Li, Rico Jonschkowski, Anelia AngelovaForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth ImageYida Wang, David Joseph Tan, Nassir Navab, Federico Tombari A Learned Representation for Scalable Vector GraphicsRaphael Gontijo Lopes, David Ha, Douglas Eck, Jonathon Shlens FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB ImageJingwei Huang, Yichao Zhou, Thomas Funkhouser, Leonidas GuibasPrior-Aware Neural Network for Partially-Supervised Multi-Organ SegmentationYuyin Zhou, Zhe Li, Song Bai, Xinlei Chen, Mei Han, Chong Wang, Elliot Fishman, Alan Yuille Boundless: Generative Adversarial Networks for Image ExtensionDilip Krishnan, Piotr Teterwak, Aaron Sarna, Aaron Maschinot, Ce Liu, David Belanger, William FreemanCap2Det: Learning to Amplify Weak Caption Supervision for Object DetectionKeren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, Jesse Berent NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi-supervised Object DetectionJiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, Ram Nevatia Object-Driven Multi-Layer Scene Decomposition from a Single ImageHelisa Dhamo, Nassir Navab, Federico Tombari Improving Adversarial Robustness via Guided Complement EntropyHao-Yun Chen, Jhao-Hong Liang, Shih-Chieh Chang, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, Da-Cheng Juan XRAI: Better Attributions Through RegionsAndrei Kapishnikov, Tolga Bolukbasi, Fernanda Viegas, Michael Terry
SegSort: Segment Sorting for Semantic SegmentationJyh-Jing Hwang, Stella Yu, Jianbo Shi, Maxwell Collins, Tien-Ju Yang, Xiao Zhang, Liang-Chieh Chen Self-Supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and CameraYuhua Chen, Cordelia Schmid, Cristian Sminchisescu VideoBERT: A Joint Model for Video and Language Representation LearningChen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid Explaining the Ambiguity of Object Detection and 6D Pose from Visual DataFabian Manhardt, Diego Martín Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, Federico Tombari Constructing Self-Motivated Pyramid Curriculums for Cross-Domain Semantic SegmentationQing Lian, Lixin Duan, Fengmao Lv, Boqing Gong Learning Shape Templates Using Structured Implicit FunctionsKyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, Thomas FunkhouserTransferable Representation Learning in Vision-and-Language NavigationHaoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, Eugene Ie Controllable Attention for Structured Layered Video DecompositionJean-Baptiste Alayrac, Joao Carreira, Relja Arandjelović, Andrew ZissermanPixel2Mesh++: Multi-view 3D Mesh Generation via DeformationChao Wen, Yinda Zhang, Zhuwen Li, Yanwei FuBeyond Cartesian Representations for Local DescriptorsPatrick Ebel, Anastasiia Mishchuk, Kwang Moo Yi, Pascal Fua, Eduard TrullsDomain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain DataXiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, Boqing Gong Evolving Space-Time Neural Architectures for VideosAJ Piergiovanni, Anelia Angelova, Alexander Toshev, Michael Ryoo Moulding Humans: Non-parametric 3D Human Shape Estimation from Single ImagesValentin Gabeur, Jean-Sebastien Franco, Xavier Martin, Cordelia Schmid, Gregory RogezMulti-view Image FusionMarc Comino Trinidad, Ricardo Martin-Brualla, Florian Kainz, Janne Kontkanen EvalNorm: Estimating Batch Normalization Statistics for EvaluationSaurabh Singh, Abhinav ShrivastavaAttention Augmented Convolutional NetworksIrwan Bello, Barret Zoph, Quoc Le, Ashish Vaswani, Jonathon Shlens Patchwork: A Patch-wise Attention Network for Efficient Object Detection and Segmentation in Video StreamsYuning ChaiWorkshopsLow-Power Computer VisionBo ChenNeural ArchitectsBarret ZophThe 3rd YouTube-8M Large-Scale Video Understanding WorkshopPaul Natsev, Cordelia Schmid, Rahul Sukthankar, Joonseok Lee, George TodericiShould We Pre-register Experiments in Computer Vision?Jack ValmadreExtreme Vision ModelingRahul SukthankarJoint COCO and Mapillary Recognition ChallengeTsung-Yi Lin, Yin CuiOpen Images ChallengeVittorio Ferrari, Alina Kuznetsova, Rodrigo Benenson, Victor Gomes, Matteo Malloci
TutorialsKevin Swersky

A New Workflow for Collaborative Machine Learning Research in Biodiversity

Friday, October 25, 2019

Posted by Serge Belongie, Visiting Faculty and Hartwig Adam, Engineering Director, Google ResearchspeciesidentificationFGVCLifeCLEFwildlife camera trap imagesherbarium sheetsscientific research publicationsdesigned with these considerations in mindBiodiversity NextGBIFiNaturalistVisipediaannouncingbillion+ species occurrence countmediated data

To assist in developing and refining machine vision models, GBIF will package datasets, taking care to ensure license and citation practice are respected. The training datasets will be issued a Digital Object Identifier (DOI), and will be linked through the DOI citation graph.

To assist application developers, Google and Visipedia will train and publish publicly accessible models with documentation on TensorFlow Hub. These models can then, in turn, be deployed in biodiversity research and citizen science efforts.

Case Study: Recognizing Fungi Species from Photos with the Interactive Mushroom RecognizerDanish Mycological SocietyTF Hub model pagelive, interactive demonstration

Illustration of live, interactive Mushroom Recognizer, powered by a publicly available model trained on a fungi dataset provided by the Danish Mycological Society.

Invitation to Participateproject pageAcknowledgementsWe’d like to thank our collaborators at GBIF, iNaturalist, and Visipedia for working together to develop this workflow. At Google we would like to thank Christine Kaeser-Chen, Chenyang Zhang, Yulong Liu, Kiat Chuan Tan, Christy Cui, Arvi Gjoka, Denis Brulé, Cédric Deltheil, Clément Beauseigneur, Grace Chu, Andrew Howard, Sara Beery, and Katherine Chou.

Learning to Smell: Using Deep Learning to Predict the Olfactory Properties of Molecules

Thursday, October 24, 2019

Posted by Alexander B Wiltschko, Senior Research Scientist, Google Researchtrigger vivid memoriesolfactory receptorsolfactory sensory neuronsolfactory epitheliumolfactory bulbVanillinsweet, vanilla, creamy,chocolatemulti-label classificationMachine Learning for Scent: Learning Generalizable Perceptual Representations of Small MoleculesgraphsGraph Neural Networks for Odor Predictionnatural model of choicemessage passing

Each node is represented as a vector, and each entry in the vector initially encodes some atomic-level information.

For each node we look at adjacent nodes and collect their information, which is then transformed with a neural network into new information for the centered node. This procedure is performed iteratively. Other variants of GNNs utilize edge and graph-level information.

Illustration of a GNN for odor prediction. We translate the structure of molecules into graphs that are fed into GNN layers to learn a better representation of the nodes. These nodes are reduced into a single vector and passed into a neural network that is used to predict multiple odor descriptors.

stereoisomers(R)- and (S)-carvonerandom forests

Example of the performance of a GNN on odor descriptors against a strong baseline, as measured by the AUROC score. Example odor descriptors are picked randomly. Closer to 1.0 means better. In the majority of cases GNNs outperform the field-standard baseline substantially, with similar performance seen against other metrics (e.g., AUPRC, recall, precision).

Learning from the Model, and Extending It to Other Taskscolor spacecommon chemoinformatic representation

2D snapshot of our embedding space with some example odors highlighted. Left: Each odor is clustered in its own space. Right: The hierarchical nature of the odor descriptor. Shaded and contoured areas are computed with a kernel-density estimate of the embeddings.

Future WorkAcknowledgementsThis early research is the result of the work and advisement of a team of talented researchers and engineers in Google Brain — Benjamin Sanchez-Lengeling, Jennifer Wei, Brian Lee, Emily Reif, Carey Radebaugh, Max Bileschi, Yoni Halpern, and D. Sculley. We are delighted to have collaborated on this work with Richard Gerkin at ASU and Alán Aspuru-Guzik at the University of Toronto. We are of course building on an enormous amount of prior work, and have benefitted particularly from work by Justin Gilmer, George Dahl and others on fundamental methodology in GNNs, among many other works in neuroscience, statistics and chemistry. We are also grateful to helpful comments from Steven Kearnes, David Belanger, Joel Mainland, and Emily Mayhew.

Quantum Supremacy Using a Programmable Superconducting Processor

Wednesday, October 23, 2019

Posted by John Martinis, Chief Scientist Quantum Hardware and Sergio Boixo, Chief Scientist Quantum Computing Theory, Google AI Quantumquantum computingquantum supremacyNatureQuantum Supremacy Using a Programmable Superconducting Processorquantum logic gates

Left: Artist's rendition of the Sycamore processor mounted in the cryostat. (Full Res Version; Forest Stearns, Google AI Quantum Artist in Residence) Right: Photograph of the Sycamore processor. (Full Res Version; Erik Lucero, Research Scientist and Lead Production Quantum Hardware)

The Experimentquantum interference

Process for demonstrating quantum supremacy.

Estimate of the equivalent classical computation time assuming 1M CPU cores for quantum supremacy circuits as a function of the number of qubits and number of cycles for the Schrödinger-Feynman algorithm. The star shows the estimated computation time for the largest experimental circuits.

extended Church-Turing thesisThe Sycamore Processortwo-qubit gatesforward compatible

Heat map showing single- (e1; crosses) and two-qubit (e2; bars) Pauli errors for all qubits operating simultaneously. The layout shown follows the distribution of the qubits on the processor. (Courtesy of Nature magazine.)

Testing Quantum Physics Prior experimentsstate-spaceApplicationsWhat’s Next?NISQ processorsAcknowledgementsWe’d like to thank our collaborators and contributors — University of California Santa Barbara, NASA Ames Research Center, Oak Ridge National Laboratory, Forschungszentrum Jülich, and many others who helped along the way.

Audio and Visual Quality Measurement Using Fréchet Distance

Tuesday, October 22, 2019

Posted by Kevin Kilgour, Software Engineer and Thomas Unterthiner, Research Software Engineer, Google Research, Zürich"I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”— William Thomson (Lord Kelvin), Lecture on "Electrical Units of Measurement" (3 May 1883), published in Popular Lectures Vol. I, p. 73deep learningImageNetPenn Treebankgenerative models

Videos generated from various models trained on sequences from the StarCraft Video (SCV) dataset.

Fréchet Audio Distance: A Metric for Evaluating Music Enhancement AlgorithmsTowards Accurate Generative Models of Video: A New Metric & ChallengesFVDFADGeneral Description of FréchetDistancevery difficultgenerative adversarial networksFréchet Inception DistanceInception object-recognition networkFréchet distance

A key component for both metrics is a pre-trained model that converts the video or audio clip into an N-dimensional embedding.

Fréchet Audio Distance and Fréchet Video Distancepeak signal-to-noise ratiostructural similarity index

Examples of videos of a robot arm, judged by the new FVD metric. FVD values were found to be approximately 2000, 1000, 600, 400, 300 and 150 (left-to-right; top-to-bottom). A lower FVD clearly correlates with higher video quality.

source-to-distortion ratiomultivariate Gaussian distributions

In the animation, we can see that as the magnitude of the distortions increases, the Gaussian distributions of the distorted audio overlaps less with the clean distribution. The magnitude of this separation is what the Fréchet distance is measuring.

EvaluationWhich audio clip sounds most like a studio-produced recording?”Plackett-Luce model

This figure compares the FAD calculated between clean background music and music distorted by a variety of methods (e.g., pitch down, Gaussian noise, etc.) to the associated worth values from human evaluation. Each type of distortion has two data points representing high and low extremes in the distortion applied. The quantization distortion (purple circles), for example, limits the audio to a specific number of bits per sample, where the two data points represent two different bitrates. Both human raters and the FAD assigned higher values (i.e., “less realistic”) to the lower bitrate quantization. Overall log FAD correlates well with human judgement — a perfect correlation between the log FAD and human perception would result in a straight line.

ConclusionAcknowledgementsThere are many people who contributed to this large effort, and we’d like to highlight some of the key contributors: Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, Sylvain Gelly, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi as well as the extended Google Brain Team in Zürich.

Blog