Google AI Blog

New Solutions for Quantum Gravity with TensorFlow

Friday, November 15, 2019

Posted by Thomas Fischbacher, Researcher in Compression, Google Research, Zürichteaching robots how to throwolfactory properties of moleculesTensorFlowquantum mechanicselectromagneticstrongweakgravitationquantum gravityM-TheoryThe Theory formerly known as Strings,”Edward WittenxyztarticleJournal of High Energy Physicsto 194tachyon-freeGoogle colabPython libraryApplying TensorFlow to M-TheoryConclusionAcknowledgementsThis research was conducted by Iulia M. Comşa, Moritz Firsching, and Thomas Fischbacher. Additional thanks go to Jyrki Alakuijala, Rahul Sukthankar, and Jay Yagnik for encouragement and support.

SPICE: Self-Supervised Pitch Estimation

Thursday, November 14, 2019

Posted by Marco Tagliasacchi, Research Scientist, Google Research
pYINSWIPECREPEpaperself-supervisionpretextunsupervised

Constant-Q transform of an audio clip, superimposed on a representation of pitch as estimated by SPICE (video).

convolutional encoderconstant-Q transformharmonic

SPICE model architecture (simplified). Two pitch-shifted versions of the same CQT frame are fed to two encoders with shared weights. The loss is designed to make the difference between the outputs of the encoders proportional to the relative pitch difference. In addition (not shown), a reconstruction loss is added to regularize the model. The model also learns to produce the confidence of the pitch estimation.

no access to ground truth labelsMIR-1k dataset

Evaluation on the MIR-1k dataset, mixing in background music at different signal-to-noise ratios.

FreddieMeterAcknowledgmentsThe work described here was authored by Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi and Mihajlo Velimirović. We are grateful for all discussions and feedback on this work that we received from our colleagues at Google. The SingingVoices dataset used for training the models in this work has been collected by Alexandra Gherghina as part of FreddieMeter, which is using SPICE and a vocal timbre similarity model to understand how closely a singer matches Freddie Mercury.

Introducing the Next Generation of On-Device Vision Models: MobileNetV3 and MobileNetEdgeTPU

Wednesday, November 13, 2019

Posted by Andrew Howard, Software Engineer and Suyog Gupta, Silicon Engineer, Google Researchalgorithmically-efficientGoogle Pixel 4Pixel Neural CoreEdge TPUedge computingMobileNetsMobileNetV3MobileNetEdgeTPUAutoMLMobileNetV2Building MobileNetV3previous version of MobileNetMnasNetNetAdaptreinforcement learning

Comparison of accuracy vs. latency for mobile models on the ImageNet classification task using the Google Pixel 4 CPU.

MobileNetV3 Search SpaceSwish

squeeze-and-excitationsigmoid functioninverted bottleneck structure

MobileNetV3 extends the MobileNetV2 inverted bottleneck structure by adding h-swish and mobile friendly squeeze-and-excitation as searchable options.

Size of expansion layer

Degree of squeeze-excite compression

Choice of activation function: h-swish or ReLU

Number of layers for each resolution block

MobileNetV3 Object Detection and Semantic SegmentationCOCO datasetsemantic segmentationCityscapes DatasetMobileNet for Edge TPUsCoralaccelerator-aware AutoMLDSPsGPUsEfficientNet-EdgeTPU

Left: Comparison of the accuracy on the ImageNet classification task between MobileNetEdgeTPU and other image classification networks designed for mobile when running on Pixel4 Edge TPU. MobileNetEdgeTPU achieves higher accuracy and lower latency compared with other models. Right: Average Edge TPU power in Watts for different classification models running at 30 frames per second (fps).

Objection Detection Using MobileNetEdgeTPUCOCO14 minival datasetThe Need for Hardware-Aware Models

MobileNetV3 is still the best performing network when using mobile CPU as the deployment target.

For Researchers and DevelopersMobileNet github pageTensorflow Object Detection APIDeepLabAcknowledgementsThis work is made possible through a collaboration spanning several teams across Google. We’d like to acknowledge contributions from Berkin Akin, Okan Arikan, Gabriel Bender, Bo Chen, Liang-Chieh Chen, Grace Chu, Eddy Hsu, John Joseph, Pieter-jan Kindermans, Quoc Le, Owen Lin, Hanxiao Liu, Yun Long, Ravi Narayanaswami, Ruoming Pang, Mark Sandler, Mingxing Tan, Vijay Vasudevan, Weijun Wang, Dong Hyuk Woo, Dmitry Kalenichenko, Yunyang Xiong, Yukun Zhu and support from Hartwig Adam, Blaise Agüera y Arcas, Chidu Krishnan and Steve Molloy.

New Insights into Human Mobility with Privacy Preserving Aggregation

Tuesday, November 12, 2019

Posted by Adam Sadilek, Software Engineer and Xerxes Dotiwalla, Product Manager, Google Researchpredicting epidemicsinfrastructure planningnatural disastersimportantdomainsbased on cell carrier logslocation "check-ins"aggregateHierarchical Organization of Urban Mobility and Its Connection with City LivabilityNature Communications

Visualization of privacy-first computation of the mobility map. Individual data points are automatically aggregated together with differential privacy noise added. Then, flows of these aggregate and obfuscated populations are studied.

Computing a Global Mobility Map While Preserving User PrivacyAI principlesdifferential privacyk-anonymityimplemented in 2014Location History, in order toS2 cellslive trafficparking availabilityMobility Map ApplicationsChristaller’s workmove

Mobility maps of Paris (left) and Los Angeles (right). Both cities have similar population sizes, but very different mobility patterns. Paris has an "onion" structure exhibiting a distinct center with a high degree of mobility (red) that progressively decreases as we move away from the center (in order: orange, yellow, green, blue). In contrast, Los Angeles has a large number of high-mobility areas scattered throughout the region.

population densitysprawl

Connecting flow hierarchy Φ with urban indicators in a sample of US cities. Proportion of trips as a function of Φ, broken down by model share: private car, public transportation, and walking. Sample city names that appear in the plot: ATL (Atlanta), CHA (Charlotte), CHI (Chicago), HOU (Houston), LA (Los Angeles), MIN (Minneapolis), NY (New York City), and SF (San Francisco). We see that cities with higher flow hierarchy exhibit significantly higher rates of public transportation use, less car use, and more walkability.

20 different variablesdebateoptimalNext StepsAI for Social Goodon-device federated learningsecure aggregation protocolsrandomized responsessecure Chrome from malicious attacksAcknowledgementsThis work resulted from a collaboration of Aleix Bassolas and José J. Ramasco from the Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), Brian Dickinson, Hugo Barbosa-Filho, Gourab Ghoshal, Surendra A. Hazarie, and Henry Kautz from the Computer Science Department and Ghoshal Lab at the University of Rochester, Riccardo Gallotti from the Bruno Kessler Foundation, and Xerxes Dotiwalla, Paul Eastham, Bryant Gipson, Onur Kucuktunc, Allison Lieber, Adam Sadilek at Google.The differential privacy library used in this work is open source and available on our GitHub repo.

Highlights from the 3rd Cohort of the Google AI Residency Program

Thursday, November 7, 2019

Posted by Katie Meckley, Program Manager, Google AI ResidencyGoogle AI Residency Program machine perception algorithms and optimization language understanding healthcare

A large-scale study on cross-lingual transfer in massive multilingual neural machine translation models (recently highlighted as part of this post), trained on billions of sentence pairs from more than 100 languages in order to significantly improve translation for both low- and high-resource languages.

Visualization of the clustering of encoder representations of all modeled languages, based on representational similarity. Encoder representations of different languages cluster according to linguistic similarity. Languages are color-coded by their linguistic family.

A generative model for Scalable Vector Graphics (SVGs), which can be used to aid designers in generating fonts.

Top: Unlike pixel representations of icons (right), in this case a "6", SVGs (left; middle) are scale-invariant representations. Bottom: By modelling SVGs directly, we can aid artists in quickly and intuitively iterating over typography designs.

A method to learn GANs using discrepancy divergence, a measure that accounts for both the loss function and hypothesis set to provide theoretical learning guarantees.

As more generators are added to the DGAN ensemble more modes in the real distribution are covered. From left to right: 1 generator, 5 generators, and 10 generators.

A likelihood ratio method for deep generative models that effectively corrects for confounding background statistics to improve out-of-distribution (OOD) detection, and a new benchmark dataset for OOD detection in genomics.

Log-likelihood (left) and log likelihood-ratio (right) of each pixel for Fashion-MNIST. The likelihood is dominated by the “background” pixels, whereas the likelihood ratio focuses on the “semantic” pixels and is thus better for OOD detection.

A study showing when label smoothing helps, focusing on its impact on calibration of predictions, representations learned by the penultimate layer and effectiveness of knowledge distillation.

2D-projection of representations of three CIFAR100 classes. Without label smoothing, examples are spread, but with label smoothing each example is encouraged to be equally distant to the clusters of the other classes, attenuating intra-class variation and inter-class similarity structure.

Organizing a workshop, bringing together experts in theoretical physics and deep learning, to explore how tools from physics can shed light on the theory of deep learning.

Founding Queer in AI, an organization for fostering a community of queer researchers and raising awareness of queer issues in AI/ML.

Organizing a hands-on Tensorflow tutorial on using Deep Learning for Natural Language Processing.

Automatically learning neural net architectures with AdaNet, an open-source, TensorFlow-based framework.

Developing Coconet, the model behind the first AI-powered Doodle (created to celebrate renowned German composer and musician Johann Sebastian Bach).

g.co/airesidency/applyg.co/airesidency

The Visual Task Adaptation Benchmark

Wednesday, November 6, 2019

Posted by Neil Houlsby, Research Scientist and Xiaohua Zhai, Research Engineer, Google Research, ZürichDeep learningcomputer visionrepresentationsTensorFlow HubPyTorch HubThe Visual Task Adaptation Benchmarkavailable on GitHubImageNetGLUEAtariThe Benchmarki) They must not be pre-trained on any of the data (labels or input images) used in the downstream evaluation tasks.ii) They must not contain hardcoded, task-specific, logic. Alternatively put, the evaluation tasks must be treated like a test set — unseen.A

The VTAB protocol. Algorithm A is applied to many tasks T, drawn from a broad distribution of vision problems P_T. In the example, pet classification, remote sensing, and maze localization are shown.

naturalspecializedstructuredNaturalSpecializedmedical imagesremote sensingstructuredDeepMind LabCLEVRdSpritesdisentangled representationsFindings Using VTABGANsVAEsImageNet

Performance of different classes of representation learning algorithms across different task groups: natural, specialized and structured. Each bar shows the average performance of all methods in that class across all tasks in the group.

NaturalS4LStructuredNatural

Top: Performance of S4L versus from-scratch training. Each bar corresponds to a task. Positive-valued bars indicate tasks where S4L outperforms from-scratch. Negative bars indicate that from-scratch performed better. Bottom: S4L versus Supervised training on ImageNet. Positive bars indicate that S4L performs better. The bar colour indicates the task group: Red=Natural, Green=Specialized, Blue=Structured. We can see that additional self-supervision tends to help on structured tasks beyond just using ImageNet labels.

SummaryGitHubpublic leaderboardTF HubTPUGPUAcknowledgementsThe core team behind this work includes Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly.

Learning to Assemble and to Generalize from Self-Supervised Disassembly

Thursday, October 31, 2019

Posted by Kevin Zakka, Research Intern and Andy Zeng, Research Scientist, Robotics at GoogleForm2Fit

Form2Fit learns to assemble a wide variety of kits by finding geometric correspondences between object surfaces and their target placement locations. By leveraging geometric information learned from multiple kits during training, the system generalizes to new objects and kits.

Data-Driven Shape Descriptors For Generalizable Assemblyorientation-sensitive geometric

Overview of Form2Fit. The suction and place networks infer candidate picking and placing locations in the scene respectively. The matching network generates pixel-wise orientation-sensitive descriptors to match picking locations to their corresponding placing locations. The planner then integrates it all to control the robot to execute the next best pick and place action.

Learning Assembly from Disassembly

An example of self-supervision through time-reversal: rewinding a disassembly sequence of a deodorant kit over time generates a valid assembly sequence.

time-reversed disassemblyGeneralization Results

Form2Fit policies are robust to a wide range of rotations and translations of the kits.

Form2Fit policies can generalize to novel kit configurations such as multiple versions of the same kit and mixtures of different kits.

Form2Fit policies can generalize to never-before-seen single and multi-object kits.

What Have the Descriptors Learned?t-SNE

The t-SNE embedding of the learned object descriptors. Similarly oriented objects of the same category display identical colors (e.g. A, B or F, G) while different objects (e.g. C, H) and same objects but different orientation (e.g. A, C, D or H, F) exhibit different colors.

Limitations & Future WorkGitHub repositoryAcknowledgmentsThis research was done by Kevin Zakka, Andy Zeng, Johnny Lee, and Shuran Song (faculty at Columbia University), with special thanks to Nick Hynes, Alex Nichol, and Ivan Krasin for fruitful technical discussions; Adrian Wong, Brandon Hurd, Julian Salazar, and Sean Snyder for hardware support; Ryan Hickman for valuable managerial support; and Chad Richards for helpful feedback on writing.

On-Device Captioning with Live Caption

Tuesday, October 29, 2019

Posted by Michelle Tadmor-Ramanovich and Nadav Bar, Senior Software Engineers, Google Research, Tel-Aviv

40%Live Caption

When media is playing, Live Caption can be launched with a single tap from the volume control to display a caption box on the screen.

Building Live Caption for Accuracy and Efficiencyrecurrent neural networksequence transductionRNN-Tunspoken punctuationconvolutional neural networksound events classification[APPLAUSE][MUSIC]

Incoming sound is processed through a Sound Recognition and ASR feedback loop. The produced text or sound label is formatted and added to the caption.

sound events detectionAudioSetautomatic speech recognition[MUSIC]optimized for edge-devicesperformswellTensorFlow LiteLooking forwardmulti-speaker contentAcknowledgementsThe core team includes Robert Berry, Anthony Tripaldi, Danielle Cohen, Anna Belozovsky, Yoni Tsafir, Elliott Burford, Justin Lee, Kelsie Van Deman, Nicole Bleuel, Brian Kemler, and Benny Schlesinger. We would like to thank the Google Speech team, especially Qiao Liang, Arun Narayanan, and Rohit Prabhavalkar for their insightful work on the ASR model as well as Chung-Cheng Chiu from Google Brain Team; Dan Ellis and Justin Paul for their help with integrating the Sound Recognition model; Tal Remez for his help in developing the punctuation model; Kevin Rocard and Eric Laurent‎ for their help with the Android audio capture API; and Eugenio Marchiori, Shivanker Goel, Ye Wen, Jay Yoo, Asela Gunawardana, and Tom Hume for their help with the Android infrastructure work.

Introducing the Schema-Guided Dialogue Dataset for Conversational Assistants

Monday, October 28, 2019

Posted by Abhinav Rastogi, Software Engineer and Pranav Khaitan, Engineering Lead, Google Researchvirtual assistantsGoogle AssistantTowards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue DatasetSchema-Guided Dialogue datasetintent predictionslot fillinglanguage generationschema-guided approach BERTopen sourceAPIsThe Datasetbankseventsmedia, calendartravelweather. Wizard-of-Ozprobabilistic automaton

Steps for obtaining dialogues, with assistant turns marked in red and user turns in blue. Left: The simulator generates a dialogue skeleton using a finite set of actions. Center: Actions are converted into utterances using templates (~50 per service) and slot values are replaced with natural variations. Right: Paraphrasing via crowdsourcing to make the flow cohesive.

The Schema-Guided Approachopen-sourced dialogue state tracking modelEighth Dialogue System Technology ChallengeDialog System Technology ChallengesSchema-Guided Dialogue State Tracking8th DSTCDSTC8 workshopAAAI-20AcknowledgementsThis post reflects the work of our co-authors Xiaoxue Zang, Srinivas Sunkara and Raghav Gupta. We also thank Amir Fayazi and Maria Wang for help with data collection and Guan-Lin Chao for insights on model design and implementation.

Blog