Google AI Blog: November 2019

Astrophotography with Night Sight on Pixel Phones

Tuesday, November 26, 2019

Posted by Florian Kainz and Kiran Murthy, Software Engineers, Google Researchexperimentspleasing resultsprevious blog postSIGGRAPH Asia 2019

The Milky Way as seen from the summit of Haleakala volcano on a cloudless and moonless September night, captured using the Google Camera App running on a Pixel 4 XL phone. The image has not been retouched or post-processed in any way. It shows significantly more detail than a person can see with the unaided eye on a night this dark. The dust clouds along the Milky Way are clearly visible, the sky is covered with thousands of stars, and unlike human night vision, the picture is colorful.

A Brief Overview of Night Sightshot noiseExperimenting with Exposure Time

Motion-blurred stars in a single-frame two-minute exposure.

Dark Current and Hot PixelsDark currentCMOS image sensors

Left: A small region of a long-exposure image with hot pixels, and warm pixels caused by dark current nonuniformity. Right: The same image after outliers have been removed. Fine details in the landscape, including small points of light, are preserved.

Scene Compositionelectronic viewfinders

Left: The live Night Sight viewfinder in a very dark outdoor environment. Except for a few points of light from distant buildings, the landscape and the sky are largely invisible. Right: The post-shutter viewfinder during a long exposure shot. The image is much clearer; it updates after every long-exposure frame.

AutofocusAutofocusSky Processingconvolutional neural network

A landscape picture taken on a bright full-moon night, without sky processing (left half), and with sky darkening (right half). Note that the landscape is not darkened.

noise reductionResults

Examples of photos taken with the Google Camera App on Pixel 4. An album with more pictures can be found here.

Tips and TrickshereAcknowledgementsNight Sight is an ongoing collaboration between several teams at Google. Key contributors to the project include from the Gcam team, Orly Liba, Nikhil Karnad, Charles He, Manfred Ernst, Michael Milne, Andrew Radin, Navin Sarma, Jon Barron, Yun-Ta Tsai, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Ryan Geiss, Sam Hasinoff, Alex Schiffhauer, Yael Pritch Knaan and Marc Levoy; from the Super Res Zoom team, Bart Wronski, Peyman Milanfar, and Ignacio Garcia Dorado; from the Google camera app team, Emily To, Gabriel Nava, Sushil Nath, Isaac Reynolds, and Michelle Chen; from the Android platform team, Ryan Chan, Ying Chen Lou, Nicholas Wilson and Bob Hung; from the Mobile Vision team, Longqi (Rocky) Cai, Huizhong Chen, Emily Manoogian, Nicole Maffeo, Tomer Meron, Weijun Wang and Andrew Howard; from Machine Perception, Elad Eban and Yair Movshovitz-Attias.

RecSim: A Configurable Simulation Platform for Recommender Systems

Tuesday, November 19, 2019

Posted by Martin Mladenov, Research Scientist and Chih-wei Hsu, Software Engineer, Google Researchrecommender systemscollaborative interactive recommenders (CIRs)deliberate sequence of interactionssequentialReinforcement learningroboticsRecSimherespecific aspects of user behaviorReinforcement Learning and Recommendation SystemsMovieLens 1MresearchOverview of RecSimuser modeldocument modeluser choice model

Data Flow through components of RecSim. Colors represent different model components — user and user-choice models (green), document model (blue), and the recommender agent (red).

Applicationsa novel decomposition techniqueQ-valuesFuture WorkTensorFlow’s probabilistic APIspreference elicitationwhite paperhereAcknowledgementsWe thank our collaborators and early adopters of RecSim, including the other members of the RecSimteam: Eugene Ie, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu and Craig Boutilier.

New Solutions for Quantum Gravity with TensorFlow

Friday, November 15, 2019

Posted by Thomas Fischbacher, Software Engineer, Google Research, Zürichteaching robots how to throwolfactory properties of moleculesTensorFlowquantum mechanicselectromagneticstrongweakgravitationquantum gravityM-TheoryThe Theory formerly known as Strings,”Edward WittenxyztarticleJournal of High Energy Physicsto 194tachyon-freeGoogle colabPython libraryApplying TensorFlow to M-TheoryConclusionAcknowledgementsThis research was conducted by Iulia M. Comşa, Moritz Firsching, and Thomas Fischbacher. Additional thanks go to Jyrki Alakuijala, Rahul Sukthankar, and Jay Yagnik for encouragement and support.

SPICE: Self-Supervised Pitch Estimation

Thursday, November 14, 2019

Posted by Marco Tagliasacchi, Research Scientist, Google ResearchpYINSWIPECREPEpaperself-supervisionpretextunsupervised

Constant-Q transform of an audio clip, superimposed on a representation of pitch as estimated by SPICE (video).

convolutional encoderconstant-Q transformharmonic

SPICE model architecture (simplified). Two pitch-shifted versions of the same CQT frame are fed to two encoders with shared weights. The loss is designed to make the difference between the outputs of the encoders proportional to the relative pitch difference. In addition (not shown), a reconstruction loss is added to regularize the model. The model also learns to produce the confidence of the pitch estimation.

no access to ground truth labelsMIR-1k dataset

Evaluation on the MIR-1k dataset, mixing in background music at different signal-to-noise ratios.

FreddieMeterAcknowledgmentsThe work described here was authored by Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi and Mihajlo Velimirović. We are grateful for all discussions and feedback on this work that we received from our colleagues at Google. The SingingVoices dataset used for training the models in this work has been collected by Alexandra Gherghina as part of FreddieMeter, which is using SPICE and a vocal timbre similarity model to understand how closely a singer matches Freddie Mercury.

Introducing the Next Generation of On-Device Vision Models: MobileNetV3 and MobileNetEdgeTPU

Wednesday, November 13, 2019

Posted by Andrew Howard and Suyog Gupta, Software Engineers, Google Researchalgorithmically-efficientGoogle Pixel 4Pixel Neural CoreEdge TPUedge computingMobileNetsMobileNetV3MobileNetEdgeTPUAutoMLMobileNetV2Building MobileNetV3previous version of MobileNetMnasNetNetAdaptreinforcement learning

Comparison of accuracy vs. latency for mobile models on the ImageNet classification task using the Google Pixel 4 CPU.

MobileNetV3 Search SpaceSwish

squeeze-and-excitationsigmoid functioninverted bottleneck structure

MobileNetV3 extends the MobileNetV2 inverted bottleneck structure by adding h-swish and mobile friendly squeeze-and-excitation as searchable options.

Size of expansion layer

Degree of squeeze-excite compression

Choice of activation function: h-swish or ReLU

Number of layers for each resolution block

MobileNetV3 Object Detection and Semantic SegmentationCOCO datasetsemantic segmentationCityscapes DatasetMobileNet for Edge TPUsCoralaccelerator-aware AutoMLDSPsGPUsEfficientNet-EdgeTPU

Left: Comparison of the accuracy on the ImageNet classification task between MobileNetEdgeTPU and other image classification networks designed for mobile when running on Pixel4 Edge TPU. MobileNetEdgeTPU achieves higher accuracy and lower latency compared with other models. Right: Average Edge TPU power in Watts for different classification models running at 30 frames per second (fps).

Objection Detection Using MobileNetEdgeTPUCOCO14 minival datasetThe Need for Hardware-Aware Models

MobileNetV3 is still the best performing network when using mobile CPU as the deployment target.

For Researchers and DevelopersMobileNet github pageTensorflow Object Detection APIDeepLabAcknowledgementsThis work is made possible through a collaboration spanning several teams across Google. We’d like to acknowledge contributions from Berkin Akin, Okan Arikan, Gabriel Bender, Bo Chen, Liang-Chieh Chen, Grace Chu, Eddy Hsu, John Joseph, Pieter-jan Kindermans, Quoc Le, Owen Lin, Hanxiao Liu, Yun Long, Ravi Narayanaswami, Ruoming Pang, Mark Sandler, Mingxing Tan, Vijay Vasudevan, Weijun Wang, Dong Hyuk Woo, Dmitry Kalenichenko, Yunyang Xiong, Yukun Zhu and support from Hartwig Adam, Blaise Agüera y Arcas, Chidu Krishnan and Steve Molloy.

New Insights into Human Mobility with Privacy Preserving Aggregation

Tuesday, November 12, 2019

Posted by Adam Sadilek, Software Engineer and Xerxes Dotiwalla, Product Manager, Google Research(Updated May 18, 2020: We have released a new paper based on this mobility dataset. In our latest work published in Nature Human Behavior, we quantify how human movement patterns vary across sociodemographic and environmental contexts and present international movement patterns across national borders.)predicting epidemicsinfrastructure planningnatural disastersimportantdomainsbased on cell carrier logslocation "check-ins"aggregateHierarchical Organization of Urban Mobility and Its Connection with City LivabilityNature Communications

Visualization of privacy-first computation of the mobility map. Individual data points are automatically aggregated together with differential privacy noise added. Then, flows of these aggregate and obfuscated populations are studied.

Computing a Global Mobility Map While Preserving User PrivacyAI principlesdifferential privacyk-anonymityimplemented in 2014Location History, in order toS2 cellslive trafficparking availabilityMobility Map ApplicationsChristaller’s workmove

Mobility maps of Paris (left) and Los Angeles (right). Both cities have similar population sizes, but very different mobility patterns. Paris has an "onion" structure exhibiting a distinct center with a high degree of mobility (red) that progressively decreases as we move away from the center (in order: orange, yellow, green, blue). In contrast, Los Angeles has a large number of high-mobility areas scattered throughout the region.

population densitysprawl

Connecting flow hierarchy Φ with urban indicators in a sample of US cities. Proportion of trips as a function of Φ, broken down by model share: private car, public transportation, and walking. Sample city names that appear in the plot: ATL (Atlanta), CHA (Charlotte), CHI (Chicago), HOU (Houston), LA (Los Angeles), MIN (Minneapolis), NY (New York City), and SF (San Francisco). We see that cities with higher flow hierarchy exhibit significantly higher rates of public transportation use, less car use, and more walkability.

20 different variablesdebateoptimalNext StepsAI for Social Goodon-device federated learningsecure aggregation protocolsrandomized responsessecure Chrome from malicious attacksAcknowledgementsThis work resulted from a collaboration of Aleix Bassolas and José J. Ramasco from the Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), Brian Dickinson, Hugo Barbosa-Filho, Gourab Ghoshal, Surendra A. Hazarie, and Henry Kautz from the Computer Science Department and Ghoshal Lab at the University of Rochester, Riccardo Gallotti from the Bruno Kessler Foundation, and Damien Desfontaines, Xerxes Dotiwalla, Paul Eastham, Bryant Gipson, Onur Kucuktunc, Allison Lieber, Adam Sadilek at Google.The differential privacy library used in this work is open source and available on our GitHub repo.

Highlights from the 3rd Cohort of the Google AI Residency Program

Thursday, November 7, 2019

Posted by Katie Meckley, Program Manager, Google AI ResidencyGoogle AI Residency Program machine perception algorithms and optimization language understanding healthcare

A large-scale study on cross-lingual transfer in massive multilingual neural machine translation models (recently highlighted as part of this post), trained on billions of sentence pairs from more than 100 languages in order to significantly improve translation for both low- and high-resource languages.

Visualization of the clustering of encoder representations of all modeled languages, based on representational similarity. Encoder representations of different languages cluster according to linguistic similarity. Languages are color-coded by their linguistic family.

A generative model for Scalable Vector Graphics (SVGs), which can be used to aid designers in generating fonts.

Top: Unlike pixel representations of icons (right), in this case a "6", SVGs (left; middle) are scale-invariant representations. Bottom: By modelling SVGs directly, we can aid artists in quickly and intuitively iterating over typography designs.

A method to learn GANs using discrepancy divergence, a measure that accounts for both the loss function and hypothesis set to provide theoretical learning guarantees.

As more generators are added to the DGAN ensemble more modes in the real distribution are covered. From left to right: 1 generator, 5 generators, and 10 generators.

A likelihood ratio method for deep generative models that effectively corrects for confounding background statistics to improve out-of-distribution (OOD) detection, and a new benchmark dataset for OOD detection in genomics.

Log-likelihood (left) and log likelihood-ratio (right) of each pixel for Fashion-MNIST. The likelihood is dominated by the “background” pixels, whereas the likelihood ratio focuses on the “semantic” pixels and is thus better for OOD detection.

A study showing when label smoothing helps, focusing on its impact on calibration of predictions, representations learned by the penultimate layer and effectiveness of knowledge distillation.

2D-projection of representations of three CIFAR100 classes. Without label smoothing, examples are spread, but with label smoothing each example is encouraged to be equally distant to the clusters of the other classes, attenuating intra-class variation and inter-class similarity structure.

Organizing a workshop, bringing together experts in theoretical physics and deep learning, to explore how tools from physics can shed light on the theory of deep learning.

Founding Queer in AI, an organization for fostering a community of queer researchers and raising awareness of queer issues in AI/ML.

Organizing a hands-on Tensorflow tutorial on using Deep Learning for Natural Language Processing.

Automatically learning neural net architectures with AdaNet, an open-source, TensorFlow-based framework.

Developing Coconet, the model behind the first AI-powered Doodle (created to celebrate renowned German composer and musician Johann Sebastian Bach).

g.co/airesidency/applyg.co/airesidency

The Visual Task Adaptation Benchmark

Wednesday, November 6, 2019

Posted by Neil Houlsby, Research Scientist and Xiaohua Zhai, Research Engineer, Google Research, ZürichDeep learningcomputer visionrepresentationsTensorFlow HubPyTorch HubThe Visual Task Adaptation Benchmarkavailable on GitHubImageNetGLUEAtariThe Benchmarki) They must not be pre-trained on any of the data (labels or input images) used in the downstream evaluation tasks.ii) They must not contain hardcoded, task-specific, logic. Alternatively put, the evaluation tasks must be treated like a test set — unseen.A

The VTAB protocol. Algorithm A is applied to many tasks T, drawn from a broad distribution of vision problems P_T. In the example, pet classification, remote sensing, and maze localization are shown.

naturalspecializedstructuredNaturalSpecializedmedical imagesremote sensingstructuredDeepMind LabCLEVRdSpritesdisentangled representationsFindings Using VTABGANsVAEsImageNet

Performance of different classes of representation learning algorithms across different task groups: natural, specialized and structured. Each bar shows the average performance of all methods in that class across all tasks in the group.

NaturalS4LStructuredNatural

Top: Performance of S4L versus from-scratch training. Each bar corresponds to a task. Positive-valued bars indicate tasks where S4L outperforms from-scratch. Negative bars indicate that from-scratch performed better. Bottom: S4L versus Supervised training on ImageNet. Positive bars indicate that S4L performs better. The bar colour indicates the task group: Red=Natural, Green=Specialized, Blue=Structured. We can see that additional self-supervision tends to help on structured tasks beyond just using ImageNet labels.

SummaryGitHubpublic leaderboardTF HubTPUGPUAcknowledgementsThe core team behind this work includes Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly.

Blog