Google AI Blog

Night Sight: Seeing in the Dark on Pixel Phones

Wednesday, November 14, 2018

Posted by Marc Levoy, Distinguished Engineer and Yael Pritch, Staff Software EngineerHDR+

Left: iPhone XS (full resolution image here). Right: Pixel 3 Night Sight (full resolution image here).

Why is Low-light Photography Hard?image noiseshot noiseread noisesignal-to-noise ratioHDR+How Dark is Dark?

Capturing the Datazero-shutter-lagHDR+Super Res Zoompositive-shutter-lagoptical flow¹

Left: 15-frame burst captured by one of two side-by-side Pixel 3 phones. Center: Night Sight shot with motion metering disabled, causing this phone to use 73ms exposures. The dog’s head is motion blurred in this crop. Right: Night Sight shot with motion metering enabled, causing this phone to notice the motion and use shorter 48ms exposures. This shot has less motion blur. (Mike Milne)

Left: Crop from a handheld Night Sight shot of the sky (full resolution image here). There was slight handshake, so Night Sight chose 333ms x 15 frames = 5.0 seconds of capture. Right: Tripod shot (full resolution image here). No handshake was detected, so Night Sight used 1.0 second x 6 frames = 6.0 seconds. The sky is cleaner (less noise), and you can see more stars. (Florian Kainz)

Alignment and Mergingexposure stackingSynthcamsteadily became cleaner as you watchedHDR+'s merging algorithmSuper Res ZoomOther Challenges1. Auto white balancing (AWB) fails in low light.color constancycolor temperatureauto white balancingill-posed problemlearning-based AWB algorithm

Left: The white balancer in the Pixel’s default camera mode doesn't know how yellow the illumination was on this shack on the Vancouver waterfront (full resolution image here). Right: Our learning-based AWB algorithm does a better job (full resolution image here). (Marc Levoy)

2. Tone mapping of scenes that are too dark to see.cone cells

Yosemite valley at nighttime, Canon DSLR, 28mm f/4 lens, 3-minute exposure, ISO 100. It's nighttime, since you can see stars, but it looks like daytime (full resolution image here). (Jesse Levinson)

A Philosopher Lecturing on the Orrery, by Joseph Wright of Derby, 1766 (image source: Wikidata). The artist uses pigments from black to white, but the scene depicted is evidently dark. How does he accomplish this? He increases contrast, surrounds the scene with darkness, and drops shadows to black, because we cannot see detail there.

S-curve

Pixel 3, 6-second Night Sight shot, with tripod (full resolution image here). (Alex Savu)

How Dark can Night Sight Go?hyperfocal distanceblog postHow to Get the Most out of Night SightexamplesA/B comparisons- Night Sight can't operate in complete darkness, so pick a scene with some light falling on it.- Soft, uniform lighting works better than harsh lighting, which creates dark shadows.- To avoid lens flare artifacts, try to keep very bright light sources out of the field of view.- To increase exposure, tap on various objects, then move the exposure slider. Tap again to disable.- To decrease exposure, take the shot and darken later in Google’s Photos editor; it will be less noisy.- If it’s so dark the camera can’t focus, tap on a high-contrast edge, or the edge of a light source.- If this won’t work for your scene, use the Near (4 feet) or Far (12 feet) focus buttons (see below).- To maximize image sharpness, brace your phone against a wall or tree, or prop it on a table or rock.- Night Sight works for selfies too, as in the A/B album, with optional illumination from the screen itself.

Manual focus buttons (Pixel 3 only).

AcknowledgementsNight Sight was a collaboration of several teams at Google. Key contributors to the project include: from the Gcam team Charles He, Nikhil Karnad, Orly Liba, David Jacobs, Tim Brooks, Michael Milne, Andrew Radin, Navin Sarma, Jon Barron, Yun-Ta Tsai, Jiawen Chen, Kiran Murthy, Tianfan Xue, Dillon Sharlet, Ryan Geiss, Sam Hasinoff and Alex Schiffhauer; from the Super Res Zoom team Bart Wronski, Peyman Milanfar and Ignacio Garcia Dorado; from the Google camera app team Gabriel Nava, Sushil Nath, Tim Smith , Justin Harrison, Isaac Reynolds and Michelle Chen.
1 By the way, the exposure time shown in Google Photos (if you press "i") is per-frame, not total time, which depends on the number of frames captured. You can get some idea of the number of frames by watching the animation while the camera is collecting light. Each tick around the circle is one captured frame.^↩

2 For a wonderful analysis of these techniques, look at von Helmholtz, "On the relation of optics to painting" (1876).^↩

Accurate Online Speaker Diarization with Supervised Learning

Monday, November 12, 2018

Posted by Chong Wang, Research Scientist, Google AISpeaker diarizationunderstanding medical conversationsvideo captioning

Online speaker diarization on streaming audio input. Different colors in the bottom axis indicate different speakers.

Fully Supervised Speaker DiarizationNIST SRE 2000 CALLHOMEprevious clustering-based methoddeep neural network embedding methodsopen sourcing the core algorithmsClustering versus Interleaved-state RNNk-meansspectral clusteringrecurrent neural networkinterleavedblueyellowpinkgreenChinese restaurant processblueyellowbluey₇green

The generative process of our model. Colors indicate labels for speaker segments.

stochastic gradient descentFuture Workour paperGithub pageAcknowledgmentsThis work was done as a close collaboration between Google AI and Speech & Assistant teams. Contributors include Aonan Zhang (intern), Quan Wang, Zhengyao Zhu and Chong Wang.

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing

Friday, November 2, 2018

Posted by Jacob Devlin and Ming-Wei Chang, Research Scientists, Google AI Languagenatural language processingbillionspre-trainingfine-tunedquestion answeringsentiment analysisopen sourcedBERTransformersBERTCloud TPUTensorFlowIn our associated paperStanford Question Answering DatasetWhat Makes BERT Different? Semi-supervised Sequence LearningGenerative Pre-TrainingELMoULMFitdeeply bidirectionalunsupervisedWikipediacontext-freecontextualcontextualunidirectionalbidirectionalword2vecGloVeword embeddingbankbank accountbank of the river.I accessed the bank accountbankI accessed theaccountbankI accessed theaccount

BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.

The Strength of Bidirectionalityand

very long timeABBA

Training with Cloud TPUsCloud TPUsTransformer model architectureopen source releasetensor2tensor libraryResults with BERTSQuAD v1.1

GLUE benchmarkimproves upon the state-of-the-art

Making BERT Work for Youhttp://goo.gl/language/bertColabBERT FineTuning with Cloud TPUsBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Google at EMNLP 2018

Wednesday, October 31, 2018

Posted by Manaal Faruqui, Senior Research Scientist and Emily Pitler, Staff Research Scientist, Google AI LanguageEmpirical Methods in Natural Language ProcessingComputational Natural Language Learningnatural language processing

Noun-Verb Ambiguity in POS Tagging Dataset: English part-of-speech taggers regularly make egregious errors related to noun-verb ambiguity, despite high accuracies on standard datasets. For example: in “Mark which area you want to distress” several state-of-the-art taggers annotate “Mark” as a noun instead of a verb. We release a new dataset of over 30,000 naturally occurring non-trivial annotated examples of noun-verb ambiguity. Taggers previously indistinguishable from each other have accuracies ranging from 57% to 75% accuracy on this challenge set.

Query Wellformedness Dataset: Web search queries are usually “word-salad” style queries with little resemblance to natural language questions (“barack obama height” as opposed to “What is the height of Barack Obama?”). Differentiating a natural language question from a query is of importance to several applications including dialogue. We annotate and release 25,100 queries from the open-source Paralex corpus with ratings on how close they are to well-formed natural language questions.

WikiSplit: Split and Rephrase Dataset Extracted from Wikipedia Edits: We extract examples of sentence splits from Wikipedia edits where one sentence gets split into two sentences that together preserve the original meaning of the sentence (E.g. “Street Rod is the first in a series of two games released for the PC and Commodore 64 in 1989.” is split into “Street Rod is the first in a series of two games.” and “It was released for the PC and Commodore 64 in 1989.”) The released corpus contains one million sentence splits with a vocabulary of more than 600,000 words.

WikiAtomicEdits: A Multilingual Corpus of Atomic Wikipedia Edits: Information about how people edit language in Wikipedia can be used to understand the structure of language itself. We pay particular attention to two atomic edits: insertions and deletions that consist of a single contiguous span of text. We extract around 43 million such edits in 8 languages and show that they provide valuable information about entailment and discourse. For example, insertion of “in 1949” adds a prepositional phrase to the sentence “She died there after a long illness” resulting in “She died there in 1949 after a long illness”.

Conceptual CaptionsGAP Coreference Resolutionpast contributionsblueLinguistically-Informed Self-Attention for Semantic Role LabelingBest Long Paper awardsArea Chairs Include:Ming-Wei Chang, Marius Pasca, Slav Petrov, Emily Pitler, Meg Mitchell, Taro WatanabeEMNLP PublicationsA Challenge Set and Methods for Noun-Verb AmbiguityAli Elkahky, Kellie Webster, Daniel Andor, Emily PitlerA Fast, Compact, Accurate Model for Language Identification of Codemixed TextYuan Zhang, Jason Riesa, Daniel Gillick, Anton Bakalov, Jason Baldridge, David WeissAirDialogue: An Environment for Goal-Oriented Dialogue Research Wei Wei, Quoc Le, Andrew Dai, Jia LiContent Explorer: Recommending Novel Entities for a Document WriterMichal Lukasik, Richard ZensDeep Relevance Ranking using Enhanced Document-Query InteractionsRyan McDonald, George Brokos, Ion AndroutsopoulosHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question AnsweringZhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. ManningIdentifying Well-formed Natural Language QuestionsManaal Faruqui, Dipanjan DasLearning To Split and Rephrase From Wikipedia Edit HistoryJan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, Dipanjan DasLinguistically-Informed Self-Attention for Semantic Role LabelingEmma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallumOpen Domain Question Answering Using Early Fusion of Knowledge Bases and TextHaitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William CohenNoise Contrastive Estimation for Conditional Models: Consistency and Statistical Efficiency Zhuang Ma, Michael CollinsPart-of-Speech Tagging for Code-Switched, Transliterated Texts without Explicit Language IdentificationKelsey Ball, Dan GarrettePhrase-Indexed Question Answering: A New Challenge for Scalable Document ComprehensionMinjoon Seo, Tom Kwiatkowski, Ankur P. Parikh, Ali Farhadi, Hannaneh HajishirziPolicy Shaping and Generalized Update Equations for Semantic Parsing from DenotationsDipendra Misra, Ming-Wei Chang, Xiaodong He, Wen-tau YihRevisiting Character-Based Neural Machine Translation with Capacity and CompressionColin Cherry, George Foster, Ankur Bapna, Orhan Firat, Wolfgang MachereySelf-governing neural networks for on-device short text classificationSujith Ravi, Zornitsa KozarevaSemi-Supervised Sequence Modeling with Cross-View TrainingKevin Clark, Minh-Thang Luong, Christopher D. Manning, Quoc LeState-of-the-art Chinese Word Segmentation with Bi-LSTMsJi Ma, Kuzman Ganchev, David WeissSubgoal Discovery for Hierarchical Dialogue Policy LearningDa Tang, Xiujun Li, Jianfeng Gao, Chong Wang, Lihong Li, Tony JebaraSwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine TranslationXinyi Wang, Hieu Pham, Zihang Dai, Graham NeubigThe Importance of Generation Order in Language ModelingNicolas Ford, Daniel Duckworth, Mohammad Norouzi, George DahlTraining Deeper Neural Machine Translation Models with Transparent AttentionAnkur Bapna, Mia Chen, Orhan Firat, Yuan Cao, Yonghui WuUnderstanding Back-Translation at ScaleSergey Edunov, Myle Ott, Michael Auli, David GrangierUnsupervised Natural Language Generation with Denoising AutoencodersMarkus Freitag, Scott RoyWikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and DiscourseManaal Faruqui, Ellie Pavlick, Ian Tenney, Dipanjan DasWikiConv: A Corpus of the Complete Conversational History of a Large Online Collaborative CommunityYiqing Hua, Cristian Danescu-Niculescu-Mizil, Dario Taraborelli, Nithum Thain, Jeffery Sorensen, Lucas DixonEMNLP DemosSentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo, John RichardsonUniversal Sentence Encoder for EnglishDaniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, Ray KurzweilCoNLL Shared TaskMultilingual Parsing from Raw Text to Universal DependenciesSlav Petrov, co-organizerUniversal Dependency Parsing with Multi-Treebank ModelsAaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, Sara StymneUniversal POS TaggingMorphological Taggingopen-sourced Meta-BiLSTM taggerCoNLL PublicationSentence-Level Fluency Evaluation: References Help, But Can Be Spared!Katharina Kann, Sascha Rothe, Katja Filippova

Introducing AdaNet: Fast and Flexible AutoML with Learning Guarantees

Tuesday, October 30, 2018

Posted by Charles Weill, Software Engineer, Google AI, NYCEnsemble learningNetflix PrizeKaggle competitionsbecome more readily availablewill grow largerAdaNetTensorFlowreinforcement learningevolutionaryselecting optimal neural network architectures

AdaNet adaptively growing an ensemble of neural networks. At each iteration, it measures the ensemble loss for each candidate, and selects the best one to move onto the next iteration.

Fast and Easy to UseTensorFlow EstimatorTensorFlow Hub modulesTensorFlow Model AnalysisGoogle Cloud’s Hyperparameter Tuner

AdaNet’s accuracy (y-axis) per train step (x-axis) on CIFAR-100. The blue line is accuracy on the training set, and red line is performance on the test set. A new subnetwork begins training every million steps, and eventually improves the performance of the ensemble. The grey and green lines are the accuracies of the ensemble before adding the new subnetwork.

TensorBoardTensorFlow ServingLearning GuaranteesAdaNet: Adaptive Structural Learning of Artificial Neural NetworksICML 2017

The generalization error of the ensemble is bounded by its training error and complexity.

By optimizing this objective, we are directly minimizing this bound.

our tutorial about the AdaNet objectiveExtensibleadanet.subnetwork.Buildertf.layersadanet.Estimatoropen-source implementationNASNet-A CIFAR architecture

Performance of a NASNet-A model as presented in Zoph et al., 2018 versus AdaNet learning to combine small NASNet-A subnetworks on CIFAR-10.

tf.contrib.estimator.Headsadanet.subnetwork.GeneratorGithub repotutorial notebooksworking examples using dense layers and convolutionsAcknowledgementsThis project was only possible thanks to the members of the core team including Corinna Cortes, Mehryar Mohri, Xavi Gonzalvo, Charles Weill, Vitaly Kuznetsov, Scott Yak, and Hanna Mazzawi. We also extend a special thanks to our collaborators, residents and interns Gus Kristiansen, Galen Chuang, Ghassen Jerfel, Vladimir Macko, Ben Adlam, Scott Yang and the many others at Google who helped us test it out.

Acoustic Detection of Humpback Whales Using a Convolutional Neural Network

Monday, October 29, 2018

Posted by Matt Harvey, Software Engineer, Google AI PerceptionGoogle AI Perceptionnon-speech captionsAudioSetmodel codeAI for Social GoodPacific Islands Fisheries Science CenterNational Oceanic and Atmospheric Administrationhumpback whale

HARP deployment locations. Green: sites with currently active recorders. Red: previous recording sites.

Passive Acoustic Monitoring and the NOAA HARP DatasetPassive acoustic monitoringhydrophonesWiggins and Hildebrand, 2007full text PDFdecimationmanuallySupervised Learning: Optimizing an Image Model for Humpback Detectionspectrogram

Example spectrograms of audio events found in the dataset, with time on the x-axis and frequency on the y-axis. Left: a humpback whale call (in particular, a tonal unit), Center: narrow-band noise from an unknown source, Right: hard disk noise from the HARP

ResNet-50success at classifying non-speech audiosupervised learningfrequency-modulatedtonal

per-channel energy normalization

Spectrograms of the same 5-unit excerpt from humpback whale song beginning at 0:06 in the above recording. Top: PCEN. Bottom: log of squared magnitude. The dark blue horizontal bar along the bottom under log compression has become much lighter relative to the whale call when using PCEN

whale songUnsupervised Learning: Representation for Finding Similar Song UnitsWhere are all the humpback sounds in this data?similarUnsupervised Learning of Semantic Audio RepresentationsanchorpositivenegativePrincipal component analysisTensorFlow Embedding ProjectorColor byclass_labelsitet-SNE

A sample of 5000 data points in the unsupervised representation. (Orange: humpback. Blue: not humpback.)

Manually chosen query units (boxed) and nearest neighbors using the unsupervised representation.

Predictions from the Supervised Classifier on the Entire DatasetDuty cycling

Time density of presence on year / month axes for the Kona and Saipan sites.

NOAA Fisheries blog postAcknowledgementsWe would like to thank Ann Allen (NOAA Fisheries) for providing the bulk of the ground truth data, for many useful rounds of feedback, and for some of the words in this post. Karlina Merkens (NOAA affiliate) provided further useful guidance. We also thank the NOAA Pacific Islands Fisheries Science Center as a whole for collecting and sharing the acoustic data.

Within Google, Jiayang Liu, Julie Cattiau, Aren Jansen, Rif A. Saurous, and Lauren Harrell contributed to this work. Special thanks go to Lauren, who designed the plots in the analysis section and implemented them using ggplot.

Curiosity and Procrastination in Reinforcement Learning

Wednesday, October 24, 2018

Posted by Nikolay Savinov, Research Intern, Google Brain Team and Timothy Lillicrap, Research Scientist, DeepMindReinforcement learningcarrot-and-stickDQNAlphaGoZeroOpenAI-Fivegrasp new objectsstruggleEpisodic Curiosity through ReachabilityGoogle Brain teamDeepMindETH Zürich

Episodic Curiosity through Reachability: Observations are added to memory, reward is computed based on how far the current observation is from the most similar observation in memory. The agent receives more reward for seeing observations which are not yet represented in memory.

Previous Curiosity Formulations[1][2][3][4]Curiosity-driven Exploration by Self-supervised Prediction

“Now I’m in the meat section, so I think the section around the corner is the fish section — those are usually adjacent in this supermarket chain”“No, it’s actually the vegetables section. I didn’t expect that!”

Agent imbued with surprise-based curiosity gets stuck when it encounters TV. GIF adopted from a video by © Deepak Pathak, used under CC BY 2.0 license.

The Dangers of “Procrastination”Large-Scale Study of Curiosity-Driven LearningOpenAIEpisodic CuriosityEpisodic Curiosity through Reachabilitysimilardeep neural network

Graph of reachabilities would determine novelty. In practice, this graph is not available — so we train a neural network approximator to estimate a number of steps between observations.

Experimental ResultsViZDoomDMLab

Surprise-based ICM method is persistently tagging the wall instead of exploring the maze.

Our method shows reasonable exploration.

Our reward visualization: red means negative reward, green means positive reward. Left to right: map with rewards, map with locations currently in memory, first-person view.

research paperAcknowledgements:This project is a result of a collaboration between the Google Brain team, DeepMind and ETH Zürich. The core team includes Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap and Sylvain Gelly. We would like to thank Olivier Pietquin, Carlos Riquelme, Charles Blundell and Sergey Levine for the discussions about the paper. We are grateful to Indira Pasko for the help with illustrations.References:Count-Based Exploration with Neural Density ModelsGeorg Ostrovski, Marc G. Bellemare, Aaron van den Oord, Remi Munos#Exploration: A Study of Count-Based Exploration for Deep Reinforcement LearningHaoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter AbbeelUnsupervised Learning of Goal Spaces for Intrinsically Motivated Goal ExplorationAlexandre Péré, Sébastien Forestier, Olivier Sigaud, Pierre-Yves OudeyerVIME: Variational Information Maximizing ExplorationRein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel