Google AI Blog: November 2018

Night Sight: Seeing in the Dark on Pixel Phones

Wednesday, November 14, 2018

Posted by Marc Levoy, Distinguished Engineer and Yael Pritch, Staff Software EngineerHDR+

Left: iPhone XS (full resolution image here). Right: Pixel 3 Night Sight (full resolution image here).

Why is Low-light Photography Hard?image noiseshot noiseread noisesignal-to-noise ratioHDR+How Dark is Dark?

Capturing the Datazero-shutter-lagHDR+Super Res Zoompositive-shutter-lagoptical flow¹

Left: 15-frame burst captured by one of two side-by-side Pixel 3 phones. Center: Night Sight shot with motion metering disabled, causing this phone to use 73ms exposures. The dog’s head is motion blurred in this crop. Right: Night Sight shot with motion metering enabled, causing this phone to notice the motion and use shorter 48ms exposures. This shot has less motion blur. (Mike Milne)

Left: Crop from a handheld Night Sight shot of the sky (full resolution image here). There was slight handshake, so Night Sight chose 333ms x 15 frames = 5.0 seconds of capture. Right: Tripod shot (full resolution image here). No handshake was detected, so Night Sight used 1.0 second x 6 frames = 6.0 seconds. The sky is cleaner (less noise), and you can see more stars. (Florian Kainz)

Alignment and Mergingexposure stackingSynthcamsteadily became cleaner as you watchedHDR+'s merging algorithmSuper Res ZoomOther Challenges1. Auto white balancing (AWB) fails in low light.color constancycolor temperatureauto white balancingill-posed problemlearning-based AWB algorithm

Left: The white balancer in the Pixel’s default camera mode doesn't know how yellow the illumination was on this shack on the Vancouver waterfront (full resolution image here). Right: Our learning-based AWB algorithm does a better job (full resolution image here). (Marc Levoy)

2. Tone mapping of scenes that are too dark to see.cone cells

Yosemite valley at nighttime, Canon DSLR, 28mm f/4 lens, 3-minute exposure, ISO 100. It's nighttime, since you can see stars, but it looks like daytime (full resolution image here). (Jesse Levinson)

A Philosopher Lecturing on the Orrery, by Joseph Wright of Derby, 1766 (image source: Wikidata). The artist uses pigments from black to white, but the scene depicted is evidently dark. How does he accomplish this? He increases contrast, surrounds the scene with darkness, and drops shadows to black, because we cannot see detail there.

S-curve

Pixel 3, 6-second Night Sight shot, with tripod (full resolution image here). (Alex Savu)

How Dark can Night Sight Go?hyperfocal distanceblog postHow to Get the Most out of Night SightexamplesA/B comparisons- Night Sight can't operate in complete darkness, so pick a scene with some light falling on it.- Soft, uniform lighting works better than harsh lighting, which creates dark shadows.- To avoid lens flare artifacts, try to keep very bright light sources out of the field of view.- To increase exposure, tap on various objects, then move the exposure slider. Tap again to disable.- To decrease exposure, take the shot and darken later in Google’s Photos editor; it will be less noisy.- If it’s so dark the camera can’t focus, tap on a high-contrast edge, or the edge of a light source.- If this won’t work for your scene, use the Near (4 feet) or Far (12 feet) focus buttons (see below).- To maximize image sharpness, brace your phone against a wall or tree, or prop it on a table or rock.- Night Sight works for selfies too, as in the A/B album, with optional illumination from the screen itself.

Manual focus buttons (Pixel 3 only).

AcknowledgementsNight Sight was a collaboration of several teams at Google. Key contributors to the project include: from the Gcam team Charles He, Nikhil Karnad, Orly Liba, David Jacobs, Tim Brooks, Michael Milne, Andrew Radin, Navin Sarma, Jon Barron, Yun-Ta Tsai, Jiawen Chen, Kiran Murthy, Tianfan Xue, Dillon Sharlet, Ryan Geiss, Sam Hasinoff and Alex Schiffhauer; from the Super Res Zoom team Bart Wronski, Peyman Milanfar and Ignacio Garcia Dorado; from the Google camera app team Gabriel Nava, Sushil Nath, Tim Smith , Justin Harrison, Isaac Reynolds and Michelle Chen.
1 By the way, the exposure time shown in Google Photos (if you press "i") is per-frame, not total time, which depends on the number of frames captured. You can get some idea of the number of frames by watching the animation while the camera is collecting light. Each tick around the circle is one captured frame.^↩

2 For a wonderful analysis of these techniques, look at von Helmholtz, "On the relation of optics to painting" (1876).^↩

Accurate Online Speaker Diarization with Supervised Learning

Monday, November 12, 2018

Posted by Chong Wang, Research Scientist, Google AISpeaker diarizationunderstanding medical conversationsvideo captioning

Online speaker diarization on streaming audio input. Different colors in the bottom axis indicate different speakers.

Fully Supervised Speaker DiarizationNIST SRE 2000 CALLHOMEprevious clustering-based methoddeep neural network embedding methodsopen sourcing the core algorithmsClustering versus Interleaved-state RNNk-meansspectral clusteringrecurrent neural networkinterleavedblueyellowpinkgreenChinese restaurant processblueyellowbluey₇green

The generative process of our model. Colors indicate labels for speaker segments.

stochastic gradient descentFuture Workour paperGithub pageAcknowledgmentsThis work was done as a close collaboration between Google AI and Speech & Assistant teams. Contributors include Aonan Zhang (intern), Quan Wang, Zhengyao Zhu and Chong Wang.

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing

Friday, November 2, 2018

Posted by Jacob Devlin and Ming-Wei Chang, Research Scientists, Google AI Languagenatural language processingbillionspre-trainingfine-tunedquestion answeringsentiment analysisopen sourcedBERTransformersBERTCloud TPUTensorFlowIn our associated paperStanford Question Answering DatasetWhat Makes BERT Different? Semi-supervised Sequence LearningGenerative Pre-TrainingELMoULMFitdeeply bidirectionalunsupervisedWikipediacontext-freecontextualcontextualunidirectionalbidirectionalword2vecGloVeword embeddingbankbank accountbank of the river.I accessed the bank accountbankI accessed theaccountbankI accessed theaccount

BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.

The Strength of Bidirectionalityand

very long timeABBA

Training with Cloud TPUsCloud TPUsTransformer model architectureopen source releasetensor2tensor libraryResults with BERTSQuAD v1.1

GLUE benchmarkimproves upon the state-of-the-art

Making BERT Work for Youhttp://goo.gl/language/bertColabBERT FineTuning with Cloud TPUsBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding