Google AI Blog: April 2017

Experimental Nighttime Photography with Nexus and Pixel

Tuesday, April 25, 2017

Posted by Florian Kainz, Software Engineer, Google DaydreamMarin Headlands

A view of the Golden Gate Bridge from the Marin Headlands, taken with a DSLR camera (Canon 1DX, Zeiss Otus 28mm f/1.4 ZE). Click here for the full resolution image.

Gcamcomputational photographythe image processing pipeline that enables the HDR+ modesurprisingly good picturesThe ChallengesSeeInTheDarkICCV 2015 Extreme Imaging Workshopcontrast detectionphase detectionExperiments & ResultsDNGPoint Reyes lighthouse

Point Reyes lighthouse at night, photographed with Google Nexus 6P (full resolution image here).

Point Reyes Lighthouse at night, hand-held HDR+ shot (full resolution image here). The inset rectangle has been brightened in Photoshop to roughly match the previous picture.

they appear to rotate around the celestial poles

The North Star above Mount Burdell, single 2-second exposure. (full resolution image here).

Mean of 32 2-second exposures (full resolution image here).

Mean of 32 2-second exposures, stars aligned (full resolution image here).

The North Star above Mount Burdell, shot with Google Pixel, final image (full resolution image here).

Using Even Less Light

Wrecked fishing boat in Inverness and the Big Dipper, 64 2-second exposures, shot with Google Pixel (full resolution image here).

Stars above Pierce Point Ranch, 64 2-second exposures, shot with Google Pixel (full resolution image here).

Pushing the LimitsRussian RiverScorpiusSagittarius

Only starlight, shot with Google Pixel (full resolution image here).

Golden Gate Bridge at night, shot with Google Nexus 6P (full resolution image here).

The Moon above San Francisco, shot with Google Nexus 6P (full resolution image here).

Here’s a Google Photos album

Research at Google and ICLR 2017

Monday, April 24, 2017

Posted by Ian Goodfellow, Staff Research Scientist, Google Brain Team5th International Conference on Learning RepresentationsMachine LearningNeural NetworksDeep LearningGoogle Brain teamGoogle Research EuropeblueArea Chairs include:George Dahl, Slav Petrov, Vikas SindhwaniProgram Chairs include:Hugo Larochelle, Tara SainathContributed TalksUnderstanding Deep Learning Requires Rethinking Generalization(Best Paper Award)Chiyuan Zhang*, Samy Bengio, Moritz Hardt, Benjamin Recht*, Oriol VinyalsSemi-Supervised Knowledge Transfer for Deep Learning from Private Training Data(Best Paper Award)Nicolas Papernot*, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal
TalwarQ-Prop: Sample-Efficient Policy Gradient with An Off-Policy CriticShixiang (Shane) Gu*, Timothy Lillicrap, Zoubin Ghahramani, Richard E.
Turner, Sergey LevineNeural Architecture Search with Reinforcement LearningBarret Zoph, Quoc LePostersAdversarial Machine Learning at ScaleAlexey Kurakin, Ian J. Goodfellow^†, Samy BengioCapacity and Trainability in Recurrent Neural NetworksJasmine Collins, Jascha Sohl-Dickstein, David SussilloImproving Policy Gradient by Exploring Under-Appreciated RewardsOfir Nachum, Mohammad Norouzi, Dale SchuurmansOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff DeanUnrolled Generative Adversarial NetworksLuke Metz, Ben Poole*, David Pfau, Jascha Sohl-DicksteinCategorical Reparameterization with Gumbel-SoftmaxEric Jang, Shixiang (Shane) Gu*, Ben Poole*Decomposing Motion and Content for Natural Video Sequence PredictionRuben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, Honglak LeeDensity Estimation Using Real NVPLaurent Dinh*, Jascha Sohl-Dickstein, Samy BengioLatent Sequence DecompositionsWilliam Chan*, Yu Zhang*, Quoc Le, Navdeep Jaitly*Learning a Natural Language Interface with Neural ProgrammerArvind Neelakantan*, Quoc V. Le, Martín Abadi, Andrew McCallum*, Dario Amodei*Deep Information PropagationSamuel Schoenholz, Justin Gilmer, Surya Ganguli, Jascha Sohl-DicksteinIdentity Matters in Deep LearningMoritz Hardt, Tengyu MaA Learned Representation For Artistic StyleVincent Dumoulin*, Jonathon Shlens, Manjunath KudlurAdversarial Training Methods for Semi-Supervised Text ClassificationTakeru Miyato, Andrew M. Dai, Ian Goodfellow^†HyperNetworksDavid Ha, Andrew Dai, Quoc V. LeLearning to Remember Rare EventsLukasz Kaiser, Ofir Nachum, Aurko Roy*, Samy BengioDeep Learning with Dynamic Computation GraphsMoshe Looks, Marcello Herreshof, DeLesley Hutchins, Peter NorvigHolStep: A Machine Learning Dataset for Higher-order Logic Theorem ProvingCezary Kaliszyk, François Chollet, Christian SzegedyHyperband: Bandit-based Configuration Evaluation for Hyperparameter Optimization Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet TalwalkarWorkshop Track AbstractsParticle Value FunctionsChris J. Maddison, Dieterich Lawson, George Tucker, Nicolas Heess, Arnaud Doucet, Andriy Mnih, Yee Whye TehNeural Combinatorial Optimization with Reinforcement LearningIrwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, Samy BengioShort and Deep: Sketching and Neural NetworksAmit Daniely, Nevena Lazic, Yoram Singer, Kunal TalwarExplaining the Learning Dynamics of Direct Feedback AlignmentJustin Gilmer, Colin Raffel, Samuel S. Schoenholz, Maithra Raghu, Jascha Sohl-DicksteinTraining a Subsampling Mechanism in ExpectationColin Raffel, Dieterich LawsonTuning Recurrent Neural Networks with Reinforcement LearningNatasha Jaques*, Shixiang (Shane) Gu*, Richard E. Turner, Douglas EckREBAR: Low-Variance, Unbiased Gradient Estimates for Discrete Latent Variable ModelsGeorge Tucker, Andriy Mnih, Chris J. Maddison, Jascha Sohl-DicksteinAdversarial Examples in the Physical WorldAlexey Kurakin, Ian Goodfellow^†, Samy BengioRegularizing Neural Networks by Penalizing Confident Output DistributionsGabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, Geoffrey HintonUnsupervised Perceptual Rewards for Imitation LearningPierre Sermanet, Kelvin Xu, Sergey LevineChanging Model Behavior at Test-time Using Reinforcement LearningAugustus Odena, Dieterich Lawson, Christopher Olah * Work performed while at Google † Work performed while at OpenAI

PhotoScan: Taking Glare-Free Pictures of Pictures

Thursday, April 20, 2017

Posted by Ce Liu, Michael Rubinstein, Mike Krainin and Bill Freeman, Research Scientistsupdate to PhotoScaniOSAndroid

Left: A regular digital picture of a physical print. Right: Glare-free digital output from PhotoScan

Left: The captured, input images (5 in total). Right: If we stabilize the images on the photo, we can see just the glare moving, covering different parts of the photo. Notice no single image is glare-free.

SIGGRAPH 2015obstruction-free photographyHow it Workssparse feature pointsORB featuresHarris cornershomographies

Left: Detected feature matches between the reference frame and each other frame (left), and the warped frames according to the estimated homographies (right).

optical flow

Comparison between the warped frames using homographies (left) and after the additional warp refinement using optical flow (right).

Comparison between the glare removal result with (right) and without (left) optical flow refinement. In the result using homographies only (left), notice artifacts around the eye, nose and teeth of the person, and duplicated stems and flower petals on the fabric.

bilinear combination

The grid setup for grid optical flow. A point p is represented as the bilinear interpolation of the four corner points of the cell that encapsulates it.

Left: Illustration of the computed flow field on one of the frames. Right: The flow color coding: orientation and magnitude represented by hue and saturation, respectively.

Szeliski and Coughlan (1997)

Flipping between the homography-registered frame and the flow-refined warped frame (using the above flow field), superimposed on the (clean) reference frame, shows how the computed flow field “snaps” image parts to their corresponding parts in the reference frame, improving the registration.

Regular minimum (left) versus soft minimum (right) over the registered frames.

Input Registered Glare-free

AndroidiOS

Teaching Machines to Draw

Thursday, April 13, 2017

Posted by David Ha, Google Brain Resident

Vector drawings produced by sketch-rnn.

A Neural Representation of Sketch Drawingsrecurrent neural networkraster images

Examples of animals generated with the wrong number of body parts, produced using previous GAN models trained on 128x128 ImageNet dataset. The image above is Figure 29 of
Generative Adversarial Networks, Ian Goodfellow, NIPS 2016 Tutorial.

sequence-to-sequencevariational inferencehypernetworkslatent

Schematic of sketch-rnn.

Reconstructions from a model trained on cat sketches.

notnew

Reconstructions from a model trained on pig sketches.

Latent space interpolations generated from a model trained on pig sketches.

Latent Space Interpolations from a model trained on sketches of both cats and pigs.

Learned relationships between abstract concepts, explored using latent vector arithmetic.

Creative Applications

Similar, but unique cats, generated from a single input sketch (green and yellow boxes).

Exploring the latent space of generated chair-cats.

Exploring the latent space of generated sketches of everyday objects.
Latent space interpolation from left to right, and then top to bottom.

The model can start with incomplete sketches (the red partial sketches to the left of the vertical line) and automatically generate different completions.

Predicting the endings of the same circle and square figures (center) using various sketch-rnn models trained to draw different objects.

A Neural Representation of Sketch DrawingsAcknowledgementsGoogle Brain Residency

Introducing tf-seq2seq: An Open Source Sequence-to-Sequence Framework in TensorFlow

Tuesday, April 11, 2017

Posted by Anna Goldie and Denny Britz, Research Software Engineer and Google Brain Resident, Google Brain Team(Crossposted on the Google Open Source Blog)Google Neural Machine Translationseq2seqtf-seq2seqMassive Exploration of Neural Machine Translation Architectures

A seq2seq model translating from Mandarin to English. At each time step, the encoder takes in one Chinese character and its own previous state (black arrow), and produces an output vector (blue arrow). The decoder then generates an English translation word-by-word, at each time step taking in the last word, the previous state, and a weighted combination of all the outputs of the encoder (aka attention [3], depicted in blue) and then producing the next English word. Please note that in our implementation we use wordpieces [4] to handle rare words.

machine translationGitHub repositoryAcknowledgments:References:Massive Exploration of Neural Machine Translation ArchitecturesDenny Britz, Anna Goldie, Minh-Thang Luong, Quoc LeSequence to Sequence Learning with Neural NetworksIlya Sutskever, Oriol Vinyals, Quoc V. Le. NIPS, 2014Neural Machine Translation by Jointly Learning to Align and TranslateDzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. ICLR, 2015Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine TranslationYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean. Technical Report, 2016Attention and Augmented Recurrent Neural NetworksChris Olah, Shan Carter. Distill, 2016Neural Machine Translation and Sequence-to-sequence Models: A TutorialGraham NeubigSequence-to-Sequence ModelsTensorFlow.org

Announcing the 2017 Google PhD Fellows for North America, Europe and the Middle East

Monday, April 10, 2017

Posted by Michael Rennaker, Program ManagerPhD Fellowship programAlgorithms, Optimizations and MarketsUniversity of California, BerkeleyHarvard UniversityUniversity of Southern CaliforniaHuman-Computer InteractionUniversity of Illinois, Urbana-ChampaignNorthwestern UniversityUniversity of Minnesota - Twin CitiesMachine LearningFondation Sciences Mathématiques de ParisColumbia UniversityUniversity College LondonNew York UniversityUniversity of AmsterdamUniversity of TorontoCornell UniversityMachine Perception, Speech Technology and Computer VisionSaarland University - Saarbrücken GSCS and MPI Institute for InformaticsImperial College LondonStanford UniversityUniversity of California, San DiegoUniversity of Texas, AustinNatural Language ProcessingPrivacy and SecurityENS - École Normale SupérieureDuke UniversityUniversity of Maryland, College ParkProgramming Languages and Software EngineeringAarhus UniversitUniversity of California, Los AngelesTel-Aviv UniversityStructured Data and Database ManagementEPFL CSUniversity of Illinois, Urbana-ChampaignSystems and NetworkingGeorgia Institute of TechnologyUniversity of California, IrvineUniversity of California, BerkeleyPrinceton University

Predicting Properties of Molecules with Machine Learning

Friday, April 7, 2017

Posted by George Dahl, Research Scientist, Google Brain Teamdrug discoverybattery designOLEDscatalystsDensity Functional TheoryRupp et al.Behler and ParrinelloQM9benchmarkGoogle Brain teamGoogle Accelerated Science teamDeepMindUniversity of Baselfirst paperNeural Message Passing for Quantum Chemistryatoms as nodes and bonds as edgesLi et al.Duvenaud et al.Kearnes et al.Defferrard et al.Google Brain residents

Federated Learning: Collaborative Machine Learning without Centralized Training Data

Thursday, April 6, 2017

Posted by Brendan McMahan and Daniel Ramage, Research ScientistsFederated LearningMobile Vision APIOn-Device Smart Replytraining

Your phone personalizes the model locally, based on your usage (A). Many users' updates are aggregated (B) to form a consensus change (C) to the shared model, after which the procedure is repeated.

Gboard on Android

Stochastic Gradient DescentFederated Averaging algorithmmuch slowercompressing updatesdesigned algorithmsTensorFlow

Your phone participates in Federated Learning only
when it won't negatively impact your experience.

Secure Aggregation protocolrecognize different dog breedsAcknowledgements

Blog