Blog

The latest news from Google AI

Google Brain Residency Program - 7 months in and looking ahead

Thursday, January 5, 2017

Posted by Jeff Dean, Google Senior Fellow and Leslie Phillips, Google Brain Residency Program Manager“Beyond being incredibly instructive, the Google Brain Residency program has been a truly affirming experience. Working alongside people who truly love what they do--and are eager to help you develop your own passion--has vastly increased my confidence in my interests, my ability to explore them, and my plans for the near future.”Google Brain ResidencyGoogle Brain teamBayLearn ConferenceMean Field Neural NetworksRegularizing Neural Networks by Penalizing Their Output DistributionNIPS 2016 Adversarial Training workshopISMIR 2016ICLR 2017

An LSTM Cell (Left) and a state of the art RNN Cell found using a neural network (Right). This is an example of a novel architecture found using the approach presented in “Neural Architecture Search with Reinforcement Learning” (B. Zoph and Q. V. Le, submitted to ICLR 2017). This paper uses a neural network to generate novel RNN cell architectures that outperform the widely used LSTM on a variety of different tasks.

The training accuracy for neural networks, colored from black (random chance) to red (high accuracy). Overlaid in white dashed lines are the theoretical predictions showing the boundary between trainable and untrainable networks. (a) Networks with no dropout. (b)-(d) Networks with dropout rates of 0.01, 0.02, 0.06 respectively. This research explores whether theoretical calculations can replace large hyperparameter searches. For more details, read “Deep Information Propagation” (S. S. Schoenholz, J. Gilmer, S. Ganguli, J. Sohl-Dickstein, submitted to ICLR 2017).

Accepted conference papers(Google Brain Residents marked with asterisks)Unrolled Generative Adversarial NetworksLuke Metz*, Ben Poole, David Pfau, Jascha Sohl-DicksteinNIPS 2016 Adversarial Training Workshop (oral presentation)Conditional Image Synthesis with Auxiliary Classifier GANsAugustus Odena*, Chris Olah, Jon ShlensNIPS 2016 Adversarial Training Workshop (oral presentation)Regularizing Neural Networks by Penalizing Their Output DistributionGabriel Pereyra*, George Tucker, Lukasz Kaiser, Geoff HintonBayLearn 2016 (oral presentation)Mean Field Neural NetworksSamuel S. Schoenholz*, Justin Gilmer*, Jascha Sohl-DicksteinBayLearn 2016 (oral presentation)Learning to RememberAurko Roy, Ofir Nachum*, Łukasz Kaiser, Samy BengioBayLearn 2016 (poster session)Towards Generating Higher Resolution Images with Generative Adversarial NetworksAugustus Odena*, Jonathon ShlensBayLearn 2016 (poster session)Multi-Task Convolutional Music ModelsDiego Ardila, Cinjon Resnick*, Adam Roberts, Douglas EckBayLearn 2016 (poster session)Audio DeepDream: Optimizing Raw Audio With Convolutional NetworksDiego Ardila, Cinjon Resnick*, Adam Roberts, Douglas EckISMIR 2016 (poster session)Papers under review(Google Brain Residents marked with asterisks)Learning to Remember Rare Events Lukasz Kaiser, Ofir Nachum*, Aurko Roy, Samy Bengio Submitted to ICLR 2017 Neural Combinatorial Optimization with Reinforcement LearningIrwan Bello*, Hieu Pham*, Quoc V. Le, Mohammad Norouzi, Samy BengioSubmitted to ICLR 2017 HyperNetworksDavid Ha*, Andrew Dai, Quoc V. LeSubmitted to ICLR 2017 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerNoam Shazeer, Azalia Mirhoseini*, Krzysztof Maziarz, Quoc Le, Jeff DeanSubmitted to ICLR 2017 Neural Architecture Search with Reinforcement LearningBarret Zoph* and Quoc LeSubmitted to ICLR 2017 Deep Information PropagationSamuel Schoenholz*, Justin Gilmer*, Surya Ganguli, Jascha Sohl-DicksteinSubmitted to ICLR 2017 Capacity and Trainability in Recurrent Neural NetworksJasmine Collins*, Jascha Sohl-Dickstein, David SussilloSubmitted to ICLR 2017 Unrolled Generative Adversarial NetworksLuke Metz*, Ben Poole, David Pfau, Jascha Sohl-DicksteinSubmitted to ICLR 2017 Conditional Image Synthesis with Auxiliary Classifier GANsAugustus Odena*, Chris Olah, Jon ShlensSubmitted to ICLR 2017
Generating Long and Diverse Responses with Neural Conversation ModelsLouis Shao, Stephan Gouws, Denny Britz*, Anna Goldie, Brian Strope, Ray KurzweilSubmitted to ICLR 2017Intelligible Language Modeling with Input Switched Affine NetworksJakob Foerster, Justin Gilmer*, Jan Chorowski, Jascha Sohl-dickstein, David SussilloSubmitted to ICLR 2017 Regularizing Neural Networks by Penalizing Confident Output DistributionsGabriel Pereyra*, George Tucker*, Jan Chorowski, Lukasz Kaiser, Geoffrey HintonSubmitted to ICLR 2017 Unsupervised Perceptual Rewards for Imitation LearningPierre Sermanet, Kelvin Xu*, Sergey LevineSubmitted to ICLR 2017 Improving policy gradient by exploring under-appreciated rewardsOfir Nachum*, Mohammad Norouzi, Dale SchuurmansSubmitted to ICLR 2017 Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step ConditioningAkosua Busia*, Jasmine Collins*, Navdeep JaitlyWe are currently accepting applications for the 2017 Google Brain Residency Programg.co/brainresidency