Blog

The latest from Google Research

AutoML-Zero: Evolving Code that Learns

Thursday, July 9, 2020

Posted by Esteban Real, Staff Software Engineer and Chen Liang, Software Engineer, Google Research, Brain Teamdeep neural networksAutoMLneural architecture searchtopicmuchresearchlearning rulesince the early 90sour researchrecent paperAutoML-Zerobackpropagationlinear regressionEvolving Learning Algorithms from Scratchvariantevolutionary methodsproved usefulsince the 80smakes them especially suitable for the discovery of learning algorithmsparentmutated

A population is initialized with empty programs. Many generations later, we see a more evolved population and two of its algorithms compete. The most accurate wins to produce a child. After many such events, the final population contains highly accurate classifiers.

Exploring a Difficult Search Space12 migrationcodestochastic gradient descent

Progress of an evolution experiment. As time passes, from left to right, we see the algorithms becoming more complex and more accurate.

The Evolved Algorithmnoise injection as data augmentationbilinear modelgradient normalizationweight averagingnoisy ReLUConclusion^*AcknowledgementsWe want to thank our co-authors, David R. So and Quoc V. Le, and the many who helped us through discussions during the project and paper writing, including Samy Bengio, Vincent Vanhoucke, Doug Eck, Charles Sutton, Yanping Huang, Jacques Pienaar, Jeff Dean, and particularly Gabriel Bender, Hanxiao Liu, Rishabh Singh, Chiyuan Zhang, and Hieu Pham. We also want to especially thank Tom Small for contributing the animations in this post.* The electricity consumption for the experiments (run in 2019) was matched with the purchase of renewable energy. ↩