AutoML-Zero aims to automatically discover computer programs that can solve machine learning tasks, starting from empty or random programs and using only basic math operations. The goal is to simultaneously search for all aspects of an ML algorithm—including the model structure and the learning strategy—while employing minimal human bias.
GIF for the experiment progress
Despite AutoML-Zero’s challenging search space, evolutionary search shows promising results by discovering linear regression with gradient descent, 2-layer neural networks with backpropagation, and even algorithms that surpass hand designed baselines of comparable complexity. The figure above shows an example sequence of discoveries from one of our experiments, evolving algorithms to solve binary classification tasks. Notably, the evolved algorithms can be interpreted. Below is an analysis of the best evolved algorithm: the search process “invented” techniques like bilinear interactions, weight averaging, normalized gradient, and data augmentation (by adding noise to the inputs).
GIF for the interpretation of the best evolved algorithm
More examples, analysis, and details can be found in the paper.
