Blog

The latest from Google Research

Predicting the Generalization Gap in Deep Neural Networks

Tuesday, July 9, 2019

Posted by Yiding Jiang, Google AI ResidentDeepneural networksimage recognitionimage segmentationmachine translationVC-dimensionRademacher complexitygeneralize poorlyrecent workmassivelygeneralization gapmargindecision boundarysupport-vector machinesmargintheoretical upper bounds

An example of a support-vector machine decision boundary. The hyperplane defined by w∙x-b=0 is the "decision boundary" of this linear classifier, i.e., every point x lying on the hyperplane is equally likely to be in either class under this classifier.

ICLR 2019Predicting the Generalization Gap in Deep Networks with Margin Distributionsnormalized margingeneralization gapGithub repository

Each plot corresponds to a convolutional neural network trained on CIFAR-10 with different classification accuracies. The probability density (y-axis) of normalized margin distributions (x-axis) at 4 layers of a network is shown for three different models with increasingly better generalization (left to right). The normalized margin distributions are strongly correlated with test accuracy, which suggests they can be used as a proxy for predicting a network's generalization gap. Please see our paper for more details on these networks.

Margin Distributions as a Predictor of Generalizationlinear regression

Predicted generalization gap (x-axis) vs. true generalization gap (y-axis) on CIFAR-100 + ResNet-32. The points lie close to the diagonal line, which indicates that the predicted values of the log linear model fit the true generalization gap very well.