Blog

The latest from Google Research

Can You Trust Your Model’s Uncertainty?

miércoles, 15 de enero de 2020

Posted by Jasper Snoek, Research Scientist and Zachary Nado, Research Engineer, Google Researchdeep learningcovariate shiftit was recently observedCan you trust your model’s uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift” NeurIPS 2019What is Out-of-Distribution Data?uncertainty out-of-distribution (OOD)related postQuantifying the Quality of Uncertaintyproper scoring rulesBrier ScoreLog Likelihoodexpected calibration errorExperimentsImageNetCorrupted Imagenet

We explore how model uncertainty behaves under changes to the data distribution, such as increasing intensities of the image perturbations used in Corrupted Imagenet. Shown here are examples of each type of image corruption, at intensity level 3 (of 5).

shifteddropouttemperature scaling

Accuracy (top) and expected calibration error (bottom; lower is better) for increasing intensities of dataset shift on ImageNet-C. We observe that the decrease in accuracy is not reflected by an increase in uncertainty of the model, indicated by both accuracy and ECE getting worse.

temperature scalingPlatt scalingDeep ensemblesSummary and Recommended Best Practices

Uncertainty under dataset shift is a real concern that needs to be considered when training models.

Improving calibration and accuracy on an in-distribution test set often does not translate to improved calibration on shifted data.

Out of all the methods we considered, deep ensembles are the most robust to dataset shift, and a relatively small ensemble size (e.g., 5) is sufficient. The effectiveness of ensembles presents interesting avenues for improving other approaches.