Timezone: »

 
The Peril of Popular Deep Learning Uncertainty Estimation Methods
Yehao Liu · Matteo Pagliardini · Tatjana Chavdarova · Sebastian Stich

@
in Bayesian Deep Learning »
Event URL: https://openreview.net/forum?id=Qm1M4AjJEzE »

Uncertainty estimation (UE) techniques---such as the Gaussian process (GP), Bayesian neural networks (BNN), Monte Carlo dropout (MCDropout)---aim to improve the interpretability of machine learning models by assigning an estimated uncertainty value to each of their prediciton outputs. However, since too high uncertainty estimates can have fatal consequences in practice, this paper analyzes the above techniques.Firstly, we show that GP methods always yield high uncertainty estimates on out of distribution (OOD) data. Secondly, we show on a 2D toy example that both BNNs and MCDropout do not give high uncertainty estimates on OOD samples. Finally, we show empirically that this pitfall of BNNs and MCDropout holds on real world datasets as well. Our insights (i) raise awareness for the more cautious use of currently popular UE methods in Deep Learning, (ii) encourage the development of UE methods that approximate GP-based methods---instead of BNNs and MCDropout, and (iii) our empirical setups can be used for verifying the OOD performances of any other UE method.

Author Information

Yehao Liu (EPFL)
Matteo Pagliardini (Swiss Federal Institute of Technology Lausanne)
Tatjana Chavdarova (UC Berkeley)
Sebastian Stich (CISPA)

Dr. [Sebastian U. Stich](https://sstich.ch/) is a faculty at the CISPA Helmholtz Center for Information Security. Research interests: - *methods for machine learning and statistics*—at the interface of theory and practice - *collaborative learning* (distributed, federated and decentralized methods) - *optimization for machine learning* (adaptive stochastic methods and generalization performance)

More from the Same Authors