Timezone: »

 
Poster
Deep Ensembles Work, But Are They Necessary?
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · Richard Zemel · John Cunningham

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #906
in Poster Session 2 »

Ensembling neural networks is an effective way to increase accuracy, and can often match the performance of individual larger models. This observation poses a natural question: given the choice between a deep ensemble and a single neural network with similar accuracy, is one preferable over the other? Recent work suggests that deep ensembles may offer distinct benefits beyond predictive power: namely, uncertainty quantification and robustness to dataset shift. In this work, we demonstrate limitations to these purported benefits, and show that a single (but larger) neural network can replicate these qualities. First, we show that ensemble diversity, by any metric, does not meaningfully contribute to an ensemble's ability to detect out-of-distribution (OOD) data, but is instead highly correlated with the relative improvement of a single larger model. Second, we show that the OOD performance afforded by ensembles is strongly determined by their in-distribution (InD) performance, and - in this sense - is not indicative of any "effective robustness." While deep ensembles are a practical way to achieve improvements to predictive power, uncertainty quantification, and robustness, our results show that these improvements can be replicated by a (larger) single model.

Keywords: uncertainty quantification · robustness · deep ensembles · Dataset Shift
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Taiga Abe (Columbia University)
Estefany Kelly Buchanan (Columbia University)
Geoff Pleiss (Columbia University)
Richard Zemel (Columbia University)
John Cunningham (Columbia University)

More from the Same Authors