Timezone: »
Ensembling neural networks is an effective way to increase accuracy, and can often match the performance of individual larger models. This observation poses a natural question: given the choice between a deep ensemble and a single neural network with similar accuracy, is one preferable over the other? Recent work suggests that deep ensembles may offer distinct benefits beyond predictive power: namely, uncertainty quantification and robustness to dataset shift. In this work, we demonstrate limitations to these purported benefits, and show that a single (but larger) neural network can replicate these qualities. First, we show that ensemble diversity, by any metric, does not meaningfully contribute to an ensemble's ability to detect out-of-distribution (OOD) data, but is instead highly correlated with the relative improvement of a single larger model. Second, we show that the OOD performance afforded by ensembles is strongly determined by their in-distribution (InD) performance, and - in this sense - is not indicative of any "effective robustness." While deep ensembles are a practical way to achieve improvements to predictive power, uncertainty quantification, and robustness, our results show that these improvements can be replicated by a (larger) single model.
Author Information
Taiga Abe (Columbia University)
Estefany Kelly Buchanan (Columbia University)
Geoff Pleiss (Columbia University)
Richard Zemel (Columbia University)
John Cunningham (Columbia University)
More from the Same Authors
-
2021 : Understanding Post-hoc Adaptation for Improving Subgroup Robustness »
David Madras · Richard Zemel -
2021 : Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data »
Sindy Löwe · David Madras · Richard Zemel · Max Welling -
2022 : Reliability benchmarks for image segmentation »
Estefany Kelly Buchanan · Michael Dusenberry · Jie Ren · Kevin Murphy · Balaji Lakshminarayanan · Dustin Tran -
2022 : The Best Deep Ensembles Sacrifice Predictive Diversity »
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · John Cunningham -
2022 : Denoising Deep Generative Models »
Gabriel Loaiza-Ganem · Brendan Ross · Luhuan Wu · John Cunningham · Jesse Cresswell · Anthony Caterini -
2023 Poster: Sharp Calibrated Gaussian Processes »
Alexandre Capone · Sandra Hirche · Geoff Pleiss -
2023 Poster: Twisting Towards Perfection: Asymptotically Exact Conditional Sampling in Diffusion Models »
Luhuan Wu · Brian Trippe · Christian Naesseth · John Cunningham · David Blei -
2023 Poster: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra »
Andres Potapczynski · Marc Finzi · Geoff Pleiss · Andrew Wilson -
2023 Poster: Distribution-Free Statistical Dispersion Control for Societal Applications »
Zhun Deng · Thomas Zollo · Jake Snell · Toniann Pitassi · Richard Zemel -
2023 Workshop: I Can’t Believe It’s Not Better (ICBINB): Failure Modes in the Age of Foundation Models »
Estefany Kelly Buchanan · Fan Feng · Andreas Kriegler · Ian Mason · Tobias Uelwer · Yubin Xie · Rui Yang -
2022 Workshop: The Symbiosis of Deep Learning and Differential Equations II »
Michael Poli · Winnie Xu · Estefany Kelly Buchanan · Maryam Hosseini · Luca Celotti · Martin Magill · Ermal Rrapaj · Qiyao Wei · Stefano Massaroli · Patrick Kidger · Archis Joglekar · Animesh Garg · David Duvenaud -
2022 Workshop: Gaussian Processes, Spatiotemporal Modeling, and Decision-making Systems »
Alexander Terenin · Elizaveta Semenova · Geoff Pleiss · Zi Wang -
2022 Poster: Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome »
Elliott Gordon-Rodriguez · Thomas Quinn · John Cunningham -
2022 Poster: Posterior and Computational Uncertainty in Gaussian Processes »
Jonathan Wenger · Geoff Pleiss · Marvin Pförtner · Philipp Hennig · John Cunningham -
2022 Poster: Implications of Model Indeterminacy for Explanations of Automated Decisions »
Marc-Etienne Brunet · Ashton Anderson · Richard Zemel -
2021 Workshop: The Symbiosis of Deep Learning and Differential Equations »
Luca Celotti · Kelly Buchanan · Jorge Ortiz · Patrick Kidger · Stefano Massaroli · Michael Poli · Lily Hu · Ermal Rrapaj · Martin Magill · Thorsteinn Jonsson · Animesh Garg · Murtadha Aldeer -
2021 Poster: The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective »
Geoff Pleiss · John Cunningham -
2021 Poster: Posterior Collapse and Latent Variable Non-identifiability »
Yixin Wang · David Blei · John Cunningham -
2021 Poster: Rectangular Flows for Manifold Learning »
Anthony Caterini · Gabriel Loaiza-Ganem · Geoff Pleiss · John Cunningham -
2021 Poster: Variational Model Inversion Attacks »
Kuan-Chieh Wang · YAN FU · Ke Li · Ashish Khisti · Richard Zemel · Alireza Makhzani -
2021 Poster: Identifying and Benchmarking Natural Out-of-Context Prediction Problems »
David Madras · Richard Zemel -
2020 : Contributed talks 5: Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification »
Elliot Creager · David Madras · Richard Zemel -
2020 Poster: Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking »
Anqi Wu · Estefany Kelly Buchanan · Matthew Whiteway · Michael Schartner · Guido Meijer · Jean-Paul Noel · Erica Rodriguez · Claire Everett · Amy Norovich · Evan Schaffer · Neeli Mishra · C. Daniel Salzman · Dora Angelaki · Andrés Bendesky · The International Brain Laboratory The International Brain Laboratory · John Cunningham · Liam Paninski -
2020 Poster: Recurrent Switching Dynamical Systems Models for Multiple Interacting Neural Populations »
Joshua Glaser · Matthew Whiteway · John Cunningham · Liam Paninski · Scott Linderman -
2020 Poster: Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax »
Andres Potapczynski · Gabriel Loaiza-Ganem · John Cunningham -
2019 Poster: Paraphrase Generation with Latent Bag of Words »
Yao Fu · Yansong Feng · John Cunningham -
2019 Poster: BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos »
Eleanor Batty · Matthew Whiteway · Shreya Saxena · Dan Biderman · Taiga Abe · Simon Musall · Winthrop Gillis · Jeffrey Markowitz · Anne Churchland · John Cunningham · Sandeep R Datta · Scott Linderman · Liam Paninski -
2019 Poster: Incremental Few-Shot Learning with Attention Attractor Networks »
Mengye Ren · Renjie Liao · Ethan Fetaya · Richard Zemel -
2019 Poster: Deep Random Splines for Point Process Intensity Estimation of Neural Population Data »
Gabriel Loaiza-Ganem · Sean Perkins · Karen Schroeder · Mark Churchland · John Cunningham -
2019 Poster: SMILe: Scalable Meta Inverse Reinforcement Learning through Context-Conditional Policies »
Kamyar Ghasemipour · Shixiang (Shane) Gu · Richard Zemel -
2019 Poster: The continuous Bernoulli: fixing a pervasive error in variational autoencoders »
Gabriel Loaiza-Ganem · John Cunningham -
2019 Poster: Efficient Graph Generation with Graph Recurrent Attention Networks »
Renjie Liao · Yujia Li · Yang Song · Shenlong Wang · Will Hamilton · David Duvenaud · Raquel Urtasun · Richard Zemel -
2018 Poster: Learning Latent Subspaces in Variational Autoencoders »
Jack Klys · Jake Snell · Richard Zemel -
2018 Poster: Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer »
David Madras · Toni Pitassi · Richard Zemel -
2018 Poster: Neural Guided Constraint Logic Programming for Program Synthesis »
Lisa Zhang · Gregory Rosenblatt · Ethan Fetaya · Renjie Liao · William Byrd · Matthew Might · Raquel Urtasun · Richard Zemel -
2017 : Contributed talk: Predict Responsibly: Increasing Fairness by Learning To Defer Abstract »
David Madras · Richard Zemel · Toni Pitassi -
2017 : 3 spotlight presentations »
Estefany Kelly Buchanan · Mathias Lechner · Kezhi Li -
2017 Poster: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Poster: Causal Effect Inference with Deep Latent-Variable Models »
Christos Louizos · Uri Shalit · Joris Mooij · David Sontag · Richard Zemel · Max Welling -
2017 Spotlight: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Poster: Few-Shot Learning Through an Information Retrieval Lens »
Eleni Triantafillou · Richard Zemel · Raquel Urtasun -
2017 Poster: Prototypical Networks for Few-shot Learning »
Jake Snell · Kevin Swersky · Richard Zemel -
2016 Poster: Linear dynamical neural population models through nonlinear embeddings »
Yuanjun Gao · Evan Archer · Liam Paninski · John Cunningham -
2016 Poster: Understanding the Effective Receptive Field in Deep Convolutional Neural Networks »
Wenjie Luo · Yujia Li · Raquel Urtasun · Richard Zemel -
2016 Poster: Learning Deep Parsimonious Representations »
Renjie Liao · Alex Schwing · Richard Zemel · Raquel Urtasun -
2016 Poster: Automated scalable segmentation of neurons from multispectral images »
Uygar Sümbül · Douglas Roossien · Dawen Cai · Fei Chen · Nicholas Barry · John Cunningham · Edward Boyden · Liam Paninski -
2015 Poster: Skip-Thought Vectors »
Jamie Kiros · Yukun Zhu · Russ Salakhutdinov · Richard Zemel · Raquel Urtasun · Antonio Torralba · Sanja Fidler -
2015 Poster: Bayesian Active Model Selection with an Application to Automated Audiometry »
Jacob Gardner · Gustavo Malkomes · Roman Garnett · Kilian Weinberger · Dennis Barbour · John Cunningham -
2015 Poster: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham -
2015 Spotlight: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham -
2015 Poster: Exploring Models and Data for Image Question Answering »
Mengye Ren · Jamie Kiros · Richard Zemel -
2014 Workshop: Representation and Learning Methods for Complex Outputs »
Richard Zemel · Dale Schuurmans · Kilian Q Weinberger · Yuhong Guo · Jia Deng · Francesco Dinuzzo · Hal Daumé III · Honglak Lee · Noah A Smith · Richard Sutton · Jiaqian YU · Vitaly Kuznetsov · Luke Vilnis · Hanchen Xiong · Calvin Murdock · Thomas Unterthiner · Jean-Francis Roy · Martin Renqiang Min · Hichem SAHBI · Fabio Massimo Zanzotto -
2014 Poster: A Multiplicative Model for Learning Distributed Text-Based Attribute Representations »
Jamie Kiros · Richard Zemel · Russ Salakhutdinov -
2013 Workshop: Output Representation Learning »
Yuhong Guo · Dale Schuurmans · Richard Zemel · Samy Bengio · Yoshua Bengio · Li Deng · Dan Roth · Kilian Q Weinberger · Jason Weston · Kihyuk Sohn · Florent Perronnin · Gabriel Synnaeve · Pablo R Strasser · julien audiffren · Carlo Ciliberto · Dan Goldwasser -
2013 Poster: A Determinantal Point Process Latent Variable Model for Inhibition in Neural Spiking Data »
Jasper Snoek · Richard Zemel · Ryan Adams -
2013 Poster: On the Expressive Power of Restricted Boltzmann Machines »
James Martens · Arkadev Chattopadhya · Toni Pitassi · Richard Zemel -
2012 Poster: Collaborative Ranking With 17 Parameters »
Maksims Volkovs · Richard Zemel -
2012 Poster: Bayesian n-Choose-k Models for Classification and Ranking »
Kevin Swersky · Danny Tarlow · Richard Zemel · Ryan Adams · Brendan J Frey -
2012 Poster: Efficient Sampling for Bipartite Matching Problems »
Maksims Volkovs · Richard Zemel -
2012 Poster: Cardinality Restricted Boltzmann Machines »
Kevin Swersky · Danny Tarlow · Ilya Sutskever · Richard Zemel · Russ Salakhutdinov · Ryan Adams -
2010 Talk: Opening Remarks and Awards »
Richard Zemel · Terrence Sejnowski · John Shawe-Taylor -
2009 Placeholder: Opening Remarks »
Richard Zemel -
2008 Poster: Comparing model predictions of response bias and variance in cue combination »
Rama Natarajan · Iain Murray · Ladan Shams · Richard Zemel -
2008 Poster: Learning Hybrid Models for Image Annotation with Partially Labeled Data »
Xuming He · Richard Zemel -
2008 Poster: Competing RBM density models for classification of fMRI images »
Tanya Schmah · Geoffrey E Hinton · Richard Zemel