Timezone: »
Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we study the extent to which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, we find that this theory does not provide a tight explanation of the out-of-domain generalization observed across a large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures—that, however, lack theory—which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the out-of-distribution generalization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.
Author Information
Ramakrishna Vedantam (Facebook AI Research)
David Lopez-Paz (Facebook AI Research)
David Schwab (CUNY Graduate Center)
More from the Same Authors
-
2021 : Learning Background Invariance Improves Generalization and Robustness in Self Supervised Learning on ImageNet and Beyond »
Chaitanya Ryali · David Schwab · Ari Morcos -
2022 : Pre-train, fine-tune, interpolate: a three-stage strategy for domain generalization »
Alexandre Rame · Jianyu Zhang · Leon Bottou · David Lopez-Paz -
2022 Workshop: INTERPOLATE — First Workshop on Interpolation Regularizers and Beyond »
Yann Dauphin · David Lopez-Paz · Vikas Verma · Boyi Li -
2021 Poster: Perturbation Theory for the Information Bottleneck »
Vudtiwat Ngampruetikorn · David Schwab -
2020 Poster: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2020 Spotlight: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2019 Poster: Learning about an exponential amount of conditional distributions »
Mohamed Ishmael Belghazi · Maxime Oquab · David Lopez-Paz -
2019 Poster: Single-Model Uncertainties for Deep Learning »
Nataša Tagasovska · David Lopez-Paz -
2018 : Opening Remarks »
David Lopez-Paz -
2018 Workshop: Causal Learning »
Martin Arjovsky · Christina Heinze-Deml · Anna Klimovskaia · Maxime Oquab · Leon Bottou · David Lopez-Paz -
2017 Poster: Gradient Episodic Memory for Continual Learning »
David Lopez-Paz · Marc'Aurelio Ranzato -
2016 : Welcome »
David Lopez-Paz · Alec Radford · Leon Bottou -
2016 Workshop: Adversarial Training »
David Lopez-Paz · Leon Bottou · Alec Radford -
2014 Workshop: Modern Nonparametrics 3: Automating the Learning Pipeline »
Eric Xing · Mladen Kolar · Arthur Gretton · Samory Kpotufe · Han Liu · Zoltán Szabó · Alan Yuille · Andrew G Wilson · Ryan Tibshirani · Sasha Rakhlin · Damian Kozbur · Bharath Sriperumbudur · David Lopez-Paz · Kirthevasan Kandasamy · Francesco Orabona · Andreas Damianou · Wacha Bounliphone · Yanshuai Cao · Arijit Das · Yingzhen Yang · Giulia DeSalvo · Dmitry Storcheus · Roberto Valerio -
2013 Workshop: Randomized Methods for Machine Learning »
David Lopez-Paz · Quoc V Le · Alexander Smola -
2013 Poster: The Randomized Dependence Coefficient »
David Lopez-Paz · Philipp Hennig · Bernhard Schölkopf -
2012 Poster: Semi-Supervised Domain Adaptation with Non-Parametric Copulas »
David Lopez-Paz · José Miguel Hernández-Lobato · Bernhard Schölkopf -
2012 Spotlight: Semi-Supervised Domain Adaptation with Non-Parametric Copulas »
David Lopez-Paz · José Miguel Hernández-Lobato · Bernhard Schölkopf