Timezone: »

 
Poster
On the detrimental effect of invariances in the likelihood for variational inference
Richard Kurle · Ralf Herbrich · Tim Januschowski · Yuyang (Bernie) Wang · Jan Gasthaus

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #322
in Poster Session 2 »

Variational Bayesian posterior inference often requires simplifying approximations such as mean-field parametrisation to ensure tractability. However, prior work has associated the variational mean-field approximation for Bayesian neural networks with underfitting in the case of small datasets or large model sizes. In this work, we show that invariances in the likelihood function of over-parametrised models contribute to this phenomenon because these invariances complicate the structure of the posterior by introducing discrete and/or continuous modes which cannot be well approximated by Gaussian mean-field distributions. In particular, we show that the mean-field approximation has an additional gap in the evidence lower bound compared to a purpose-built posterior that takes into account the known invariances. Importantly, this invariance gap is not constant; it vanishes as the approximation reverts to the prior. We proceed by first considering translation invariances in a linear model with a single data point in detail. We show that, while the true posterior can be constructed from a mean-field parametrisation, this is achieved only if the objective function takes into account the invariance gap. Then, we transfer our analysis of the linear model to neural networks. Our analysis provides a framework for future work to explore solutions to the invariance problem.

Keywords: Variational Inference · Invariance · Variational Bayes · Bayesian Neural Networks · posterior collapse · Symmetry

Author Information

Richard Kurle (Technical University of Munich)
Ralf Herbrich (Hasso Plattner Institute)
Ralf Herbrich

I am Professor and Chair for Artificial Intelligence and Sustainability research group at the Hasso-Plattner institute.

Tim Januschowski (Zalando SE)

- Director Pricing Platform, Zalando SE - Head of Time Series ML at AWS AI

Yuyang (Bernie) Wang (AWS AI Labs)
Jan Gasthaus

More from the Same Authors