Timezone: »
Stochastic gradient Markov chain Monte Carlo (SGMCMC) is considered the gold standard for Bayesian inference in large-scale models, such as Bayesian neural networks. Since practitioners face speed versus accuracy tradeoffs in these models, variational inference (VI) is often the preferable option. Unfortunately, VI makes strong assumptions on both the factorization and functional form of the posterior. In this work, we propose a new non-parametric variational approximation that makes no assumptions about the approximate posterior's functional form and allows practitioners to specify the exact dependencies the algorithm should respect or break. The approach relies on a new Langevin-type algorithm that operates on a modified energy function, where parts of the latent variables are averaged over samples from earlier iterations of the Markov chain. This way, statistical dependencies can be broken in a controlled way, allowing the chain to mix faster. This scheme can be further modified in a ``dropout'' manner, leading to even more scalability. By implementing the scheme on a ResNet-20 architecture, we obtain better predictive likelihoods and larger effective sample sizes than full SGMCMC.
Author Information
Antonios Alexos (University of California, Irvine)
Alex Boyd (UC Irvine)
Stephan Mandt (University of California, Irvine)
More from the Same Authors
-
2021 : Analyzing High-Resolution Clouds and Convection using Multi-Channel VAEs »
Harshini Mangipudi · Griffin Mooers · Mike Pritchard · Tom Beucler · Stephan Mandt -
2022 : Probabilistic Querying of Continuous-Time Sequential Events »
Alex Boyd · Yuxin Chang · Stephan Mandt · Padhraic Smyth -
2022 : An Unsupervised Learning Perspective on the Dynamic Contribution to Extreme Precipitation Changes »
Griffin Mooers · Tom Beucler · Mike Pritchard · Stephan Mandt -
2022 Panel: Panel 5B-4: Predictive Querying for… & On the difficulty… »
Alex Boyd · Jonas Mikhaeil -
2022 : Q & A »
Karen Ullrich · Yibo Yang · Stephan Mandt -
2022 Tutorial: Data Compression with Machine Learning »
Karen Ullrich · Yibo Yang · Stephan Mandt -
2022 : Tutorial part 1 »
Yibo Yang · Karen Ullrich · Stephan Mandt -
2022 Poster: Predictive Querying for Autoregressive Neural Sequence Models »
Alex Boyd · Samuel Showalter · Stephan Mandt · Padhraic Smyth -
2021 Poster: Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning »
Aodong Li · Alex Boyd · Padhraic Smyth · Stephan Mandt -
2020 : Q/A and Discussion for ML Theory Session »
Karthik Kashinath · Mayur Mudigonda · Stephan Mandt · Rose Yu -
2020 : Stephan Mandt »
Stephan Mandt -
2020 Poster: User-Dependent Neural Sequence Models for Continuous-Time Event Data »
Alex Boyd · Robert Bamler · Stephan Mandt · Padhraic Smyth