Timezone: »
Recent research has seen several advances relevant to black-box VI, but the current state of automatic posterior inference is unclear. One such advance is the use of normalizing flows to define flexible posterior densities for deep latent variable models. Another direction is the integration of Monte-Carlo methods to serve two purposes; first, to obtain tighter variational objectives for optimization, and second, to define enriched variational families through sampling. However, both flows and variational Monte-Carlo methods remain relatively unexplored for black-box VI. Moreover, on a pragmatic front, there are several optimization considerations like step-size scheme, parameter initialization, and choice of gradient estimators, for which there are no clear guidance in the existing literature. In this paper, we postulate that black-box VI is best addressed through a careful combination of numerous algorithmic components. We evaluate components relating to optimization, flows, and Monte-Carlo methods on a benchmark of 30 models from the Stan model library. The combination of these algorithmic components significantly advances the state-of-the-art "out of the box" variational inference.
Author Information
Abhinav Agrawal (UMass Amherst)
Ph.D. student working to scale probabilistic inference.
Daniel Sheldon (University of Massachusetts Amherst)
Justin Domke (University of Massachusetts, Amherst)
More from the Same Authors
-
2023 Poster: Provable convergence guarantees for black-box variational inference »
Justin Domke · Robert Gower · Guillaume Garrigos -
2023 Poster: Discriminative Calibration »
Yuling Yao · Justin Domke -
2022 Spotlight: Kernel Interpolation with Sparse Grids »
Mohit Yadav · Daniel Sheldon · Cameron Musco -
2022 Poster: Kernel Interpolation with Sparse Grids »
Mohit Yadav · Daniel Sheldon · Cameron Musco -
2021 Poster: Relaxed Marginal Consistency for Differentially Private Query Answering »
Ryan McKenna · Siddhant Pradhan · Daniel Sheldon · Gerome Miklau -
2021 Poster: MCMC Variational Inference via Uncorrected Hamiltonian Annealing »
Tomas Geffner · Justin Domke -
2021 Poster: Amortized Variational Inference for Simple Hierarchical Models »
Abhinav Agrawal · Justin Domke -
2020 Poster: Approximation Based Variance Reduction for Reparameterization Gradients »
Tomas Geffner · Justin Domke -
2020 Poster: Permute-and-Flip: A new mechanism for differentially private selection »
Ryan McKenna · Daniel Sheldon -
2020 Spotlight: Permute-and-Flip: A new mechanism for differentially private selection »
Ryan McKenna · Daniel Sheldon -
2019 Poster: Thompson Sampling and Approximate Inference »
My Phan · Yasin Abbasi Yadkori · Justin Domke -
2019 Poster: Provable Gradient Variance Guarantees for Black-Box Variational Inference »
Justin Domke -
2019 Poster: Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation »
Justin Domke · Daniel Sheldon -
2019 Spotlight: Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation »
Justin Domke · Daniel Sheldon -
2019 Poster: Differentially Private Bayesian Linear Regression »
Garrett Bernstein · Daniel Sheldon -
2018 Poster: Differentially Private Bayesian Inference for Exponential Families »
Garrett Bernstein · Daniel Sheldon -
2018 Poster: Using Large Ensembles of Control Variates for Variational Inference »
Tomas Geffner · Justin Domke -
2018 Poster: Importance Weighting and Variational Inference »
Justin Domke · Daniel Sheldon -
2018 Poster: Inferring Latent Velocities from Weather Radar Data using Gaussian Processes »
Rico Angell · Daniel Sheldon -
2016 Poster: Probabilistic Inference with Generating Functions for Poisson Latent Variable Models »
Kevin Winner · Daniel Sheldon -
2014 Poster: Stochastic Network Design in Bidirected Trees »
Xiaojian Wu · Daniel Sheldon · Shlomo Zilberstein -
2013 Workshop: Machine Learning for Sustainability »
Edwin Bonilla · Thomas Dietterich · Theodoros Damoulas · Andreas Krause · Daniel Sheldon · Iadine Chades · J. Zico Kolter · Bistra Dilkina · Carla Gomes · Hugo P Simao -
2011 Poster: Collective Graphical Models »
Daniel Sheldon · Thomas Dietterich -
2007 Spotlight: Collective Inference on Markov Models for Modeling Bird Migration »
Daniel Sheldon · M.A. Saleh Elmohamed · Dexter Kozen -
2007 Poster: Collective Inference on Markov Models for Modeling Bird Migration »
Daniel Sheldon · M.A. Saleh Elmohamed · Dexter Kozen