Timezone: »

VarGrad: A Low-Variance Gradient Estimator for Variational Inference
Lorenz Richter · Ayman Boustati · Nikolas Nüsken · Francisco Ruiz · Omer Deniz Akyildiz

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #855

We analyse the properties of an unbiased gradient estimator of the ELBO for variational inference, based on the score function method with leave-one-out control variates. We show that this gradient estimator can be obtained using a new loss, defined as the variance of the log-ratio between the exact posterior and the variational approximation, which we call the log-variance loss. Under certain conditions, the gradient of the log-variance loss equals the gradient of the (negative) ELBO. We show theoretically that this gradient estimator, which we call VarGrad due to its connection to the log-variance loss, exhibits lower variance than the score function method in certain settings, and that the leave-one-out control variate coefficients are close to the optimal ones. We empirically demonstrate that VarGrad offers a favourable variance versus computation trade-off compared to other state-of-the-art estimators on a discrete VAE.

Author Information

Lorenz Richter (Freie Universität Berlin, BTU Cottbus-Senftenberg, dida Datenschmiede GmbH)
Ayman Boustati (University of Warwick)
Nikolas Nüsken (Universität Potsdam)
Francisco Ruiz (DeepMind)
Omer Deniz Akyildiz (University of Warwick and The Alan Turing Institute)

More from the Same Authors