Poster
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
James Lucas · George Tucker · Roger Grosse · Mohammad Norouzi

Wed Dec 11th 10:45 AM -- 12:45 PM @ East Exhibition Hall B + C #123
in Deep Learning -- Generative Models »

Posterior collapse in Variational Autoencoders (VAEs) with uninformative priors arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers a uniquely identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.

Keywords: Deep Learning -> Generative Models · Probabilistic Methods; Probabilistic Methods -> Latent Variable Models

Author Information

James Lucas (University of Toronto)
George Tucker (Google Brain)
Roger Grosse (University of Toronto)
Mohammad Norouzi (Google Brain)

More from the Same Authors