Poster
Improving Variational Autoencoders with Inverse Autoregressive Flow
Durk Kingma · Tim Salimans · Rafal Jozefowicz · Peter Chen · Xi Chen · Ilya Sutskever · Max Welling

Mon Dec 5th 06:00 -- 09:30 PM @ Area 5+6+7+8 #83 #None
in Monday Posters »

We propose a simple and scalable method for improving the flexibility of variational inference through a transformation with autoregressive neural networks. Autoregressive neural networks, such as RNNs or the PixelCNN, are very powerful models and potentially interesting for use as variational posterior approximation. However, ancestral sampling in such networks is a long sequential operation, and therefore typically very slow on modern parallel hardware, such as GPUs. We show that by inverting autoregressive neural networks we can obtain equally powerful posterior models from which we can sample efficiently on modern hardware. We show that such data transformations, inverse autoregressive flows (IAF), can be used to transform a simple distribution over the latent variables into a much more flexible distribution, while still allowing us to compute the resulting variables' probability density function. The method is simple to implement, can be made arbitrarily flexible and, in contrast with previous work, is well applicable to models with high-dimensional latent spaces, such as convolutional generative models. The method is applied to a novel deep architecture of variational auto-encoders. In experiments with natural images, we demonstrate that autoregressive flow leads to significant performance gains.

Keywords: Deep Learning or Neural Networks

Author Information

Durk Kingma (Google)
Tim Salimans (Algoritmica)
Rafal Jozefowicz (OpenAI)
Peter Chen (UC Berkeley and OpenAI)
Xi Chen (UC Berkeley and OpenAI)
Ilya Sutskever (Google)
Max Welling (University of Amsterdam / Qualcomm AI Research)

More from the Same Authors