Timezone: »

 
Poster
Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
Guy Lorberbom · Andreea Gane · Tommi Jaakkola · Tamir Hazan

Tue Dec 10 05:30 PM -- 07:30 PM (PST) @ East Exhibition Hall B + C #75
in Deep Learning -- Generative Models »
Reparameterization of variational auto-encoders with continuous random variables is an effective method for reducing the variance of their gradient estimates. In the discrete case, one can perform reparametrization using the Gumbel-Max trick, but the resulting objective relies on an $\arg \max$ operation and is non-differentiable. In contrast to previous works which resort to \emph{softmax}-based relaxations, we propose to optimize it directly by applying the \emph{direct loss minimization} approach. Our proposal extends naturally to structured discrete latent variable models when evaluating the $\arg \max$ operation is tractable. We demonstrate empirically the effectiveness of the direct loss minimization technique in variational autoencoders with both unstructured and structured discrete latent variables.
Keywords: Variational Inference · Deep Learning · Deep Learning -> Generative Models; Probabilistic Methods
[ Paper]  [ Slides] 

Author Information

Guy Lorberbom (Technion)
Andreea Gane (Google AI)
Tommi Jaakkola (MIT)

Tommi Jaakkola is a professor of Electrical Engineering and Computer Science at MIT. He received an M.Sc. degree in theoretical physics from Helsinki University of Technology, and Ph.D. from MIT in computational neuroscience. Following a Sloan postdoctoral fellowship in computational molecular biology, he joined the MIT faculty in 1998. His research interests include statistical inference, graphical models, and large scale modern estimation problems with predominantly incomplete data.

Tamir Hazan (Technion)

More from the Same Authors