Timezone: »

 
Poster
Embrace the Gap: VAEs Perform Independent Mechanism Analysis
Patrik Reizinger · Luigi Gresele · Jack Brady · Julius von Kügelgen · Dominik Zietlow · Bernhard Schölkopf · Georg Martius · Wieland Brendel · Michel Besserve

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #403
in Poster Session 2 »

Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood. While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder---a commonly used but unproven conjecture---which we refer to as self-consistency. Leveraging self-consistency, we show that the ELBO converges to a regularized log-likelihood. This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds an inductive bias towards decoders with column-orthogonal Jacobians, which helps recovering the true latent factors. The gap between ELBO and log-likelihood is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning. In experiments on synthetic and image data, we show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.

Keywords: Variational Inference · Representation Learning · variational autoencoder · ELBO · independent mechanism analysis
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Patrik Reizinger (University of Tübingen)
Patrik Reizinger

Ph.D. student at IMPRS-IS and ELLIS from 2021, a former student of Budapest Univeristy of Technology and Economics.

Luigi Gresele (MPI for Intelligent Systems, Tübingen)
Jack Brady (Texas A&M)
Julius von Kügelgen (Max Planck Institute for Intelligent Systems Tübingen & University of Cambridge)
Dominik Zietlow (Max Planck Institute for Intelligent Systems, Max-Planck Institute)
Bernhard Schölkopf (MPI for Intelligent Systems, Tübingen)

Bernhard Scholkopf received degrees in mathematics (London) and physics (Tubingen), and a doctorate in computer science from the Technical University Berlin. He has researched at AT&T Bell Labs, at GMD FIRST, Berlin, at the Australian National University, Canberra, and at Microsoft Research Cambridge (UK). In 2001, he was appointed scientific member of the Max Planck Society and director at the MPI for Biological Cybernetics; in 2010 he founded the Max Planck Institute for Intelligent Systems. For further information, see www.kyb.tuebingen.mpg.de/~bs.

Georg Martius (Max Planck Institute for Intelligent Systems)
Wieland Brendel (AG Bethge, University of Tübingen)
Michel Besserve (MPI for Intelligent Systems)

More from the Same Authors