Poster
Practical and Consistent Estimation of f-Divergences
Paul Rubenstein · Olivier Bousquet · Josip Djolonga · Carlos Riquelme · Ilya Tolstikhin

Tue Dec 10th 10:45 AM -- 12:45 PM @ East Exhibition Hall B + C #51
in Algorithms -- Density Estimation »

The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning. Most works study this problem under very weak assumptions, in which case it is provably hard. We consider the case of stronger structural assumptions that are commonly satisfied in modern machine learning, including representation learning and generative modelling with autoencoder architectures. Under these assumptions we propose and study an estimator that can be easily implemented, works well in high dimensions, and enjoys faster rates of convergence. We verify the behavior of our estimator empirically in both synthetic and real-data experiments, and discuss its direct implications for total correlation, entropy, and mutual information estimation.

Keywords: Theory -> Frequentist Statistics · Deep Learning -> Deep Autoencoders; Probabilistic Methods -> Latent Variable Models; Theory -> Learning Theory

Author Information

Paul Rubenstein (MPI for IS)
Olivier Bousquet (Google Brain (Zurich))
Josip Djolonga (Google Research, Brain Team)
Carlos Riquelme (Google Brain)
Ilya Tolstikhin (MPI for Intelligent Systems)

More from the Same Authors