f-GANs in an Information Geometric Nutshell

Nowozin \textit{et al} showed last year how to scale the GANs \textit{principle} to all $f$-divergences. The approach is elegant but falls short of a full description of the supervised game, and says nothing about the key player, the generator: for example, what does the generator actually fit if solving the GAN game means convergence in some space of parameters? How does that hint on the generator's design and compare to the flourishing, essentially experimental literature on the subject? In this paper, we unveil the broad class of densities for which such convergence happens and show tight connections with the three other key GAN parameters: loss, game and model. In particular, we show that current deep architectures are able to factor a potentially very large number of such densities, hence displaying the power of deep architectures and their adequation to the $f$-GAN game. This result holds provided a sufficient condition on \textit{activation functions} is satisfied --- and it turns out to be satisfied by most popular choices. The key to our results is a variational generalization of an old theorem that relates the KL divergence between regular exponential families and divergences between their natural parameters. We complete this picture with additional results and experimental insights on how these results may be used to ground further improvements of GAN architectures.