Skip to yearly menu bar Skip to main content


Sinkhorn Natural Gradient for Generative Models

Zebang Shen · Zhenfu Wang · Alejandro Ribeiro · Hamed Hassani

Poster Session 6 #1852

Keywords: [ Probabilistic Methods ] [ Bayesian Nonparametrics ] [ Information Theory ] [ Algorithms -> Online Learning; Theory -> Computational Complexity; Theory ]

We consider the problem of minimizing a functional over a parametric family of probability measures, where the parameterization is characterized via a push-forward structure. 
An important application of this problem is in training generative adversarial networks.  
In this regard, we propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence.
We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically with respect to the desired accuracy. This is in sharp contrast to  existing natural gradient methods that can only be carried out approximately.
Moreover, in practical applications when only Monte-Carlo type integration is available, we design an empirical estimator for SIM and provide the stability analysis.
In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.

Chat is not available.