Learning features to compare distributions

An important component of GANs is the discriminator, which tells apart samples from the generator and samples from a reference set. Discriminators implement empirical approximations to various divergence measures between probability densities (originally Jensen-Shannon, and more recently other f-divergences and integral probability metrics). If we think about this problem in the setting of hypothesis testing, a good discriminator can tell generator samples from reference samples with high probability: in other words, it maximizes the test power. A reasonable goal then becomes to learn a discriminator to directly maxmize test power (we will briefly look at relations between test power and classifier performance).

I will demonstrate ways of training a discriminator with maximum test power using two divergence measures: the maximum mean discrepancy (MMD), and differences of learned smooth features (the ME test, NIPS 2016). In both cases, the key point is that variance matters: it is not enough to have a large empirical divergence; we also need to have high confidence in the value of our divergence. Using an optimized MMD discriminator, we can detect subtle differences in the distribution of GAN outputs and real hand-written digits which humans are unable to find (for instance, small imbalances in the proportions of certain digits, or minor distortions that are implausible in normal handwriting).