Interpretable Comparison of Distributions and Models

Modern machine learning has seen the development of models of increasing complexity for high-dimensional real-world data, such as documents and images. Some of these models are implicit, meaning they generate samples without specifying a probability distribution function (e.g. GANs), and some are explicit, specifying a distribution function – with a potentially quite complex structure which may not admit efficient sampling or normalization. This tutorial will provide modern nonparametric tools for evaluating and benchmarking both implicit and explicit models. For implicit models, samples from the model are compared with real-world samples; for explicit models, a Stein operator is defined to compare the model to data samples without requiring a normalized probability distribution. In both cases, we also consider relative tests to choose the best of several incorrect models. We will emphasize interpretable tests throughout, where the way in which the model differs from the data is conveyed to the user.