Timezone: »

 
Poster
Are GANs overkill for NLP?
David Alvarez-Melis · Vikas Garg · Adam Kalai

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #334
in Poster Session 5 »

This work offers a novel theoretical perspective on why, despite numerous attempts, adversarial approaches to generative modeling (e.g., GANs) have not been as successful for certain generation tasks, particularly sequential tasks such as Natural Language Generation, as they have in others, such as Computer Vision. In particular, on sequential data such as text, maximum-likelihood approaches are significantly more utilized than GANs. We show that, while it may seem that maximizing likelihood is inherently different than minimizing distinguishability, this distinction is largely an artifact of the limited representational capacity of the model family, for a wide class of adversarial objectives. We give a theoretical model in which minimizing KL-divergence (i.e., maximizing likelihood) is a more efficient approach to effectively minimizing the same distinguishability criteria that adversarial models seek to optimize. Reductions show that minimizing distinguishability can be seen as simply boosting likelihood for certain families of models including n-gram models and neural networks with a softmax output layer. To achieve a full polynomial-time reduction, a novel next-token distinguishability model is considered. Some preliminary empirical evidence is also provided to substantiate our theoretical analyses.

Keywords: language model · generative modeling · GAN · maximum likelihood estimation · polynomial reduction

Author Information

David Alvarez-Melis (Microsoft)
Vikas Garg (Aalto University/YaiYai Ltd)
Adam Kalai (Microsoft Research New England (-(-_(-_-)_-)-))

More from the Same Authors