Timezone: »
The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments. Our code is available at https://github.com/cunningham-lab/igr.
Author Information
Andres Potapczynski (Columbia University)
Gabriel Loaiza-Ganem (Layer 6 AI)
John Cunningham (University of Columbia)
More from the Same Authors
-
2021 : Entropic Issues in Likelihood-Based OOD Detection »
Anthony Caterini · Gabriel Loaiza-Ganem -
2021 : Entropic Issues in Likelihood-Based OOD Detection »
Anthony Caterini · Gabriel Loaiza-Ganem -
2022 : Relating Regularization and Generalization through the Intrinsic Dimension of Activations »
Bradley Brown · Jordan Juravsky · Anthony Caterini · Gabriel Loaiza-Ganem -
2022 : CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds »
Jesse Cresswell · Brendan Ross · Gabriel Loaiza-Ganem · Humberto Reyes-Gonzalez · Marco Letizia · Anthony Caterini -
2022 : Relating Regularization and Generalization through the Intrinsic Dimension of Activations »
Bradley Brown · Jordan Juravsky · Anthony Caterini · Gabriel Loaiza-Ganem -
2022 : The Union of Manifolds Hypothesis »
Bradley Brown · Anthony Caterini · Brendan Ross · Jesse Cresswell · Gabriel Loaiza-Ganem -
2022 : The Best Deep Ensembles Sacrifice Predictive Diversity »
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · John Cunningham -
2022 : Denoising Deep Generative Models »
Gabriel Loaiza-Ganem · Brendan Ross · Luhuan Wu · John Cunningham · Jesse Cresswell · Anthony Caterini -
2022 : Spotlight 5 - Gabriel Loaiza-Ganem: Denoising Deep Generative Models »
Gabriel Loaiza-Ganem -
2022 Poster: Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome »
Elliott Gordon-Rodriguez · Thomas Quinn · John Cunningham -
2022 Poster: Posterior and Computational Uncertainty in Gaussian Processes »
Jonathan Wenger · Geoff Pleiss · Marvin Pförtner · Philipp Hennig · John Cunningham -
2022 Poster: Deep Ensembles Work, But Are They Necessary? »
Taiga Abe · Estefany Kelly Buchanan · Geoff Pleiss · Richard Zemel · John Cunningham -
2021 Poster: The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective »
Geoff Pleiss · John Cunningham -
2021 Poster: Posterior Collapse and Latent Variable Non-identifiability »
Yixin Wang · David Blei · John Cunningham -
2021 Poster: Rectangular Flows for Manifold Learning »
Anthony Caterini · Gabriel Loaiza-Ganem · Geoff Pleiss · John Cunningham -
2020 Poster: Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking »
Anqi Wu · Estefany Kelly Buchanan · Matthew Whiteway · Michael Schartner · Guido Meijer · Jean-Paul Noel · Erica Rodriguez · Claire Everett · Amy Norovich · Evan Schaffer · Neeli Mishra · C. Daniel Salzman · Dora Angelaki · Andrés Bendesky · The International Brain Laboratory The International Brain Laboratory · John Cunningham · Liam Paninski -
2020 Poster: Recurrent Switching Dynamical Systems Models for Multiple Interacting Neural Populations »
Joshua Glaser · Matthew Whiteway · John Cunningham · Liam Paninski · Scott Linderman -
2019 Poster: Paraphrase Generation with Latent Bag of Words »
Yao Fu · Yansong Feng · John Cunningham -
2019 Poster: BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos »
Eleanor Batty · Matthew Whiteway · Shreya Saxena · Dan Biderman · Taiga Abe · Simon Musall · Winthrop Gillis · Jeffrey Markowitz · Anne Churchland · John Cunningham · Sandeep R Datta · Scott Linderman · Liam Paninski -
2019 Poster: Deep Random Splines for Point Process Intensity Estimation of Neural Population Data »
Gabriel Loaiza-Ganem · Sean Perkins · Karen Schroeder · Mark Churchland · John Cunningham -
2019 Poster: The continuous Bernoulli: fixing a pervasive error in variational autoencoders »
Gabriel Loaiza-Ganem · John Cunningham -
2016 Poster: Linear dynamical neural population models through nonlinear embeddings »
Yuanjun Gao · Evan Archer · Liam Paninski · John Cunningham -
2016 Poster: Automated scalable segmentation of neurons from multispectral images »
Uygar Sümbül · Douglas Roossien · Dawen Cai · Fei Chen · Nicholas Barry · John Cunningham · Edward Boyden · Liam Paninski -
2015 Poster: Bayesian Active Model Selection with an Application to Automated Audiometry »
Jacob Gardner · Gustavo Malkomes · Roman Garnett · Kilian Weinberger · Dennis Barbour · John Cunningham -
2015 Poster: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham -
2015 Spotlight: High-dimensional neural spike train analysis with generalized count linear dynamical systems »
Yuanjun Gao · Lars Busing · Krishna V Shenoy · John Cunningham