Timezone: »
Poster
Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation
Indraneel Mukherjee · David Blei
Hierarchical probabilistic modeling of discrete data has emerged as a powerful tool for text analysis. Posterior inference in such models is intractable, and practitioners rely on approximate posterior inference methods such as variational inference or Gibbs sampling. There has been much research in designing better approximations, but there is yet little theoretical understanding of which of the available techniques are appropriate, and in which data analysis settings. In this paper we provide the beginnings of such understanding. We analyze the improvement that the recently proposed collapsed variational inference (CVB) provides over mean field variational inference (VB) in latent Dirichlet allocation. We prove that the difference in the tightness of the bound on the likelihood of a document decreases as $O(k-1) + \log m /m$, where $k$ is the number of topics in the model and $m$ is the number of words in a document. As a consequence, the advantage of CVB over VB is lost for long documents but increases with the number of topics. We demonstrate empirically that the theory holds, using simulated text data and two text corpora. We provide practical guidelines for choosing an approximation.
Author Information
Indraneel Mukherjee (Princeton University)
David Blei (Columbia University)
Related Events (a corresponding poster, oral, or spotlight)
-
2008 Spotlight: Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation »
Tue. Dec 9th 04:31 -- 04:32 AM Room
More from the Same Authors
-
2014 Workshop: Advances in Variational Inference »
David Blei · Shakir Mohamed · Michael Jordan · Charles Blundell · Tamara Broderick · Matthew D. Hoffman -
2014 Poster: A Filtering Approach to Stochastic Variational Inference »
Neil Houlsby · David Blei -
2014 Poster: Smoothed Gradients for Stochastic Variational Inference »
Stephan Mandt · David Blei -
2014 Poster: Content-based recommendations with Poisson factorization »
Prem Gopalan · Laurent Charlin · David Blei -
2013 Workshop: Topic Models: Computation, Application, and Evaluation »
David Mimno · Amr Ahmed · Jordan Boyd-Graber · Ankur Moitra · Hanna Wallach · Alexander Smola · David Blei · Anima Anandkumar -
2013 Workshop: Probabilistic Models for Big Data »
Neil D Lawrence · Joaquin QuiƱonero-Candela · Tianshi Gao · James Hensman · Zoubin Ghahramani · Max Welling · David Blei · Ralf Herbrich -
2013 Poster: Efficient Online Inference for Bayesian Nonparametric Relational Models »
Dae Il Kim · Prem Gopalan · David Blei · Erik Sudderth -
2013 Poster: Modeling Overlapping Communities with Node Popularities »
Prem Gopalan · Chong Wang · David Blei -
2012 Poster: Truncation-free Online Variational Inference for Bayesian Nonparametric Models »
Chong Wang · David Blei -
2012 Poster: Scalable Inference of Overlapping Communities »
Prem Gopalan · David Mimno · Sean Gerrish · Michael Freedman · David Blei -
2012 Spotlight: Scalable Inference of Overlapping Communities »
Prem Gopalan · David Mimno · Sean Gerrish · Michael Freedman · David Blei -
2012 Poster: How They Vote: Issue-Adjusted Models of Legislative Behavior »
Sean Gerrish · David Blei -
2011 Poster: Spatial distance dependent Chinese Restaurant Process for image segmentation »
Soumya Ghosh · Andrei B Ungureanu · Erik Sudderth · David Blei -
2010 Session: Oral Session 18 »
David Blei -
2010 Spotlight: Online Learning for Latent Dirichlet Allocation »
Matthew D. Hoffman · David Blei · Francis Bach -
2010 Poster: Online Learning for Latent Dirichlet Allocation »
Matthew D. Hoffman · David Blei · Francis Bach -
2010 Oral: A Theory of Multiclass Boosting »
Indraneel Mukherjee · Robert E Schapire -
2010 Poster: Nonparametric Density Estimation for Stochastic Optimization with an Observable State Variable »
Lauren A Hannah · Warren B Powell · David Blei -
2010 Poster: A Theory of Multiclass Boosting »
Indraneel Mukherjee · Robert E Schapire -
2009 Workshop: Applications for Topic Models: Text and Beyond »
David Blei · Jordan Boyd-Graber · Jonathan Chang · Katherine Heller · Hanna Wallach -
2009 Poster: Reading Tea Leaves: How Humans Interpret Topic Models »
Jonathan Chang · Jordan Boyd-Graber · Sean Gerrish · Chong Wang · David Blei -
2009 Oral: Reading Tea Leaves: How Humans Interpret Topic Models »
Jonathan Chang · Jordan Boyd-Graber · Sean Gerrish · Chong Wang · David Blei -
2009 Poster: Decoupling Sparsity and Smoothness in the Discrete Hierarchical Dirichlet Process »
Chong Wang · David Blei -
2009 Spotlight: Decoupling Sparsity and Smoothness in the Discrete Hierarchical Dirichlet Process »
Chong Wang · David Blei -
2009 Poster: Variational Inference for the Nested Chinese Restaurant Process »
Chong Wang · David Blei -
2009 Poster: A Bayesian Analysis of Dynamics in Free Recall »
Richard Socher · Samuel J Gershman · Adler Perotte · Per Sederberg · David Blei · Kenneth Norman -
2008 Workshop: Analyzing Graphs: Theory and Applications »
Edo M Airoldi · David Blei · Jake M Hofman · Tony Jebara · Eric Xing -
2008 Poster: Mixed Membership Stochastic Blockmodels »
Edo M Airoldi · David Blei · Stephen E Fienberg · Eric Xing -
2008 Spotlight: Mixed Membership Stochastic Blockmodels »
Edo M Airoldi · David Blei · Stephen E Fienberg · Eric Xing -
2008 Poster: Syntactic Topic Models »
Jordan Boyd-Graber · David Blei -
2008 Spotlight: Syntactic Topic Models »
Jordan Boyd-Graber · David Blei -
2007 Poster: Supervised Topic Models »
David Blei · Jon McAuliffe