Timezone: »
We describe a new model for learning meaningful representations of text documents from an unlabeled collection of documents. This model is inspired by the recently proposed Replicated Softmax, an undirected graphical model of word counts that was shown to learn a better generative model and more meaningful document representations. Specifically, we take inspiration from the conditional mean-field recursive equations of the Replicated Softmax in order to define a neural network architecture that estimates the probability of observing a new word in a given document given the previously observed words. This paradigm also allows us to replace the expensive softmax distribution over words with a hierarchical distribution over paths in a binary tree of words. The end result is a model whose training complexity scales logarithmically with the vocabulary size instead of linearly as in the Replicated Softmax. Our experiments show that our model is competitive both as a generative model of documents and as a document representation learning algorithm.
Author Information
Hugo Larochelle (Google DeepMind)
Stanislas Lauly (NYU)
More from the Same Authors
-
2017 Workshop: Workshop on Meta-Learning »
Roberto Calandra · Frank Hutter · Hugo Larochelle · Sergey Levine -
2014 Session: Oral Session 3 »
Hugo Larochelle -
2014 Poster: An Autoencoder Approach to Learning Bilingual Word Representations »
Sarath Chandar · Stanislas Lauly · Hugo Larochelle · Mitesh Khapra · Balaraman Ravindran · Vikas C Raykar · Amrita Saha -
2013 Workshop: Deep Learning »
Yoshua Bengio · Hugo Larochelle · Russ Salakhutdinov · Tomas Mikolov · Matthew D Zeiler · David Mcallester · Nando de Freitas · Josh Tenenbaum · Jian Zhou · Volodymyr Mnih -
2013 Session: Spotlight Session 10 »
Hugo Larochelle -
2013 Session: Spotlight Session 9 »
Hugo Larochelle -
2013 Session: Spotlight Session 8 »
Hugo Larochelle -
2013 Session: Spotlight Session 7 »
Hugo Larochelle -
2013 Session: Spotlight Session 6 »
Hugo Larochelle -
2013 Session: Spotlight Session 5 »
Hugo Larochelle -
2013 Poster: RNADE: The real-valued neural autoregressive density-estimator »
Benigno Uria · Iain Murray · Hugo Larochelle -
2013 Session: Spotlight Session 4 »
Hugo Larochelle -
2013 Session: Spotlight Session 3 »
Hugo Larochelle -
2013 Session: Spotlight Session 2 »
Hugo Larochelle -
2013 Session: Spotlight Session 1 »
Hugo Larochelle -
2012 Poster: Practical Bayesian Optimization of Machine Learning Algorithms »
Jasper Snoek · Hugo Larochelle · Ryan Adams -
2010 Oral: Learning to combine foveal glimpses with a third-order Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton -
2010 Poster: Learning to combine foveal glimpses with a third-order Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton -
2006 Poster: Greedy Layer-Wise Training of Deep Networks »
Yoshua Bengio · Pascal Lamblin · Dan Popovici · Hugo Larochelle -
2006 Talk: Greedy Layer-Wise Training of Deep Networks »
Yoshua Bengio · Pascal Lamblin · Dan Popovici · Hugo Larochelle