Timezone: »
Complex probabilistic models of unlabeled data can be created by combining simpler models. Mixture models are obtained by averaging the densities of simpler models and "products of experts" are obtained by multiplying the densities together and renormalizing. A far more powerful type of combination is to form a "composition of experts" by treating the values of the latent variables of one model as the data for learning the next model. The first half of the tutorial will show how deep belief nets  directed generative models with many layers of hidden variables  can be learned one layer at a time by composing simple, undirected, product of expert models that only have one hidden layer. It will also explain why composing directed models does not work.
Deep belief nets are trained as generative models on large, unlabeled datasets, but once multiple layers of features have been created by unsupervised learning, they can be finetuned to give excellent discrimination on small, labeled datasets. The second half of the tutorial will describe applications of deep belief nets to several tasks including object recognition, nonlinear dimensionality reduction, document retrieval, and the interpretation of medical images. It will also show how the learning procedure for deep belief nets can be extended to highdimensional time series and hierarchies of Conditional Random Fields.
Author Information
Geoffrey E Hinton (Google & University of Toronto)
Geoffrey Hinton received his PhD in Artificial Intelligence from Edinburgh in 1978 and spent five years as a faculty member at CarnegieMellon where he pioneered backpropagation, Boltzmann machines and distributed representations of words. In 1987 he became a fellow of the Canadian Institute for Advanced Research and moved to the University of Toronto. In 1998 he founded the Gatsby Computational Neuroscience Unit at University College London, returning to the University of Toronto in 2001. His group at the University of Toronto then used deep learning to change the way speech recognition and object recognition are done. He currently splits his time between the University of Toronto and Google. In 2010 he received the NSERC Herzberg Gold Medal, Canada's top award in Science and Engineering.
More from the Same Authors

2019 Poster: Lookahead Optimizer: k steps forward, 1 step back »
Michael Zhang · James Lucas · Jimmy Ba · Geoffrey E Hinton 
2019 Poster: Stacked Capsule Autoencoders »
Adam Kosiorek · Sara Sabour · Yee Whye Teh · Geoffrey E Hinton 
2019 Poster: When does label smoothing help? »
Rafael Müller · Simon Kornblith · Geoffrey E Hinton 
2019 Spotlight: When does label smoothing help? »
Rafael Müller · Simon Kornblith · Geoffrey E Hinton 
2018 Poster: Assessing the Scalability of BiologicallyMotivated Deep Learning Algorithms and Architectures »
Sergey Bartunov · Adam Santoro · Blake Richards · Luke Marris · Geoffrey E Hinton · Timothy Lillicrap 
2017 Poster: Dynamic Routing Between Capsules »
Sara Sabour · Nicholas Frosst · Geoffrey E Hinton 
2017 Spotlight: Dynamic Routing Between Capsules »
Sara Sabour · Nicholas Frosst · Geoffrey E Hinton 
2016 Poster: Attend, Infer, Repeat: Fast Scene Understanding with Generative Models »
S. M. Ali Eslami · Nicolas Heess · Theophane Weber · Yuval Tassa · David Szepesvari · koray kavukcuoglu · Geoffrey E Hinton 
2016 Poster: Using Fast Weights to Attend to the Recent Past »
Jimmy Ba · Geoffrey E Hinton · Volodymyr Mnih · Joel Leibo · Catalin Ionescu 
2016 Oral: Using Fast Weights to Attend to the Recent Past »
Jimmy Ba · Geoffrey E Hinton · Volodymyr Mnih · Joel Leibo · Catalin Ionescu 
2015 Poster: Grammar as a Foreign Language »
Oriol Vinyals · Łukasz Kaiser · Terry Koo · Slav Petrov · Ilya Sutskever · Geoffrey Hinton 
2015 Tutorial: Deep Learning »
Geoffrey E Hinton · Yoshua Bengio · Yann LeCun 
2014 Workshop: Deep Learning and Representation Learning »
Andrew Y Ng · Yoshua Bengio · Adam Coates · Roland Memisevic · Sharanyan Chetlur · Geoffrey E Hinton · Shamim Nemati · Bryan Catanzaro · Surya Ganguli · Herbert Jaeger · Phil Blunsom · Leon Bottou · Volodymyr Mnih · ChenYu Lee · Rich M Schwartz 
2012 Poster: ImageNet Classification with Deep Convolutional Neural Networks »
Alex Krizhevsky · Ilya Sutskever · Geoffrey E Hinton 
2012 Invited Talk: Dropout: A simple and effective way to improve neural networks »
Geoffrey E Hinton · George Dahl 
2012 Poster: A Better Way to PreTrain Deep Boltzmann Machines »
Russ Salakhutdinov · Geoffrey E Hinton 
2012 Spotlight: ImageNet Classification with Deep Convolutional Neural Networks »
Alex Krizhevsky · Ilya Sutskever · Geoffrey E Hinton 
2010 Workshop: Deep Learning and Unsupervised Feature Learning »
Honglak Lee · Marc'Aurelio Ranzato · Yoshua Bengio · Geoffrey E Hinton · Yann LeCun · Andrew Y Ng 
2010 Talk: A Probabilistic Approach to Data Visualization »
Geoffrey E Hinton 
2010 Oral: Learning to combine foveal glimpses with a thirdorder Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton 
2010 Poster: Learning to combine foveal glimpses with a thirdorder Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton 
2010 Poster: Generating more realistic images using gated MRF's »
Marc'Aurelio Ranzato · Volodymyr Mnih · Geoffrey E Hinton 
2010 Poster: Phone Recognition with the MeanCovariance Restricted Boltzmann Machine »
George Dahl · Marc'Aurelio Ranzato · Abdelrahman Mohamed · Geoffrey E Hinton 
2010 Poster: Gated Softmax Classification »
Roland Memisevic · Christopher Zach · Geoffrey E Hinton · Marc Pollefeys 
2009 Workshop: Deep Learning for Speech Recognition and Related Applications »
Li Deng · Dong Yu · Geoffrey E Hinton 
2009 Poster: Replicated Softmax: an Undirected Topic Model »
Russ Salakhutdinov · Geoffrey E Hinton 
2009 Poster: 3D Object Recognition with Deep Belief Nets »
Vinod Nair · Geoffrey E Hinton 
2009 Spotlight: 3D Object Recognition with Deep Belief Nets »
Vinod Nair · Geoffrey E Hinton 
2009 Invited Talk: Deep Learning with Multiplicative Interactions »
Geoffrey E Hinton 
2009 Poster: Zeroshot Learning with Semantic Output Codes »
Mark M Palatucci · Dean Pomerleau · Geoffrey E Hinton · Tom Mitchell 
2008 Poster: Using matrices to model symbolic relationship »
Ilya Sutskever · Geoffrey E Hinton 
2008 Demonstration: Visualizing NIPS Cooperations using Multiple Maps tSNE »
Laurens van der Maaten · Geoffrey E Hinton 
2008 Spotlight: Using matrices to model symbolic relationship »
Ilya Sutskever · Geoffrey E Hinton 
2008 Poster: The Recurrent Temporal Restricted Boltzmann Machine »
Ilya Sutskever · Geoffrey E Hinton · Graham W Taylor 
2008 Poster: A Scalable Hierarchical Distributed Language Model »
Andriy Mnih · Geoffrey E Hinton 
2008 Poster: Implicit Mixtures of Restricted Boltzmann Machines »
Vinod Nair · Geoffrey E Hinton 
2008 Poster: Competing RBM density models for classification of fMRI images »
Tanya Schmah · Geoffrey E Hinton · Richard Zemel 
2007 Poster: Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes »
Russ Salakhutdinov · Geoffrey E Hinton 
2007 Poster: Modeling image patches with a directed hierarchy of Markov random fields »
Simon Osindero · Geoffrey E Hinton 
2006 Poster: Modeling Human Motion Using Binary Latent Variables »
Graham W Taylor · Geoffrey E Hinton · Sam T Roweis 
2006 Spotlight: Modeling Human Motion Using Binary Latent Variables »
Graham W Taylor · Geoffrey E Hinton · Sam T Roweis