Workshop
Bayesian Deep Learning
Yarin Gal · José Miguel Hernández-Lobato · Christos Louizos · Andrew Wilson · Andrew Wilson · Durk Kingma · Zoubin Ghahramani · Kevin Murphy · Max Welling

Sat Dec 9th 08:00 AM -- 06:30 PM @ Hall C
Event URL: http://bayesiandeeplearning.org/ »

While deep learning has been revolutionary for machine learning, most modern deep learning models cannot represent their uncertainty nor take advantage of the well studied tools of probability theory. This has started to change following recent developments of tools and techniques combining Bayesian approaches with deep learning. The intersection of the two fields has received great interest from the community over the past few years, with the introduction of new deep learning models that take advantage of Bayesian techniques, as well as Bayesian models that incorporate deep learning elements [1-11]. In fact, the use of Bayesian techniques in deep learning can be traced back to the 1990s’, in seminal works by Radford Neal [12], David MacKay [13], and Dayan et al. [14]. These gave us tools to reason about deep models’ confidence, and achieved state-of-the-art performance on many tasks. However earlier tools did not adapt when new needs arose (such as scalability to big data), and were consequently forgotten. Such ideas are now being revisited in light of new advances in the field, yielding many exciting new results.

Extending on last year’s workshop’s success, this workshop will again study the advantages and disadvantages of such ideas, and will be a platform to host the recent flourish of ideas using Bayesian approaches in deep learning and using deep learning tools in Bayesian modelling. The program includes a mix of invited talks, contributed talks, and contributed posters. It will be composed of five main themes: deep generative models, variational inference using neural network recognition models, practical approximate inference techniques in Bayesian neural networks, applications of Bayesian neural networks, and information theory in deep learning. Future directions for the field will be debated in a panel discussion.

Topics:
Probabilistic deep models for classification and regression (such as extensions and application of Bayesian neural networks),
Generative deep models (such as variational autoencoders),
Incorporating explicit prior knowledge in deep learning (such as posterior regularization with logic rules),
Approximate inference for Bayesian deep learning (such as variational Bayes / expectation propagation / etc. in Bayesian neural networks),
Scalable MCMC inference in Bayesian deep models,
Deep recognition models for variational inference (amortized inference),
Model uncertainty in deep learning,
Bayesian deep reinforcement learning,
Deep learning with small data,
Deep learning in Bayesian modelling,
Probabilistic semi-supervised learning techniques,
Active learning and Bayesian optimization for experimental design,
Applying non-parametric methods, one-shot learning, and Bayesian deep learning in general,
Implicit inference,
Kernel methods in Bayesian deep learning.


References:
[1] - Kingma, DP and Welling, M, ‘’Auto-encoding variational bayes’’, 2013.
[2] - Rezende, D, Mohamed, S, and Wierstra, D, ‘’Stochastic backpropagation and approximate inference in deep generative models’’, 2014.
[3] - Blundell, C, Cornebise, J, Kavukcuoglu, K, and Wierstra, D, ‘’Weight uncertainty in neural network’’, 2015.
[4] - Hernandez-Lobato, JM and Adams, R, ’’Probabilistic backpropagation for scalable learning of Bayesian neural networks’’, 2015.
[5] - Gal, Y and Ghahramani, Z, ‘’Dropout as a Bayesian approximation: Representing model uncertainty in deep learning’’, 2015.
[6] - Gal, Y and Ghahramani, G, ‘’Bayesian convolutional neural networks with Bernoulli approximate variational inference’’, 2015.
[7] - Kingma, D, Salimans, T, and Welling, M. ‘’Variational dropout and the local reparameterization trick’’, 2015.
[8] - Balan, AK, Rathod, V, Murphy, KP, and Welling, M, ‘’Bayesian dark knowledge’’, 2015.
[9] - Louizos, C and Welling, M, “Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors”, 2016.
[10] - Lawrence, ND and Quinonero-Candela, J, “Local distance preservation in the GP-LVM through back constraints”, 2006.
[11] - Tran, D, Ranganath, R, and Blei, DM, “Variational Gaussian Process”, 2015.
[12] - Neal, R, ‘’Bayesian Learning for Neural Networks’’, 1996.
[13] - MacKay, D, ‘’A practical Bayesian framework for backpropagation networks‘’, 1992.
[14] - Dayan, P, Hinton, G, Neal, R, and Zemel, S, ‘’The Helmholtz machine’’, 1995.
[15] - Wilson, AG, Hu, Z, Salakhutdinov, R, and Xing, EP, “Deep Kernel Learning”, 2016.
[16] - Saatchi, Y and Wilson, AG, “Bayesian GAN”, 2017.
[17] - MacKay, D.J.C. “Bayesian Methods for Adaptive Models”, PhD thesis, 1992.

08:05 AM Deep Probabilistic Programming (Invited talk) Dustin Tran
08:30 AM TBD 1 (Contributed talk)
08:45 AM Automatic Model Selection in BNNs with Horseshoe Priors (Invited talk) Finale Doshi-Velez
09:10 AM Deep Bayes for Distributed Learning, Uncertainty Quantification and Compression (Special talk) Max Welling
09:40 AM Poster spotlights (Spotlights)
09:55 AM Discussion over coffee and poster session 1 (Break)
10:55 AM Stochastic Gradient Descent as Approximate Bayesian Inference (Invited talk)
11:20 AM TBD 2 (Contributed talk)
11:35 AM TBD 2.5 (Invited talk) Nal Kalchbrenner
12:00 PM Lunch (Break)
01:35 PM Deep Kernel Learning (Invited talk) Ruslan Salakhutdinov
02:00 PM TBD 3 (Contributed talk)
02:15 PM Bayes by Backprop (Invited talk) Meire Fortunato
02:40 PM Discussion over coffee and poster session 2 (Break)
03:35 PM How do the Deep Learning layers converge to the Information Bottleneck limit by Stochastic Gradient Descent? (Invited talk) Naftali Tishby
04:00 PM Panel Session <span> <a href="#"></a> </span> Neil Lawrence, Finale Doshi-Velez, Zoubin Ghahramani, Yann LeCun, Max Welling, Yee Whye Teh, Ole Winther
05:00 PM Poster session (Break)
Xun Zheng, Tim G. J. Rudner, Christopher Tegho, Patrick McClure, Yunhao Tang, ASH D'CRUZ, Juan Camilo Gamboa Higuera, Chandra Seelamantula, Jhosimar Arias Figueroa, Andy Berlin, Maxime Voisin, Alexander Amini, Thang DOAN, Hengyuan Hu, Alex Botev, Niko Sünderhauf, CHI ZHANG, John Lambert

Author Information

Yarin Gal (University of Oxford)
José Miguel Hernández-Lobato (University of Cambridge)
Christos Louizos (University of Amsterdam)
Andrew Wilson (Cornell University)
Andrew Wilson (Cornell University)

I am a professor of machine learning at New York University.

Durk Kingma (Google)
Zoubin Ghahramani (Uber and University of Cambridge)

Zoubin Ghahramani is Professor of Information Engineering at the University of Cambridge, where he leads the Machine Learning Group. He studied computer science and cognitive science at the University of Pennsylvania, obtained his PhD from MIT in 1995, and was a postdoctoral fellow at the University of Toronto. His academic career includes concurrent appointments as one of the founding members of the Gatsby Computational Neuroscience Unit in London, and as a faculty member of CMU's Machine Learning Department for over 10 years. His current research interests include statistical machine learning, Bayesian nonparametrics, scalable inference, probabilistic programming, and building an automatic statistician. He has held a number of leadership roles as programme and general chair of the leading international conferences in machine learning including: AISTATS (2005), ICML (2007, 2011), and NIPS (2013, 2014). In 2015 he was elected a Fellow of the Royal Society.

Kevin Murphy (Google)
Max Welling (University of Amsterdam / Qualcomm AI Research)

More from the Same Authors