Timezone: »

 
Poster
Bayesian dark knowledge
Anoop Korattikara Balan · Vivek Rathod · Kevin Murphy · Max Welling

Thu Dec 10 08:00 AM -- 12:00 PM (PST) @ 210 C #21
in Poster Session »

We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities p(y|x, D), e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time).We describe a method for “distilling” a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [HLA15] and an approach based on variational Bayes [BCKW15]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

[ PDF] 

Author Information

Anoop Korattikara Balan (Google)
Vivek Rathod (Google)
Kevin Murphy (Google)
Max Welling (Microsoft Research AI4Science / University of Amsterdam)

More from the Same Authors