Timezone: »

 
Poster
Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
Hippolyt Ritter · Aleksandar Botev · David Barber

Thu Dec 06 02:00 PM -- 04:00 PM (PST) @ Room 517 AB #135
in Thu Poster Session B »

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

Keywords: Online Learning · Supervised Deep Networks · Probabilistic Methods
[ Paper] 

Author Information

Hippolyt Ritter (University College London)
Aleksandar Botev (University College London)
David Barber (University College London)

More from the Same Authors