Confluence between Kernel Methods and Graphical Models

Website: https://sites.google.com/site/kernelgraphical/

Kernel methods and graphical models are two important families of techniques for machine learning. Our community has witnessed many major but separate advances in the theory and applications of both subfields. For kernel methods, the advances include kernels on structured data, Hilbert-space embeddings of distributions, and applications of kernel methods to multiple kernel learning, transfer learning, and multi-task learning. For graphical models, the advances include variational inference, nonparametric Bayes techniques, and applications of graphical models to topic modeling, computational biology and social network problems.

This workshop addresses two main research questions: first, how may kernel methods be used to address difficult learning problems for graphical models, such as inference for multi-modal continuous distributions on many variables, and dealing with non-conjugate priors? And second, how might kernel methods be advanced by bringing in concepts from graphical models, for instance by incorporating sophisticated conditional independence structures, latent variables, and prior information?


Kernel algorithms have traditionally had the advantage of being solved via convex optimization or eigenproblems, and having strong statistical guarantees on convergence. The graphical model literature has focused on modelling complex dependence structures in a flexible way, although approximations may be required to make inference tractable. Can we develop a new set of methods which blend these strengths?

There has recently been a number of publications combining kernel and graphical model techniques, including kernel hidden Markov models [SBSGS08], kernel belief propagation [SGBLG11], kernel Bayes rule [FSG11], kernel topic models [HSHG12], kernel variational inference [GHB12], kernel herding as Bayesian quadrature [HD12], kernel beta processes [RWDC08], a connection between kernel k-means and Bayesian nonparametrics [KJ12] and kernel determinantal point processes for recommendations [KT12]. Each of these results deals with different inference tasks, and makes use of a range of RKHS properties. We propose this workshop so as to "connect the dots" and develop a unified toolkit to address a broad range of learning problems, to the mutual benefit of researchers in kernels and graphical models. The goals of the workshop are thus twofold: first, to provide an accessible review and synthesis of recent results combining graphical models and kernels. Second, to provide a discussion forum for open problems and technical challenges.

Selected bibliography:
%%%%%%%%%%%%%

[SBSGS08] Song, L. and Boots, B. and Siddiqi, S. and Gordon, G. and Smola, A., Hilbert space embeddings of hidden Markov models, ICML'10.

[SGBLG11] Song, L. and Gretton, A. and Bickson, D. and Low, Y. and Guestrin, C., Kernel belief propagation, AISTATS'11.

[FSG11] Fukumizu, K. and Song, L. and Gretton, A., Kernel Bayes rules, NIPS'11.

[HSHG12] Hennig, P. and Stern, D. and Herbrich, R. and Graepel, T., Kernel topic models, AISTATS'12.

[HD12] Huszar, F. and Duvenaud, D., Optimally-weighted herding is Bayesian quadrature, UAI'12.

[RWDC08] Ren, L. and Wang, Y. and Dunson, D.B. and Carin, L., Kernel beta processes, NIPS'11.

[GHB12] Gershman, S. and Hoffman, M. and Blei, D., Nonparametric variational inference, ICML'12.

[KJ12] Kulis, B. and Jordan, M., Revisiting k-means: new algorithms via Bayesian nonparametrics, ICML'12.

[KT12] Kulesza, A. and Taskar, B., Determinantal point processes for machine learning, arXiv:1207.6083