Timezone: »
We show that convex KL-regularized objective functions are obtained from a PAC-Bayes risk bound when using convex loss functions for the stochastic Gibbs classifier that upper-bound the standard zero-one loss used for the weighted majority vote. By restricting ourselves to a class of posteriors, that we call quasi uniform, we propose a simple coordinate descent learning algorithm to minimize the proposed KL-regularized cost function. We show that standard ellp-regularized objective functions currently used, such as ridge regression and ellp-regularized boosting, are obtained from a relaxation of the KL divergence between the quasi uniform posterior and the uniform prior. We present numerical experiments where the proposed learning algorithm generally outperforms ridge regression and AdaBoost.
Author Information
Pascal Germain (Université Laval)
Alexandre Lacasse (Universite Laval)
Francois Laviolette (Université Laval)
Mario Marchand (Université Laval)
Sara Shanian (Laval University)
More from the Same Authors
-
2021 Spotlight: Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis »
Qi CHEN · Changjian Shui · Mario Marchand -
2021 Poster: Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis »
Qi CHEN · Changjian Shui · Mario Marchand -
2021 Poster: Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound »
Valentina Zantedeschi · Paul Viallard · Emilie Morvant · Rémi Emonet · Amaury Habrard · Pascal Germain · Benjamin Guedj -
2020 Poster: Decision trees as partitioning machines to characterize their generalization properties »
Jean-Samuel Leboeuf · Frédéric LeBlanc · Mario Marchand -
2019 Poster: Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks »
Gaël Letarte · Pascal Germain · Benjamin Guedj · Francois Laviolette -
2017 : Neil Lawrence, Francis Bach and François Laviolette »
Neil Lawrence · Francis Bach · Francois Laviolette -
2017 : François Laviolette - A Tutorial on PAC-Bayesian Theory »
Francois Laviolette -
2017 Workshop: (Almost) 50 shades of Bayesian Learning: PAC-Bayesian trends and insights »
Benjamin Guedj · Pascal Germain · Francis Bach -
2017 Poster: Maximum Margin Interval Trees »
Alexandre Drouin · Toby Hocking · Francois Laviolette -
2014 Workshop: Second Workshop on Transfer and Multi-Task Learning: Theory meets Practice »
Urun Dogan · Tatiana Tommasi · Yoshua Bengio · Francesco Orabona · Marius Kloft · Andres Munoz · Gunnar Rätsch · Hal Daumé III · Mehryar Mohri · Xuezhi Wang · Daniel Hernández-lobato · Song Liu · Thomas Unterthiner · Pascal Germain · Vinay P Namboodiri · Michael Goetz · Christopher Berlind · Sigurd Spieckermann · Marta Soare · Yujia Li · Vitaly Kuznetsov · Wenzhao Lian · Daniele Calandriello · Emilie Morvant -
2014 Poster: Multilabel Structured Output Learning with Random Spanning Trees of Max-Margin Markov Networks »
Mario Marchand · Hongyu Su · Emilie Morvant · Juho Rousu · John Shawe-Taylor -
2012 Workshop: Multi-Trade-offs in Machine Learning »
Yevgeny Seldin · Guy Lever · John Shawe-Taylor · Nicolò Cesa-Bianchi · Yacov Crammer · Francois Laviolette · Gabor Lugosi · Peter Bartlett -
2011 Workshop: New Frontiers in Model Order Selection »
Yevgeny Seldin · Yacov Crammer · Nicolò Cesa-Bianchi · Francois Laviolette · John Shawe-Taylor -
2011 Poster: PAC-Bayesian Analysis of Contextual Bandits »
Yevgeny Seldin · Peter Auer · Francois Laviolette · John Shawe-Taylor · Ronald Ortner -
2008 Poster: A Transductive Bound for the Voted Classifier with an Application to Semi-supervised Learning »
Massih R Amini · Nicolas Usunier · Francois Laviolette -
2008 Spotlight: A Transductive Bound for the Voted Classifier with an Application to Semi-supervised Learning »
Massih R Amini · Nicolas Usunier · Francois Laviolette -
2006 Poster: A PAC-Bayes Risk Bound for General Loss Functions »
Pascal Germain · Alexandre Lacasse · Francois Laviolette · Mario Marchand -
2006 Poster: PAC-Bayes Bounds for the Risk of the Majority Vote and the Variance of the Gibbs Classifier »
Alexandre Lacasse · Francois Laviolette · Mario Marchand · Pascal Germain · Nicolas Usunier -
2006 Spotlight: PAC-Bayes Bounds for the Risk of the Majority Vote and the Variance of the Gibbs Classifier »
Alexandre Lacasse · Francois Laviolette · Mario Marchand · Pascal Germain · Nicolas Usunier