Timezone: »

 
Poster
Generalized Boosting
Arun Suggala · Bingbin Liu · Pradeep Ravikumar

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #364
in Poster Session 2 »

Boosting is a widely used learning technique in machine learning for solving classification problems. In boosting, one predicts the label of an example using an ensemble of weak classifiers. While boosting has shown tremendous success on many classification problems involving tabular data, it performs poorly on complex classification tasks involving low-level features such as image classification tasks. This drawback stems from the fact that boosting builds an additive model of weak classifiers, each of which has very little predictive power. Often, the resulting additive models are not powerful enough to approximate the complex decision boundaries of real-world classification problems. In this work, we present a general framework for boosting where, similar to traditional boosting, we aim to boost the performance of a weak learner and transform it into a strong learner. However, unlike traditional boosting, our framework allows for more complex forms of aggregation of weak learners. In this work, we specifically focus on one form of aggregation - \emph{function composition}. We show that many popular greedy algorithms for learning deep neural networks (DNNs) can be derived from our framework using function compositions for aggregation. Moreover, we identify the drawbacks of these greedy algorithms and propose new algorithms that fix these issues. Using thorough empirical evaluation, we show that our learning algorithms have superior performance over traditional additive boosting algorithms, as well as existing greedy learning techniques for DNNs. An important feature of our algorithms is that they come with strong theoretical guarantees.

Author Information

Arun Suggala (Carnegie Mellon University)
Bingbin Liu (Carnegie Mellon University)

Bingbin is a PhD student at the Machine Learning Department of Carnegie Mellon University co-advised by Prof. Pradeep Ravikumar and Prof. Andrej Risteski. Her current research focus is the theoretical understanding of self-supervised learning and representation learning, often motivated by findings in vision and language.

Pradeep Ravikumar (Carnegie Mellon University)

More from the Same Authors