New Optimization Methods for Very Large Scale SVMs

We revisit the classical support vector machine (SVM) from a duality perspective and propose new optimization methods that yield both theoretical and practical advantages. For binary L2 SVM, we utilize proximal gradient method where in each iteration we derive a novel dual formulation to deal with equation constraint. The Lagrangian multiplier can be obtained fast due to its monotone property. For the multiclass setting, we introduce a scalable optimization algorithm based on the Frank-Wolfe method applied to a structured dual of the multiclass SVM problem. Unlike reformulating into a classical quadratic programming (QP), our method is scalable to problems with many samples and classes. Empirical experiments demonstrate that our methods is significantly superior in terms of way less time consumption on large-scale datasets.