Poster
Sub-sampled Newton Methods with Non-uniform Sampling
Peng Xu · Jiyan Yang · Farbod Roosta-Khorasani · Christopher Ré · Michael W Mahoney

Tue Dec 6th 06:00 -- 09:30 PM @ Area 5+6+7+8 #158 #None
in Tuesday Posters »
We consider the problem of finding the minimizer of a convex function $F: \mathbb R^d \rightarrow \mathbb R$ of the form $F(w) \defeq \sum_{i=1}^n f_i(w) + R(w)$ where a low-rank factorization of $\nabla^2 f_i(w)$ is readily available.We consider the regime where $n \gg d$. We propose randomized Newton-type algorithms that exploit \textit{non-uniform} sub-sampling of $\{\nabla^2 f_i(w)\}_{i=1}^{n}$, as well as inexact updates, as means to reduce the computational complexity, and are applicable to a wide range of problems in machine learning. Two non-uniform sampling distributions based on {\it block norm squares} and {\it block partial leverage scores} are considered. Under certain assumptions, we show that our algorithms inherit a linear-quadratic convergence rate in $w$ and achieve a lower computational complexity compared to similar existing methods. In addition, we show that our algorithms exhibit more robustness and better dependence on problem specific quantities, such as the condition number. We numerically demonstrate the advantages of our algorithms on several real datasets.
Keywords: Large Scale Learning and Big Data · Stochastic Methods · Convex Optimization

Author Information

Peng Xu (Stanford University)
Jiyan Yang (Stanford University)
Farbod Roosta-Khorasani (University of California Berkeley)
Chris Ré (Stanford)
Michael W Mahoney (UC Berkeley)

More from the Same Authors