Timezone: »

 
Poster
StopWasting My Gradients: Practical SVRG
Reza Babanezhad Harikandeh · Mohamed Osama Ahmed · Alim Virani · Mark Schmidt · Jakub Konečný · Scott Sallinen

Mon Dec 07 04:00 PM -- 08:59 PM (PST) @ 210 C #79
in Poster Session »

We present and analyze several strategies for improving the performance ofstochastic variance-reduced gradient (SVRG) methods. We first show that theconvergence rate of these methods can be preserved under a decreasing sequenceof errors in the control variate, and use this to derive variants of SVRG that usegrowing-batch strategies to reduce the number of gradient calculations requiredin the early iterations. We further (i) show how to exploit support vectors to reducethe number of gradient computations in the later iterations, (ii) prove that thecommonly–used regularized SVRG iteration is justified and improves the convergencerate, (iii) consider alternate mini-batch selection strategies, and (iv) considerthe generalization error of the method.

[ PDF] 

Author Information

Reza Babanezhad Harikandeh (UBC)
Mohamed Osama Ahmed
Alim Virani
Mark Schmidt (University of British Columbia)
Jakub Konečný
Scott Sallinen (UBC)

More from the Same Authors