Poster

Optimal Rates for Random Order Online Optimization

Uri Sherman · Tomer Koren · Yishay Mansour

Keywords: [ Optimization ] [ Online Learning ]

[ Abstract ]
Thu 9 Dec 8:30 a.m. PST — 10 a.m. PST
 
Oral presentation: Oral Session 4: Vision Applications and Optimization
Fri 10 Dec midnight PST — 1 a.m. PST

Abstract:

We study online convex optimization in the random order model, recently proposed by Garber et al. (2020), where the loss functions may be chosen by an adversary, but are then presented to the online algorithm in a uniformly random order. Focusing on the scenario where the cumulative loss function is (strongly) convex, yet individual loss functions are smooth but might be non-convex, we give algorithms that achieve the optimal bounds and significantly outperform the results of Garber et al. (2020), completely removing the dimension dependence and improve their scaling with respect to the strong convexity parameter. Our analysis relies on novel connections between algorithmic stability and generalization for sampling without-replacement analogous to those studied in the with-replacement i.i.d. setting, as well as on a refined average stability analysis of stochastic gradient descent.

Chat is not available.