Workshop: OPT2020: Optimization for Machine Learning

Contributed Video: Variance Reduction on Adaptive Stochastic Mirror Descent, Wenjie Li

Wenjie Li


We study the application of the variance reduction technique on general adaptive stochastic mirror descent algorithms in nonsmooth nonconvex optimization problems. We prove that variance reduction helps to reduce the gradient complexity of most general stochastic mirror descent algorithms, so it works well with time-varying steps sizes and adaptive optimization algorithms such as AdaGrad. We check the validity of our claims using experiments in deep learning.