Talk
in
Workshop: OPT2020: Optimization for Machine Learning

Contributed Video: Variance Reduction on Adaptive Stochastic Mirror Descent, Wenjie Li

Wenjie Li

2020 Talk
in
Workshop: OPT2020: Optimization for Machine Learning

Project Page

Abstract

We study the application of the variance reduction technique on general adaptive stochastic mirror descent algorithms in nonsmooth nonconvex optimization problems. We prove that variance reduction helps to reduce the gradient complexity of most general stochastic mirror descent algorithms, so it works well with time-varying steps sizes and adaptive optimization algorithms such as AdaGrad. We check the validity of our claims using experiments in deep learning.

Video

Chat is not available.