Timezone: »

Continuous-time Models for Stochastic Optimization Algorithms
Antonio Orvieto · Aurelien Lucchi

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #208

We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these continuous-time models, together with simple Lyapunov analysis as well as tools from stochastic calculus, in order to derive convergence bounds for various types of non-convex functions. Guided by such analysis, we show that the same Lyapunov arguments hold in discrete-time, leading to matching rates. In addition, we use these models and Ito calculus to infer novel insights on the dynamics of SGD, proving that a decreasing learning rate acts as time warping or, equivalently, as landscape stretching.

Author Information

Antonio Orvieto (ETH Zurich)

Phd Student at ETH Zurich. I’m interested in the design and the analysis of adaptive stochastic momentum optimization algorithms for non-convex machine learning problems. Publications: 2Neurips, 1ICML, 1AISTATS, 1UAI. 1 Patent on a learning algorithm for an impact screwdriver. Besides my PhD research, I am involved in several computational systems biology projects at ETH Zurich, such as SignalX. I also work as a biomedical data analyst (genetic rare diseases research) at the University of Padua.

Aurelien Lucchi (ETH Zurich)

More from the Same Authors