This is the public, feature-limited version of the conference webpage. After Registration and login please visit the full version.

Workshop: OPT2020: Optimization for Machine Learning

Courtney Paquette, Mark Schmidt, Sebastian Stich, Quanquan Gu, Martin Takac

2020-12-11T03:15:00-08:00 - 2020-12-11T16:30:00-08:00
Abstract: Optimization lies at the heart of many machine learning algorithms and enjoys great interest in our community. Indeed, this intimate relation of optimization with ML is the key motivation for the OPT series of workshops.

Looking back over the past decade, a strong trend is apparent: The intersection of OPT and ML has grown to the point that now cutting-edge advances in optimization often arise from the ML community. The distinctive feature of optimization within ML is its departure from textbook approaches, in particular, its focus on a different set of goals driven by "big-data, nonconvexity, and high-dimensions," where both theory and implementation are crucial.

We wish to use OPT 2020 as a platform to foster discussion, discovery, and dissemination of the state-of-the-art in optimization as relevant to machine learning. And well beyond that: as a platform to identify new directions and challenges that will drive future research, and continue to build the OPT+ML joint research community.

**Invited Speakers**
Volkan Cevher (EPFL)
Michael Friedlander (UBC)
Donald Goldfarb (Columbia)
Andreas Krause (ETH, Zurich)
Suvrit Sra (MIT)
Rachel Ward (UT Austin)
Ashia Wilson (MSR)
Tong Zhang (HKUST)

Please join us in for all breaks and poster sessions (for link see any abstract for a break or poster session, opens on December 11).


To ask questions please use rocketchat, available only upon registration and login.


2020-12-11T03:15:00-08:00 - 2020-12-11T03:50:00-08:00
Welcome event (
Quanquan Gu, Courtney Paquette, Mark Schmidt, Sebastian Stich, Martin Takac
2020-12-11T03:50:00-08:00 - 2020-12-11T04:00:00-08:00
Welcome remarks to Session 1
Sebastian Stich
2020-12-11T04:00:00-08:00 - 2020-12-11T04:20:00-08:00
Invited speaker: The Convexity of Learning Infinite-width Deep Neural Networks, Tong Zhang
Tong Zhang
2020-12-11T04:20:00-08:00 - 2020-12-11T04:30:00-08:00
Live Q&A with Tong Zhang (Zoom)
Sebastian Stich
2020-12-11T04:30:00-08:00 - 2020-12-11T04:50:00-08:00
Invited speaker: Adaptation and universality in first-order methods, Volkan Cevher
Volkan Cevher
2020-12-11T04:50:00-08:00 - 2020-12-11T05:00:00-08:00
Live Q&A with Volkan Cevher (Zoom)
Sebastian Stich
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed Video: Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions?, Ohad Shamir
Ohad Shamir
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed Video: Distributed Proximal Splitting Algorithms with Rates and Acceleration, Laurent Condat
Laurent Condat
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed Video: Employing No Regret Learners for Pure Exploration in Linear Bandits, Mohammadi Zaki
Mohammadi Zaki
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed Video: Constraint-Based Regularization of Neural Networks, Tiffany Vlaar
Tiffany Vlaar
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed Video: PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization, Zhize Li
Zhize Li
2020-12-11T05:00:00-08:00 - 2020-12-11T05:30:00-08:00
Contributed talks in Session 1 (Zoom)
Sebastian Stich, Laurent Condat, Zhize Li, Ohad Shamir, Tiffany Vlaar, Mohammadi Zaki
2020-12-11T05:30:00-08:00 - 2020-12-11T06:00:00-08:00
Break (
2020-12-11T06:00:00-08:00 - 2020-12-11T06:50:00-08:00
Poster Session 1 (
Laurent Condat, Tiffany Vlaar, Ohad Shamir, Mohammadi Zaki, Zhize Li, Guan-Horng Liu, Samuel Horváth, Mher Safaryan, Yoni Choukroun, kumar Shridhar, Nabil Kahale, Jikai Jin, Pratik Kumar Jawanpuria, Gaurav Kumar Yadav, Kazuki Koyama, Junyoung Kim, Xiao Li, Saugata Purkayastha, Adil Salim, Dighanchal Banerjee, Peter Richtarik, Lakshman Mahto, Tian Ye, Bamdev Mishra
2020-12-11T06:50:00-08:00 - 2020-12-11T07:00:00-08:00
Welcome remarks to Session 2
Martin Takac
2020-12-11T07:00:00-08:00 - 2020-12-11T07:20:00-08:00
Invited speaker: Adaptive Sampling for Stochastic Risk-Averse Learning, Andreas Krause
Andreas Krause
2020-12-11T07:20:00-08:00 - 2020-12-11T07:30:00-08:00
Live Q&A with Andreas Krause (Zoom)
Martin Takac
2020-12-11T07:30:00-08:00 - 2020-12-11T07:50:00-08:00
Invited speaker: Practical Kronecker-factored BFGS and L-BFGS methods for training deep neural networks, Donald Goldfarb
Donald Goldfarb
2020-12-11T07:50:00-08:00 - 2020-12-11T08:00:00-08:00
Live Q&A with Donald Goldfarb (Zoom)
Martin Takac
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed Video: Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search), Sharan Vaswani
Sharan Vaswani
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed Video: How to make your optimizer generalize better, Sharan Vaswani
Sharan Vaswani
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed talks in Session 2 (Zoom)
Martin Takac, Samuel Horváth, Guan-Horng Liu, Nicolas Loizou, Sharan Vaswani
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed Video: Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence, Nicolas Loizou
Nicolas Loizou
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed Video: DDPNOpt: Differential Dynamic Programming Neural Optimizer, Guan-Horng Liu
Guan-Horng Liu
2020-12-11T08:00:00-08:00 - 2020-12-11T08:30:00-08:00
Contributed Video: Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization, Samuel Horvath
Samuel Horváth
2020-12-11T08:30:00-08:00 - 2020-12-11T09:00:00-08:00
Break (
2020-12-11T09:00:00-08:00 - 2020-12-11T09:05:00-08:00
Intro to Invited Speaker 5
Martin Takac
2020-12-11T09:00:00-08:00 - 2020-12-11T09:20:00-08:00
Invited speaker: SGD without replacement: optimal rate analysis and more, Suvrit Sra
Suvrit Sra
2020-12-11T09:20:00-08:00 - 2020-12-11T09:30:00-08:00
Live Q&A with Suvrit Sra (Zoom)
Martin Takac
2020-12-11T09:45:00-08:00 - 2020-12-11T10:50:00-08:00
Poster Session 2 (
Sharan Vaswani, Nicolas Loizou, Wenjie Li, Preetum Nakkiran, Zhan Gao, Sina Baghal, Jingfeng Wu, Roozbeh Yousefzadeh, Jinyi Wang, Jing Wang, Cong Xie, Anastasia Borovykh, Stanislaw Jastrzebski, Soham Dan, Yiliang Zhang, Mark Tuddenham, Sarath Pattathil, Ievgen Redko, Jeremy Cohen, Yasaman Esfandiari, Zhanhong Jiang, Mostafa ElAraby, Chulhee Yun, Jia-Jie Zhu, Michael Psenka, Robert Gower, Xiaoyu Wang
2020-12-11T10:50:00-08:00 - 2020-12-11T11:00:00-08:00
Welcome remarks to Session 3
Mark Schmidt
2020-12-11T11:00:00-08:00 - 2020-12-11T11:20:00-08:00
Invited speaker: Stochastic Geodesic Optimization, Ashia Wilson
Ashia Wilson
2020-12-11T11:20:00-08:00 - 2020-12-11T11:30:00-08:00
Live Q&A with Ashia Wilson (Zoom)
Mark Schmidt
2020-12-11T11:30:00-08:00 - 2020-12-11T11:50:00-08:00
Invited speaker: Concentration for matrix products, and convergence of Oja’s algorithm for streaming PCA, Rachel Ward
Rachel Ward
2020-12-11T11:50:00-08:00 - 2020-12-11T12:00:00-08:00
Live Q&A with Rachel Ward (Zoom)
Mark Schmidt
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed Video: Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems, Preetum Nakkiran
Preetum Nakkiran
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed Video: TenIPS: Inverse Propensity Sampling for Tensor Completion, Chengrun Yang
Chengrun Yang
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed Video: Variance Reduction on Adaptive Stochastic Mirror Descent, Wenjie Li
Wenjie Li
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed Video: Incremental Greedy BFGS: An Incremental Quasi-Newton Method with Explicit Superlinear Rate, Zhan Gao
Zhan Gao
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed talks in Session 3 (Zoom)
Mark Schmidt, Zhan Gao, Wenjie Li, Preetum Nakkiran, Denny Wu, Chengrun Yang
2020-12-11T12:00:00-08:00 - 2020-12-11T12:30:00-08:00
Contributed Video: When Does Preconditioning Help or Hurt Generalization?, Denny Wu
Denny Wu
2020-12-11T12:30:00-08:00 - 2020-12-11T13:30:00-08:00
Break (
2020-12-11T13:30:00-08:00 - 2020-12-11T13:50:00-08:00
Invited speaker: Fast convergence of stochastic subgradient method under interpolation, Michael Friedlander
Michael Friedlander
2020-12-11T13:30:00-08:00 - 2020-12-11T13:35:00-08:00
Intro to Invited Speaker 8
Mark Schmidt
2020-12-11T13:50:00-08:00 - 2020-12-11T14:00:00-08:00
Live Q&A with Michael Friedlander (Zoom)
Mark Schmidt
2020-12-11T14:00:00-08:00 - 2020-12-11T14:50:00-08:00
Poster Session 3 (
Denny Wu, Chengrun Yang, Tolga Ergen, sanae lotfi, Charles Guille-Escuret, Boris Ginsburg, Hanbake Lyu, Cong Xie, David Newton, Debraj Basu, Yewen Wang, James Lucas, MAOJIA LI, Lijun Ding, Jose Javier Gonzalez Ortiz, Reyhane Askari Hemmat, Zhiqi Bu, Neal Lawton, Kiran Thekumparampil, Jiaming Liang, Lindon Roberts, Jingyi Zhu
2020-12-11T14:50:00-08:00 - 2020-12-11T15:00:00-08:00
Welcome remarks to Session 4
Quanquan Gu
2020-12-11T15:00:00-08:00 - 2020-12-11T15:20:00-08:00
Invited speaker: Online nonnegative matrix factorization for Markovian and other real data, Deanna Needell and Hanbaek Lyu
Hanbake Lyu, Deanna Needell
2020-12-11T15:20:00-08:00 - 2020-12-11T15:30:00-08:00
Live Q&A with Deanna Needell and Hanbake Lyu (Zoom)
Quanquan Gu
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed Video: A Study of Condition Numbers for First-Order Optimization, Charles Guille-Escuret
Charles Guille-Escuret
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed talks in Session 4 (Zoom)
Quanquan Gu, sanae lotfi, Charles Guille-Escuret, Tolga Ergen, Dongruo Zhou
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed Video: Stochastic Damped L-BFGS with controlled norm of the Hessian approximation, Sanae Lotfi
sanae lotfi
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed Video: On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization, Dongruo Zhou
Dongruo Zhou
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed Video: Convex Programs for Global Optimization of Convolutional Neural Networks in Polynomial-Time, Tolga Ergen
Tolga Ergen
2020-12-11T15:30:00-08:00 - 2020-12-11T16:00:00-08:00
Contributed Video: Affine-Invariant Analysis of Frank-Wolfe on Strongly Convex Sets, Lewis Liu
2020-12-11T16:00:00-08:00 - 2020-12-11T16:05:00-08:00
Closing remarks
Quanquan Gu, Courtney Paquette, Mark Schmidt, Sebastian Stich, Martin Takac