Beyond first order methods in machine learning systems
Anastasios Kyrillidis · Albert Berahas · Fred Roosta · Michael W Mahoney

Fri Dec 13th 08:00 AM -- 06:00 PM @ West 211 - 214
Event URL: »

Optimization lies at the heart of many exciting developments in machine learning, statistics and signal processing. As models become more complex and datasets get larger, finding efficient, reliable and provable methods is one of the primary goals in these fields.

In the last few decades, much effort has been devoted to the development of first-order methods. These methods enjoy a low per-iteration cost and have optimal complexity, are easy to implement, and have proven to be effective for most machine learning applications. First-order methods, however, have significant limitations: (1) they require fine hyper-parameter tuning, (2) they do not incorporate curvature information, and thus are sensitive to ill-conditioning, and (3) they are often unable to fully exploit the power of distributed computing architectures.

Higher-order methods, such as Newton, quasi-Newton and adaptive gradient descent methods, are extensively used in many scientific and engineering domains. At least in theory, these methods possess several nice features: they exploit local curvature information to mitigate the effects of ill-conditioning, they avoid or diminish the need for hyper-parameter tuning, and they have enough concurrency to take advantage of distributed computing environments. Researchers have even developed stochastic versions of higher-order methods, that feature speed and scalability by incorporating curvature information in an economical and judicious manner. However, often higher-order methods are “undervalued.”

This workshop will attempt to shed light on this statement. Topics of interest include --but are not limited to-- second-order methods, adaptive gradient descent methods, regularization techniques, as well as techniques based on higher-order derivatives.

08:00 AM Opening Remarks (Opening remarks) Anastasios Kyrillidis, Albert Berahas, Fred Roosta, Michael W Mahoney
08:15 AM Talk 1 (Plenary talk - Coralia Cartis)
09:00 AM Spotlight talks (Spotlight talks from paper submissions) Diego Granziol, Fabian Pedregosa, Hilal Asi, Leonard Adolphs, Si Yi (Cathy) Meng
09:45 AM Poster Session <span> <a href="#"></a> </span>
Eduard Gorbunov, Alexandre d'Aspremont, Lingxiao Wang, Liwei Wang, Boris Ginsburg, Alessio Quaglino, Camille Castera, Saurabh Adya, Diego Granziol, Rudrajit Das, Raghu Bollapragada, Fabian Pedregosa, Martin Takac, Majid Jahani, Sai Praneeth Karimireddy, Hilal Asi, Balint Daroczy, Leonard Adolphs, Aditya Rawal, Nicolas Brandt, Minhan Li, Giuseppe Ughi, Orlando Romero, Ivan Skorokhodov, Damien Scieur, Kiwook Bae, Konstantin Mishchenko, Rohan Anil, Vatsal Sharan, Aditya Balu, Chao Chen, Zhewei Yao, Tolga Ergen, Paul Grigas, Chris Junchi Li, Jimmy Ba, Stephen J Roberts, Sharan Vaswani, Armin Eftekhari, Chhavi Sharma
10:30 AM Economical use of second-order information in training machine learning models (Plenary talk) Donald Goldfarb
11:15 AM Adaptive gradient methods: efficient implementation and generalization (Plenary talk - Elad Hazan)
12:00 PM Lunch break <span> <a href="#"></a> </span>
02:00 PM K-FAC: Extensions, improvements, and applications (Plenary talk) James Martens
02:45 PM Spotlight talks (Spotlight talks from paper submissions) Paul Grigas, Zhewei Yao, Rohan Anil, Konstantin Mishchenko, Damien Scieur
03:30 PM Poster Session (same as above) (Poster Session)
04:15 PM Analysis of linear search methods for various gradient approximation schemes for noisy derivative free optimization. (Plenary talk) Katya Scheinberg
05:00 PM Second-order methods for nonconvex optimization with complexity guarantees (Plenary talk) Stephen Wright
05:45 PM Final remarks <span> <a href="#"></a> </span> Anastasios Kyrillidis, Albert Berahas, Fred Roosta, Michael W Mahoney

Author Information

Anastasios Kyrillidis (Rice University)
Albert Berahas (Lehigh University)
Fred Roosta (University of Queensland)
Michael W Mahoney (UC Berkeley)

More from the Same Authors