NeurIPS 2022

Timezone: »

Practical Structured Riemannian Optimization with Momentum by using Generalized Normal Coordinates

Wu Lin · Valentin Duruisseaux · Melvin Leok · Frank Nielsen · Mohammad Emtiyaz Khan · Mark Schmidt

in Symmetry and Geometry in Neural Representations (NeurReps) »

Event URL: https://openreview.net/forum?id=1aybhSfabqh »

Adding momentum into Riemannian optimization is computationally challenging due to the intractable ODEs needed to define the exponential and parallel transport maps. We address these issues for Gaussian Fisher-Rao manifolds by proposing new local coordinates to exploit sparse structures and efficiently approximate the ODEs, which results in a numerically stable update scheme. Our approach extends the structured natural-gradient descent method of Lin et al. (2021a) by incorporating momentum into it and scaling the method for large-scale applications arising in numerical optimization and deep learning

Keywords: Riemannian Manifolds · Numerical Optimization · Matrix Lie Groups

[ Poster] [ OpenReview]

Author Information

Wu Lin (University of British Columbia)

Valentin Duruisseaux (University of California, San Diego)

Melvin Leok (University of California, San Diego)

Frank Nielsen (Sony Computer Science Laboratories Inc (Tokyo))

Mohammad Emtiyaz Khan (RIKEN)

Emtiyaz Khan (also known as Emti) is a team leader at the RIKEN center for Advanced Intelligence Project (AIP) in Tokyo where he leads the Approximate Bayesian Inference Team. He is also a visiting professor at the Tokyo University of Agriculture and Technology (TUAT). Previously, he was a postdoc and then a scientist at Ecole Polytechnique Fédérale de Lausanne (EPFL), where he also taught two large machine learning courses and received a teaching award. He finished his PhD in machine learning from University of British Columbia in 2012. The main goal of Emti’s research is to understand the principles of learning from data and use them to develop algorithms that can learn like living beings. For the past 10 years, his work has focused on developing Bayesian methods that could lead to such fundamental principles. The approximate Bayesian inference team now continues to use these principles, as well as derive new ones, to solve real-world problems.