Timezone: »

 
Poster
Amortized Proximal Optimization
Juhan Bae · Paul Vicol · Jeff Z. HaoChen · Roger Grosse

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #502
in Poster Session 6 »

We propose a framework for online meta-optimization of parameters that govern optimization, called Amortized Proximal Optimization (APO). We first interpret various existing neural network optimizers as approximate stochastic proximal point methods which trade off the current-batch loss with proximity terms in both function space and weight space. The idea behind APO is to amortize the minimization of the proximal point objective by meta-learning the parameters of an update rule. We show how APO can be used to adapt a learning rate or a structured preconditioning matrix. Under appropriate assumptions, APO can recover existing optimizers such as natural gradient descent and KFAC. It enjoys low computational overhead and avoids expensive and numerically sensitive operations required by some second-order optimizers, such as matrix inverses. We empirically test APO for online adaptation of learning rates and structured preconditioning matrices for regression, image reconstruction, image classification, and natural language translation tasks. Empirically, the learning rate schedules found by APO generally outperform optimal fixed learning rates and are competitive with manually tuned decay schedules. Using APO to adapt a structured preconditioning matrix generally results in optimization performance competitive with second-order methods. Moreover, the absence of matrix inversion provides numerical stability, making it effective for low-precision training.

Keywords: Meta-Learning · second-order optimization · proximal point method · learning rate adaptation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Juhan Bae (University of Toronto, Vector Institute)
Paul Vicol (University of Toronto)
Jeff Z. HaoChen (Stanford University)
Roger Grosse (University of Toronto)

More from the Same Authors