Timezone: »

 
Ricky T. Q. Chen---Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Tian Qi Chen

Sat Dec 12 12:15 PM -- 12:30 PM (PST) @
in I Can’t Believe It’s Not Better! Bridging the gap between theory and empiricism in probabilistic machine learning »

Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters. Based on this intuition, we explore the use of exact per-sample Hessian-vector products and gradients to construct optimizers that are self-tuning and hyperparameter-free. Based on a dynamics model of the gradient, we derive a process which leads to a curvature-corrected, noise-adaptive online gradient estimate. The smoothness of our updates makes it more amenable to simple step size selection schemes, which we also base off of our estimates quantities. We prove that our model-based procedure converges in the noisy quadratic setting. Though we do not see similar gains in deep learning tasks, we can match the performance of well-tuned optimizers and ultimately, this is an interesting step for constructing self-tuning optimizers.

Author Information

Tian Qi Chen (U of Toronto)

More from the Same Authors