Poster

Mechanic: A Learning Rate Tuner

Ashok Cutkosky · Aaron Defazio · Harsh Mehta

Great Hall & Hall B1+B2 (level 1) #1219
[ ]
Thu 14 Dec 3 p.m. PST — 5 p.m. PST

Abstract:

We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call Mechanic. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate Mechanic on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, Mechanic either comes very close to, matches or even improves upon manual tuning of learning rates.

Chat is not available.