Timezone: »

 
Poster
Does Momentum Change the Implicit Regularization on Separable Data?
Bohan Wang · Qi Meng · Huishuai Zhang · Ruoyu Sun · Wei Chen · Zhi-Ming Ma · Tie-Yan Liu

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #617
in Poster Session 5 »
The momentum acceleration technique is widely adopted in many optimization algorithms. However, there is no theoretical answer on how the momentum affects the generalization performance of the optimization algorithms. This paper studies this problem by analyzing the implicit regularization of momentum-based optimization. We prove that on the linear classification problem with separable data and exponential-tailed loss, gradient descent with momentum (GDM) converges to the $L^2$ max-margin solution, which is the same as vanilla gradient descent. That means gradient descent with momentum acceleration still converges to a low-complexity model, which guarantees their generalization. We then analyze the stochastic and adaptive variants of GDM (i.e., SGDM and deterministic Adam) and show they also converge to the $L^2$ max-margin solution. Technically, the implicit regularization of SGDM is established based on a novel convergence analysis of SGDM under a general noise condition called affine noise variance condition. To the best of our knowledge, we are the first to derive SGDM’s convergence under such an assumption. Numerical experiments are conducted to support our theoretical results.
Keywords: Optimization · implicit regularization · Momentum-based Optimizers

Author Information

Bohan Wang (USTC)
Qi Meng (Microsoft)
Huishuai Zhang (Microsoft Research Asia)
Ruoyu Sun (Chinese University of Hong Kong (Shenzhen))
Wei Chen ( Chinese Academy of Sciences)
Zhi-Ming Ma
Tie-Yan Liu (Microsoft Research)

More from the Same Authors