A theoretical study of the $(L_0,L_1)$-smoothness condition in deep learning

We study the $(L_0,L_1)$-smoothness condition introduced by Zhang-He-Sra-Jadbabai in 2020 in the setting of loss functions arising in deep learning. Theoretical work on $(L_0,L_1)$-smoothnes has focused on convergence guarantees for functions which satisfy this condition. In this paper we provide theoretical analysis of the condition in the setting of feedforward neural networks of depth at least 2, with either $L2$ or cross-entropy loss, and find the $(L_0,L_1)$-smoothness condition is not satisfied.