Timezone: »

 
Poster
Gradient Dynamics of Shallow Univariate ReLU Networks
Francis Williams · Matthew Trager · Daniele Panozzo · Claudio Silva · Denis Zorin · Joan Bruna

Thu Dec 12 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #240
in Theory -- Spaces of Functions and Kernels »

We present a theoretical and empirical study of the gradient dynamics of overparameterized shallow ReLU networks with one-dimensional input, solving least-squares interpolation. We show that the gradient dynamics of such networks are determined by the gradient flow in a non-redundant parameterization of the network function. We examine the principal qualitative features of this gradient flow. In particular, we determine conditions for two learning regimes: \emph{kernel} and \emph{adaptive}, which depend both on the relative magnitude of initialization of weights in different layers and the asymptotic behavior of initialization coefficients in the limit of large network widths. We show that learning in the kernel regime yields smooth interpolants, minimizing curvature, and reduces to \emph{cubic splines} for uniform initializations. Learning in the adaptive regime favors instead \emph{linear splines}, where knots cluster adaptively at the sample points.

Keywords: Optimization for Deep Networks · Spaces of Functions and Kernels · Deep Learning · Algorithms -> Kernel Methods; Optimization -> Non-Convex Optimization; Theory
[ Paper]  [ Poster] 

Author Information

Francis Williams (New York University)
Matthew Trager (NYU)
Daniele Panozzo (NYU)
Claudio Silva (New York University)
Denis Zorin (New York University)
Joan Bruna (NYU)

More from the Same Authors