`

Timezone: »

 
Poster
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
Basri Ronen · David Jacobs · Yoni Kasten · Shira Kritchman

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #236

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.

Author Information

Basri Ronen (Weizmann Inst.)
David Jacobs (University of Maryland, USA)
Yoni Kasten (Weizmann Institute)
Shira Kritchman (Weizmann Institute)

More from the Same Authors