Timezone: »
Poster
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Libin Zhu · Chaoyue Liu · Misha Belkin
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their ``width'' approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.
Author Information
Libin Zhu (UC San Diego)
Chaoyue Liu (University of California, San Diego)
Misha Belkin (Ohio State University)
More from the Same Authors
-
2023 Poster: When does over-parameterized SGD take as few iterations to converge as gradient descent? »
Chaoyue Liu · Dmitriy Drusvyatskiy · Yian Ma · Damek Davis · Misha Belkin -
2022 Poster: Benign, Tempered, or Catastrophic: Toward a Refined Taxonomy of Overfitting »
Neil Mallinar · James Simon · Amirhesam Abedsoltan · Parthe Pandit · Misha Belkin · Preetum Nakkiran -
2022 Poster: Benign Overfitting in Two-layer Convolutional Neural Networks »
Yuan Cao · Zixiang Chen · Misha Belkin · Quanquan Gu -
2020 Poster: On the linearity of large non-linear models: when and why the tangent kernel is constant »
Chaoyue Liu · Libin Zhu · Misha Belkin -
2020 Spotlight: On the linearity of large non-linear models: when and why the tangent kernel is constant »
Chaoyue Liu · Libin Zhu · Misha Belkin