Timezone: »
As a popular approach to modeling the dynamics of training overparametrized neural networks (NNs), the neural tangent kernels (NTK) are known to fall behind real-world NNs in generalization ability. This performance gap is in part due to the \textit{label agnostic} nature of the NTK, which renders the resulting kernel not as \textit{locally elastic} as NNs~\citep{he2019local}. In this paper, we introduce a novel approach from the perspective of \emph{label-awareness} to reduce this gap for the NTK. Specifically, we propose two label-aware kernels that are each a superimposition of a label-agnostic part and a hierarchy of label-aware parts with increasing complexity of label dependence, using the Hoeffding decomposition. Through both theoretical and empirical evidence, we show that the models trained with the proposed kernels better simulate NNs in terms of generalization ability and local elasticity.
Author Information
Shuxiao Chen (University of Pennsylvania)
Hangfeng He (University of Pennsylvania)
Weijie Su (The Wharton School, University of Pennsylvania)
More from the Same Authors
-
2021 Spotlight: A Central Limit Theorem for Differentially Private Query Answering »
Jinshuo Dong · Weijie Su · Linjun Zhang -
2022 Poster: The alignment property of SGD noise and how it helps select flat minima: A stability analysis »
Lei Wu · Mingze Wang · Weijie Su -
2021 Poster: A Central Limit Theorem for Differentially Private Query Answering »
Jinshuo Dong · Weijie Su · Linjun Zhang -
2021 Poster: You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism »
Weijie Su -
2021 Poster: Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations »
Jiayao Zhang · Hua Wang · Weijie Su -
2020 Poster: A Group-Theoretic Framework for Data Augmentation »
Shuxiao Chen · Edgar Dobriban · Jane Lee -
2020 Oral: A Group-Theoretic Framework for Data Augmentation »
Shuxiao Chen · Edgar Dobriban · Jane Lee -
2020 Poster: The Complete Lasso Tradeoff Diagram »
Hua Wang · Yachong Yang · Zhiqi Bu · Weijie Su -
2020 Spotlight: The Complete Lasso Tradeoff Diagram »
Hua Wang · Yachong Yang · Zhiqi Bu · Weijie Su -
2019 Poster: Algorithmic Analysis and Statistical Estimation of SLOPE via Approximate Message Passing »
Zhiqi Bu · Jason Klusowski · Cynthia Rush · Weijie Su -
2019 Poster: Acceleration via Symplectic Discretization of High-Resolution Differential Equations »
Bin Shi · Simon Du · Weijie Su · Michael Jordan