Toward Understanding How the Data Affects Neural Collapse: A Kernel-Based Approach

Recently, a vast amount of literature has focused on the "Neural Collapse" (NC) phenomenon, which emerges when training neural network (NN) classifiers beyond the zero training error point. The core component of NC is the decrease in the within-class variability of the network's deepest features, dubbed as NC1. The theoretical works that study NC are typically based on simplified unconstrained features models (UFMs) that mask any effect of the data on the extent of collapse. In this paper, we take a step toward addressing this limitation by analyzing NC1 using kernels associated with shallow NNs. By considering the NN Gaussian Process kernel (NNGP), and the complement Neural Tangent Kernel (NTK), we show that the NTK surprisingly does not represent more collapsed features than the NNGP for gaussian data of arbitrary dimensions. We then consider an alternative to NTK: the recently proposed adaptive kernel, which generalizes NNGP to model the feature mapping learned from the training data. Through this kernel vs. kernel analysis, we present insights into the settings (data dimension, sample size, width) under which the kernel based NC1 aligns with that of shallow NNs.