Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

We study the nuclear-norm update concurrently considered as SLORR-Nuc through the lens of decoupled weight decay and proximal optimization. We call this update spectral weight decay. Unlike standard weight decay, it shrinks singular values additively rather than multiplicatively, concentrating the spectrum. We interpret the update as an approximate nuclear-norm proximal step and compute its polar factor with Newton–Schulz iterations. Across $124$M to $500$M-parameter LLaMA models, it lowers effective rank and improves SVD-LLM compression at matched validation loss. Under fixed-horizon training with $60\%$ label noise, it also improves final clean-test accuracy over $L_2$-SP by $1.33$ to $4.76$ points across four BERT-base tasks. At $500$M and a $4\%$ distortion budget, it achieves $1.89\times$ compression and $1.18\times$ H100 inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Code is available at https://anonymous.4open.science/r/SpectralWD-2E77.