SLORR: Simple and Efficient In-Training Low-Rank Regularization

Direct spectral regularization is a natural way to induce low-rank weights, but common objectives are costly at modern scale because they typically require repeated singular value decompositions (SVDs). Existing alternatives reduce this cost through reformulations that introduce architectural reparameterization, cached spectral state, predefined target ranks, or periodic decompositions, thus hindering practicality. We introduce SLORR to make direct spectral regularization practical without these restrictions while retaining competitive post-compression performance. Our SVD-free, stateless, architecture-preserving framework uses GPU-friendly approximations of the spectral quantities needed for regularizer forward and backward passes. SLORR supports different spectral objectives, instantiated here with Hoyer spectral sparsity and nuclear norm regularization. We evaluate post-compression performance of regularized models under different low-rank compression procedures, using ImageNet-1K pretraining and continued training of ResNets and vision transformers, as well as LLM pretraining at 135M, 560M, and 1B parameters. Across vision settings and for 135M LLMs, SLORR achieves competitive compression–quality tradeoffs with recent strong low-rank regularizers while incurring very low training overhead and avoiding their practical limitations. We further scale SLORR to 560M and 1B LLMs: it substantially improves post-compression performance over matched unregularized training. These compressibility gains persist across different language modeling token budgets and four optimizers.