Residual-Aware Sparsification and Pruning: Protecting Principal Subspaces During Sparsification or Pruning

Linear layers dominate LLM parameter count and memory traffic, making them a major computational bottleneck. Existing structured pruning and activation-sparsity methods reduce these costs by removing or skipping channels according to saliency or activation-based criteria. However, such methods can discard dominant singular directions of the original projection, making performance sensitive to the scoring rule or sparsification criterion. We propose \textbf{Residual-Aware Sparsification and Pruning} (\RASP{}), a training-free drop-in wrapper for pruning and sparsity methods. For each targeted weight (W), \RASP{} decomposes (W = Lr + Rr), where (Lr) captures the top-(r) singular directions and (Rr) is the residual. \RASP{} then confines pruning or sparsification to (R_r), preserving the dominant projection directions while bounding approximation error by the residual norm, rather than the full weight norm. At 50\% sparsity, \RASP{} consistently improves generation scores -- by up to +7.33 points over \griffin{}, +7.93 over \wanda{}, +2.10 over \teal{}. Interestingly, \RASP{} surpasses dense inference on several Gemma 2 9B IT generation benchmarks. Finally, \RASP{} is able to achieve 40\%-46\% speed-ups compared to a dense model. Our code is publicly available \href{https://anonymous.4open.science/r/RASP-EB5E}{\textbf{here}}.