Spectral Compensation for Inversion-Free One-Shot Pruning of Diffusion Models

Structured pruning of text-to-image diffusion models yields immediate wall-clock speedups, but removing attention heads or feed-forward channels degrades generation quality without careful weight repair. One-shot second-order methods adapt Optimal Brain Surgeon (OBS) to diffusion backbones, using the layer's estimated curvature both to rank weights by the output error pruning induces and to redistribute that error onto the surviving weights. Yet both steps require inverting the activation Gram matrix, an operation whose result is dominated by the matrix's smallest eigenvalues. These are precisely the directions a small calibration set estimates worst, so the correction is driven by what the data cannot resolve. Ridge damping softens this but still admits every direction, while diagonal approximations avoid the inversion at the cost of the cross-channel correlations OBS depends on. We propose an inversion-free non-uniform structured pruning framework that circumvents this instability. Rather than inverting the Gram submatrix over surviving channels, we restrict the OBS weight-repair update to the subspace spanned by its leading eigenvectors, computed by randomized subspace iteration. In this basis the matrix to be inverted is exactly diagonal, so a dense factorization collapses into scalar reciprocals and the update becomes a low-rank sum of outer products. Each retained direction absorbs only the residual component lying along it, confining estimation error to directions the calibration data actually supports and making truncation at a variance threshold an exact projection rather than an approximation. On SDXL-Base-1.0 we surpass baselines on FID, Inception Score (IS), CLIP score and SSIM at every sparsity target. The margin is widest on SSIM, indicating our pruned models preserve substantially more of the dense model's structure.