PRISM: Principal Subspace Alignment for Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation have become essential for adapting large language models to downstream tasks. Existing approaches often apply low-rank updates uniformly across all layers, ignoring the structural coupling inherent in transformer architectures. We observe that transformers contain \emph{weight pairs} whose interactions govern model behavior including query-key products, value-output products, and feedforward networks. We propose PRincipal Interaction Subspace Matching (PRISM), a principled framework that exploits this coupling through spectral parameterization. For each weight pair, we parameterize adaptations using the spectral basis of the paired matrix, ensuring that updates occur in the subspace where the paired matrix has maximal effect. We validate our approach through controlled experiments on synthetic models, demonstrating faster convergence. Empirical evaluations on arithmetic reasoning, code generation, and vision tasks show that PRISM outperforms existing PEFT methods.