LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

Low-rank adaptation (LoRA) fine-tunes large models through two low-rank factors, but independent factor updates ignore the geometry of their induced weight update. We introduce LoRA-TSD, which performs Muon-style spectral-norm steepest descent in the tangent space of the fixed-rank manifold. A LoRA-native retraction maps this direction back to the factors without full weight matrices and is up to $2.8\times$ cheaper than the truncated-SVD retraction used by prior manifold methods. The Frobenius-norm surrogate recovers LoRA-Pro. We prove the first global convergence guarantees for LoRA-Pro and LoRA-TSD in terms of the tangent-projected gradient, which is computable from factor gradients alone. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B, and Qwen3-32B, LoRA-TSD outperforms all evaluated LoRA optimizers and remains robust across adapter ranks. Code is available at https://anonymous.4open.science/r/LoRA-TSD-36D3.