Riemannian Optimization for Low-Rank Adaptation via Desingularization

Low-rank Adaptation (LoRA) has been widely used as a parameter-efficient method in fine-tuning large language models (PEFT). However, trained adapters often exhibit low stable rank and small trailing singular values. Near such rank-deficient regions, fixed-rank formulations suffer from severe Hessian bounds and local Lipschitz constants that grow with $\mathcal{O}\big(\sigma^{-1}_r(X)\big)$. To mitigate this issue, we propose ROLAND (**R**iemannian **O**ptimization for **L**ow-Rank **A**daptatio**N** via **D**esingularization), a desingularization method that lifts LoRA to a smooth bounded-rank geometry with both left and right null-space projectors. ROLAND replaces rank-deficient singularities with compact and smooth fibers while retaining a low-rank representation of the adapter update. Within this framework, we equip the manifold optimization with a new Riemannian metric and retraction mechanism, which allow us to establish tighter Hessian bounds and milder condition-number dependence in the local convergence rate. Empirical results demonstrate ROLAND's superiority across a wide range of experimental settings compared with other state-of-the-art PEFT methods.