DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning

Kronecker-factored preconditioners enable memory-efficient optimization by approximating Adagrad preconditioner. Yet, existing updates for the Kronecker factors are heuristic and fail to minimize the true approximation error, yielding suboptimal curvature estimates and limited efficiency. We propose DyKAF, a principled framework that casts the Kronecker approximation as a dynamical low-rank approximation problem and solves it with a projector-splitting integrator. This is provably more accurate and plugs directly into existing optimizers. Both theory and experiments confirm that DyKAF improves approximation quality, translating into measurable gains across pretraining and fine-tuning tasks.