Power Steepest Mirror Flow: Disentangling Mirror and Steepest Geometries in Deep Linear Networks

Mirror descent and steepest descent impose geometry differently: one moves the gradient into a dual space determined by a potential; the other reshapes the update direction through regularization. When the mirror potential and the steepest regularizer are both drawn from the Schatten-power family, with power exponents chosen independently, what does each geometry control? We answer this in an exactly solvable deep linear matrix factorization model, deriving the spectral dynamics of power steepest mirror flow (Power-SMF), a two-parameter family that combines both. The resulting dynamics reveal complementary roles: the steepest exponent governs convergence near the target, whereas, at fixed steepest exponent and depth, the mirror exponent tunes emergence from small initialization and the order in which modes are learned.