A Brief Benchmark of Spectral-Whitening Optimizers

A family of recent optimizers have emerged that share a similar spectral-whitening transformation. We perform a controlled empirical comparison, concluding that under optimal hyperparameters, such optimizers outperform Adam across the board, and SOAP does so to the highest degree. This trend remains true under increasing batch size. We empirically show that whitening independent parameter subsets results in roughly additive benefits, and that only left-preconditioning recovers a majority of the performance gain.