SRC as a Convergent Dynamical Process]{Sleep Replay Consolidation Is a Convergent Dynamical Process Whose Attractor Is A Flat Loss Region
Giri Krishnan ⋅ Krishi Chawda ⋅ Jean Erik Delanois ⋅ Maksim Bazhenov
Abstract
A core challenge for deep learning is achieving both strong generalization under distributional shifts and stable continual learning: models that rapidly fit new tasks often converge to sharp minima that are sensitive to input perturbations and that overwrite earlier knowledge. We argue that Sleep Replay Consolidation (SRC), a biologically inspired unsupervised replay algorithm, is best understood as a \emph{dynamical process}: iterating its sleep operator repeatedly drives the weight update toward zero, while loss-landscape sharpness converges onto a stable plateau - a strong indicator of convergence to a fixed-point attractor. Further, this attractor corresponds to a flatter region of the loss landscape, with sharpness more uniform across tasks, than the pre-sleep state. This attractor dynamics explains why SRC improves robustness to input distortions and, in class-incremental training, produces flatter joint solutions that mitigate catastrophic forgetting: across a 5-task run, the number of tasks recovered rises in lockstep with the fall in mean sharpness as the attractor is approached ($r=-0.99$). Extending sleep past the point of best task performance keeps flattening the landscape while accuracy declines, indicating that SRC's benefit comes from partial, not complete, convergence. Characterizing SRC as a convergent dynamical process points to principled stopping rules for sleep-like regularizers and testable hypotheses about the computational role of biological sleep.
Chat is not available.
Successful Page Load