Test-Time Continual Learning via Metric-Topological Factorization

Catastrophic forgetting is conventionally treated as an optimization problem \cite{de2021continual}. We argue it is a \emph{factorization} failure: every classifier must decide \emph{which context} (topological) and \emph{which label within it} (metric). Inspired by complementary learning systems, we develop \textbf{Metric-Topological Factorization} (MTF) for test-time continual learning (TTCL) in this paper. Our contributions are threefold. \emph{Theory:} we introduce predictive $(\gamma,\delta)$-width, a covering quantity that lower-bounds the number of experts required; prove prediction invariance for frozen experts together with a measurable zero-forgetting routing margin; and derive an addressing decomposition showing that, once experts are frozen, retained performance is governed by oracle expert quality and errors in the index $\Sigma$. \emph{Algorithm:} we propose the Topological Trinity Transform (TTT), an Evaluate-Detect-Transform cycle that discovers, recognizes, and reuses contexts from the stream without supplied task identity, together with a per-sample test-time addressing map that selects which frozen expert answers each query. \emph{Experiments:} Our experimental results identify a boundary of factorized continual learning: protecting expert knowledge can solve the storage problem, but the system succeeds only when the active predictive regime can be inferred reliably and more cheaply than the target label itself.