Controlling Temporal Pseudo-Label Marginals for Stable Online Test-Time Adaptation

Continual test-time adaptation (CTTA) updates a model online on a stream of unlabeled data under distribution shift. In practice, CTTA often becomes unstable: small early prediction biases are reused as supervision, causing pseudo-labels to progressively concentrate on a few classes and eventually leading to drift or collapse. We observe that this instability is closely linked to the temporal contraction of pseudo-label class usage during adaptation. Existing CTTA methods often address reliability or diversity through sample selection or loss weighting, but they do not explicitly control how pseudo-label mass is allocated across classes over time. We introduce SCALAR (Sinkhorn Class-Marginal Regularization), a lightweight plug-in module that implements this temporal control over a short window of recent predictions. SCALAR maintains a buffer of recent predictions, applies Sinkhorn balancing to project the window-level class marginal toward a target distribution, and then uses the resulting balanced soft targets for adaptation. SCALAR requires no architectural changes and adds less than 1\% runtime and memory overhead. SCALAR is most beneficial under skewed, non-stationary, and aggressive-adaptation regimes, where it substantially mitigates drift and collapse, even on top of entropy- and diversity-aware stabilization mechanisms.