Beyond Confidence: Density-Based Routing in Probing Continual Learners

Continual learning with pretrained models increasingly relies on frozen representations and lightweight task-specific classifiers, reducing representation forgetting while shifting the focus toward reliable cross-task prediction and alleviating classifier bias. We show that, in this regime, the conventional characterization of classifier bias as task-recency bias is insufficient: errors do not consistently favor recent tasks, but instead arise from competition among independently trained classifier scores. We formalize this phenomenon through an outscoring analysis that characterizes how routing error accumulates as new task classifiers are introduced. We show that logit- and confidence-based routing provide no distribution-dependent control over the set of tasks that can meaningfully outscore the true task. In contrast, density-based routing admits a pairwise error bound determined by task-distribution overlap and density-estimation error. We instantiate this approach with task-specific normalizing flows over frozen representations. Experiments on CIFAR-100 and ImageNet-R with ViT-B/16 and ResNet-50 demonstrate that density-based routing generally achieves higher accuracy and substantially less degradation as the number of tasks grows. Our findings suggest density estimation as a promising foundation for test-time continual learning, where density-derived task assignments and uncertainty can guide selective online adaptation.