What Should Titans Forget? Disentangling Magnitude, Distribution, and Assignment in Test-Time Memory
Beloslava Malakova
Abstract
Test-time neural memory must continually balance retaining useful information against forgetting previous updates. Titans addresses this through learned, data-dependent decay, yet it remains unclear which properties of this mechanism account for its effectiveness. Learned forgetting is studied through controlled test-time interventions on a Titans Memory-as-Context model, decomposing forgetting into its magnitude, distribution, and content-dependent assignment. Removing decay severely degrades next-token prediction, and hand-designed policies based on surprise, novelty, recency, and their combination also underperform native decay, even when matched to its mean. The recorded native decay explains why: two of three memory layers clear their memory at almost every chunk, while the third retains information for about $1.4$ chunks, mostly following position in the sequence. A constant decay per layer closes about $99.5\%$ of the gap left by global mean matching, and a fixed per-layer, per-position schedule matches native loss to within $0.0001$ nats. Native input-dependent decay helps slightly on typical sequences but hurts on rare out-of-distribution ones, with no measurable net benefit. In this setting, learned forgetting is effective because it sets the right timescale in each layer and a positional schedule, rather than because it selects what to forget.
Chat is not available.
Successful Page Load