Continuous Memory Machines

Biological memory pairs fast-changing traces with persistent stores, arising from the coordinated dynamics of heterogeneous neurons. Existing recurrent architectures capture only one side of this picture. Prior memory-augmented networks add a persistent store but do not separate short- and long-term memory into distinct matrix-valued components that can update at different rates. The Continuous Thought Machine (CTM), meanwhile, offers rich neuron-level short-term dynamics but has no long-term memory and cannot retain information across long horizons. We bridge this gap by introducing the Continuous Memory Machine (CMM), which pairs the CTM's sliding window short-term memory with an explicit long-term memory, jointly updated at every timestep via a Transformer. The CMM can outperform architectures designed specifically for algorithmic memory tasks, achieves strong results on few-shot in-context regression, and improves the CTM's interpretable recurrent reasoning behaviour. These results show that timescale-separated memory and neuron-level dynamics combine into a more capable recurrent architecture.