Informing generative replay for continual learning with long-term memory formation in the fruit fly

Continual learning without catastrophic forgetting is a challenge for artificial systems but it is done naturally across a range of biological systems, including in insects. A recurrent circuit has been identified in the fruit fly mushroom body to consolidate long term memories (LTM), but there is not currently an algorithmic understanding of this LTM formation. We hypothesize that generative replay is occurring to consolidate memories in this recurrent circuit, and find anatomical evidence in synapse-level connectivity that supports this hypothesis. Next, we introduce a computational model which combines a short-term memory (STM) and LTM phase to perform generative replay based continual learning. When evaluated on a CIFAR-100 class-incremental continual learning task, the modeled LTM phase increases classification performance by 20% and approaches within 2% of the performance for a non-incremental upper bound baseline. Unique elements of the proposed generative replay model include: 1) coupling high dimensional sparse activation patterns with generative replay and 2) sampling and reconstructing higher level representations for training generative replay (as opposed to reconstructing sensory-level or processed sensory-level representations). Additionally, we make the experimentally testable prediction that a specific set of synapses would need to undergo experience-dependent plasticity during LTM formation to support our generative replay-based model.