Modality-Adaptive Memory Gating with Embedding Replay for Continual Learning in Spiking Neural Networks

Deploying spiking neural networks (SNNs) in multi-sensory edge systems requires sequential acquisition of new modalities without forgetting previously learned ones. However, visual and auditory neuromorphic data exhibit fundamentally different spike statistics event-camera recordings (N-MNIST produce spatially structured, temporally compact patterns, while cochlea-inspired encodings (SHD) yield dense temporal sequences across hundreds of frequency channels. Prior work has shown that memory mechanisms are modality-dependent: Hopfield networks excel at spatial visual tasks (97.68%) but degrade on temporal auditory tasks (76.15%), while hierarchical gating (HGRN) provides balanced cross-modal performance. This raises the question: can a single SNN architecture adapt its internal processing to each modality while mitigating catastrophic forgetting during sequential cross-modal training? We show that modality-adaptive memory routing with compact embedding replay can preserve previously learned representations while encouraging specialized processing of visual and auditory spike data.