On the geometry of recurrent spiking networks

Biological neural networks are recurrent and transmit information via discrete spikes. However, neural network theories largely focus on deep feedforward architectures with continuous activations, and it is not clear to what extent these theories are relevant to the analysis and optimization of recurrent spiking neural networks (SNNs). We propose to study the geometry of SNNs as piecewise-constant functions, which we visualize with our new algorithm, SplineCam-SNN. We first study how the parameters of deep SNNs affect their input-output geometry. In contrast to deep networks with continuous activations, recurrent (synaptic) weights appear to have a more limited influence on the geometry than skip connections, leakages, and delays. With these insights, we compare two plasticity methods: gradient descent aligns input-output geometry around data, while STDP, being focused on recurrent weights, cannot. Our findings emphasize the importance of skip connections, leakages, and delays for training SNNs, and suggest that these parameters may be promising targets for future plasticity algorithms.