Topological defects propagate information in deep neural networks

We study the effect of spontaneously broken discrete symmetries on the flow of information through deep neural networks. In physical systems, a spontaneously broken symmetry allows for the formation of topological defects, stable localized excitations that arise from the existence of distinct degenerate ground states. We demonstrate a similar phenomenon in deep learning. In particular, we study a image manipulation task solved by a simple toy model with recurrent dynamics inspired by Kuramoto oscillators. We show that a spontaneously broken $\mathbb{Z}_2$ symmetry in the internal representation space allows for the formation of a $\mathbb{Z}_2$ topological defect, or domain wall. These defects carry information in a novel manner through the layers of the network, providing potential advantages during training. We also demonstrate that under the addition of noise at each layer of the trained network, the spontaneous symmetry breaking is destroyed at a continuous phase transition, at which the RMSE loss displays indications of finite-size scaling laws that are familiar from critical phenomena in statistical physics. We speculate on further applications.