Functionally Equivalent, Structurally Distinct Solutions in Low-Rank Recurrent Networks

How does learning select among functionally equivalent neural representations? We study a minimal solvable rank-1 recurrent network in which an infinite family of Gaussian connectivity solutions implements the same bistable dynamics. A symmetry of the learning dynamics biases gradient descent toward a bimodal solution that appears more complex than the task requires, whereas noisy learning favors flatter regions of the solution manifold and selects a simpler single-Gaussian solution instead. These results show that complex connectivity structure, often attributed to the computation being performed, can also arise from the learning dynamics themselves.