The balance between feature learning and collapse in generative dynamical systems

Over the past decade, a growing body of work has established that implicit biases in learning dynamics fundamentally shape the solutions found by neural networks, governing their learned features and generalisation performance. However, despite these strides in deep learning theory, far less is known about representation learning in generative models based on dynamical systems, from normalising flows to diffusion. Importantly, continuous-time generative models can be prone to mode collapse, yet the learning dynamics underpinning such biases remain elusive. To address this, we use tools from dynamical systems and operator theory to show that bounds on the rank of the gradient of the model weights can explain how collapse occurs in generative models. We show that, in both variational inference and flow matching tasks, optimisation induces low-rank biases that encourage parsimonious representation learning but may also cause the learned distribution to collapse. Together these findings point to a trade-off between promoting feature learning while avoiding both memorisation and collapse in dynamical systems-based generative models.