Task-Level Insights from Eigenvalues across Sequence Models

Although softmax attention drives state-of-the-art performance for sequence models, its quadratic complexity limits scalability, motivating linear alternatives such as state space models (SSMs). While these alternatives improve efficiency, their fundamental differences in information processing remain poorly understood. In this work, we leverage the recently proposed dynamical systems framework (DSF) to represent softmax and linear attention as dynamical systems, enabling a structured comparison with SSMs, building upon the systems’ eigenvalues. Our results demonstrate that eigenvalues influence essential aspects of memory and long-range dependency modeling. Furthermore, we identify spectral signatures that correlate with task requirements, motivating eigenvalue analysis as a principled tool for understanding sequence model capabilities.