On the Dissociation of Order and Identity in Number Manifold Representations

Representations in neural networks can take the form of linear directions or low-dimensional manifolds. In this work, we question the computational utility of the manifold structure: if it possesses an underlying linear representation, what role does the additional nonlinear structure play? Using a minimal single-layer attention model trained on running maxi- mum, we show that comparison admits a one-dimensional ordered solution, but such a representation cannot decode multiple number identities with a linear readout. We derive an explicit interference budget governing how additional transverse structure can support identity without disrupting comparison, providing a task-level account of the curvature– discriminability tradeoff observed in prior work. In the trained model, number order is concentrated along a single linear direction, while structure orthogonal to it supports number identity. Intervening on the two components produces distinct behavioral effects, allowing comparison and identity to be steered separately. Thus, linear and manifold structure can implement different parts of the same computation.