Disagreement Geometry in Hyper-Connections

Hyper-Connections replace a single residual stream with multiple interacting streams, and recent work constrains the residual mixer to stabilize training. These constraints are certified by the spectral norm of the composed mixer, but that norm is at least $1$ for any mixer preserving the stream mean, and therefore says nothing about the orthogonal subspace where the streams differ from that mean. A mean-preserving spectral-norm constraint is in fact satisfied by the matrix that erases this subspace entirely. Restricting the same norm to it, we find that four constraint families span a factor of $4.5$ in how much disagreement survives to the final layer, from $0.220$ to $1.000$, while their validation losses are comparable. The isometric endpoint is the mean-preserving isometric subgroup $\mathcal{G}_n$, which uses three dynamic scalars per layer at $n = 4$ and is contained in an existing spectral-sphere parameterization; fixing its singular values to one gives a controlled ablation in which that parameterization contracts disagreement during training while the isometric restriction preserves it exactly.