SheafSeq: Recurrent, Attention, and State-Space Models on Cellular Sheaves

Graph and cell-complex data often contain features that are only comparable after a local change of coordinates: neighboring cells may represent measurements in different frames, incompatible modalities, or heterophilic relational regimes. Cellular sheaves provide a principled language for such local vector spaces and their restriction or transport maps, while modern sequence architectures provide powerful mechanisms for content-dependent memory and long-range selection. We introduce SheafSeq, a unified framework that first transports contextual tokens into a target stalk of a cellular sheaf and then processes the transported sequence using a recurrent, attention, or state-space backbone. The framework yields SheafSeq-GRU, SheafSeq-LSTM, SheafSeq-Transformer, and SheafMamba as special cases. We prove that sheaf tokenization is gauge-covariant and complete for local connection-sheaf configurations up to a root gauge, that any local gauge-covariant map factors through this tokenization, and that sheaf diffusion is a linear order-insensitive member of the same family. We also prove an information-theoretic separation: raw-coordinate sequence models are Bayes-limited on twisted-frame last-event tasks, while sheaf-transported sequence models solve them exactly. Experiments on a connection-sheaf benchmark support the theory: raw GRU, LSTM, Transformer, and SSM models yield limited performance, while their sheaf-transported counterparts become highly accurate. Among sheafified models, recurrent baselines achieve the highest accuracy on the task, while SheafMamba offers a strong accuracy-efficiency tradeoff and stable length transfer. Overall, SheafSeq shows that sheaf transport can serve as a general interface between geometry-aware relational representation and scalable sequence computation, making modern sequence models applicable to heterophilic domains with local coordinate frames, where raw coordinates are coordinate-frame dependent.