ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

Convolutional networks, recurrent networks, and transformers each encode different inductive biases, locality, sequential memory, and content-dependent pairwise interaction, and have remained mathematically distinct since their inception. We show that this fragmentation reflects not a fundamental diversity in how signals should be processed, but rather incomplete views of a single underlying mathematical object: a learnable integral transform. We introduce the Integral Transform Network (ITNet), a unified architecture built around a learnable kernel that depends jointly on positions and features. This kernel is implemented as a small neural network, specifically an MLP, that models pairwise interactions, enabling the model to adapt its behavior from data. We show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as exact special cases within one explicit operator class - a pointwise kernel closed under a single query normalization or a single causal gated accumulation, and that this class is a genuine restriction rather than a class that can express everything. ITNet is also a universal approximator of continuous operators. To make this practical, we develop tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization, enabling efficient and scalable computation. A single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on ImageNet-1K, GLUE, ModelNet40, VQAv2, NLVR2, Long Range Arena, and WikiText-103 autoregressive language modeling. Measuring the trained kernels shows that they specialize per modality, local on images, content-driven on text, geometry-aware on point clouds, without any architectural change, demonstrating that a single learned interaction mechanism can recover the behavior of all three architectural families from data.