Tokenization as Capacity Allocation: Matched Budgets, Communication Cuts, and Task Demand

Two tokenizers with the same active vocabulary, sequence length, token-length histogram, embedding allowance, and downstream architecture can expose different raw-variable interactions to a shallow Transformer. We formalize this effect for lossless exact-substring chunk tokenizers on a full binary product domain. Distributed encoding cost then equals boundary load across a raw-coordinate cut. Composing this identity with a one-layer Transformer simulation yields a lower-bound certificate that charges the encoder before assigning the remaining burden to the network. Our matched family moves one boundary between $AB$ and $BC$ groupings while every scalar budget remains fixed. At either endpoint, the aligned task has a constant-width one-layer construction. The opposite task requires $m(p+\log n)H=\Omega(n)$, and a second layer recovers $O(\log n)$ width. This allocation also yields a robust design rule. Known demand selects an endpoint. Worst-case uncertainty selects the nearest feasible half allocation; task-specific loss weights move the threshold, and adjustment costs yield partial adjustment. The conclusions concern communication certificates for exact computation, not centralized runtime, energy, or monetary cost. Deployment quantities require separate empirical calibration.