Multiscale Residual Tokenization for Continuous Spectral Coordinates: Semantics and Physical Stability
Abstract
Spectral coordinates are continuous, but many foundation-model pipelines require discrete vocabularies. We introduce multiscale residual tokenization (MRT), which represents each coordinate by level-specific coarse-to-fine residual digits. On frozen NMR Sim→Exp retrieval, MRT improves ¹³C Top-1 from 0.1363 to 0.2006, compared with 0.1071 for fixed-width quantization, and improves ¹H Top-1 from 0.0141 to 0.0281. We use controlled coordinate jitter to isolate why this representation transfers. MRT reduces degradation relative to adaptive quantization, but the absolute comparison depends on perturbation scale: at the smallest tested jitter, post-perturbation kNN is tied on ¹H and IR and lower on ¹³C; by jitter comparable to the finest residual scale, MRT overtakes adaptive in absolute kNN across all three coordinates. For example, at ρ = 1.50 on ¹H, ρ = 1.00 on ¹³C, and ρ = 1.00 on IR, MRT is ahead by 0.0243, 0.0202, and 0.0032 kNN Tanimoto, respectively. Paired bootstrap intervals on the jitter degradation difference exclude zero at every tested NMR and IR jitter level. A packed-token ablation removes most of this advantage, supporting the role of explicit level-wise geometry. These results identify a tradeoff between clean semantic geometry and stability in spectral tokenization.