Are Diffusion Language Models Invertible?
Preeti ⋅ KIRAN RAVISH ⋅ Ankita Kushwaha ⋅ Pawan Kumar
Abstract
Deterministic language models can be injective at the hidden-state level, but diffusion language models insert a stochastic corruption channel between clean sequences and observed states. We formalize this distinction through four notions of inversion: distributional injectivity, single-shot exact recovery, pathwise recovery with revealed forward randomness, and denoiser injectivity. We then prove a sharp separation. Positive-noise channels are generally not exactly invertible from one sample; for masked diffusion, revealing the full mask pattern restores recovery only on the no-mask event, while a closed-form Bayes lower bound and an exact repeated-sample coverage law quantify how much information the channel preserves. In contrast, deterministic Transformer denoisers are generically injective on finite noisy-token domains under analytic residual assumptions, so denoiser injectivity and channel invertibility are logically distinct. The separation has direct deployment consequences: noisy diffusion states and deterministic interfaces (hidden states, embeddings, output distributions) are different audit surfaces, masked corruption remains structurally harder to invert than additive corruption even with full side information, and repeated-query interfaces admit a closed-form sample budget for the coverage event under which independent corruptions become uniformly exactly recoverable. We package the theory into a two-stage audit protocol with theorem-driven channel certificates and a finite-domain hidden-state margin scan, and validate it with synthetic recovery experiments and representation audits on GPT-2, MDLM, RADD, LLaDA-8B, and Dream-7B, where empirical recovery curves match the closed-form predictions within mean absolute deviations below $10^{-3}$ and maximum deviations below $8\times 10^{-3}$, and verified hidden-state collision counts are zero across all audited layers.
Chat is not available.
Successful Page Load