Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

In-context learning allows transformers to adapt from examples in a prompt without parameter updates. We ask whether this capability extends from prediction to data generation. We prove that frozen transformers can exactly realize closed-form and smoothed closed-form diffusion samplers from in-context samples: softmax attention computes Gaussian-mixture responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. In pretrained language models, normalized hidden states exhibit a two-stage geometry, approaching a uniform spherical reference in intermediate layers and returning to structured representations near the output. The corresponding interacting-particle energy has the same U-shaped profile. Motivated by this observation, we further prove that transformers can approximate Estimation-Free Sampling (EFS), an energy-based forward--backward particle sampler.