Disaggregated Quantization: Specializing LLM Prefill and Decode

The prefill and decode phases of LLM inference reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computational formats, weights and storage placement to both of these phases. Starting with format disaggregation, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Moving to full disaggregation, training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while exceeding weight-only accuracy on both decode-heavy and prefill-heavy tasks for 2-3-bit quantization. To accommodate the extra prefill checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. Using released Qwen3.8-27B GGUF decoders, training an auxiliary NVFP4 prefill model in a fully-disaggregated fashion improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro, without modifying the decode checkpoint. Combined with ODP, it delivers a 1.78x prefill speedup over the weight-only baseline at 8K prompt length in llama.cpp at no device memory overhead. Additional PTQ validation includes models with up to 2.8T parameters.