PageGauge: Attention-Factorized KV Quantization for Faster Long-Context LLM Decoding

KV-cache compression reduces memory traffic during long-context LLM inference, but conventional affine quantization still requires per-element reconstruction inside the attention path. PageGauge co-designs the quantized representation with softmax attention so that shared key centers cancel as a constant logit shift, shared value centers are restored once after normalization, and scalar page/head scales are applied at fragment granularity around the main contractions. The resulting factorization is exact in real arithmetic relative to the reconstructed mixed-precision cache. We implement PageGauge in a stateful paged decoder using INT8 historical KV, exact FP16 regions, online-softmax merging, and architecture-specific GPU scheduling. On Mistral-7B-v0.3 with a 20,480-token prefix, PageGauge reduces served KV storage by 36.86%. Eight-block fresh-process measurements show sustained complete-decoder speedups of 1.1326x on an RTX 5090 and 1.1093x on an A100-SXM4-40GB relative to optimized FP16 FlashInfer. Independent post-freeze PG-19 validation gives minimum logit cosine 0.999231 and top-1 agreement 0.998264, while runtime-appended pages are finalized, quantized, and later reread through the INT8 path.