Breaking the Bottlenecks of Autoregressive LLM Inference
-From INT4 KV-Cache Compression to Multi-Token Prediction and DFlash

Autoregressive language-model inference is increasingly constrained by two systems bottlenecks: a KV cache that grows with context length and token-by-token decoding that limits hardware utilization. This talk examines complementary techniques for addressing both constraints.

We begin with KV-cache quantization and compression, focusing on INT4 representations that reduce memory capacity and bandwidth requirements in long-context and high-concurrency workloads. We then examine methods that generate more than one useful token per target-model step. Conventional speculative decoding uses a lightweight draft model to propose candidate tokens for parallel verification. Multi-Token Prediction extends this idea by using auxiliary prediction modules associated with the target model to propose several future tokens, reducing the need for a separate drafter. We also discuss DFlash, which replaces sequential autoregressive drafting with a lightweight block-diffusion drafter, conditioned on target-model features, to propose an entire candidate block in parallel
The talk concludes with real performance numbers and experimental results, evaluated through a unified framework that covers time to first token, inter-token latency, throughput, memory footprint, energy consumption, acceptance behavior, and output quality.