A Latency Budget for LLM Inference: Predicting Optimization Benefits From a Single Trace

There are several ways to optimize a LLM inference workload: capture CUDA graphs, fuse the operators with a compiler, quantize the weights or the KV cache, or write better kernels. Each takes days of engineering, and a profiler does not help you choose, because it reports where the time was spent rather than how much a particular change would improve performance. We describe a latency budget that estimates the savings in advance. It takes one profiler trace of the unoptimized workload plus four constants measured once per GPU, and splits the measured forward latency into six terms, each corresponding to one optimization and carrying an estimated saving. No term is measured by applying the optimization it recommends: idle time comes from the gaps between kernels in the trace, not from timing a CUDA graph. We test it on 72 predicted/measured pairs across six decoder stacks and vLLM serving Qwen3-4B, applying 4 optimizations one at a time. Two of the four estimates are accurate: CUDA-graph capture to a median of 207 μs and fusion to 228 μs, which is 7.7× and 3.4× closer than baselines fitted on the test measurements themselves. The estimates for fp8 weights and an fp8 KV cache are not, and a constant beats both, because the budget counts the traffic removed but not the overhead added. We report those two terms as a limitation. The budget also reproduces three effects we did not build into it, including that quantizing before the host is off the critical path returns nothing at all.