Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Speculative decoding (SD) accelerates large language model inference through a \textit{draft-then-verify} paradigm. To improve acceptance, recent methods build expansive draft trees, but these trees incur substantial VRAM bandwidth and computation overhead that limits end-to-end speedup. Dynamic-depth pruning reduces this latency by halting expansion at low-confidence branches, yet it may also discard valid continuations and lower the accepted length below the dense-tree upper bound. We identify a resource-allocation opportunity: pruning frees computational budget that can be repurposed. To exploit this budget, we introduce \method, a compensation framework that \textbf{couples pruning and retrieval as mutually reinforcing operations.} Pruning supplies budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. Through a sequential `prune-then-graft' mechanism, \method attaches predictive retrieved tokens into slots left by early-stopped drafting, filling topological gaps with near-zero overhead. \method is entirely training-free and lossless. Comprehensive evaluations show that \method establishes a \textbf{new Pareto frontier} across various models and benchmarks. On short-context benchmarks, it achieves up to \textbf{5.41$\times$} speedup and improves average speedup over EAGLE3 by up to \textbf{21.8\%} on Qwen3-235B. On long-context, \method outperforms EAGLE3-64K by \textbf{16.6\%} on Qwen3-14B. We also explore applying \method to the DFlash-style block drafting paradigm.