SpecRouter: Learned Cost-Aware Selection Among Heterogeneous Speculative Drafters

Speculative decoding accelerates language-model inference by drafting tokens cheaply and verifying them in parallel. The available drafting mechanisms are heterogeneous --- retrieval, small draft models and learned tree drafters differ by an order of magnitude in what a proposal costs and disagree about when it is accepted --- and deployments typically fix one of them. Yet the mechanism that pays off changes \emph{within} a single generation: a retrieval drafter is nearly free while the model quotes its prompt and worthless once it starts composing, and a learned tree drafter behaves in the opposite way. We present SpecRouter, a compact router that selects among heterogeneous drafters before every draft--verify block. It predicts, from the target model's own hidden state, how many tokens each mechanism would get accepted, prices each prediction with a calibrated cost table, and takes the action minimising cost per committed token; reading the target state is what lets it compare mechanisms without running them. On sixteen workloads and two 8B targets, SpecRouter reaches $2.90\times$ and $2.95\times$ over autoregressive decoding in end-to-end wall-clock time, beating the best single mechanism by $29.8\%$ and $26.6\%$.