ETA: Exit on Tool Arrival in Speculative Agent Loops
SAPTARSHI MITRA ⋅ Xuan Zhu ⋅ Rohith Kukkala ⋅ Sitao Huang ⋅ Karthik Vijayan
Abstract
Tool-using language agents in explicit thinking modes generate long chains of thought before acting, making each turn inherently serial: later tokens depend on the tool observation. Recent speculative tool dispatch methods overlap tool execution with reasoning, but offers limited benefit when reasoning dominates latency. We present ETA, that treats the speculative fork not as an actuator but as a sensor of the model's decision state, using it to terminate the reasoning of the main decode. It truncates reasoning only when the prediction is stable, agrees with the dispatched call, passes a free probe measuring whether a call will be emitted at all, and the tool result has arrived: a deadline that bounds per-turn regret by the remaining decode cost. ETA achieves speedups of $1.33\times$ on $\tau^2$ airline/retail, $1.26\times$ on $\tau^2$ telecom, and $1.13\times$ on BFCL v4 multi-turn. On $\tau^2$, it decodes 27% fewer tokens at a nearly unchanged per-token rate without dropping action quality.
Chat is not available.
Successful Page Load