When Tools Fail, Agents Fabricate: Silent Failure and Verification-First Oversight in Tool-Using LLM Agents
Gün Kaynar ⋅ Boshen Yan ⋅ Carl Kingsford
Abstract
Tool-using LLM agents must recover when a tool call fails, by retrying, switching tools, or revising their plan. We study how agents fail when tools fail, using RetryRealistic, a sandboxed yet realistic testbed for controlled fault injection in which agents operate on a real OS filesystem with genuine permission enforcement, SQLite databases, a deterministic local HTTP server, and a constrained shell, across 8 tools, 9 fault types with controlled persistence, and 30 multi-step tasks. Evaluating four agents over 2{,}100 fault-fired runs against ground truth, we find that recovery is unreliable (27--54\% correct) and frequently silent: 19--30\% of runs end in a confident fabrication, a wrong answer asserted as fact. The failure mode is governed by the fault type rather than the model: faults that return partial or misleading data drive fabrication (up to 46\%), while hard denials drive honest failure. The mechanism is what we call \emph{silent ingestion}: the agent accepts faulted data as valid on first contact without retrying or flagging the problem, then builds a fabricated answer on top of it. We show that fabrication is reducible: a verification-first Monitor, grounding checks in structured tool outputs and issuing corrective directives rather than hard blocks, raises correct recovery from 42.5\% to 48.0\% and cuts fabrication from 24.7\% to 20.3\%. Removing chain-of-thought increases fabrication ($+7.7$pp), so deliberation guards against silent ingestion. We release RetryRealistic, the fault-injection toolkit, and all annotated traces.
Chat is not available.
Successful Page Load