When Verifiers Drift: Benchmarking Safety Critic Faithfulness Under Tool-Ecosystem Growth

LLM-based critics are widely deployed as safety guardrails for agentic systems, yet their reliability under realistic operating conditions remains underexplored. We demonstrate that critics calibrated against a fixed system snapshot degrade as tools are added and application domains expand, with no external signal indicating weakened detection capability. We formalize this problem along controlled axes and introduce DRIFT (Detecting Reliability Issues From Tool-scaling), a benchmark for measuring critic robustness under environment change. Building on SafeToolBench's multi-domain tool ecosystem, we construct verified cross-app composition scenarios, both violating and compliant, where policy violations emerge only from data flow across application boundaries, and evaluate 9 models across 6 provider families on a two-dimensional scaling matrix separating intra-app noise from cross-app noise. We find that (1) all 9 models degrade under cross-app scaling, with UPR increases ranging from +0.083 to +0.296, (2) tool scaling erodes critic faithfulness: missed violations rise while false blocks simultaneously fall, a loss of judgment that standard accuracy metrics cannot detect, and (3) cross-app noise consistently dominates intra-app noise, with cross-app deltas positive for all nine models while intra-app deltas are small and directionless. We release DRIFT as an open benchmark to help practitioners evaluate and monitor critic robustness as tool ecosystems evolve.