Passing Without Repairing: Verifier Blind Spots in a Live SRE Agent Benchmark

Agentic benchmarks report a single number, and that number is only as meaningful as the verifier that produced it. Recent audits of code, web, and terminal benchmarks recommend a simple diagnostic: run an agent that takes no action, and treat any nonzero score as a defect. We apply that check to a class it has not been applied to, a live-system benchmark whose verifier reads Kubernetes cluster state, and then go further, because real agents act and act imperfectly. On the missing_service problems of a recent live SRE benchmark, an agent that does nothing is graded a success: across eighteen faulted-state measurements spanning two applications and three provenance categories, including the benchmark's own conductor, the stock mitigation oracle reported success on every unrepaired system it was asked about, while the benchmark's own workload generator was failing 9.86% and 59.74% of user requests. The cause is structural: the oracle observes only Deployment replica counts and Pod readiness, while the injector deletes the Service and then, as its final act, restarts every pod in the namespace and waits for stability, restoring the exact surface the oracle measures. We then mutation-test the verifier with three operational Kubernetes mutants, each modelling a plausible but wrong repair and each frozen with its expected outcome under a third-party RFC 3161 timestamp before execution, run three times apiece. The stock oracle accepted all nine faulted states; an independently frozen operational contract rejected all nine and separated the mutants by four, three and two violated invariants. The mutant that best models a wrong repair left about a tenth of the healthy system's successful responses and received the same verdict as one that left nine tenths. A census of the 123 registered problems identifies at least eight whose oracle cannot observe the resource its injector perturbs, and we sketch a fix built from machinery the benchmark already contains, pre-registered and then evaluated as an in-process prototype on the live cluster: attached in the obvious way it rejects correct repairs, and only once the attribute it reads is supplied does it accept every control while rejecting all three mutants.