Would Your LLM Trader Beat a Placebo? Displaced-zone controls, power limits, and evidence contracts for finance agents
Abstract
Backtests of trading agents usually compare a final strategy with cash or a broad benchmark. That comparison does not test whether the information cited by the strategy matters. We introduce a matched placebo for level-based trading: replay the same trigger, stop, target, timing and costs after moving only the cited price level. Applied to a session-extreme reversal rule over 225 Nasdaq-100 futures sessions, the test finds no usable location effect. Across 2,491 development signals, the rule returns -0.112R at a fixed 1R target. Displaced zones return -0.139R, leaving a real-minus-placebo estimate of +0.027R with a 95% interval of [-0.038,+0.087]. Its upper endpoint is below the median round-trip cost of 0.117R. A later validation gain of +167.75 points is also fragile: one session supplies 80% of it. The frozen high/low-of-day band contains the realized extreme in only 1–2% of development sessions, and three numerical outcome models fail to establish positive expectancy. The paper contributes the displaced-zone gate, a power budget for short evaluations, and an evidence contract that ties each agent claim to information available at decision time. The same gate is designed for language-model trading agents: given matched real and placebo packets, an agent whose decisions do not change with the cited level is using information it does not need, whatever its explanation says. We specify that test, the evidence contract and a preregistered sample size, so the first agent evaluated under them can be credited or rejected on terms fixed in advance.