MarketScientistBench: Measuring the Decision Value of Experiments for LLM Agents

Does an AI agent make better decisions when it can experiment first? Answering this requires a measure of the losses its experiments help avoid. We introduce MarketScientistBench, a market laboratory that connects experimental evidence to counterfactual intervention utilities. An agent tests interventions in one market realization and chooses a leverage cap or liquidity backstop for an independently seeded target. Regret is computed by the simulator, without human or LLM scoring. A prospectively specified study on 24 new worlds finds that experimental access reduces Luna’s mean regret from 8.878 to 3.785, a paired reduction of 5.092 units (nominal 95% interval: 1.028–9.507). Two other deployments have incomplete prospective comparisons; all four deployments improve descriptively in a separate exploratory study. Increasing laboratory adaptation from fixed to evolving strategy shares yields no established additional benefit. MarketScientistBench makes the value of experimental access measurable as avoided decision loss, and exposes whether additional simulator detail improves the decision it is meant to support.