One Order, One Vote: How GRPO's Reward Normalization Reweights an Execution Desk, Measured Exactly

Reinforcement learning is now used across quantitative finance, from execution and market making to hedging and to the LLM agents that increasingly assist quantitative research, and its most popular modern recipe, Group Relative Policy Optimization (GRPO), standardizes rewards within each training instance before every update. We show that this default quietly changes what a trading desk optimizes. An execution desk that trains one liquidation policy across its order book cares about total, dollar-weighted cost, so large and risky orders should dominate training. GRPO’s update is invariant to each order’s cost scale: every order gets one vote, much like an inverse-volatility (naive risk-parity) weighting of the book. We make this precise in the Almgren-Chriss model of optimal execution, where the expected cost of any policy, its gradient, its variance, and the optimal schedule are all available in closed form, so we can compute exactly which policy GRPO settles on and what it costs the desk. In a noise-free benchmark where a single schedule model must serve every order, the exact GRPO limit costs the desk 5.0% more than its optimum, and trained GRPO raises total risk-adjusted execution cost by 12.7% (95% CI 4.6-19.6%) relative to an unbiased estimator (RLOO). The loss falls on the three most urgent orders, which carry half of the desk’s cost, while the average order does no worse. The theory identifies which orders pay (correlation 0.998, or 0.993 without the three largest losers); its prediction holds at practical group sizes, though trained GRPO overshoots it on the most urgent orders. With market price noise the effect shrinks to +2.1% (not significant), in line with the exact prediction of +2.6%. Normalizing once across the whole book removes the effect, and pass/fail execution benchmarks largely neutralize it. Whether execution quality is judged per order or per dollar is a business decision; it should not be made silently by an optimizer default.