An AI shopping assistant can find the right product and still fail the shopper: miss a stated constraint, recommend something that's out of stock, or lose the thread across a dozen turns and tool calls. At Shopify, we're developing an approach to turn these failures into training signal for open-source models tailored to Shop Agent and our other commerce assistants. This talk outlines the foundations: the agent environment, multi-turn groupwise rollouts, and an evaluation framework that distinguishes a well-written response from a completed task. Deterministic checks assess tool-call correctness, consistency with live catalog and inventory data, and structured UI validity, while rubric-based LLM judges assess groundedness, interaction quality, and whether the shopper's goal was met.

Our adaptive learning loop uses the distribution of production failures to shape hybrid rewards and guide reinforcement learning and on-policy self-distillation. We examine how to assign credit across decisions, detect and mitigate reward hacking, test transfer to unfamiliar tasks, and preserve existing capabilities through repeated policy updates. Finally, we outline how to decide when an updated policy is ready for production, weighing offline improvements against serving cost and latency, and how online metrics can inform subsequent training cycles.