Do Coding Agents Keep Their Word? Design-Commitment Drift and Context Management in Multi-Turn Repository Work
Abstract
Software engineering with coding agents is increasingly becoming conversational. Software requirements arrive over the course of development across multiple turns, agents may make design choices as they work, and later arriving requirements can force earlier design choices to evolve. Existing coding benchmarks usually evaluate the final state of code for test accuracy (pass/fail), which makes it difficult to differentiate between cases where design decisions are and aren't abided by. We introduce design-intent fidelity, a framework that treats any design commitments stated by the agent as checkable constraints against produced code. We apply this novel framework to eight SWE-bench Verified tasks across 144 agent trajectories, with 1,137 design commitments retained for analysis, and find that final commitment violations can arise in two different ways: some commitments are never realized in code, while others are initially satisfied and later changed. Among these later design changes, most reflect changes made to satisfy later requirements or explicit design revisions, while unacknowledged design revisions are comparatively less frequent. We also test various context-management strategies to expose the agent to information from its own earlier design decisions, and introduce DCC: expose the agent to a ledger of its own prior commitments. Among the multi-turn conditions, DCC has the lowest observed design-change rate (1.6%), compared with the other evaluated strategies (6.9-10.6%).