Natural-Context Redirection in Decision Models

Dedicated decision models such as Jev map unstructured language to probability distributions over finite choices, allowing their outputs to route requests, select tools, and trigger actions. Yet real-world inputs rarely arrive in isolation: they come with background details and surrounding context. We find that short additions that fit naturally into this context can redirect an otherwise correct decision, even when the correct answer remains unchanged. We study this natural-context redirection by fixing a wrong target option for each initially correct item and using option probabilities to refine fluent additions that preserve the source, question, choices, and gold answer. Within 64 accepted target evaluations, the optimizer identifies contexts that redirect Jev on 312 of 508 initially correct decisions (61.4%); in 229 cases, Jev assigns at least 0.7 probability to the fixed wrong option. Across seven datasets, three additional decision systems show targeted flip rates of 64.9%–73.2% on their initially correct decisions. These findings expose fragility in current decision models: short, ordinary-looking context can shift a correct choice to a high-confidence wrong one, raising concerns about treating their probability outputs as reliable decision interfaces.