Cue2Act: Contract-Bounded Vision-Language Coaching for Novice Drivers Under Scene Ambiguity and Model Uncertainty
Elahe Yahyapour ⋅ Mahmoud Pourmehrab ⋅ Chengbo Ai
Abstract
New drivers often find spotting a hazard simpler than deciding how to respond. A pedestrian near a crosswalk, a closing gap, a car hidden behind a truck, or an unreadable signal can demand action before the threat is obvious. Most driver-assistance work targets perception or vehicle control. Less thought goes to the person behind the wheel, who must read the situation, choose between slowing and stopping, and understand why. We present \textbf{Cue2Act}, a contract-bounded vision-language coaching framework that turns scene cues into a compact recommendation, \textsc{Proceed}, \textsc{Slow}, or \textsc{Stop}, with a brief explanation aimed at novice drivers. Cue2Act separates two functions that VLM-based driving systems often blur. A frozen vision-language model, Qwen2.5-VL, handles semantic interpretation: it identifies salient cues and writes a readable recommendation card. A deterministic physical governor enforces a safety floor using time-to-collision, gap size, stopping distance, right-of-way, crosswalk presence, traffic-light state, and occlusion. The VLM can add context and explanation but cannot override the physical contract. This lets us ask a narrower question: not whether a VLM can replace physical reasoning, but whether VLM reasoning can be useful without ever issuing unsafe coaching advice. That distinction matters for the benchmark results. The physics-policy labels come from the same safety rules the governor applies, so near-perfect rule-only performance is expected, we treat it as a safety-envelope reference, not a learned baseline. A gradient-boosted model using physical features alone also clears 98\% accuracy on both the nuScenes-derived benchmark and the PhysicalAI stress test, confirming that most label structure is recoverable from scene geometry. Cue2Act's point is not to beat physics on physics-derived labels. It is to keep the safety envelope intact while adding grounded language reasoning, uncertainty-aware caution, and explanations that rules alone cannot supply. \vspace{0.35em} \begin{center} \scriptsize \setlength{\tabcolsep}{3.2pt} \renewcommand{\arraystretch}{1.0} \begin{tabular}{llccc} \toprule \textbf{Dataset} & \textbf{System} & \makecell{\textbf{Unsafe}\\\textbf{\textsc{Proceed}}} & \makecell{\textbf{Safety}\\\textbf{cost}} & \makecell{\textbf{Readable}\\\textbf{explanation}} \\ \midrule NuScenes-2K & Unconstrained VLM & 44.6\% & 2486 & Yes \\ & Rule-only floor & 0.0\% & 7 & No \\ & \textbf{Cue2Act} & \textbf{0.0\%} & \textbf{13} & \textbf{Yes} \\ \midrule PhysicalAI-500 & Unconstrained VLM & 61.0\% & 312 & Yes \\ & Rule-only floor & 0.0\% & 0 & No \\ & \textbf{Cue2Act} & \textbf{0.0\%} & \textbf{8} & \textbf{Yes} \\ \bottomrule \end{tabular} \end{center} \vspace{0.15em} The numbers show why the safety boundary is necessary. When Qwen2.5-VL reasons without the contract, it often recommends \textsc{Proceed} where physical margins require intervention. Adding ambiguity awareness alone trims some errors but does not reliably remove this failure mode. With the governor enabled, unsafe \textsc{Proceed} recommendations drop to zero on both benchmarks. Remaining errors are mostly overcautious \textsc{Slow} or \textsc{Stop} calls rather than missed interventions, the intended trade-off for novice-driver coaching. Safety cost serves as an asymmetric metric penalizing missed interventions more than unnecessary cautions, sensitivity analysis reaches the same conclusion because Cue2Act removes the dominant cost term. To avoid relying only on rule-derived labels, we ran expert audits. Three experienced drivers and traffic-safety professionals independently labeled 120 difficult NuScenes scenes, chosen to be hard rather than representative. Experts reached a clear majority on 117 of 120 scenes, but pairwise agreement sat at only 50.0-60.8\%, a reminder that borderline driving judgment genuinely varies even among experienced raters. Against the expert majority, Cue2Act matched 62.4\% of decisions and missed only 3 of 103 scenes where experts called for \textsc{Slow} or \textsc{Stop}, on the unanimous subset, it missed 0 of 37 required interventions. In a separate 30-scene explanation audit, three experts rated Cue2Act's cards as moderately faithful ($3.54 \pm 1.11$/5), helpful ($3.54 \pm 1.02$/5), and suitable for novice drivers ($3.33 \pm 1.08$/5), with 74 of 90 ratings indicating the guidance was not excessively conservative. Cue2Act does not claim a language model can out-predict a physical safety rule on a benchmark built from those rules. The claim is narrower and more useful: a generative VLM can sit inside a hard physical safety boundary and still add what rules alone cannot, scene-grounded explanations, uncertainty-aware caution, and coaching language for human drivers. The expert disagreement also points toward future personalization: not changing the safety floor, but adapting how and when guidance is delivered over time.
Chat is not available.
Successful Page Load