Does the Verifier Listen? Stress-Testing Counterfactual Acoustic Necessity for Adaptive Speech Reasoning

# Does the Verifier Listen? Stress-Testing Counterfactual Acoustic Necessity for Adaptive Speech Reasoning **Keywords:** speech-language models; acoustic grounding; counterfactual verification; test-time scaling; self-distillation Speech-language models can produce fluent reasoning while relying on lexical priors instead of the speech signal. This creates a missing control variable for test-time scaling: before allocating or removing inference compute, can we verify that a model's decision actually depends on the relevant acoustic evidence? Answer agreement alone cannot resolve this question because repeated traces may reproduce the same language shortcut. We study whether counterfactual acoustic necessity is identifiable enough to support safe adaptive reasoning. Given speech $x$, question $q$, trace $r_t$, answer $y_t$, and a trace-cited interval $S_t$, we compare an attribute-targeted intervention $T_{t,m}$ with an equal-duration matched control $C_{t,m}$. For each valid intervention $m$, the necessity margin is $$N_{t,m} = \\log p_{\\theta}\\!\\left(y_t \\mid q, C_{t,m}(x)\\right) - \\log p_{\\theta}\\!\\left(y_t \\mid q, T_{t,m}(x)\\right).$$ A positive margin means that removing the cited attribute reduces support for the answer more than a matched control. This comparison is designed to separate evidence removal from generic perturbation sensitivity. **Pilot evidence.** We ran a pre-specified, five-fold cross-fitted study with Phi-4 Multimodal on 200 stratified VoxParadox examples spanning ten paralinguistic tasks, where lexical content conflicts with acoustic ground truth [1]. Fixed-eight self-consistency achieved only 23.5% acoustic accuracy while agreeing with the misleading lexical label on 56.5% of items. More importantly, the targeted-minus-control verifier was uninformative (AUROC 0.500; 95% cluster-bootstrap CI 0.329–0.660), and only 12.8% of traces admitted both a valid targeted intervention and control. A necessity-aware sequential controller reduced end-to-end GPU time by 48.3%, but accuracy fell from 23.5% to 14.5% (−9.0 points; 95% CI −16.0 to −2.0). Thus, fewer traces can conceal systematic under-thinking when verifier reliability or coverage is weak. **Proposed direction: Identifiability-Gated Efficient Acoustic Reasoning (I-GEAR).** We will replace single, often off-manifold perturbations with ensembles of transcript-preserving interventions that modify only the queried acoustic attribute (e.g., prosody, pitch, rate, emphasis, or emotion), paired with duration-, speaker-, and spectrum-matched controls. For $M_t$ valid interventions, I-GEAR uses a conservative lower confidence bound $$L_t = N_{t,avg} - z_{1-\\delta}s(N_t)M_t^{-1/2},$$ rather than trusting a single necessity estimate. A reliability-and-coverage gate $G(x)$ opens only when held-out lower bounds on verifier discrimination and target-control effect exceed predeclared thresholds and the current item has sufficient valid interventions. Verified votes then enter a sequential probability ratio test [2], while the compute policy is null-safe: $K(x)=K_{adaptive}(x)$ if $G(x)=1$, and $K(x)=K_{max}$ otherwise. The model therefore receives the full reasoning budget whenever acoustic dependence is not demonstrably identifiable. Finally, only acoustically verified on-policy traces will be distilled, using a contrastive margin that rewards answers supported by the original speech more than by its counterfactual. We will test I-GEAR across model families and held-out paralinguistic tasks using verifier AUROC and coverage, acoustic-ground-truth accuracy, lexical-shortcut agreement, and measured end-to-end GPU time. Success requires a Pareto improvement over fixed-budget self-consistency [3]: non-inferior accuracy with lower compute, not trace reduction alone. The pilot therefore supplies both a falsifiable motivation and the safety criterion for the proposed algorithm: adaptive speech reasoning should stop early only when its verifier has first demonstrated that it listens. ## References [1] J. Pang, A. Chaubey, and M. Soleymani. *Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox.* 2026. [2] A. Wald. “Sequential Tests of Statistical Hypotheses.” *Annals of Mathematical Statistics*, 1945. [3] X. Wang et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” *ICLR*, 2023.