Rank-Preserving, Threshold-Breaking: Post-Training Silently Decalibrates Inherited Safety Monitors in VLA Policies
Aahlad Bysani
Abstract
Runtime failure detectors for vision-language-action (VLA) policies are lightweight probes over internal features, deployed with an alert threshold calibrated by conformal prediction. Recent work in the language-model setting shows that such monitors go stale when the underlying model is updated, and measures staleness as a drop in probe AUC; under that criterion, quantization is benign. We show that in the VLA setting the failure takes a different form that this criterion cannot see. Using matched fp32 and simulated 4-bit weight-quantized checkpoints of SmolVLA on LIBERO-Spatial (200 episodes, 6,367 activation pairs collected under identical observations), we find that quantization leaves detection AUC unchanged to three decimals ($0.766 \rightarrow 0.765$) while realized conformal coverage of the inherited threshold falls from $0.77$ to $0.41$ against a nominal target of $0.90$ — a target our fp32 baseline does not itself reach, for reasons we detail. The cause is that quantization approximately translates the score distribution while preserving its ordering: mean predicted failure probability moves from $0.68$ to $0.03$ at one horizon and from $0.71$ to $0.88$ at another, so a fixed threshold under- or over-fires with uncontrolled sign. Across eight layer–horizon settings, coverage shifts significantly in five, by up to $0.67$, while AUC never moves by more than $0.19$ and usually by less than $0.05$. Applying the label-free activation realignment proposed for language models, we recover coverage to within $0.06$ of the original checkpoint in all eight settings and to statistical indistinguishability in six. We additionally find that 8-bit quantization leaves the VLM text stream numerically untouched while perturbing the action expert two orders of magnitude more — an architectural asymmetry with no language-model analogue. Our conclusion is a measurement one: AUC does not certify that an inherited monitor survived a policy update, and realized coverage should be reported alongside it.
Chat is not available.
Successful Page Load