Do Emotion-Aware Agents Know When They Are Wrong? Distilling SLM Trust for Real-Time Perception

Emotion-aware agents require both an estimate of a user’s affective state and an indication of how reliable that estimate is. However, invoking a language model at every interaction can introduce substantial latency. In this paper, we present TRUSTDISTIL, an offline teacher–student pipeline for continuous speech emotion recognition (SER) that separates prediction distillation from trust distillation. To obtain the acoustic representations, we used a frozen Wav2Vec2-Large-Robust encoder. These features are then reduced by PCA and provided to a lightweight teacher VAD predictor. The same representation is used by a compact student, which learns the teacher’s VAD predictions without access to ground-truth labels. Qwen2.5-0.5B-Instruct uses the teacher predictions and their uncertainty to gen- erate trust targets offline. These targets are distilled into the student. Our system was evaluated on a benchmark dataset, IEMOCAP. Prediction distillation retains 97.98%. The mean concordance correlation coefficient (CCC) achieved by the teacher was 0.6650, while the student achieved a Mean of CCC score 0.6515. In contrast, applying the distilled trust signal consistently reduces performance, with mean CCC decreasing from 0.6515 without trust to 0.5919 at full trust strength. The predicted trust values also collapse into a narrow range around 0.75, suggesting that they behave more like a global shrinkage factor than a discriminative confidence estimate. Nevertheless, offline distillation provides a substantial computational benefit: the student trust head is approximately 24,553 times faster than a per-turn Qwen2.5-0.5B-Instruct query. These results show that an SLM-derived trust signal can be efficiently distilled for real-time use, but efficient trust estimation does not necessarily imply useful correctness-aware calibration.