PAIR: Perceptual Affective Inference and Regulation in a Real-Time Multimodal Conversational Agent

Sustained emotional support requires generative agents to connect momentary emotion inference and regulation with continuity across encounters. We present PAIR (Perceptual Affective Inference and Regulation), a real-time multimodal generative agent that uses appraisal reasoning to infer emotion from event narrations and select regulation guidance. Guidance unfolds through conversation with coordinated speech, color, and avatar cues, while rolling memory carries context and feedback into subsequent encounters. In a 14-day deployment with 19 participants, 1,093 sessions paired initial and post-guidance estimates with unanchored self-reports, complemented by feedback and interviews. Text-based inference tracked valence and dominance more closely than arousal, and conversations were followed by the largest valence gains at negative initial emotional states. Participants described feeling understood through contextual clarification and emotional acknowledgment before advice. Across encounters, participants linked personalization to relevant recall, updated context, and persistent corrections, while accumulated records and familiar dialogue supported reflection and companionship. These findings position sustained emotional support as an ongoing process of connecting momentary inference and regulation with conversational fit and continuity across encounters, examined through per-event, first-person evaluation.