Multimodal Foundation Models for Group AI Assistants

Collaboration drives progress in nearly every field, yet its quality is often invisible to the people involved, or becomes apparent only after it is too late. AI assistants could help address this blind spot, but today’s assistants primarily support individuals, and approaches for supporting groups remain limited. This gap persists in part because group behavior is dynamic and relational, and cannot be reduced to the sum of its members. We propose multimodal, foundation-model methods for group AI assistants that sense group state, predict how coordination is likely to unfold, and intervene to help steer the group. We represent group state with three sociograms that capture conversation, proximity, and shared attention, and predict the next 32-second interaction structure. An LSTM baseline saturates at 0.29 weighted Jaccard similarity. Encoding group state as natural language at the individual, group, and temporal levels shows that performance depends strongly on modality. Conversation reaches 0.958 structural similarity, a 3.2x improvement over the baseline, while shared attention remains at 0.00 across configurations and proximity degrades under fine-tuning. Ongoing work explores visual encodings and structured representations of group interactions as model input, and adds a human-in-the-loop intervention policy.