Evaluating Clinical Safety of Multimodal LLMs Across Text and Speech in Low-Resource Languages.

In many rural settings, Community Health Workers (CHWs) are the first—and often only—point of care, yet they frequently face cases beyond their training and cannot wait for delayed physician support. LLMs are increasingly being explored to fill this gap, but their safety is typically evaluated on English, text-only benchmarks that poorly reflect multilingual, spoken primary care. We study this gap using more than 20,000 expert ratings of LLM responses to 7,000+ spoken and written point-of-care questions from 283 CHWs in Nigeria across five languages. Reliability drops sharply under real deployment conditions: low-resource languages introduce a \textbf{language tax}, with harm severity rising up to 8--10$\times$ relative to English, while speech adds an \textbf{audio penalty} that further increases unsafe responses. Similar degradation appears in hallucination, factuality, and bias, translating into frontline risks such as unsafe triage, missed referrals, and misplaced reassurance. These findings suggest that standard benchmarks overestimate deployment readiness and motivate safety evaluations that jointly account for language, modality, and clinical risk.