Evaluating and Enhancing Large Language Models for Amharic Maternal and Child Health Question Answering

Large Language Models (LLMs) have shown remarkable promise in medical question answering and patient education, yet these advances remain disproportionately concentrated on high-resource languages, leaving billions of low-resource language speakers underserved. Amharic, the official language of Ethiopia with over 30 million native speakers, exemplifies this disparity: its rich root-and-pattern morphology, complex affixation, and limited annotated data make it especially challenging for NLP, while Ethiopia's maternal and child health (MCH) outcomes remain among the most critical in the world due to limited healthcare access, provider shortages, and language barriers. Culturally and linguistically sensitive conversational AI could help close these information gaps for pregnant women, mothers, and caregivers, but no benchmark exists to evaluate whether current LLMs are safe or effective for Amharic MCH question answering; existing multilingual QA datasets (TyDi QA, XQuAD, MLQA) lack MCH specificity, and existing medical QA datasets (PubMedQA, MedQA) are English-centric. We introduce the first expert-reviewed QA benchmark for Amharic MCH, combining 150 locally collected and expert-validated QA pairs, 300 translated English public-health records (verified via SSA-COMET and manual review), and 2,000 synthetic QA pairs generated via the Gemini API and manually inspected. Using EleutherAI's lm-eval harness, we evaluate eight general-purpose and medical-specialized open-source LLMs under English, Amharic, and mixed-language prompting, measuring multiple-choice accuracy (MCQA) and generative fluency (chrF++). Our results reveal a substantial cross-lingual gap: every model's performance drops sharply moving from English to Amharic, showing that medical knowledge learned in high-resource languages transfers poorly. Prompting language matters differently across tasks: English instructions recover much of the lost MCQA accuracy (e.g., Gemma-3-27B-IT rises from 48.22% to 96.89%), but the same substitution degrades generative fluency (chrF++ falls from 13.66 to 4.59), revealing a disconnect between selecting a correct answer and generating fluent, clinically reliable Amharic text. Performance also varies by care stage: Fetal questions are answered most accurately (50.0% average), reflecting more universal anatomical knowledge, while Postpartum and Early Childhood questions – more culturally specific – score lowest (44.5% and 45.7%), posing the greatest safety risk. Counter to expectations, general-purpose multilingual models outperform English-specialized medical LLMs by up to 29 points in Amharic (48.22% vs. 19.56%); within the same model family, English-only medical fine-tuning (medgemma-27b-it) underperforms its multilingual base (gemma-3-27b-it), suggesting that domain-specific fine-tuning on English medical text induces catastrophic forgetting of multilingual capability, whereas scaling multilingual pretraining offers a safer path to cross-lingual clinical reasoning. Building on this baseline, our ongoing work investigates whether culturally grounded adaptation can close these gaps. We are fine-tuning Gemma-3-27B-IT and Llama-3.1-8B with LoRA on a curated Amharic MCH corpus of roughly 1,000 synthetic question-answer pairs, to be evaluated on a held-out set of 150 authentic community questions using MCQA accuracy, chrF++, and expert assessment of clinical correctness and cultural appropriateness. This work establishes the first benchmark and baseline for Amharic MCH question answering, demonstrates that current medical AI systems are not yet safe or reliable for this setting, and lays the groundwork for developing inclusive, culturally grounded healthcare AI for underrepresented languages.