Building an LLM Powered Speech-to-Speech Translation System for Low-Resource Ethiopian Healthcare Communication

Language barriers between patients and healthcare professionals in Ethiopia’s multilingual health system compromise diagnostic accuracy, treatment quality, and patient trust. Despite recent advances in speech technologies, most African languages remain critically underrepresented in speech-to-speech translation (S2ST) research due to severe data scarcity and lack of task specific datasets. This work presents the first bidirectional S2ST system specifically designed for clinical communication between Amharic and Afaan Oromo, two of Ethiopia’s most widely spoken yet computationally underserved languages. The system follows a cascaded architecture integrating Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS), optimized for real-world healthcare interaction. A major contribution of this work is the creation of large scale, healthcare focused datasets built from scratch, including over 250 hours of Amharic speech, 200 hours of Afaan Oromo speech, approximately 200 hours of studio-quality multi-speaker TTS recordings, millions of monolingual sentences, and 667,021 parallel sentence pairs. All datasets were carefully curated, normalized, and validated by native-language experts to ensure linguistic and clinical quality. For ASR, we develop hybrid HMM-GMM and CNN-TDNNf-based models using the Kaldi toolkit, achieving Word Error Rates of 12.36% for Amharic and 19.6% for Afaan Oromo. For machine translation, we adapt mT5-Large using a two-stage training strategy: continued pretraining on large-scale monolingual corpora followed by fine-tuning on parallel healthcare domain data. This approach is particularly important for Afaan Oromo, which is absent in the original mT5 pretraining corpus. The resulting system achieves BLEU scores of 26.3 for Amharic→Afaan Oromo and 20.79 for the reverse direction. For speech synthesis, we fine-tune SpeechT5 with speaker embeddings to enable multi-speaker, controllable synthesis, achieving Mean Opinion Scores (MOS) of 4.65/5 for Amharic and 4.43/5 for Afaan Oromo in terms of naturalness, intelligibility, and pronunciation quality. Beyond individual components, we integrate all models into a real-time, deployable mobile and web-based application using a Flutter frontend and a Next.js backend, enabling end-to-end speech-to-speech translation for clinical use. The system demonstrates practical round-trip performance suitable for structured medical dialogue, while highlighting the impact of error propagation across cascaded pipelines. Beyond model development, this work provides foundational speech and language resources for low-resource Ethiopian languages and demonstrates a scalable blueprint for building AI-driven healthcare communication systems in under-resourced multilingual regions. It represents a step toward reducing linguistic inequality in healthcare and enabling more equitable access to medical services through AI.