Empowering East African Voices: A Matched-Scale Comparison of Multilingual ASR Models

Automatic Speech Recognition (ASR) is a technology that converts spoken language into written text. However, developing ASR systems for many East African languages remains difficult because sufficient speech and text data are often unavailable. Although East Africa has more than 150 languages, previous ASR research has focused mainly on widely spoken languages such as Swahili. Other languages, including Kikuyu, Luo, Somali, Maasai, and Kalenjin, have received much less attention. Collecting and labeling large amounts of speech data is also expensive and time-consuming. In this study, we developed and evaluated a multilingual ASR system for six East African languages: Swahili, Kikuyu, Luo, Somali, Maasai, and Kalenjin. These languages represent three major language families in the region: Bantu, Nilotic, and Cushitic. We collected a total of 1,182 hours of labeled speech data across the six languages. The audio was converted into a standard format of 16 kHz, 16-bit WAV files. We also cleaned and normalized the text by correcting spelling, grammar, abbreviations, and numbers. To build the ASR systems, we compared two pretrained speech models, MMS-300M and XLS-R-300M, both of which have approximately 300 million parameters. We fine-tuned both models using the same labeled dataset and training settings to ensure a fair comparison. The models were trained to recognize characters from speech, and their performance was measured using Word Error Rate (WER), where a lower WER indicates better recognition accuracy. The results showed that XLS-R-300M performed better overall, achieving an average WER of 32.98%, compared with 35.65% for MMS-300M. XLS-R performed better for five of the six languages. For example, it achieved WERs of 25.90% for Kikuyu and 24.37% for Somali, demonstrating strong recognition performance for these languages. However, MMS-300M performed better for Swahili, with a WER of 15.06%, compared with 16.50% for XLS-R. Both models had more difficulty recognizing the Nilotic languages, particularly Maasai and Kalenjin. XLS-R achieved WERs of 49.60% for Maasai and 57.54% for Kalenjin. These higher error rates may be related to the limited training data available for these languages and differences in their linguistic structures. Overall, this study provides a comparative evaluation of two pretrained speech models for multilingual ASR across six under-resourced East African languages. The results show that XLS-R-300M provides a strong baseline, outperforming MMS-300M across most of the evaluated languages. The findings provide useful benchmarks for future ASR research and the development of speech technologies for East African languages. In future work, we plan to evaluate larger models such as MMS-1B, and explore language-model-based rescoring. These improvements could especially help reduce recognition errors for highly under-resourced languages such as Maasai and Kalenjin and support the development of more accurate and accessible speech technologies for East African communities.