A Unified Sparse Mixture-of-Experts Architecture for Sheng-Swahili-English Code-Switched Speech

ASR in low-resource settings is complicated by natural code switching that happens especially in East African where languages such as Sheng, Kiswahili, and English among others are spoken within the same utterance, and thus, pose acoustic and linguistic variability that challenge existing multilingual and sparse Mixture-of-Experts (MoE). We propose a boundary-aware, shared-router sparse MoE architecture for Sheng–Kiswahili–English code-switched ASR. A multilingual speech encoder produces frame-level representations $h_t$; a CTC loss offers temporal token evidence without the need for manually labeling frame boundaries. Aligned transcriptions within a span that have validated language labels are used to offer supervision for routing and switch boundary learning on reliable frames while masking uncertain frames. Encoder states, language-ID posteriors $p_t^{\text{lang}}$, switch-boundary probabilities $b_t$, and acoustic context features $c_t$ form a unified representation: $$z_t = [h_t \parallel p_t^{\text{lang}} \parallel b_t \parallel c_t]$$ $z_t$ is conditioned by a shared router to produce sparse top-$k$ expert assignments across MoE layers, routing between language-specific experts (Sheng, Kiswahili, English) and shared linguistic, multilingual, and acoustic/phonetic experts. Expert utilization is controlled through lossless load control, obviating any requirement for balancing losses. The training objective includes ASR loss together with routing, boundary, and expert-level CTC supervision, which are introduced incrementally to determine the impact of each loss term independently. We intend to benchmark the model architecture against dense and sparse-MoE baselines that are parameter-equivalent, using the Mozilla Data Collective Sheng-Swahili-English corpus, where we will perform cross-lingual transfer from Common Voice (Swahili/English) and compare on AfriSwitch, to determine whether boundary-aware routing leads to better switch region accuracy without sacrificing non-switch performance or on-device latency.