Cross-Lingual Text-to-Ultrasound Synthesis for Swahili Silent Speech via Articulatory Feature Transfer

After a total laryngectomy, or in the late stages of amyotrophic lateral sclerosis (ALS), a person may still be able to move their tongue while producing no voice at all. In Kenya, laryngeal cancer is among the most common head and neck malignancies, and total laryngectomy is routinely performed at national referral hospitals, leaving patients to recover in Swahili-speaking households with no speech technology designed for their language. Silent Speech Interfaces (SSIs) reconstruct speech from residual tongue movement, and ultrasound tongue imaging is a promising non-invasive way to capture it. But building these systems requires paired data: ultrasound recordings matched to transcripts. For Swahili, no such dataset is publicly available, and the specialized setup needed to create one, a stabilized probe with synchronized audio and video capture, is largely absent from East African clinics. Roughly 200 million speakers are left without access to an entire category of assistive technology. We ask: if articulatory features like place of articulation, manner, and voicing are shared across human languages, can a model that learns to map phonemes to tongue movement in English then generate plausible tongue ultrasound for Swahili, without ever seeing Swahili ultrasound data? We propose a framework and evaluation methodology for generating articulatory training data in the absence of target-language recordings. Our framework, Text-to-Ultrasound Synthesis (TUS), converts English and Standard Swahili text to International Phonetic Alphabet (IPA) sequences through Epitran and generates grayscale mid-sagittal ultrasound frames of the tongue. The model combines a Transformer encoder with a learned duration predictor and a convolutional decoder, with ConvLSTM for post-decoding temporal refinement. The core design choice is in the embedding layer: each IPA token is represented as a learned vector concatenated with a fixed articulatory feature vector from panphon, encoding place, manner, voicing, and tongue root position in a representation that carries no language-specific information. Training uses English ultrasound video from a single speaker in the SSR7000 corpus; transcripts were recoverable for the 2,342 utterances in which the speaker reads TIMIT sentences, giving 2,170 training utterances after length filtering. That the training source is a single speaker is a constraint we lean into rather than around. If the model generalizes from one person's vocal tract anatomy to Swahili phonemes it has never encountered, that would suggest the articulatory feature space captures lingual dynamics at a level above individual anatomy. Held-out English evaluation uses Mean Sum of Distances and Hausdorff distance between predicted and ground-truth tongue contours. Preliminary results from a capacity-matched ablation show that, at this data scale, panphon features give a statistically detectable but negligible pixel-level gain, no change in Mean Sum of Distances, and slightly worse Hausdorff distance. Motion analysis shows that generated sequences in both English and Swahili move roughly 14× less than real English ultrasound; the collapse is not specific to Swahili, pointing to temporal modeling and data scale as the current bottleneck. Zero-shot Swahili synthesis is evaluated on a probe set of words and sentences targeting phenomena absent from English, including prenasalized stops (/mb/, /nd/, /ŋg/), currently represented as nasal + stop sequences, syllabic nasals, and the language's predominantly vowel-final word structure (a, e, i, o, u), using phone coverage, analysis of the shared articulatory feature space, and motion statistics. Planned evaluation will compare the generated frames against published articulatory descriptions of these phenomena, and will train a phoneme classifier on synthetic frames alone to measure its accuracy against real Swahili ultrasound recordings collected through a clinical partnership we are pursuing. The central question is not whether the generated frames are pixel-perfect, but whether synthetic data carries enough articulatory signal to bootstrap SSI development for low-resource languages lacking articulatory data.