Lightweight Cross-Lingual Speaker Adaptation for Indic TTS

Text-to-speech has improved quickly, but most work targets high-resource languages, and Indian languages remain under-served. Voice-cloning models trained on Indic data drop words and are slow to run, which makes them hard to deploy. We present a lightweight speaker adaptation pipeline that needs only a 10-second reference sample. Synthetic speech from a voice-cloning model is passed through a four-stage quality-control pipeline and used to fine-tune a FastSpeech2 model with dual-site speaker conditioning and a cosine consistency loss. Phoneme unification through the Common Label Set (CLS) lets the adapted voice speak Hindi, Marathi, Tamil and Telugu without additional recordings. The resulting system is deterministic, runs 53× faster than the cloning baseline at 4.7× fewer parameters, and improves intelligibility and naturalness while preserving speaker characteristics in cross-lingual synthesis. Pretraining takes about 16 hours on a single A6000, against roughly three days on 32 H100 GPUs for the cloning baseline. Audio samples are available online.