Diagnosing and Mitigating Script-Based Representational Divergence in Multilingual Encoders

Pretrained multilingual models are assumed to learn representations invariant to surface form, yet the script-based representational divergence and its consequences for downstream tasks remain poorly understood. We systematically diagnose and mitigate this divergence in six typologically diverse language pairs, which are Kazakh, Serbian, Chinese, Uyghur, Japanese, and Hindi--Urdu, and probe four multilingual encoders, which are XLM-R, ByT5, mT5, and LaBSE, through layer-wise cosine similarity, script cluster analysis, masked language model probing, and cross-script retrieval. XLM-R cross-script retrieval Mean Reciprocal Rank (MRR) falls below 0.05 for Kazakh, Uyghur, and Hindi--Urdu, and a consistent gap in silhouette and cosine similarity scores for Kazakh and Serbian is consistent with pretraining data volume as a contributor to divergence beyond tokenization. LaBSE's retrieval-specialized training achieves a mean MRR of 0.905 across all six language pairs, with Hindi--Urdu remaining the hardest case at 0.722. Script normalization and whitening greatly improve retrieval and are computationally inexpensive. We release parallel corpora, code, and a token overlap diagnostic that predicts cross-script retrieval failure without model inference. We hope to encourage future work in NLP on low-resource and digraphic languages.