Position: Script Is the Missing Interface in Multilingual Evaluation
Md Rashad Al Hasan Rony
Abstract
Multilingual natural language processing (NLP) has quietly equated 'supporting a language' with supporting its standard native script. Yet for very large user populations the language as typed is romanized, noisily spelled and code-mixed, most visibly across South Asia and again in Arabizi, Greeklish and transliterated Slavic. Recent work has begun to measure the cost. Deployed clinical triage degrades by up to 24 points on romanized queries, code-switched prompts bypass refusals, and new benchmarks target romanized code-mixed instructions. We argue that these are symptoms of one structural error no prior work has named, namely that the training and evaluation loop is closed over an orthographic register that is not the one users produce. Corpora are filtered by script-based language identification that discards romanized text, instruction data is authored in native script, benchmarks inherit the same convention, and leaderboards certify the result as 'support.' No stage measures the excluded register, so the failure is invisible by construction and specialized models inherit it. The deeper error is conceptual. Typically, the orthographic variation is treated as perturbation of a canonical form when these communities operate a repertoire of written forms with no privileged member. Our inference-only demonstration comprises 43,200 generations over six models, three of them Indic-specialized, with identical items in five orthographic conditions. A matched native-script noise control isolates script change from spelling instability. Models retain only 37 to 40\% of their correct native-script answers once romanized, against 68\% under comparable noise. A within-language tokenization analysis adds that six of eight tokenizers spend \emph{fewer} tokens on the romanized form, so the excluded register is usually cheaper to serve. ``Supporting language $L$'' must therefore be redefined as supporting $L$'s attested written repertoire, enforced by script-stratified evaluation as a release gate.
Chat is not available.
Successful Page Load