Now You See It, Now You Don’t: An Arabic Case Study of Data Contamination Through Translation

Large language models are increasingly evaluated on multilingual tasks, yet benchmark contamination is still commonly audited through the surface form in which a benchmark is tested. This creates a blind spot: a test item may appear during training in another language, preserve its benchmark-specific information, and still influence evaluation even when the original-language string is absent. We study this failure mode by fine-tuning four open-weight instruction-tuned models, LLaMA-3.2-1B, Qwen3-1.7B, Gemma-3-1B, and Mistral-7B, on Arabic translations of held-out English MMLU and XQuAD items at 0%, 10%, 50%, and 100% exposure, then evaluating on the untouched English benchmarks. On MMLU, translated exposure improves English accuracy by 2.8 to 11.3 percentage points from the clean to fully contaminated conditions, despite the English evaluation strings never appearing in the contamination data. The effect is less uniform on XQuAD, suggesting that cross-lingual contamination depends on task and model behavior. We then ask whether an English-only detector reveals the same exposure. Using TS-Guessing, a prompt-based memorization probe, we find highly model-dependent behavior: LLaMA produces a strong contamination signal, while Qwen and Mistral remain at or below the random baseline and Gemma weakens as exposure increases. Contamination can therefore improve benchmark performance without producing a reliable same-language memorization trace. We next examine cross-lingual consistency using semantically equivalent English, Arabic, and French versions of MMLU items. Agreement rises substantially after translated exposure for several models, from 0.274 to 0.851 for Gemma and from 0.090 to 0.799 for Qwen at its peak, revealing a behavioral trace that is less visible from English alone. Building on concerns about benchmark leakage and evidence that contamination can cross language barriers , these results support a translation-aware view of contamination auditing: translation can alter where leakage is visible without eliminating the information that affects benchmark scores. This is particularly relevant for multilingual and Global South AI evaluation, where models are trained on heterogeneous multilingual data. We argue that the absence of same-language overlap should not be treated as evidence of a clean evaluation. Audits should instead consider translated exposure and compare behavior across semantically equivalent language views, complementing existing contamination checks.