Robustness Does Not Transfer: Culturally Grounded Red-Teaming of Arabic LLMs

Safety robustness in Arabic large language models (LLMs) does not transfer across adversarial framings: models that resist false-premise, invented-authority, and multi-turn escalation attacks can fail almost completely under persona-based framing. We show this using a red-teaming framework spanning seven Arabic-capable LLMs, five adversarial attack families, and three Arabic dialects (Darija, Egyptian, Najdi) in Islamic jurisprudence (fiqh), a domain offering structured expert ground truth, legitimate scholarly pluralism, and genuine native dialectal use, evaluated across a 414-prompt bank spanning all five attack families and benign controls. Claude Opus 4.7 and Gemini 3.1 Pro show attack success below 5% against false-premise, invented-authority, and escalation attacks, yet collapse to 96.7% and 99.2% under a named-scholar persona attack; across the full seven-model panel, rankings on four attack families correlate strongly with one another (Spearman ρ=0.85 to 0.89, five of six pairwise correlations at p<0.02) while the persona attack correlates with none of them at conventional significance, a pattern consistent with strongly attack-specific robustness. Separately, models show substantial miscalibration on two proper scoring rules and Expected Calibration Error (0.144 to 0.774 across the panel, roughly a five-fold gap), and raw confidence tracks correctness poorly even in aggregate (r=0.007) and inverts under false-premise framing (r=-0.243). Aggregate vulnerability varies strongly by attack type and model; no statistically significant aggregate dialect effect was detected (χ²=2.77, p=0.25), and the sole Arabic-native model tested is both the least calibrated and most vulnerable overall, though this single-model comparison cannot establish a general property of Arabic-native models. These results show that high-stakes AI in underrepresented linguistic and cultural contexts requires locally grounded, multi-attack evaluation rather than translated single-benchmark testing, and that linguistic localization alone is not evidence of safety.