WAHM: Measuring Hallucination Drift Across Arabic Varieties

Arabic language models are commonly evaluated in Modern Standard Arabic (MSA), although everyday communication spans regional varieties with substantial lexical and syntactic differences. We introduce WAHM, a paired benchmark for measuring factual hallucination under Arabic dialect shift. WAHM aligns 300 factual questions across MSA, Gulf, Egyptian, Levantine, and Sudanese Arabic and evaluates six Arabic-centric and multilingual language models. We measure changes in hallucination frequency using the Hallucination Drift Score (HDS) and changes in error-set identity using hallucination-set intersection-over-union (IoU). Responses are evaluated by a two-stage automatic pipeline that screens for degeneration before applying a question-aware AraBERT factuality judge, with reliability assessed through blinded human validation. Results show that MSA factual reliability and dialect robustness are distinct: models with similar aggregate hallucination rates can differ substantially in which questions they answer incorrectly, and several models exhibit their largest positive drift on Sudanese Arabic. These findings highlight the need to evaluate Arabic language models beyond MSA and to treat dialect robustness as a separate dimension of factual reliability.