Listening or Reciting? Evaluating Audio LLMs for Qur'anic Mispronunciation Detection

Existing methods for Qur'anic Mispronunciation Detection and Diagnosis (MDD) rely predominantly on task-specific, fine-tuned CTC-based architectures, leaving the zero-shot capabilities of modern Large Audio-Language Models (LALMs) largely unexplored. In this work, we present a systematic empirical study evaluating off-the-shelf LALMs on phoneme-level Qur'anic MDD. We evaluate 20 open- and closed-weight audio models across 31 evaluation strategies and prompting configurations on the QuranMB.v2 benchmark without fine-tuning. Our best-performing zero-shot configuration achieves an MDD F1 of 0.5110, clearing the official mHuBERT baseline (0.4414) although it remains short of the best fine-tuned approach (0.7201). Additionally, we show that standard evaluation metrics fail to capture critical behavioral nuances. We show that Phoneme Error Rate (PER) is easily saturated by memorization: unconditionally outputting canonical phoneme strings, with no access to the audio at all, yields a PER of 6.82\%, lower than that of most evaluated systems, and our lowest-PER configuration reproduces the canonical reference in 81.2\% of cases while attaining an F1 of only 0.17. To systematically disentangle genuine acoustic perception from prior linguistic conditioning, we introduce two diagnostic metrics: Canonical Copy Rate (CCR) and Canonical Lean (CL). These diagnostics reveal clear behavioral differences between open and closed models. Frontier models hallucinate canonical text on approximately one-third of pure-audio input, while open and closed models exhibit opposing trade-offs between consonant and vowel error detection. Our findings expose key vulnerabilities in standard MDD metrics and provide rigorous zero-shot baselines.