QLR: An Auditable Benchmark for Fidelity in Qur’anic Pronunciation Respelling
Nicholas Cheng ⋅ Liam K Bray ⋅ Rishan Checker ⋅ Abdikadir Moalin
Abstract
Learner-facing Qur'anic pronunciation respellings must preserve pronunciation-relevant distinctions while remaining readable and auditable. We introduce \qlr\ (Qur'anic Learner Respelling), a constrained ASCII representation and deterministic benchmark for measuring fidelity of generated respellings to a phonemizer-derived reference. \qlr\ uses explicit symbols for selected consonantal, vowel-length, gemination, nasalization, emphasis, and pausing distinctions, together with a fixed decoder that permits automatic comparison without an LLM judge. Across 446 released contexts, the encoder--decoder pair exactly reconstructs the normalized reference representation. We evaluate valid Hafs contexts under four generation conditions: a deterministic letter-based baseline, Arabic-only generation, generation grounded in the phonemizer-derived analysis, and grounded generation followed by critique and revision. In the original 200-context evaluation, grounding reduced mean phoneme error rate from 12.63\% to 1.14\%, whereas critique and revision produced a smaller, statistically inconclusive reduction to 0.82\% (paired difference $-0.31$ percentage points, 95\% CI $[-0.92,+0.28]$, $p=0.375$). We therefore treat reflection as exploratory rather than as an established improvement. The primary contribution is an auditable evaluation framework that separates fidelity to a fixed symbolic reference from the linguistic validity and pedagogical usefulness of that reference. We release context-level targets, deterministic encoding and decoding, prompts, model outputs, and scoring procedures to support reproducible future studies. Expert validation of the phonetic targets and human evaluation of learner outcomes remain necessary before \qlr\ can be interpreted as an educational intervention.
Chat is not available.
Successful Page Load