Beyond Exact Match: Unmasking WER Inflation in Akan ASR through Variant-Aware Evaluation

Standard Automatic Speech Recognition (ASR) evaluation relies on Word Error Rate (WER), which treats every token-level mis- match as an error regardless of linguistic validity. For dialectally diverse languages like Akan, phonological variations, spelling discrepancies, and code-switching markers are penalized identically to genuine missrecognitions, inflating reported error rates. We introduce a variant-aware WER (va-WER) that normalizes predictions and references through a canonical lexicon compiled exclusively from training and validation data before scoring. Our framework classifies near-miss word pairs into a five-tier linguistic taxonomy using an LLM-as-a-Judge approach validated against native-speaker human annotators, then compiles frequency-based canonical lexicons. Across three Akan ASR models (Whisper-large-v3, Gemma- 3n-2B, MMS-1b-all), we demonstrate that: (1) standard WER is inflated by 15.0%–17.0% (relative) due strictly to legitimate phonological and code-switching variations, rising to 22.4%–25.4% when human spelling inconsistencies are included; (2) the LLM judge achieves 81.4% agree- ment with native speakers when orthographic categories are collapsed, and sensitivity simulations (N = 50 Monte Carlo trials) confirm that metric reduction remains statistically robust (p < 10−4) even under substantial boundary noise; and (3) lexicons from spontaneous speech corpora transfer to independent Akan datasets, confirming that captured variants represent systematic linguistic phenomena rather than corpus- specific noise. We advocate reporting va-WER alongside standard WER in orthographically fluid languages to distinguish genuine model failures from benign dialectal variations.