Task-Critical Word Error Rate: Extending Entity-Focused Evaluation to Voice Commands in African-Accented Speech

Voice assistants increasingly mediate everyday tasks, making accurate speech recognition important not only for transcription but also for preserving user intent. Yet conventional Word Error Rate (WER) treats all transcription errors equally, although errors in command-critical words can change or invalidate an intended action. This raises an important question for equitable voice interfaces: does aggregate WER capture the errors that matter for command execution? Recent work has moved beyond aggregate WER. AfriSpeech-MultiBench [1] evaluates African-accented English across application-specific domains, including named-entity and command utterances, while AfriVox-v2 [2] introduces Entity Error Rate (EWER) and Numeric Error Rate (NWER) for entity- and number-containing utterances. These approaches provide finer-grained evaluation but do not explicitly capture the broader vocabulary required for command execution, including action verbs (e.g., “call,” “play,” “set”) and state or temporal attributes (e.g., “on/of” ,"today/tomorrow”). We propose Task-Critical Word Error Rate (TC-WER), extending entity- and numeric-focused evaluation to tokens required to preserve a command’s intended action. We situate TC-WER within a three-level hierarchy: (1) transcription accuracy (WER), (2) task-critical recognition error (TC-WER), and (3) functional command success. This pilot evaluates the first two levels directly; we additionally propose simulated voice commands as a controlled setting for examining whether task-critical recognition errors translate into command-level failure. Because commercial assistants do not expose their internal ASR pipelines, we use Whisper [3] as an accessible evaluation testbed rather than as a proxy for any specific assistant architecture. We construct country-stratified evaluation subsets from AfriSpeech-200 [4], focusing on Nigeria, Kenya, South Africa, Ghana, and Uganda, with matched utterance counts where possible. We evaluate the same speech using WER and TC-WER, with entity- and number-dense utterances enabling comparison with EWER/NWER. Ultimately, this work investigates whether comparable WER can mask disparities in task-critical recognition across African English varieties and whether extending entity- and numeric-focused evaluation to command-execution vocabulary reveals them. We hypothesize that TC-WER offers a more actionable measure of voice-assistant equity than existing error metrics. Thus, we contribute TC-WER, a country-stratified evaluation protocol, and a framework for ASR equity auditing when downstream systems are inaccessible.