Real-Time Edge ASR in Virtual Reality: Adapting Lightweight Speech Models for Stroke Rehabilitation in Aphasia

Aphasia affects roughly one-third of stroke survivors, presenting a critical barrier to poststroke recovery. Beyond language impairment, communication breakdown carries a severe psychosocial cost, with post-stroke depression affecting up to 70% of individuals with aphasia, compared to roughly one-third of the general stroke population. While automatic speech recognition (ASR) could provide people with aphasia a direct channel to communicate without relying on a listener’s familiarity with disordered speech, general-purpose ASR degrades sharply on aphasic speech, and no commercially available system targets this population. We fine-tuned Whisper-tiny, OpenAI’s smallest Whisper variant, on aphasic speech drawn from AphasiaBank, with an explicit deployment target: the virtual reality-based stroke rehabilitation program at the Centre for Immersive Technologies and AI Development (CITAID, “Braining”), where low-latency, offline-capable inference is required for real-time interaction during virtual reality (VR) therapy sessions. Unlike prior aphasia-ASR work, which has largely relied on larger Whisper variants unsuited to edge and mobile VR hardware, we prioritize the smallest model capable of meaningful clinical adaptation. On mild, moderate, and severe aphasia, the fine-tuned model achieved a word error rate (WER) of 45%, 52%, and 64%, respectively. On the target hardware, the system yielded an average real-time factor (RTF) of 0.1, well below the standard 0.5 RTF deployability threshold for on-device ASR and consistent with prior edge-deployed Whisper benchmarks. This work demonstrates that fine-tuning lightweight, edge-deployable ASR models for aphasic speech is both technically feasible and clinically viable within a VR rehabilitation setting. Considering the severe psychosocial burden of communication breakdown, on-device speech tools represent an important, under-explored clinical intervention. Future efforts will benchmark the accuracy–deployability trade-offs across Whisper model sizes, expand evaluations to cross-lingual and cross-disorder settings, and measure longitudinal patient outcomes within the CITAID VR platform.