Evaluating AI Safety in Yoruba: A Capability-Aware Approach to Local-Language Safety

Safety evaluation of large language models in African languages often relies on refusal-based metrics, but these metrics may conflate safety behaviour with failures in language generation. We investigate this problem in Yorùbá using 225 matched English–Yorùbá prompt pairs from AdvBench, HarmBench, and XSTest. We evaluate Gemma-2-9B-IT and Gemini-2.5-Pro, classifying responses as refusal, harmful compliance, or capability failure. For Gemma, refusal on harmful prompts falls from 98.99% in English to 6.87% in Yorùbá, but 92.32% of Yorùbá responses are capability failures and only 0.81% are harmful compliance. In contrast, Gemini shows only 1.55% capability failure in Yorùbá, while harmful compliance rises from 7.53% to 15.98%. These results show that refusal-rate evaluation can produce fundamentally different interpretations of safety across models. We therefore propose capability-aware three-way classification as a simple complement to refusal-based evaluation for local-language AI safety.