AfriLCB: Benchmarking Language Confusion in Large Language Models Across African Languages

Large language models (LLMs) frequently fail to remain in the language requested by a user, yet existing language-confusion evaluations substantially underrepresent African languages. We present the design and ongoing development of AfriLCB, a reproduction and adaptation of the Language Confusion Benchmark for 18 African languages: Acholi, Amharic, Chichewa, Fongbe, Ga, Hausa, Igbo, Kinyarwanda, Lingala, Malagasy, Ndebele, Sepedi, Shona, Swahili, Tigrinya, Twi, Wolof, and Yoruba. The benchmark will evaluate nine open- and closed-weight model releases: GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Llama-3.3-70B-Instruct, Qwen3-235B-A22B-Instruct-2507, Mistral Large (2512), Command R+ (08-2024), Kimi-K2-Instruct (0905), and GLM-5.2. AfriLCB retains the original benchmark’s monolingual and cross-lingual generation settings. Monolingual prompts are drawn from Aya, Dolly, PolyWrite, and AfriQA. For each available language–source pair, we sample up to 100 Aya prompts, 200 Dolly prompts, 100 PolyWrite prompts, and 250 AfriQA prompts using a fixed random seed. Where fewer examples are available, we retain all available examples and report the shortfall without oversampling or padding. For cross-lingual generation, matched English prompts request responses in each of the 18 target languages using four instruction variants placed at either the beginning or end of the prompt. Model responses will be evaluated using GlotLID-based line- and span-level language identification, language-confusion pass rates, and native-speaker validation where available. The benchmark is motivated by a completed native-speaker pilot involving 30 African languages and three widely used AI assistants. The pilot revealed five recurring failure types: wrong-dialect generation, substitution or mixing of another language, failure to recognize the target language, target-language responses containing incorrect linguistic forms, and prompt-dependent behavior. The observed failures were structured rather than random. All three assistants substituted Twi for Fante and mixed Zulu into Ndebele, while Tigrinya was confused with Amharic. English and French also repeatedly appeared as fallback languages. For very low-resource languages such as Bum and Saar, models alternated between acknowledging that they did not know the language and generating confident but nonsensical responses. English instructions occasionally corrected language confusion but, in other cases, increased language mixing or hallucination. The ongoing AfriLCB evaluation will examine whether these pilot patterns persist across a broader set of model families and how language confusion varies by prompting setting, dataset, language-resource availability, and linguistic or regional proximity. The study aims to produce a reproducible African-language benchmark, a failure taxonomy grounded in native-speaker observations, and evidence for developing mitigation strategies for language confusion in African-language generation.