How many lives does your cat have? Cultural Bias and the Cost of Cross-Lingual Consistency in LLMs

Most modern LLMs claim to be multilingual. However, studies show that answer accuracy drops drastically when questions are asked in languages other than English, and that models struggle with culturally specific questions. Furthermore, it was shown that models tend to think in English and only translate the answer into the target language at the end. Across all of this prior work, however, the correct answer is assumed to remain the same regardless of the question's language. To the best of our knowledge, we are the first to investigate how LLMs behave on language-dependent questions. In this work, we investigate questions whose answers objectively vary with the language in which they are asked. One example is the proverb "A cat has nine lives." In German, the proverb goes as follows: "Eine Katze hat sieben Leben" (A cat has seven lives.) Therefore, the answer to the question depends on which language it is asked in. The same pattern can be seen for further proverbs (ENG: "Two birds with one stone", GER: "Zwei Fliegen mit einer Klappe"), movie titles (ENG: "Fast X", GER: "Fast \& Furious 10"), animal sounds (ENG: "Woof", GER: "Wau"), medical products (ENG: "Band-Aid", GER: "Hansaplast"), or units (ENG: "Letter" as the standard document format, GER: "A4"). We investigate 14 open- and closed-weight models on how well they account for these language-dependent answers. We therefore create a dataset consisting of 96 questions, with answers varying depending on whether they are asked in German or English. The dataset is constructed and validated by two native German speakers with high English proficiency. Furthermore, the movies and the medical questions were validated by two separate domain experts. Every question is asked in 3 different variations: Once without any language cue, once with an English cue (e.g., "How many lives does a cat have according to an English proverb?"), and once with a German cue, resulting in 288 questions overall. Furthermore, every question was asked in English and German, with 3 repetitions to account for non-determinism in LLMs, yielding 24,192 model responses. This design lets us distinguish whether a model lacks the knowledge or gives the correct answer for the wrong language (e.g., nine lives in German). Our results show that in 13.5\% of German questions without a language cue, the English answer is given instead of the German one, whereas the reverse occurs for only 5\% of the questions. Without a cue, 11 out of 14 models show a significant anglocentric asymmetry, while "gpt-5.6-sol" is significantly reversed. Across all conditions, models like "gpt-5" have the highest overall accuracy at 86.3\% but also the highest share of language-dependent errors, with 50.7\% of its German errors on language-dependent items being the English answer. For smaller models in the same family, both the accuracy and the share of language-dependent errors decrease: "gpt-5-mini" (75.1\% $\rightarrow$ 49.3\%), "gpt-5-nano" (68.1\% $\rightarrow$ 44.4\%). Similar values are found for, e.g., "gpt-5.4-mini" (80.2\% $\rightarrow$ 44.7\%) or "deepseek-v4-flash" (75.5\% $\rightarrow$ 44.9\%). For "mistral-small-2603" the share of language-dependent errors is 15.8\% while the overall accuracy is 53.5\%. Therefore, the stronger the model, the fewer errors it makes in German, but if it gives an incorrect answer, it is more likely to be the English one. The only exception is again "gpt-5.6-sol" (85.6\% $\rightarrow$ 7\%). Without a cue, the models are more likely to give English answers to pop culture questions or proverbs, while German answers are more likely for units or conventions. These differences could stem from, for example, °C being used worldwide outside the USA, and the training data therefore being more likely to contain these units. Unlike the anglocentric direction, this metric default does not diminish with model strength. Furthermore, the confusions are consistent across all three repetitions for 41.8\% of the affected model-item pairs, and for 67.5\% of them in at least two repetitions. We can also see that explicit cues are more effective than the prompt language itself, with accuracy rising from 70\% to 76.4\%, while mismatched cues almost double the confusion rate (9.3\% $\rightarrow$ 16\%). Lastly, even when prompt language and cue both point to German, the English answer still appears 224 times, versus 86 in the mirrored configuration. Our results suggest that cross-lingual consistency, usually treated as a desirable property, can itself be an error, and that current multilingual evaluations cannot detect this. We are planning to release the dataset and encourage extension to additional languages.