Privacy for Some: LLM Privacy Protection Is Uneven Across Names and Breaks Down in Low-Resource Languages
Abstract
Large language model (LLM) privacy protection is not a universal safeguard applied equally to every user. We motivate it from prior work and test it with a small controlled experiment on two open models. Prior work supports each piece of the concern in isolation: LLMs memorize and leak personally identifiable information (PII); this leakage and the protections against it are unevenly effective across languages and across minority subpopulations within a single language; LLM providers construct materially different, jurisdiction-specific privacy commitments in their policy documents; and LLMs separately exhibit systematic geographic bias against lower-resource regions in their outputs. What has not been tested is whether protective behavior at inference time varies with user-identity cues that carry no change in content. In a paired 2x2 study (Western vs. Rwandan name x English vs. Kinyarwanda) using unique planted canaries so that any disclosure is a provable leak, we find two failures. First, one of two models (Qwen2.5-3B) discloses a confidential record for a Rwandan-named person 31.3% of the time versus 1.3% for an identical English-named record when both conversations are in fluent English (McNemar p<0.001); every such leak follows a "refuse-but-echo" pattern in which the model states a confidentiality refusal and then reveals the value. Second, in Kinyarwanda both models cease to comprehend the task entirely, so protective behavior is not merely weaker but undefined for the lowest-resource users -- an equity failure by incapacity. We respond to alternative explanations and close with recommendations, with attention to low-resource African languages and Sub-Saharan jurisdictions that sit outside the enforcement regimes current LLM privacy engineering is built around.