Toward Culturally- and Linguistically-Aware Benchmarks for AI Risk and Reliability

As language and vision-language models are deployed globally, evaluating their safety requires more than a single, English-centric notion of harm. This work synthesizes findings from three recent AI risk and reliability benchmarking effort: MLCommons' AILuminate, the culture- and modality-aware Pluralis benchmark, and the multilingual jailbreak study Mātṛka, to show that alignment safeguards trained mainly on English data do not transfer uniformly across languages, modalities, or cultural contexts. Across these efforts, three findings recur: automated safety evaluators remain the dominant source of measurement uncertainty outside English and high-resource scripts; safety and cultural appropriateness behave as distinguishable, only weakly correlated axes; and model scale does not reliably predict robustness to multilingual or symbolic adversarial attacks. We argue these results motivate benchmarks that natively source hazards and annotations from diverse linguistic and cultural communities, report safety and cultural-appropriateness scores separately, and explicitly quantify evaluator uncertainty, work that stands to benefit directly from broader participation by underrepresented communities in the ML research pipeline.