Visual Checksum: Using Translation to Diagnose Multilingual Grounding Failures in VLMs

Multilingual evaluation with translated captions conflates changes to the ranking task with failures to ground target-language text. We introduce Visual Checksum, which evaluates translated SugarCrepe positive and hard-negative pairs through three paths: original English, direct target-language scoring, and matched translate-test. Translate-test diagnoses whether the contrast remains recoverable through translation; it does not directly measure translation quality. Across 500 RE7 PLACE/SWAP items, five languages, and seven encoders, SigLIP 2 base achieves 69.6–70.0% direct accuracy in French and Hindi, but only 51.0–58.8% in Swahili, Amharic, and Quechua, despite matched GPT-5.5 translate-test accuracy of 72.4–75.4% in these three languages. NLLB-CLIP achieves 69.4–72.6% direct accuracy across all five languages. Its advantages on Swahili and Amharic persist across two translators, whereas Quechua is more translator-sensitive. Round-trip improvements over English are consistent with caption canonicalization. Translation affects absolute performance but does not fully explain the observed direct grounding deficits.