BEYOND SAFETY SCORES: ALIGNING VISION-LANGUAGE MODELS WITH HUMAN VISUAL PERCEPTION

Vision-language models (VLMs) can assess urban scenes, but their judgements may not be grounded in the visual cues used by human observers. We examine this gap using street-view scenes and perception data from participants in India and Singapore. We collected human perception data on overall safety,four planning dimensions: Visibility, Surveillance, Accessibility, and Social Activity and 15 constituent environmental cues. This representation provides a common basis for comparing the visual cues identified by humans and VLMs. We first evaluate recent open-source VLMs on approximately 5,000 street-view images using cue-level measures of recall, hallucination, confusion, variance, and groundedness. The models show substantial limitations in visual grounding, with hallu- cination rates ranging from 0.122 to 0.708 and groundedness from 0.027 to 0.453. LLaVA-13B achieves the strongest baseline performance (recall = 0.430; groundedness = 0.453), but still shows substantial hallucination and confusion. We therefore introduce Visual Perception Chain-of-Thought (VP-CoT), which separates visual evidence extraction (SEE), planning-informed reasoning (THINK), and visual verification (CONFIRM). We evaluate VP-CoT on a stratified set of 50 scenes with human perception data. VP-CoT reduces hallucination and makes the visual evidence underlying each judgement explicit. However, its cue usage remains narrower than that of human observers, with 7.4 cues identified per image compared with 13.2 cues identified by humans. Agreement also varies across the four planning dimensions. The results indicate that structured visual reasoning improves the grounding and interpretability of VLM-based urban perception, but does not by itself produce human-aligned perception. VLMs may therefore be better suited to scalable extraction of observable environmental characteristics than to directly representing population-level safety perception.