Probing Negation Understanding in Vision-Language Models With Object Attention

Despite their strong performance across various multi-modal tasks, vision-language models (VLMs) still struggle with negation. Investigating the ability of models to process negation is an emerging field with benchmarks to test negation accuracy being proposed and explorations into how models handle negated captions. Comparison with human performance on these benchmarks indicates that model performance is generally lower, in some cases very substantially. Since many of the errors that VLMs make in processing negation is to overpredict the affirmative, it is important to understand whether there are parallels between the way that VLMs and humans process negation. We investigate how VLMs process negation by analysing attention patterns of different models, drawing com- parisons between where humans and models focus when processing affirmative vs negated statements. We base our experiments on Orenes et al. [2014], where an eye-tracking experiment is performed to investigate how negation is processed in humans. They use 2D images of coloured shapes with 4 shapes in an image. Each participant is presented with one of these images and 2 spoken sentences. In the binary experiment, the sentence mentions two possible colours eg. “The figure could be red or blue”. The second sentence is then either affirmative: “The figure was blue” or negated: “The figure was not blue”. The multary experiment considers four colours: “The figure could be red, green, blue or yellow”. In the binary case, eye-gaze is initially on the mentioned colour and shifts after a short time delay to the alternative colour as the negation is processed. In the multary case, however, the gaze stays focused on the mentioned colour. We replicate this binary and multary experiment with VLMs by creating our own binary and multary datasets. Since the models cannot process audio, and LLaVA and PaliGemma models accept only image and text inputs, we replicate the original priming effect in a static domain using images containing either two shapes or four shapes, mirroring the number of alternatives introduced in the original experiment. We generate images containing a plain white background and two 2D objects. We consider the colours blue, red, green and yellow and the shapes circle, square, triangle and diamond. We additionally expand the binary setting using reasistic images from the What’sUp Kamath et al. [2023] dataset, which contain images of only two objects. We compare the attention patterns for 3 models: LLaVA-1.5-7b, InternVL-3.5-4B and PaliGemma2-3B-mix-224. We analyse the cross-modal attention by capturing the weights assigned to image tokens during the generation of the output, which we limit to a single-token response. We use prompts that specify the colour of one of the image objects e.g. “The figure is {colour}” or “The figure is not {colour}”. We calculate normalised attention scores for each object within an image, averaging across all layers and heads. We identify the image tokens based on the relevant marker for each model (e.g. < image > for LLaVA) and re-construct the 1D image token sequence to their original 2D coordinates. Object bounding boxes are then projected onto this grid and we calculate normalised attention by summing attention within the bounding box and dividing by the number of tokens within the bounding box Figure: Mean object attention for each dataset and model. For a human-like response we expect Mentioned > Alternative throughout, Alternative Negated > Mentioned Negated for the Binary and What’sUp datasets, and Mentioned Negated > Alternative Negated for the Multary dataset. Error bars represent the 95% confidence interval. We find that only PaliGemma2 consistently exhibits human-like patterns across all datasets, whereas other models appear to process negation outside the attention mechanism.