Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency
NeurIPS 2026
Contributions
- An evaluation methodology : A black-box and model-agnostic method to evaluate the importance of objects in scene perception for both human and model.
- A dataset : We establish a human benchmark dataset comprising 16,603 valid responses across 307 factual scenes and 1,306 counterfactual variants from 227 independent participants.
- The discovery : We conducted the first systematic quantification of the semantic alignment gap between modern VLMs and human observers in scene description, revealing a pervasive divergence in the preference over low-level attributes and top-down priors between VLMs and humans.
The evaluation methodology: Counterfactual Semantic Saliency
Counterfactual Semantic Saliency (CSS) defines object importance by measuring the semantic shift induced by its removal as a counterfactual intervention.
Workflow of Counterfactual Semantic Saliency. (1) Counterfactual Stimuli Generation Stage: A large vision-language model generates a list of object names in the scene. For each identified object, SAM3 generates a corresponding segmentation. Following mask preprocessing, an image inpainting model (Nano Banana 2) removes the target object. In the illustration, the TV, along with the pink LED light are removed from the scene. (2) Scene Description Stage: Descriptions are sampled from the target evaluation model for both the original image and all counterfactual variants. (3) Similarity Rating Stage: Counterfactual Semantic Saliency is quantified as the mean pairwise cosine similarity between the sentence embeddings of the factual and counterfactual descriptions.
The dataset: Human Psychophysics data for counterfactual scene
Between-subjects experimental design for human psychophysics. A scene containing N target objects (1 factual image ℐ and N counterfactual variants ℐdo(oi=∅)) is distributed across N+1 mutually exclusive stimulus sets. Each set is evaluated by an independent group of 10 participants, ensuring no individual is exposed to multiple variants of the same scene. 227 Subjects are recruited to finish a total of 16,665 trials.
The discoveries
1. The Gap exists between Human and Model Scene Description
Top-1 Accuracy at predicting the critical object. Top-1 Accuracy representing the proportion of scenes where the evaluated model successfully identified the most semantically critical object, as determined by human consensus.
Mean Kendall's rank correlation coefficient (τ) across all scenes, capturing the overall ordinal alignment of object importance hierarchies between models and humans.
2. Divergence in Perceptual Biases between Human and Models
We investigated four primary factors: size, location (center), low-level saliency, and whether the object is a person, which were established in the former vision science and cognitive science research, hypothesized to influence the importance of the object in the scene. Each bias is measured by using the correlation between the CSS of the masked objects and the attribute related to the objects across all 1,306 counterfactual scenes.
We computed the correlation between specific object attributes and the resulting CSS score upon ablation. Significance tests indicate the statistical deviation of each model's correlation coefficient from the human baseline (* = p < .05; ** = p < .01; *** = p < .001; **** = p < .0001; derived from a one-tailed bootstrap test).
For vision scientists
The evaluation methodology presented here can also serve as a saliency model for predicting human fixations. Please refer to this page for another application of this methodology.
Want more details about this work? Please read our paper.
BibTeX
@article{wen2026css,
title={Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency},
author={Wen, Ziqi and Madinei, Parsa and Eckstein, Miguel P.},
journal={arXiv preprint arXiv:2605.13047},
year={2026}
}