MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation

Evaluating single-concept personalization in text-to-image diffusion has seen two categories of quantitative metrics: Concept Preservation (CP) measures identity fidelity to a reference while Prompt Following (PF) measures whether the generated scene matches the prompt. Personalization papers have commonly computed these signals using three separate backbones: CLIP-I and DINO for CP, CLIP-T for PF. In this work, we show that existing metrics fall short of correlating with human perception because they attend to the image as a whole, instead of distinguishing the concept subject from the background. This distinction is important from a human perception point of view as the concept subject in the output image should be very similar to the input concept, whereas the output background should adhere closely to the text prompt. To improve personalization evaluation in this way, we introduce MaSC, a unified metric that attends to CP and PF by differentiating concept subject regions. Specifically, given an externally provided foreground concept mask, MaSC computes the CP and PF scores from a single forward pass of a frozen SigLIP2 encoder per image. On DreamBench++ human ratings, MaSC reaches Krippendorff $\alpha = +0.471$ and Spearman $\rho = +0.611$ on CP, the highest point estimate among the evaluated non-LLM metrics, exceeding GPT-4V and falling $\Delta\alpha = 0.028$ below GPT-4o. An exploratory reference-mask sensitivity analysis reaches $\alpha = +0.506$ with $32\%$ dilation. To distinguish human perception bias we also evaluate our method, as well as common SOTA baselines, on ORIDa, a real-photo benchmark of identity preservation across physical environments. In this experiment MaSC reaches AUC $= 0.992$ almost perfectly identifying concept subjects. The PF score, obtained without a second vision-encoder forward pass, achieves higher rank correlation with human ratings than the CLIP-T baseline shipped with DreamBench++. Together, these results support foreground-local aggregation for concept preservation and show how a shared encoder can provide complementary concept-preservation and prompt-following scores. We release MaSC as a pip-installable Python package, alongside independent reproductions of every comparator's published numbers.