Penalize to Verify: A Multi-Domain Benchmark for Trustworthy LLM Evaluation

As Large Language Models are increasingly deployed as automated evaluators (LLM-as-a-Judge), ensuring the reliability of their judgments has become a critical challenge. Existing benchmarks primarily focus on human-model agreement, largely overlooking the evaluators' robustness against deceptive contexts, their inherent self-evaluation biases, and their pervasive tendency toward overconfident hallucination. To comprehensively evaluate the model's verification capability, we propose Penalize-to-Verify Benchmark (PVBench), a comprehensive multi-domain benchmark (encompassing Math, Code, and Knowledge) designed to rigorously stress-test the validation capabilities of LLMs. We propose a novel semi-automated framework to construct paired test cases featuring both correct and highly plausible misleading Chain-of-Thought trajectories, naturally derived from the errors of peer models. Furthermore, to mitigate blind guesses, we introduce a tiered penalty metric that rewards accurate verification, heavily penalizes incorrect judgments, and assigns zero score to abstentions, thereby forcing models to calibrate their true confidence. Extensive experiments reveal that state-of-the-art LLMs suffer catastrophic performance degradation when confronted with misleading CoTs, exhibit strong self-judge biases, and demonstrate distinct verification vulnerabilities across different domains. By shifting the evaluation paradigm from simple accuracy to risk-aware verification, our benchmark provides a scalable and rigorous foundation for developing trustworthy LLM evaluators. Our benchmark is available here: https://anonymous.4open.science/r/PVBench-CC63.