Compliance–Correctness Decomposition for Post-Training Evaluation of Small Language Models

Automatic evaluation of language models often conflates output compliance with task correctness, although these properties can change differently after post training. This makes strict accuracy difficult to interpret when fine tuning changes both response format and task performance. We identify this gap and introduce the Compliance–Correctness Decomposition (CCD), an item paired framework that separates responses into invalid, valid but wrong, and valid and correct states. CCD requires no additional model generations and enables direct comparison of compliance, strict success, and the valid but wrong rate before and after supervised fine tuning. Across 49,140 matched response pairs spanning 14 small language models, four benchmarks, and nine inference configurations, compliance increases by 34.78 percentage points, while strict success increases by 12.12 points. The resulting 22.66 point increase in valid but wrong responses is consistent across the evaluated models, benchmarks, and inference configurations. On MBPP, compliance increases by 27.21 points, whereas execution based correctness decreases from 18.99% to 15.33%. These results indicate that improvements in evaluator facing compliance do not necessarily correspond to improved task correctness, supporting separate reporting of compliance and correctness in post training evaluation.