The Complexity Kink: LLM Rubric Instruments for Causal Inference on Code Generation Reliability
Michael Hernandez ⋅ Tian Zhao
Abstract
Measuring complexity from a model's own code is biased by failure: a hard prompt can produce a short, broken program that looks simple. We instead score each prompt's structure on six dimensions from the prompt alone, never from generated code or test results. We draw 5,000 Python prompts across six bands of a preliminary single-rater score, and four out-of-panel LLM judges rescore every prompt (19,997 scores; composite ICC(2,1) = 0.872 on the 4,998 prompts with all four). Twenty-one models answer every prompt (105,000 generations), and an independent LLM auditor, checked against known-answer cases, reviews each unit-test verdict. Averaged over models, pass rate has a nonmonotone breakpoint at composite $\hat{\gamma}=14.0$: 83.4\% at or below, 90.1\% above. That jump is not a law of difficulty. With task-type fixed effects the breakpoint stays at 14.0 but the adjusted jump shrinks from 6.8 to 0.4 points, and neither of the benchmark's two prompt sources reproduces the jump on its own. Nineteen models pass more often above their own breakpoint and two less often, and one of those two declines, Gemini 3.1 Pro Preview's, comes from 156 API calls that returned nothing. In post hoc follow-up, Kimi K2.5 shows the clearest decline: with a prompt-source control, and within the later source alone, its break is at 14.25, where its later-source pass rate falls from 0.848 to 0.808 (95\% interval $[14.00,14.75]$). A 365-prompt audit-clean extension matches the original raw test outcomes at bins 15 and 16 but adds only 14 prompts above bin 16. Among failed generations with measurable output complexity, 25.0\% pair a prompt composite above 8 with output complexity of at most 10, twice the share among passing ones. The pooled breakpoint moves to 11.25 on raw test outcomes and 13.75 under the partial audit used at submission. Human agreement with the index is moderate and rater-dependent, paraphrase and cross-language rescoring preserve its ordering, and overidentification tests reject the six dimensions as instruments, so we treat the composite as an index and make no causal claim. The contribution is a prompt-side way to locate where each model's pass rate changes and to test how much that depends on the prompt mix and on how success is scored.
Chat is not available.
Successful Page Load