Government AI Policy: Outcome Maturity, Calibration, and Threshold Governance in Credit Decisions
Abstract
\begin{abstract} AI credit models are often governed through aggregate accuracy tests, although their operational consequences also depend on outcome definition, probability calibration, operating thresholds, and portfolio composition. Using 2{,}703{,}430 US peer-to-peer loans evaluated chronologically and restricted to outcome-mature observations, we show that only 41{,}145 of 401{,}169 test-period loans (10.3\%) are outcome-mature, and their default rate is 14.64\% versus 2.96\% when immature labels are treated as observed. Across three seeds, regularised logistic regression matches or exceeds a multilayer perceptron and histogram gradient boosting on out-of-time ROC-AUC (0.6762 vs.\ 0.6715 and 0.6706), so added capacity yields no measurable discrimination premium. The validation-selected gradient-boosting model under-predicts non-stress default risk by roughly a factor of two (mean prediction 0.0702 against observed 0.1469; ten-bin Expected Calibration Error 0.0768). The stress subset contains only 641 loans, so regime comparisons are underpowered and are reported as inconclusive. Validation-based cost optimisation selects thresholds from 0.0292 to 0.1984 as the assumed false-negative to false-positive cost ratio varies from 20:1 to 5:1; the associated review rate moves from 78.2\% to 1.3\% and the false-negative rate from 7.90\% to 96.95\%. In an exact-matched audit, grade-A 36-month loans show FNR of 1.0000 in both regimes at the 10:1 threshold, with seven realised defaults per matched cell. We consolidate the evidence into controls for outcome readiness, calibration, regime monitoring, threshold governance, worst-slice assurance, human oversight, and lineage, and discuss how these controls would need to be adapted for Shariah-compliant lenders serving Muslim communities in Australia. The empirical work is on US data; the Australia discussion is presented as implications, not findings. \end{abstract}