From Stacked Predictions to Decisions: A Contextual Optimization Approach

Optimization models and decision frameworks have seen increased use of the combination of multiple first-level learners using ensemble methods to improve predictive performance. In this work, we cast stacking as a contextual optimization problem, where a collection of $T$ first-level learners $\{h_t\}_{t=1}^{T}$ serve as context, a meta-learner estimates the target $Y$ conditional on this context, and decisions are selected to minimize the resulting conditional expected cost. We propose an empirical residual-based sample average approximation (ER-SAA) pipeline that incorporates the predictive uncertainty into the decision stage. Specifically, we (i) obtain out-of-sample first-level learners through cross-fitting, (ii) fit a meta-learner and estimate a possibly heteroskedastic scale map, and (iii) add back cross-fitted residuals as decision scenarios. This construction preserves stacking’s modeling flexibility and at the same time aligns learning with the conditional decision objective.