Adaptive Hierarchical Robust Model Selection: The Cost of Certification under Grouped Evaluation

Selecting a probabilistic classifier is difficult when evaluation labels are costly and related observations must be assessed together. We introduce Adaptive Hierarchical Robust Model Selection (A-HRMS), which selects from a fixed model library using Brier risk. It treats related observations as indivisible groups, allows unequal weights within a finite evaluation set, and protects against prespecified changes in the relative importance of biological contexts. Time-uniform confidence bounds support three outputs: all retained models, one fixed model, or a randomized choice among models. Randomization draws one model rather than averaging predictions, and its guarantee controls expected robust regret. At the primary tolerance $\epsilon=.05$, where $\epsilon$ is the maximum allowed worst-case excess Brier risk relative to the best library candidate, randomized output reduced mean evaluation from 82.6\% to 79.3\% of roots in a transposable-element frame and from 80.8\% to 75.2\% and 84.1\% to 79.2\% of wells in two RxRx1 replays. Savings occurred in every tested order. The randomized strategy did not consistently improve held-out risk, and simpler acquisition rules were sometimes cheaper. A-HRMS therefore lowers certification cost by changing the permitted output, rather than by universally improving acquisition or prediction.