CoCoA: Cohort-Aware Certification or Abstention for Precision Under Shift

Standard evaluation tools typically assume independently and identically distributed sampling, an assumption that has been observed to fail in a range of real deployment settings. Moreover, existing shift approaches, including conformal prediction, importance weighting, and distributionally robust optimization (DRO), each address parts of this problem, but there is currently no unified protocol that returns formal certify-or-abstain decisions on PPV under shift with multiple-testing control. As such, we present CoCoA, a benchmarking protocol that, given a classifier, a calibration set, and a target distribution, returns a per-cohort certify-or-abstain decision on precision with tunable controls over the family-wise false-certification rate across resulting (cohort, 𝜏) hypotheses. In doing so, we also find that a standard shift-aware tool becomes anti-conservative, certifying mathematically false claims when repurposed for certification.