Active-Set Identification by Stochastic and Noisy Optimization Algorithms for Constrained Learning

Identifying active constraints from a point near a solution is both theoretically and practically important in constrained continuous optimization, as it can help identify optimal Lagrange multipliers and allows the reduction of an inequality-constrained problem to an equality-constrained one. Furthermore, in the particular setting of constrained learning, the identification of active constraints can correspond to the identification of enlightening aspects, such as data points that lie on decision-boundary thresholds. Traditional guarantees for active-set identification assume exact function and derivative values under smoothness and constraint qualification conditions. This work extends these guarantees to settings with deterministic or stochastic noise in both objective and constraint evaluations, which is a critical development for the use of such techniques in machine learning. Two strategies are proposed where, under mild conditions, accurate active-set identification remains possible when a point is close to a local minimizer and noise is sufficiently small. High-probability guarantees are also established for the use of active-set identification strategies within a stochastic algorithm. We demonstrate our findings with illustrative examples and a constrained neural-network training task.