Agreement-Gated Distillation for Selective Forgetting in Knowledge Revision

We formalize \emph{Selective Forgetting for Knowledge Revision} (SF-KR): correcting a class's learned mapping when higher-quality data arrives. Continual-learning protection assumes old knowledge should be uniformly preserved; revision breaks that assumption: some old behavior is precisely what must be overwritten. Standard Knowledge Distillation (KD) toward a frozen pre-revision teacher protects everything equally. Reducing the weight of the KD loss does not recover effective revision. This is a failure mode we call \emph{Revision Deadlock}. The key question shifts from \emph{how strongly} to protect old knowledge to \emph{which} old knowledge should be protected. We propose, \emph{Revision-Masked Distillation} (RMD), which is an oracle reference, assuming privileged knowledge of the revised classes. As this information may not be available in practice, we also propose \emph{Agreement-Gated Distillation} (AGD), which attempts to estimate which knowledge should be retained and which should be revised based on how strongly the teacher's own prediction agrees with each incoming label, without access to a revised-class list. Compared by their \emph{Pareto frontier} over Revision and Retention Accuracy rather than a single operating point, RMD's frontier dominates KD's entirely and AGD's substantially improves on KD's under equal information, on both CIFAR-100 and CUB-200-2011.