Aligning Task Gradients to a Frozen Encoder: Regularizing Semi-supervised Segmentation Without Freezing

Semi-supervised semantic segmentation trains on a small labeled set and a large unlabeled set, typically via weak-to-strong pseudo-labeling as in FixMatch and UniMatch. Such methods are prone to confirmation bias: high-confidence incorrect predictions are repeatedly reinforced during training, causing errors to propagate. UniMatch V2 reports that in the scarcest-label regime, freezing the pretrained encoder outperforms fine-tuning it, suggesting the frozen encoder is a strong regularizer against both confirmation bias and overfitting. We ask whether that regularization can be retained while still fine-tuning. Casting supervised and pseudo-label training as two conflicting tasks, we align each task’s backbone gradient against a reference gradient computed through a frozen copy of the pretrained encoder: a direction anchored to the pretrained weights that cannot have drifted with the student. On PASCAL VOC this raises best-checkpoint mIoU in the two scarcest splits (+0.95 at 92 labels, +0.64 at 183, three paired seeds) and reduces seed variance there, is neutral or slightly negative with more labels.