Feature Space Adaptation for Mitigating Forgetting in Few-Shot Fine-Tuning

Few-shot fine-tuning adapts large-scale pre-trained models using limited downstream supervision, but may degrade previously acquired knowledge. Existing forgetting-mitigation methods mainly limit parameter drift, while recent work suggests that forgetting is also related to adaptation direction. We propose feature space adaptation, which constrains adapted weights to remain within the dominant column subspace of the pre-trained weights. We develop Low-rank Feature Adaptation (LoRFA) and Vector-based Feature Adaptation (VeFA), both satisfying this constraint by construction. On seven 16-shot CLIP image classification datasets, both methods maintain competitive fine-tuning performance while reducing forgetting on held-out tasks.