Gated Knowledge-Guided Prototype Transfer for Zero-Support Medical Vision-Language Adaptation
Abstract
Realistic few-shot adaptation of medical vision-language models (VLMs) can leave low-prevalence classes without any labeled support. The training-free ss-text+ solver remains operational in this setting, but its zero-support prototypes receive no visual adaptation and fall back to a completely textual class prototype. We introduce gated Knowledge-Guided Prototype Transfer gated KGPT, a pre-repulsion module that transfers supported-class visual residuals to missing classes using VLM similarity and an ontology-based clinical distance. A validation-free evidence gate scales transfer based on semantic relevance and residual agreement, thereby limiting unsupported extrapolation. In 50 paired realistic one-shot trials on MESSIDOR with FLAIR features, severe and proliferative diabetic retinopathy were absent from every support set. Relative to ss-text+, gated KGPT improved missing-class recall by 5.23\%, average class accuracy by 1.37\%, and the missing/supported recall harmonic mean by 1.83\%. It reduced negative-transfer frequency from 46 to 48\% for ungated transfer to 2\%, while leads to 1.20\% supported-class recall reduction and a 0.45\% top-1 reduction. These results identify that evidence-based gating provides a safety mechanism for the knowledge source.