Residual Prototypes for Open-World Multi-Label Recognition

Automated retinal screening is almost always posed as classification over a fixed disease vocabulary. Deployment violates that assumption in a clinically consequential way: the dangerous image is rarely one that is wholly unfamiliar, but one that is partly familiar. A patient presenting diabetic retinopathy, which the model knows, alongside a second pathology absent from the training taxonomy. A closed-set model then fails silently, forcing the unexplained evidence onto the nearest known label or reporting high confidence on the label it did recognise, implying the read is complete. Open-set recognition does not resolve this: the image is not out-of-distribution, part of it is squarely in-distribution. Existing retinal uncertainty methods decide at the image level, so they must reject the whole image, discarding correct predictions. We formulate this as mixed known/unknown multi-label recognition, asking not whether the image is unfamiliar but which evidence is explained by known disease concepts and whether clinically meaningful residual evidence remains. Our method represents each disease $c$ by a learned prototype $p_c$, retrieves evidence with label queries over patch tokens from a LoRA-adapted DINOv2 backbone, and decomposes those tokens into an explained and a residual stream. Each token $z_i$ is scored against the best-matching active prototype, $$e_i = \max_{c \in \mathcal{K}} \langle \bar{z}_i, \bar{p}_c \rangle,$$ which a sigmoid turns into an explained fraction $w_i = \sigma((e_i - \delta)/\tau_e)$ and an unexplained remainder $r_i = (1 - w_i) z_i$. The residual stream is pooled by additive attention into an unknown-pathology score $\hat{u}$ and a spatial heatmap showing where the known vocabulary fails.Since the dataset provides no unknown labels, we synthesise the supervision. Each episode designates a frequency-stratified subset $M_e$ of diseases as pseudo-unknown, masks their prototypes, and sets the target $$u_e(x) = \mathbb{1}[\, Y(x) \cap M_e \neq \emptyset \,]$$ while retaining supervision for $Y(x) \setminus M_e$. This asymmetry separates episodic withholding from label dropout: an image with one withheld and one active disease becomes a synthetic mixed case, its evidence routed to the residual branch while the other is still predicted correctly. We evaluate on RFMiD (3,200 images, 45 diseases) under five pre-registered, frequency-stratified label-holdout folds, with RFMiD 2.0 and JSIEC held out as external cohorts. Results are means over five seeds; unknown AUPRC and known mAP were nominated in advance as confirmatory endpoints under Benjamini–Hochberg control. The method reaches 0.767 unknown AUROC and 0.464 AUPRC, against 0.738/0.426 for the strongest of five baselines, while known-label mAP stays within the band spanned by the closed-set architectures themselves. Ablations localise the effect by removing the residual branch costs 0.149 AUPRC ($p < 0.001$), disabling episodic withholding 0.095 ($p < 0.001$). The latter is parameter-matched and keeps the branch and its losses, so its mechanism is informative without withholding, $\hat{u}$ collapses to 33 distinct values near zero and restored, it spans 510. The branch dies rather than deteriorating. We also report components that did not work, with mechanisms. Anatomy-grounded coordinates gave no measurable benefit ($p= 0.191$): acquisition and field-of-view cropping already canonicalise the disc–fovea axis to $7.6^{\circ}$, about 1.5 patches on a $37 \times 37$ grid, leaving almost no variance to remove. And the residual score alone is the weakest detector (0.626 AUROC, below energy at 0.663) . The gain comes from its anticorrelation with energy ($r = -0.68$), complementarity rather than raw detector strength.