Does Patient-Level Splitting Change What a Colposcopy Classifier Looks At? A Grad-CAM Shortcut Audit

In sub-Saharan Africa, cervical cancer kills more women than any other cancer, and deep learning classifiers on colposcopy images have reported near-expert diagnostic accuracy. Colposcopy datasets routinely contain several correlated images per woman, a structural property that lets an image-level train/test split leak patient identity rather than disease signal. Using a public, multi-magnification Nigerian colposcopy dataset (3,356 images, 332 women), we confirm this directly: an identical pipeline scoring AUC 0.91–0.93 under a naive image-level split collapses to near chance (AUC 0.50–0.55) once no woman's images cross the train/test boundary. This raises a further question: when a model is leaking, what is it actually looking at, and does fixing the split change the answer? We audit this with Grad-CAM, training a softmax baseline and a class-weighted Dirichlet evidential classifier under both split protocols, then measuring shortcut reliance as the fraction of attention falling on non-tissue pixels. The result is architecture-dependent: the softmax model shows significantly more shortcut reliance when leaky, while the evidential model shows the reverse. Across all four conditions, however, 38–45% of attention lands on non-tissue pixels regardless of split, even accuracy-inflated models are not reliably reasoning over colposcopic tissue.