Evaluating Skin Tone Fairness in an Open Vision-Language Model for Dermatological Triage

Introduction. It is estimated that an average of 3 billion people lack access to dermatological care (Daneshjou et al., 2022). Dermatology AI performs unevenly across skin tones: Daneshjou et al. (2022) found ROC-AUC drops of 27–36% on dark skin on the Diverse Dermatology Images (DDI) benchmark, and Patel Housley et al. (2025) found significantly lower sensitivity, specificity, and accuracy for ChatGPT-4o melanoma detection in darker Fitzpatrick types (FST III–VI) versus lighter (FST I–II). Both evaluated a purpose-trained classifier or a closed proprietary model. Open-weight vision language models (VLMs) proposed as low-cost diagnostic aids for specialists in regions with scarce specialists including much of sub-Saharan Africa remain untested on this axis, despite being the most realistic candidates for deployment where skin-tone bias would cause the most harm. Methodology & Data. We evaluated Qwen2-VL-2B-Instruct, an open-weight VLM, on zero-shot benign-versus-malignant lesion classification with a fixed diagnostic prompt, no fine-tuning, and no in-context examples, comparing outputs to biopsy confirmed ground truth. We used the DDI dataset (Daneshjou et al., 2022): 656 biopsy confirmed images from 570 patients, curated for balanced Fitzpatrick (FST) representation via dermatologist chart review (208 light FST I–II, 241 medium FST III–IV, 207 dark FST V–VI). Accuracy, sensitivity, specificity, and F1-score were computed per group. Results Accuracy declined 14 points from light to dark skin (72% vs. 58%), while sensitivity declined by a much larger 23 points (61% vs. 38%); specificity moved oppositely (76%→82%). The model's errors on dark skin are thus not evenly distributed: rather than misclassifying broadly, it systematically under-detects malignancy in darker skin while more readily defaulting to “benign.” Conclusion & Recommendation. Consistent with prior findings on purpose trained classifiers (Daneshjou et al., 2022) and a proprietary model (Patel Housley et al., 2025), this open weight VLM also shows substantial skin-tone disparity specifically a bias toward missing malignancies in darker skin, the costlier error. This matters because open VLMs are realistically deployable at low cost precisely in the specialist-scarce settings, including across much of Africa, where this bias would cause the most harm and go most easily undetected. Fairness auditing attending to sensitivity/specificity asymmetry, not only aggregate accuracy should precede any recommendation of open VLMs as dermatological triage aids in global health contexts. We recommend: (1) targeted fine-tuning on underrepresented skin tones prior to deployment; (2) skin-tone-aware prompting as a lower cost interim mitigation; and (3) mandatory disaggregated performance reporting as standard practice for dermatology AI proposed for diverse-skin-tone populations.