Evaluating Foundation Models in Support of Climate-Resilient Plant Breeding

Automated plant phenotyping from real-world agricultural imagery is critical for breeding climate-resilient crops, yet computer vision models struggle in open fields due to dense canopies, weeds, and structural diversity across crops. To assess whether emerging foundation models can overcome this bottleneck without crop-specific retraining, we evaluate five multimodal vision-language configurations---OWLViT+SAM, PaliGemma+SAM, SAM+Gemma 3, SAM+Gemini, and direct Gemini segmentation---in a zero-shot setting on plant organ and weed instance segmentation across 4,786 mobile field images (1,101,930 annotations) spanning nine crops in Malawi and Tanzania. Our findings reveal a substantial in-the-wild multi-class segmentation challenge (peaking at $13.0\% \pm 7.0\%$ unweighted multi-crop aggregate mIoU for our strongest configuration, direct Gemini 2.5 Flash): while models achieve moderate zero-shot segmentation on broad leaves and flowers ($50.0\%\text{--}65.0\%$ IoU on staples), performance drops sharply on fine-grained non-leaf structures (stems, buds, weeds). Crucially, our benchmark uncovers a pronounced \textit{representation disparity}: models achieve $1.7\times$--$2.3\times$ higher mIoU on globally cultivated staples (\eg, $26.0\%$ on Irish potato, $22.0\%$ on maize) than on drought-hardy orphan crops vital for smallholder climate resilience (\eg, $4.0\%$ on pearl millet, $7.0\%$ on pigeon pea). These results suggest that pre-training representation biases in web-scale corpora, compounded by divergent morphological architectures in orphan species, constrain zero-shot transfer on regional climate-critical crops, highlighting the need for accessible domain adaptation in public breeding programs.