Timezone: »

 
Poster
ReCo: Retrieve and Co-segment for Zero-shot Transfer
Gyungin Shin · Weidi Xie · Samuel Albanie

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #926
in Poster Session 5 »

Semantic segmentation has a broad range of applications, but its real-world impact has been significantly limited by the prohibitive annotation costs necessary to enable deployment. Segmentation methods that forgo supervision can side-step these costs, but exhibit the inconvenient requirement to provide labelled examples from the target distribution to assign concept names to predictions. An alternative line of work in language-image pre-training has recently demonstrated the potential to produce models that can both assign names across large vocabularies of concepts and enable zero-shot transfer for classification, but do not demonstrate commensurate segmentation abilities.We leverage the retrieval abilities of one such language-image pre-trained model, CLIP, to dynamically curate training sets from unlabelled images for arbitrary collections of concept names, and leverage the robust correspondences offered by modern image representations to co-segment entities among the resulting collections. The synthetic segment collections are then employed to construct a segmentation model (without requiring pixel labels) whose knowledge of concepts is inherited from the scalable pre-training process of CLIP. We demonstrate that our approach, termed Retrieve and Co-segment (ReCo) performs favourably to conventional unsupervised segmentation approaches while inheriting the convenience of nameable predictions and zero-shot transfer. We also demonstrate ReCo’s ability to generate specialist segmenters for extremely rare objects.

Keywords: zero-shot transfer · semantic segmentation · co-segmentation · vision-language models · image retrieval
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Gyungin Shin (Visual Geometry Group, Oxford)
Weidi Xie (University of Oxford)
Samuel Albanie (Oxford University)

More from the Same Authors