Timezone: »

 
Poster
Contrastive Language-Image Pre-Training with Knowledge Graphs
Xuran Pan · Tianzhu Ye · Dongchen Han · Shiji Song · Gao Huang

@

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

Keywords: Knowledge graph · vision-language pre-training
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Xuran Pan (Tsinghua University)
Tianzhu Ye (Tsinghua University, Tsinghua University)
Dongchen Han (Tsinghua University)
Shiji Song (Department of Automation, Tsinghua University)
Gao Huang (Cornell University)

More from the Same Authors