Timezone: »

 
Oral
Causal Intervention for Weakly-Supervised Semantic Segmentation
Dong Zhang · Hanwang Zhang · Jinhui Tang · Xian-Sheng Hua · Qianru Sun

Tue Dec 08 06:15 AM -- 06:30 AM (PST) @ Orals & Spotlights: Vision Applications
in Orals & Spotlights Track 07: Vision Applications »

We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels -- the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of "horse" and "person" may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.

Keywords: Generative Models · Algorithms · Uncertainty Estimation · Algorithms -> Active Learning; Algorithms -> Regression; Deep Learning

Author Information

Dong Zhang (Nanjing University of Science and Technology)
Hanwang Zhang (NTU)
Jinhui Tang (Nanjing University of Science and Technology)
Xian-Sheng Hua (Damo Academy, Alibaba Group)
Qianru Sun (Singapore Management University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors