Skip to yearly menu bar Skip to main content


Oral Poster

Bayesian-guided Label Mapping for Visual Reprogramming

Chengyi Cai · Zesheng Ye · Lei Feng · Jianzhong Qi · Feng Liu

East Exhibit Hall A-C #1901
[ ]
Fri 13 Dec 4:30 p.m. PST — 7:30 p.m. PST
 
Oral presentation: Oral Session 6B: Safety, New Data
Fri 13 Dec 3:30 p.m. PST — 4:30 p.m. PST

Abstract:

Visual reprogramming (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing a gradient-free one-to-one correspondence between the two sets of labels.However, in this paper, we reveal that one-to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a Bayesian-guided Label Mapping (BLM) method. BLM constructs an iteratively-updated probabilistic label mapping matrix, with each element quantifying a pairwise relationship between pretrained and downstream labels.The assignment of values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g., ResNeXt) and vision-language models (e.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers a probabilistic lens through which to understand and analyze the effectiveness of VR.Our code is available at https://github.com/tmlr-group/BayesianLM.

Live content is unavailable. Log in and register to view live content