Timezone: »

 
Poster
All Tokens Matter: Token Labeling for Training Better Vision Transformers
Zi-Hang Jiang · Qibin Hou · Li Yuan · Daquan Zhou · Yujun Shi · Xiaojie Jin · Anran Wang · Jiashi Feng

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @ Virtual
in Poster Session 3 »

In this paper, we present token labeling---a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pretrained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and model are publiclyavailable at https://github.com/zihangJiang/TokenLabeling.

Keywords: Deep Learning · Machine Learning · Transformers · Vision

Author Information

Zi-Hang Jiang (National University of Singapore)
Qibin Hou (Nankai University)
Li Yuan (National University of Singapore)
Daquan Zhou (National University of Singapore)
Yujun Shi (Pensees Pte. Ltd.)
Xiaojie Jin (National University of Singapore & Snap Research)
Anran Wang (Nanyang Technological University)
Jiashi Feng (UC Berkeley)

More from the Same Authors