Timezone: »

 
Poster
SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
Chuanyang Zheng · zheyang li · Kai Zhang · Zhi Yang · Wenming Tan · Jun Xiao · Ye Ren · Shiliang Pu

@

Vision Transformers (ViTs) yield impressive performance across various vision tasks. However, heavy computation and memory footprint make them inaccessible for edge devices. Previous works apply importance criteria determined independently by each individual component to prune ViTs. Considering that heterogeneous components in ViTs play distinct roles, these approaches lead to suboptimal performance. In this paper, we introduce joint importance, which integrates essential structural-aware interactions between components for the first time, to perform collaborative pruning. Based on the theoretical analysis, we construct a Taylor-based approximation to evaluate the joint importance. This guides pruning toward a more balanced reduction across all components. To further reduce the algorithm complexity, we incorporate the interactions into the optimization function under some mild assumptions. Moreover, the proposed method can be seamlessly applied to various tasks including object detection. Extensive experiments demonstrate the effectiveness of our method. Notably, the proposed approach outperforms the existing state-of-the-art approaches on ImageNet, increasing accuracy by 0.7% over the DeiT-Base baseline while saving 50% FLOPs. On COCO, we are the first to show that 70% FLOPs of FasterRCNN with ViT backbone can be removed with only 0.3% mAP drop. The code is available at https://github.com/hikvision-research/SAViT.

Keywords: Pruning · compression · Vision transformer
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Chuanyang Zheng (Hikvision Research Institute)
zheyang li (Hikvision Research Institute)
Kai Zhang (Hikvision Research Institute)
Zhi Yang (University of Science and Technology of China)
Wenming Tan (Hikvision Research Institute)
Jun Xiao (Zhejiang University)
Ye Ren (Zhejiang University)
Shiliang Pu (Zhejiang University)

More from the Same Authors