Timezone: »

 
Spotlight
Sparse Winning Tickets are Data-Efficient Image Recognizers
Mukund Varma T · Xuxi Chen · Zhenyu Zhang · Tianlong Chen · Subhashini Venugopalan · Zhangyang Wang

Tue Dec 06 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 2A-3 »

Improving the performance of deep networks in data-limited regimes has warranted much attention. In this work, we empirically show that “winning tickets” (small sub-networks) obtained via magnitude pruning based on the lottery ticket hypothesis, apart from being sparse are also effective recognizers in data-limited regimes. Based on extensive experiments, we find that in low data regimes (datasets of 50-100 examples per class), sparse winning tickets substantially outperform the original dense networks. This approach, when combined with augmentations or fine-tuning from a self-supervised backbone network, shows further improvements in performance by as much as 16% (absolute) on low-sample datasets and long-tailed classification. Further, sparse winning tickets are more robust to synthetic noise and distribution shifts compared to their dense counterparts. Our analysis of winning tickets on small datasets indicates that, though sparse, the networks retain density in the initial layers and their representations are more generalizable. Code is available at https://github.com/VITA-Group/DataEfficientLTH.

Author Information

Mukund Varma T (Indian Institute of Technology Madras)
Xuxi Chen (University of Texas at Austin)
Zhenyu Zhang (University of Texas at Austin)
Tianlong Chen (Unversity of Texas at Austin)
Subhashini Venugopalan (Google)
Zhangyang Wang (University of Texas at Austin)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors