Timezone: »
Poster
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
Dinghuai Zhang · Tianyuan Zhang · Yiping Lu · Zhanxing Zhu · Bin Dong
Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #145
Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin’s Maximum Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (\textbf{Y}ou \textbf{O}nly \textbf{P}ropagate \textbf{O}nce). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with \textbf{approximately 1/5 $\sim$ 1/4 GPU time} of the projected gradient descent (PGD) algorithm~\cite{kurakin2016adversarial}.
Author Information
Dinghuai Zhang (Peking University)
Tianyuan Zhang (Peking University)
I am a last-year undergraduate at Peking University. I will apply for Ph.D this year. Broadly interested in vision and learning.
Yiping Lu (Peking University)
Zhanxing Zhu (Peking University)
Bin Dong (Peking University)
More from the Same Authors
-
2021 Spotlight: Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay »
Ruosi Wan · Zhanxing Zhu · Xiangyu Zhang · Jian Sun -
2023 Poster: Unsupervised Image Denoising with Score Function »
Yutong Xie · Mingze Yuan · Bin Dong · Quanzheng Li -
2023 Poster: Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling »
ting li · Jianguo Li · Zhanxing Zhu -
2023 Poster: Implicit Bias of (Stochastic) Gradient Descent for Rank-1 Linear Neural Network »
Bochen Lv · Zhanxing Zhu -
2022 Spotlight: Meta-Auto-Decoder for Solving Parametric Partial Differential Equations »
Xiang Huang · Zhanhong Ye · Hongsheng Liu · Shi Ji · Zidong Wang · Kang Yang · Yang Li · Min Wang · Haotian CHU · Fan Yu · Bei Hua · Lei Chen · Bin Dong -
2022 Spotlight: Lightning Talks 4B-1 »
Alexandra Senderovich · Zhijie Deng · Navid Ansari · Xuefei Ning · Yasmin Salehi · Xiang Huang · Chenyang Wu · Kelsey Allen · Jiaqi Han · Nikita Balagansky · Tatiana Lopez-Guevara · Tianci Li · Zhanhong Ye · Zixuan Zhou · Feng Zhou · Ekaterina Bulatova · Daniil Gavrilov · Wenbing Huang · Dennis Giannacopoulos · Hans-peter Seidel · Anton Obukhov · Kimberly Stachenfeld · Hongsheng Liu · Jun Zhu · Junbo Zhao · Hengbo Ma · Nima Vahidi Ferdowsi · Zongzhang Zhang · Vahid Babaei · Jiachen Li · Alvaro Sanchez Gonzalez · Yang Yu · Shi Ji · Maxim Rakhuba · Tianchen Zhao · Yiping Deng · Peter Battaglia · Josh Tenenbaum · Zidong Wang · Chuang Gan · Changcheng Tang · Jessica Hamrick · Kang Yang · Tobias Pfaff · Yang Li · Shuang Liang · Min Wang · Huazhong Yang · Haotian CHU · Yu Wang · Fan Yu · Bei Hua · Lei Chen · Bin Dong -
2022 Poster: Meta-Auto-Decoder for Solving Parametric Partial Differential Equations »
Xiang Huang · Zhanhong Ye · Hongsheng Liu · Shi Ji · Zidong Wang · Kang Yang · Yang Li · Min Wang · Haotian CHU · Fan Yu · Bei Hua · Lei Chen · Bin Dong -
2021 : Layer-Parallel Training of Residual Networks with Auxiliary Variables »
Qi Sun · Hexin Dong · Zewei Chen · WeiZhen Dian · Jiacheng Sun · Yitong Sun · Zhenguo Li · Bin Dong -
2021 Poster: Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay »
Ruosi Wan · Zhanxing Zhu · Xiangyu Zhang · Jian Sun -
2020 Poster: Black-Box Certification with Randomized Smoothing: A Functional Optimization Based Framework »
Dinghuai Zhang · Mao Ye · Chengyue Gong · Zhanxing Zhu · Qiang Liu -
2020 Poster: Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher »
Guangda Ji · Zhanxing Zhu -
2018 Poster: Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning »
Rui Luo · Jianhong Wang · Yaodong Yang · Jun WANG · Zhanxing Zhu -
2018 Poster: Reinforced Continual Learning »
Ju Xu · Zhanxing Zhu -
2018 Poster: Bayesian Adversarial Learning »
Nanyang Ye · Zhanxing Zhu -
2017 Poster: Langevin Dynamics with Continuous Tempering for Training Deep Neural Networks »
Nanyang Ye · Zhanxing Zhu · Rafal Mantiuk