Timezone: »

 
Poster
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
Dinghuai Zhang · Tianyuan Zhang · Yiping Lu · Zhanxing Zhu · Bin Dong

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #145
in Deep Learning -- Optimization for Deep Networks »
Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin’s Maximum Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (\textbf{Y}ou \textbf{O}nly \textbf{P}ropagate \textbf{O}nce). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with \textbf{approximately 1/5 $\sim$ 1/4 GPU time} of the projected gradient descent (PGD) algorithm~\cite{kurakin2016adversarial}.
Keywords: Optimization for Deep Networks · Control Theory · Deep Learning · Deep Learning -> Adversarial Networks; Theory
[ Paper]  [ Poster] 

Author Information

Dinghuai Zhang (Peking University)
Tianyuan Zhang (Peking University)

I am a last-year undergraduate at Peking University. I will apply for Ph.D this year. Broadly interested in vision and learning.

Yiping Lu (Peking University)
Zhanxing Zhu (Peking University)
Bin Dong (Peking University)

More from the Same Authors