Timezone: »
Model-free deep reinforcement learning (RL) algorithms have been widely used for a range of complex control tasks. However, slow convergence and sample inefficiency remain challenging problems in RL, especially when handling continuous and high-dimensional state spaces. To tackle this problem, we propose a general acceleration method for model-free, off-policy deep RL algorithms by drawing the idea underlying regularized Anderson acceleration (RAA), which is an effective approach to accelerating the solving of fixed point problems with perturbations. Specifically, we first explain how policy iteration can be applied directly with Anderson acceleration. Then we extend RAA to the case of deep RL by introducing a regularization term to control the impact of perturbation induced by function approximation errors. We further propose two strategies, i.e., progressive update and adaptive restart, to enhance the performance. The effectiveness of our method is evaluated on a variety of benchmark tasks, including Atari 2600 and MuJoCo. Experimental results show that our approach substantially improves both the learning speed and final performance of state-of-the-art deep RL algorithms.
Author Information
Wenjie Shi (Tsinghua University)
Wenjie Shi received the B.S. degree from Huazhong University of Science and Technology, Wuhan, China, in 2016. He is currently pursuing the Ph.D. degree in control science and engineering with the Department of Automation, Tsinghua University, Beijing, China. His current research interests include deep reinforcement learning and robot control.
Shiji Song (Department of Automation, Tsinghua University)
Hui Wu (Tsinghua University)
Ya-Chu Hsu (Tsinghua University)
Cheng Wu (Tsinghua)
Gao Huang (Tsinghua)
More from the Same Authors
-
2021 Spotlight: Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning »
Yiqin Yang · Xiaoteng Ma · Chenghao Li · Zewu Zheng · Qiyuan Zhang · Gao Huang · Jun Yang · Qianchuan Zhao -
2022 Poster: Contrastive Language-Image Pre-Training with Knowledge Graphs »
Xuran Pan · Tianzhu Ye · Dongchen Han · Shiji Song · Gao Huang -
2022 Poster: Efficient Knowledge Distillation from Model Checkpoints »
Chaofei Wang · Qisen Yang · Rui Huang · Shiji Song · Gao Huang -
2022 Spotlight: Lightning Talks 1B-3 »
Chaofei Wang · Qixun Wang · Jing Xu · Long-Kai Huang · Xi Weng · Fei Ye · Harsh Rangwani · shrinivas ramasubramanian · Yifei Wang · Qisen Yang · Xu Luo · Lei Huang · Adrian G. Bors · Ying Wei · Xinglin Pan · Sho Takemori · Hong Zhu · Rui Huang · Lei Zhao · Yisen Wang · Kato Takashi · Shiji Song · Yanan Li · Rao Anwer · Yuhei Umeda · Salman Khan · Gao Huang · Wenjie Pei · Fahad Shahbaz Khan · Venkatesh Babu R · Zenglin Xu -
2022 Spotlight: Efficient Knowledge Distillation from Model Checkpoints »
Chaofei Wang · Qisen Yang · Rui Huang · Shiji Song · Gao Huang -
2022 Poster: Latency-aware Spatial-wise Dynamic Networks »
Yizeng Han · Zhihang Yuan · Yifan Pu · Chenhao Xue · Shiji Song · Guangyu Sun · Gao Huang -
2021 Poster: Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning »
Yiqin Yang · Xiaoteng Ma · Chenghao Li · Zewu Zheng · Qiyuan Zhang · Gao Huang · Jun Yang · Qianchuan Zhao -
2021 Poster: Searching Parameterized AP Loss for Object Detection »
Tao Chenxin · Zizhang Li · Xizhou Zhu · Gao Huang · Yong Liu · jifeng dai -
2021 Poster: Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition »
Yulin Wang · Rui Huang · Shiji Song · Zeyi Huang · Gao Huang -
2020 Poster: Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification »
Yulin Wang · Kangchen Lv · Rui Huang · Shiji Song · Le Yang · Gao Huang -
2019 Poster: Implicit Semantic Data Augmentation for Deep Networks »
Yulin Wang · Xuran Pan · Shiji Song · Hong Zhang · Gao Huang · Cheng Wu -
2019 Poster: Asymmetric Valleys: Beyond Sharp and Flat Local Minima »
Haowei He · Gao Huang · Yang Yuan -
2019 Spotlight: Asymmetric Valleys: Beyond Sharp and Flat Local Minima »
Haowei He · Gao Huang · Yang Yuan