Timezone: »
We propose and study a general framework for regularized Markov decision processes (MDPs) where the goal is to find an optimal policy that maximizes the expected discounted total reward plus a policy regularization term. The extant entropy-regularized MDPs can be cast into our framework. Moreover, under our framework, many regularization terms can bring multi-modality and sparsity, which are potentially useful in reinforcement learning. In particular, we present sufficient and necessary conditions that induce a sparse optimal policy. We also conduct a full mathematical analysis of the proposed regularized MDPs, including the optimality condition, performance error, and sparseness control. We provide a generic method to devise regularization forms and propose off-policy actor critic algorithms in complex environment settings. We empirically analyze the numerical properties of optimal policies and compare the performance of different sparse regularization forms in discrete and continuous environments.
Author Information
Wenhao Yang (Peking University)
Xiang Li (Peking University)
Zhihua Zhang (Peking University)
More from the Same Authors
-
2022 Poster: Personalized Federated Learning towards Communication Efficiency, Robustness and Fairness »
Shiyun Lin · Yuze Han · Xiang Li · Zhihua Zhang -
2022 Poster: Asymptotic Behaviors of Projected Stochastic Approximation: A Jump Diffusion Perspective »
Jiadong Liang · Yuze Han · Xiang Li · Zhihua Zhang -
2022 Poster: Semi-infinitely Constrained Markov Decision Processes »
Liangyu Zhang · Yang Peng · Wenhao Yang · Zhihua Zhang -
2022 Spotlight: Personalized Federated Learning towards Communication Efficiency, Robustness and Fairness »
Shiyun Lin · Yuze Han · Xiang Li · Zhihua Zhang -
2022 Spotlight: Lightning Talks 3A-1 »
Shu Ding · Wanxing Chang · Jiyang Guan · Mouxiang Chen · Guan Gui · Yue Tan · Shiyun Lin · Guodong Long · Yuze Han · Wei Wang · Zhen Zhao · Ye Shi · Jian Liang · Chenghao Liu · Lei Qi · Ran He · Jie Ma · Zemin Liu · Xiang Li · Hoang Tuan · Luping Zhou · Zhihua Zhang · Jianling Sun · Jingya Wang · LU LIU · Tianyi Zhou · Lei Wang · Jing Jiang · Yinghuan Shi -
2022 Poster: A Statistical Online Inference Approach in Averaged Stochastic Approximation »
Chuhan Xie · Zhihua Zhang