Timezone: »

 
Poster
Iteratively Learn Diverse Strategies with State Distance Information
Wei Fu · Weihua Du · Jingwei Li · Sunli Chen · Jingzhao Zhang · YI WU

Tue Dec 12 03:15 PM -- 05:15 PM (PST) @ Great Hall & Hall B1+B2 #808
in Poster Session 2 »
Event URL: https://sites.google.com/view/diversity-sipo »

In complex reinforcement learning (RL) problems, policies with similar rewards may have substantially different behaviors. It remains a fundamental challenge to optimize rewards while also discovering as many diverse strategies as possible, which can be crucial in many practical applications. Our study examines two design choices for tackling this challenge, i.e., diversity measure and computation framework. First, we find that with existing diversity measures, visually indistinguishable policies can still yield high diversity scores. To accurately capture the behavioral difference, we propose to incorporate the state-space distance information into the diversity measure. In addition, we examine two common computation frameworks for this problem, i.e., population-based training (PBT) and iterative learning (ITR). We show that although PBT is the precise problem formulation, ITR can achieve comparable diversity scores with higher computation efficiency, leading to improved solution quality in practice. Based on our analysis, we further combine ITR with two tractable realizations of the state-distance-based diversity measures and develop a novel diversity-driven RL algorithm, State-based Intrinsic-reward Policy Optimization (SIPO), with provable convergence properties. We empirically examine SIPO across three domains from robot locomotion to multi-agent games. In all of our testing environments, SIPO consistently produces strategically diverse and human-interpretable policies that cannot be discovered by existing baselines.

Author Information

Wei Fu (Institute for Interdisciplinary Information Sciences, Tsinghua University, Tsinghua University)
Weihua Du (IIIS, Tsinghua University)
Jingwei Li (Tsinghua University, Tsinghua University)
Sunli Chen (Tsinghua University)
Jingzhao Zhang (Tsinghua University, Tsinghua University)
YI WU (Shanghai Qi Zhi Institute & Tsinghua University)

More from the Same Authors