Self-Supervised Bisimulation Action Chunk Representation for Efficient RL

Action chunking in reinforcement learning is a promising approach, as it significantly reduces decision-making frequency and leads to more consistent behavior. However, due to inherent differences between the action chunk space and the original action space, uncovering its underlying structure is crucial. Previous works cope with this challenge of single-step action space through action representation methods, but directly applying these methods to action chunk space fails to capture the semantic information of multi-step behaviors. In this paper, we introduce \textbf{A}ction \textbf{C}hunk \textbf{R}epresentation (\textbf{ACR}), a self-supervised representation learning framework for uncovering the underlying structure of the action chunk space to achieve efficient RL. To build the framework, we propose the action chunk bisimulation metric to measure the principled distance between action chunks. With this metric, ACR encodes action chunks with a Transformer that extracts the temporal structure and learns a latent representation space where action churns with similar bisimulation behavior semantics are close to each other. The latent policy is then trained in the representation space, and the selected latent action chunk is decoded back into the original space to interact with the environment. We flexibly integrate ACR with various DRL algorithms and evaluate it on a range of continuous manipulation and navigation tasks. Experiments show that ACR surpasses existing action representation baselines in terms of both learning efficiency and performance.