Timezone: »
Low-level sensory and motor signals in the high-dimensional spaces (e.g., image observations or motor torques) in deep reinforcement learning are complicated to understand or harness for downstream tasks directly. While sensory representations have been widely studied, the representations of actions that form motor skills are yet under exploration. In this work, we find that when a multi-task policy network takes as input states and task embeddings, a space based on the task embeddings emerges to contain meaningful action representations with moderate constraints. Within this space, interpolated or composed embeddings can serve as a high-level interface to instruct the agent to perform meaningful action sequences. Empirical results not only show that the proposed action representations have efficacy for intra-action interpolation and inter-action composition with limited or no learning, but also demonstrate their superior ability in task adaptation to strong baselines in Mujoco locomotion tasks. The evidence elucidates that learning action representations is a promising direction toward efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Anonymous project page: https://sites.google.com/view/emergent-action-representation
Author Information
Pu Hua (Electronic Engineering, Tsinghua University, Tsinghua University)
Yubei Chen (Data Science, NYU Meta AI (FAIR))

I received my bachelor’s degree from the Electrical Engineering department at Tsinghua University, Beijing, in 2012. Then, I joined the EECS department and Berkeley AI Research (BAIR) at UC Berkeley to pursue my Ph.D. study on unsupervised learning and generative models, advised by Professor Bruno Olshausen. Along the way, I received my M.S. degree in EECS and M.A. degree in mathematics at Berkeley. In 2012, I was awarded the NSF GRFP fellowship. In 2019, I got my Ph.D. from the EECS department. In late 2020, I started to work with Yann LeCun at Meta AI (FAIR) and the Center for Data Science at NYU as a postdoctoral scholar, where I continued to work on unsupervised representation learning. I also serve as a reviewer for NeurIPS, ICLR, ICML, AAAI, CVPR, ECCV, Neural Computation, etc.
Huazhe Xu (Tsinghua University)
More from the Same Authors
-
2022 Poster: Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning »
Zhecheng Yuan · Zhengrong Xue · Bo Yuan · Xueqian Wang · YI WU · Yang Gao · Huazhe Xu -
2022 : Disentangling Images with Lie Group Transformations and Sparse Coding »
Ho Yin Chau · Frank Qiu · Yubei Chen · Bruno Olshausen -
2022 : Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation »
Linfeng Zhao · Huazhe Xu · Lawson Wong -
2023 Poster: CEIL: Generalized Contextual Imitation Learning »
Jinxin Liu · Li He · Yachen Kang · Zifeng Zhuang · Donglin Wang · Huazhe Xu -
2023 Poster: H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation »
Yanjie Ze · Yuyao Liu · Ruizhe Shi · Jiaxin Qin · Zhecheng Yuan · Jiashun Wang · Xiaolong Wang · Huazhe Xu -
2023 Poster: Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning? »
Jialu Gao · Kaizhe Hu · Guowei Xu · Huazhe Xu -
2023 Poster: $\texttt{TACO}$: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning »
Ruijie Zheng · Xiyao Wang · Yanchao Sun · Shuang Ma · Jieyu Zhao · Huazhe Xu · Hal Daumé III · Furong Huang -
2023 Poster: MoVie: Visual Model-Based Policy Adaptation for View Generalization »
Sizhe Yang · Yanjie Ze · Huazhe Xu -
2023 Poster: RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization »
Zhecheng Yuan · Sizhe Yang · Pu Hua · Can Chang · Kaizhe Hu · Xiaolong Wang · Huazhe Xu -
2022 : Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function »
Ruijie Zheng · Xiyao Wang · Huazhe Xu · Furong Huang -
2022 Spotlight: Lightning Talks 5A-3 »
Minting Pan · Xiang Chen · Wenhan Huang · Can Chang · Zhecheng Yuan · Jianzhun Shao · Yushi Cao · Peihao Chen · Ke Xue · Zhengrong Xue · Zhiqiang Lou · Xiangming Zhu · Lei Li · Zhiming Li · Kai Li · Jiacheng Xu · Dongyu Ji · Ni Mu · Kun Shao · Tianpei Yang · Kunyang Lin · Ningyu Zhang · Yunbo Wang · Lei Yuan · Bo Yuan · Hongchang Zhang · Jiajun Wu · Tianze Zhou · Xueqian Wang · Ling Pan · Yuhang Jiang · Xiaokang Yang · Xiaozhuan Liang · Hao Zhang · Weiwen Hu · Miqing Li · YAN ZHENG · Matthew Taylor · Huazhe Xu · Shumin Deng · Chao Qian · YI WU · Shuncheng He · Wenbing Huang · Chuanqi Tan · Zongzhang Zhang · Yang Gao · Jun Luo · Yi Li · Xiangyang Ji · Thomas Li · Mingkui Tan · Fei Huang · Yang Yu · Huazhe Xu · Dongge Wang · Jianye Hao · Chuang Gan · Yang Liu · Luo Si · Hangyu Mao · Huajun Chen · Jianye Hao · Jun Wang · Xiaotie Deng -
2022 Spotlight: E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidance »
Can Chang · Ni Mu · Jiajun Wu · Ling Pan · Huazhe Xu -
2022 Spotlight: Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning »
Zhecheng Yuan · Zhengrong Xue · Bo Yuan · Xueqian Wang · YI WU · Yang Gao · Huazhe Xu -
2022 Poster: E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidance »
Can Chang · Ni Mu · Jiajun Wu · Ling Pan · Huazhe Xu