Timezone: »
Poster
Reinforcement Learning with Automated Auxiliary Loss Search
Tairan He · Yuge Zhang · Kan Ren · Minghuan Liu · Che Wang · Weinan Zhang · Yuqing Yang · Dongsheng Li
A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learning informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automatically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size $7.5 \times 10^{20}$ and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2-winner) significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability to different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance.
Author Information
Tairan He (Shanghai Jiao Tong University)

I am an undergraduate student at Shanghai Jiao Tong University (SJTU), majoring in Computer Science & Technology. I have been working as a research intern at APEX Lab since 2019, advised by Prof. Weinan Zhang. I am now a visiting student at Intelligent Control Lab in the Robotics Institute at Carnegie Mellon University, advised by Prof. Changliu Liu. Prior to that, I was research intern at Microsoft Research.
Yuge Zhang (Microsoft)
Kan Ren (Microsoft)
Minghuan Liu (Shanghai Jiao Tong University)
Che Wang (New York University)
Weinan Zhang (Shanghai Jiao Tong University)
Yuqing Yang (Fudan University)
Dongsheng Li (IBM Research - China)
More from the Same Authors
-
2022 Poster: Learning Enhanced Representation for Tabular Data via Neighborhood Propagation »
Kounianhua Du · Weinan Zhang · Ruiwen Zhou · Yangkun Wang · Xilong Zhao · Jiarui Jin · Quan Gan · Zheng Zhang · David P Wipf -
2022 Poster: Parameter-free Dynamic Graph Embedding for Link Prediction »
Jiahao Liu · Dongsheng Li · Hansu Gu · Tun Lu · Peng Zhang · Ning Gu -
2022 : Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance »
Yanqiu Wu · Xinyue Chen · Che Wang · Yiming Zhang · Keith Ross -
2022 : Visual Imitation Learning with Patch Rewards »
Minghuan Liu · Tairan He · Weinan Zhang · Shuicheng Yan · Zhongwen Xu -
2022 : Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents »
Minghuan Liu · Zhengbang Zhu · Menghui Zhu · Yuzheng Zhuang · Weinan Zhang · Jianye Hao -
2023 Poster: HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face »
Yongliang Shen · Kaitao Song · Xu Tan · Dongsheng Li · Weiming Lu · Yueting Zhuang -
2023 Poster: Lending Interaction Wings to Recommender Systems with Conversational Agents »
Jiarui Jin · Xianyu Chen · Fanghua Ye · Mengyue Yang · Yue Feng · Weinan Zhang · Yong Yu · Jun Wang -
2023 Poster: Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling »
Ke Yi · Yansen Wang · Kan Ren · Dongsheng Li -
2023 Poster: Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning »
Haoran He · Chenjia Bai · Kang Xu · Zhuoran Yang · Weinan Zhang · Dong Wang · Bin Zhao · Xuelong Li -
2023 Poster: Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models »
Yubin Shi · Yixuan Chen · Mingzhi Dong · Xiaochen Yang · Dongsheng Li · Yujiang Wang · Robert Dick · Qin Lv · Yingying Zhao · Fan Yang · Tun Lu · Ning Gu · Li Shang -
2023 Poster: ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling »
Yuqi Chen · Kan Ren · Yansen Wang · Yuchen Fang · Weiwei Sun · Dongsheng Li -
2023 Poster: ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation »
ya sheng sun · Yifan Yang · Houwen Peng · Yifei Shen · Yuqing Yang · Han Hu · Lili Qiu · Hideki Koike -
2022 Poster: Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learning »
Hua Wei · Jingxiao Chen · Xiyang Ji · Hongyang Qin · Minwen Deng · Siqin Li · Liang Wang · Weinan Zhang · Yong Yu · Liu Linc · Lanxiao Huang · Deheng Ye · Qiang Fu · Wei Yang -
2022 Poster: Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling »
Kaitao Song · Yichong Leng · Xu Tan · Yicheng Zou · Tao Qin · Dongsheng Li -
2022 Poster: Bootstrapped Transformer for Offline Reinforcement Learning »
Kerong Wang · Hanye Zhao · Xufang Luo · Kan Ren · Weinan Zhang · Dongsheng Li -
2022 Poster: PerfectDou: Dominating DouDizhu with Perfect Information Distillation »
Guan Yang · Minghuan Liu · Weijun Hong · Weinan Zhang · Fei Fang · Guangjun Zeng · Yue Lin -
2022 Poster: NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning »
Rong-Jun Qin · Xingyuan Zhang · Songyi Gao · Xiong-Hui Chen · Zewen Li · Weinan Zhang · Yang Yu -
2022 Poster: Multi-Agent Reinforcement Learning is a Sequence Modeling Problem »
Muning Wen · Jakub Kuba · Runji Lin · Weinan Zhang · Ying Wen · Jun Wang · Yaodong Yang -
2022 Poster: VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning »
Che Wang · Xufang Luo · Keith Ross · Dongsheng Li -
2021 Poster: Curriculum Offline Imitating Learning »
Minghuan Liu · Hanye Zhao · Zhengyu Yang · Jian Shen · Weinan Zhang · Li Zhao · Tie-Yan Liu -
2021 Poster: Reinforcement Learning Enhanced Explainer for Graph Neural Networks »
Caihua Shan · Yifei Shen · Yao Zhang · Xiang Li · Dongsheng Li -
2021 Poster: On Effective Scheduling of Model-based Reinforcement Learning »
Hang Lai · Jian Shen · Weinan Zhang · Yimin Huang · Xing Zhang · Ruiming Tang · Yong Yu · Zhenguo Li -
2021 Poster: Recognizing Vector Graphics without Rasterization »
XINYANG JIANG · LU LIU · Caihua Shan · Yifei Shen · Xuanyi Dong · Dongsheng Li -
2020 Poster: Efficient Projection-free Algorithms for Saddle Point Problems »
Cheng Chen · Luo Luo · Weinan Zhang · Yong Yu -
2020 Poster: BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning »
Xinyue Chen · Zijian Zhou · Zheng Wang · Che Wang · Yanqiu Wu · Keith Ross -
2020 Poster: Model-based Policy Optimization with Unsupervised Model Adaptation »
Jian Shen · Han Zhao · Weinan Zhang · Yong Yu -
2020 Spotlight: Model-based Policy Optimization with Unsupervised Model Adaptation »
Jian Shen · Han Zhao · Weinan Zhang · Yong Yu -
2017 Demonstration: MAgent: A Many-Agent Reinforcement Learning Research Platform for Artificial Collective Intelligence »
Lianmin Zheng · Jiacheng Yang · Han Cai · Weinan Zhang · Jun Wang · Yong Yu -
2017 Poster: Mixture-Rank Matrix Approximation for Collaborative Filtering »
Dongsheng Li · Chao Chen · Wei Liu · Tun Lu · Ning Gu · Stephen Chu