Timezone: »

 
Poster
Reinforcement Learning with Automated Auxiliary Loss Search
Tairan He · Yuge Zhang · Kan Ren · Minghuan Liu · Che Wang · Weinan Zhang · Yuqing Yang · Dongsheng Li

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #127
in Poster Session 5 »
A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learning informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automatically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size $7.5 \times 10^{20}$ and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2-winner) significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability to different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance.
Keywords: Representation Learning · Reinforcement Learning · Auxiliary Loss
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Tairan He (Shanghai Jiao Tong University)
Tairan He

I am an undergraduate student at Shanghai Jiao Tong University (SJTU), majoring in Computer Science & Technology. I have been working as a research intern at APEX Lab since 2019, advised by Prof. Weinan Zhang. I am now a visiting student at Intelligent Control Lab in the Robotics Institute at Carnegie Mellon University, advised by Prof. Changliu Liu. Prior to that, I was research intern at Microsoft Research.

Yuge Zhang (Microsoft)
Kan Ren (Microsoft)
Minghuan Liu (Shanghai Jiao Tong University)
Che Wang (New York University)
Weinan Zhang (Shanghai Jiao Tong University)
Yuqing Yang (Fudan University)
Dongsheng Li (IBM Research - China)

More from the Same Authors