Timezone: »
We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards. Because the different datasets may have state-action distributions with large divergence, the task inference module can learn to ignore the rewards and spuriously correlate \textit{only} state-action pairs to the task identity, leading to poor test time performance. To robustify task inference, we propose a novel application of the triplet loss. To mine hard negative examples, we relabel the transitions from the training tasks by approximating their reward functions. When we allow further training on the unseen tasks, using the trained policy as an initialization leads to significantly faster convergence compared to randomly initialized policies (up to 80% improvement and across 5 different Mujoco task distributions). We name our method \textbf{MBML} (\textbf{M}ulti-task \textbf{B}atch RL with \textbf{M}etric \textbf{L}earning).
Author Information
Jiachen Li (University of California, San Diego)
Quan Vuong (University of California San Diego)
Shuang Liu (University of California, San Diego)
Minghua Liu (UCSD)
Kamil Ciosek (Microsoft)
Henrik Christensen (UC San Diego)
Hao Su (UCSD)
More from the Same Authors
-
2020 Poster: Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous GNNs »
Hao Tang · Zhiao Huang · Jiayuan Gu · Bao-Liang Lu · Hao Su -
2020 Poster: Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals »
Tongzhou Mu · Jiayuan Gu · Zhiwei Jia · Hao Tang · Hao Su -
2019 Poster: Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck »
Maximilian Igl · Kamil Ciosek · Yingzhen Li · Sebastian Tschiatschek · Cheng Zhang · Sam Devlin · Katja Hofmann -
2019 Poster: Better Exploration with Optimistic Actor Critic »
Kamil Ciosek · Quan Vuong · Robert Loftin · Katja Hofmann -
2019 Spotlight: Better Exploration with Optimistic Actor Critic »
Kamil Ciosek · Quan Vuong · Robert Loftin · Katja Hofmann -
2018 Poster: Deep Functional Dictionaries: Learning Consistent Semantic Structures on 3D Models from Functions »
Minhyuk Sung · Hao Su · Ronald Yu · Leonidas Guibas -
2017 Poster: Approximation and Convergence Properties of Generative Adversarial Learning »
Shuang Liu · Olivier Bousquet · Kamalika Chaudhuri -
2017 Spotlight: Approximation and Convergence Properties of Generative Adversarial Learning »
Shuang Liu · Olivier Bousquet · Kamalika Chaudhuri -
2017 Poster: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space »
Charles Ruizhongtai Qi · Li Yi · Hao Su · Leonidas Guibas -
2016 Poster: FPNN: Field Probing Neural Networks for 3D Data »
Yangyan Li · Soeren Pirk · Hao Su · Charles R Qi · Leonidas Guibas -
2010 Poster: Object Bank: A High-Level Image Representation for Scene Classification & Semantic Feature Sparsification »
Li-Jia Li · Hao Su · Eric Xing · Li Fei-Fei