Timezone: »

 
Poster
Guide Your Agent with Adaptive Multimodal Rewards
Changyeon Kim · Younggyo Seo · Hao Liu · Lisa Lee · Jinwoo Shin · Honglak Lee · Kimin Lee

Tue Dec 12 08:45 AM -- 10:45 AM (PST) @ Great Hall & Hall B1+B2 #1415
in Poster Session 1 »
Event URL: https://sites.google.com/view/2023arp »

Developing an agent capable of adapting to unseen environments remains a difficult challenge in imitation learning. This work presents Adaptive Return-conditioned Policy (ARP), an efficient framework designed to enhance the agent's generalization ability using natural language task descriptions and pre-trained multimodal encoders. Our key idea is to calculate a similarity between visual observations and natural language instructions in the pre-trained multimodal embedding space (such as CLIP) and use it as a reward signal. We then train a return-conditioned policy using expert demonstrations labeled with multimodal rewards. Because the multimodal rewards provide adaptive signals at each timestep, our ARP effectively mitigates the goal misgeneralization. This results in superior generalization performances even when faced with unseen text instructions, compared to existing text-conditioned policies. To improve the quality of rewards, we also introduce a fine-tuning method for pre-trained multimodal encoders, further enhancing the performance. Video demonstrations and source code are available on the project website: \url{https://sites.google.com/view/2023arp}.

Author Information

Changyeon Kim (KAIST)

My research interest lies in applying RL algorithms to challenging tasks where reward specification is burdensome. To this end, I am focusing on designing RL algorithms to tackle practical and challenging scenarios like unseen novel environments and environments without well-shaped rewards. Especially, I am interested in human preference-based reinforcement learning. I am also broadly interested in areas related to RL, including RL leveraging pre-trained representation learning, language-conditioned RL, and offline RL.

Younggyo Seo (Dyson)
Hao Liu (University of California Berkeley)
Lisa Lee (Google Brain)
Jinwoo Shin (KAIST)
Honglak Lee (LG AI Research / U. Michigan)
Kimin Lee (KAIST)

More from the Same Authors