Timezone: »

 
Spotlight
Provable General Function Class Representation Learning in Multitask Bandits and MDP
Rui Lu · Andrew Zhao · Simon Du · Gao Huang

Wed Dec 07 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 4A-4 »
While multitask representation learning has become a popular approach in reinforcement learning (RL) to boost the sample efficiency, the theoretical understanding of why and how it works is still limited. Most previous analytical works could only assume that the representation function is already known to the agent or from linear function class, since analyzing general function class representation encounters non-trivial technical obstacles such as generalization guarantee, formulation of confidence bound in abstract function space, etc. However, linear-case analysis heavily relies on the particularity of linear function class, while real-world practice usually adopts general non-linear representation functions like neural networks. This significantly reduces its applicability. In this work, we extend the analysis to general function class representations. Specifically, we consider an agent playing $M$ contextual bandits (or MDPs) concurrently and extracting a shared representation function $\phi$ from a specific function class $\Phi$ using our proposed Generalized Functional Upper Confidence Bound algorithm (GFUCB). We theoretically validate the benefit of multitask representation learning within general function class for bandits and linear MDP for the first time. Lastly, we conduct experiments to demonstrate the effectiveness of our algorithm with neural net representation.
Keywords: Theory · Representation Learning · Reinforcement Learning · multi-task

Author Information

Rui Lu (Tsinghua University)
Andrew Zhao (Tsinghua University)

Andrew Zhao is a PhD student at Tsinghua University. He obtained his masters degree from USC in 2020, and undergrad degree from UBC in 2017. His research interests are in machine learning and reinforcement learning.

Simon Du (University of Washington)
Gao Huang (Cornell University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors