Timezone: »

 
Poster
Preference-based Reinforcement Learning with Finite-Time Guarantees
Yichong Xu · Ruosong Wang · Lin Yang · Aarti Singh · Artur Dubrawski

Tue Dec 08 09:00 PM -- 11:00 PM (PST) @ Poster Session 2 #601
in Poster Session 3 »
Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning by preferences to better elicit human opinion on the target objective, especially when numerical reward values are hard to design or interpret. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. In this paper, we present the first finite-time analysis for general PbRL problems. We first show that a unique optimal policy may not exist if preferences over trajectories are deterministic for PbRL. If preferences are stochastic, and the preference probability relates to the hidden reward values, we present algorithms for PbRL, both with and without a simulator, that are able to identify the best policy up to accuracy $\varepsilon$ with high probability. Our method explores the state space by navigating to under-explored states, and solves PbRL using a combination of dueling bandits and policy search. Experiments show the efficacy of our method when it is applied to real-world problems.

Author Information

Yichong Xu (Microsoft)
Ruosong Wang (Carnegie Mellon University)
Lin Yang (UCLA)
Aarti Singh (CMU)
Artur Dubrawski (Carnegie Mellon University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors