Timezone: »

 
Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion
Utkarsh Soni · Sarath Sreedharan · Mudit Verma · Lin Guan · Matthew Marquez · Subbarao Kambhampati

@
in Human in the Loop Learning (HiLL) Workshop at NeurIPS 2022 »

There is a growing interest in developing automated agents that can work alongside humans. In addition to completing the assigned task, such an agent will undoubtedly be expected to behave in a manner that is preferred by the human. This requires the human to communicate their preferences to the agent. To achieve this, the current approaches either require the users to specify the reward function or the preference is interactively learned from queries that ask the user to compare trajectories. The former approach can be challenging if the internal representation used by the agent is inscrutable to the human while the latter is unnecessarily cumbersome for the user if their preference can be specified more easily in symbolic terms. In this work, we propose PRESCA (PREference Specification through Concept Acquisition), a system that allows users to specify their preferences in terms of concepts that they understand. PRESCA maintains a set of such concepts in a shared vocabulary. If the relevant concept is not in the shared vocabulary, then it is learned. To make learning a new concept more efficient, PRESCA leverages causal associations between the target concept and concepts that are already known. Additionally, the effort of learning the new concept is amortized by adding the concept to the shared vocabulary for supporting preference specification in future interactions. We evaluate PRESCA by using it on a Minecraft environment and show that it can be effectively used to make the agent align with the user's preference.

[ Poster] 

Author Information

Utkarsh Soni (Arizona State University)

I am a fourth-year Computer Science Ph.D. student and a Graduate Research Assistant at Arizona State University. My research focuses on human-in-the-loop reinforcement learning systems and explainable AI (XAI). More specifically, I am developing neuro-symbolic frameworks to enable preference specification, incorporation, and explanation generation in human-in-loop RL systems. I have publications in IROS, ICLR, and PacificVis conferences and workshop papers in NeurIPS and ICAPS. I also have experience conducting large-scale user studies. Following is the summary of some of my research projects: 1) Developed a framework that learns user’s vocabulary online (mapping user’s vocabulary to state images) so that the user can communicate their preferences to a reinforcement learning agent. The framework, additionally, incorporates the preference in the agent’s policy. 2) Developed a framework that generates explanations in the user’s vocabulary for the actions taken by a reinforcement learning agent operating on image observation. Verified the effectiveness of the framework through user studies. 3) Developed an AI system that personalizes explanations produced by a robot based on the type of user. The system was end-to-end, where it learns all possible user types and then identifies their type while interacting with a particular user and personalizes the robot’s explanation. Verified the effectiveness of the framework through user studies.

Sarath Sreedharan (Colorado State University)
Mudit Verma (Arizona State University)
Lin Guan (Arizona State University)
Matthew Marquez (Arizona State University)
Subbarao Kambhampati (Arizona State University)

More from the Same Authors