Timezone: »
While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HAZARDWORLD, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HAZARDWORLD, we show that our method achieves higher rewards (up to11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work.
Author Information
Tsung-Yen Yang (Princeton University)
I am a graduate student in the Department of Electrical Engineering at Princeton University, working with Prof. Peter Ramadge and Prof. Karthik Narasimhan since September 2017. My research interests lie at the intersection of machine learning, reinforcement learning, and natural language processing. Specifically, I work on safe reinforcement learning, focusing on building autonomous systems that acquire knowledge by interacting with the world, and providing provable safety guarantees during training and deployment.
Michael Y Hu (Princeton University)
Yinlam Chow (Google Research)
Peter J. Ramadge (Princeton)
Karthik Narasimhan (Princeton University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: Safe Reinforcement Learning with Natural Language Constraints »
Dates n/a. Room
More from the Same Authors
-
2021 : ProBF: Probabilistic Safety Certificates with Barrier Functions »
Sulin Liu · Athindran Ramesh Kumar · Jaime Fisac · Ryan Adams · Peter J. Ramadge -
2022 : A Mixture-of-Expert Approach to RL-based Dialogue Management »
Yinlam Chow · Azamat Tulepbergenov · Ofir Nachum · Dhawal Gupta · Moonkyung Ryu · Mohammad Ghavamzadeh · Craig Boutilier -
2022 Poster: KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation »
Ta-Chung Chi · Ting-Han Fan · Peter J. Ramadge · Alexander Rudnicky -
2022 Poster: Using natural language and program abstractions to instill human inductive biases in machines »
Sreejan Kumar · Carlos G. Correa · Ishita Dasgupta · Raja Marjieh · Michael Y Hu · Robert Hawkins · Jonathan D Cohen · nathaniel daw · Karthik Narasimhan · Tom Griffiths -
2022 Poster: Learning Physics Constrained Dynamics Using Autoencoders »
Tsung-Yen Yang · Justinian Rosca · Karthik Narasimhan · Peter J. Ramadge -
2022 Poster: Efficient Risk-Averse Reinforcement Learning »
Ido Greenberg · Yinlam Chow · Mohammad Ghavamzadeh · Shie Mannor -
2021 Poster: SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark »
Victor Zhong · Austin W. Hanjie · Sida Wang · Karthik Narasimhan · Luke Zettlemoyer -
2020 : Invited talk - Bringing Back Text Understanding into Text-based Games - Karthik Narasimhan »
Karthik Narasimhan -
2020 Poster: Task-Agnostic Amortized Inference of Gaussian Process Hyperparameters »
Sulin Liu · Xingyuan Sun · Peter J. Ramadge · Ryan Adams -
2020 Poster: Multimodal Graph Networks for Compositional Generalization in Visual Question Answering »
Raeid Saqur · Karthik Narasimhan -
2020 Poster: Latent Bandits Revisited »
Joey Hong · Branislav Kveton · Manzil Zaheer · Yinlam Chow · Amr Ahmed · Craig Boutilier -
2020 Poster: Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation »
Zhiwei Deng · Karthik Narasimhan · Olga Russakovsky -
2020 Poster: CoinDICE: Off-Policy Confidence Interval Estimation »
Bo Dai · Ofir Nachum · Yinlam Chow · Lihong Li · Csaba Szepesvari · Dale Schuurmans -
2020 Spotlight: CoinDICE: Off-Policy Confidence Interval Estimation »
Bo Dai · Ofir Nachum · Yinlam Chow · Lihong Li · Csaba Szepesvari · Dale Schuurmans -
2019 : Poster and Coffee Break 2 »
Karol Hausman · Kefan Dong · Ken Goldberg · Lihong Li · Lin Yang · Lingxiao Wang · Lior Shani · Liwei Wang · Loren Amdahl-Culleton · Lucas Cassano · Marc Dymetman · Marc Bellemare · Marcin Tomczak · Margarita Castro · Marius Kloft · Marius-Constantin Dinu · Markus Holzleitner · Martha White · Mengdi Wang · Michael Jordan · Mihailo Jovanovic · Ming Yu · Minshuo Chen · Moonkyung Ryu · Muhammad Zaheer · Naman Agarwal · Nan Jiang · Niao He · Nikolaus Yasui · Nikos Karampatziakis · Nino Vieillard · Ofir Nachum · Olivier Pietquin · Ozan Sener · Pan Xu · Parameswaran Kamalaruban · Paul Mineiro · Paul Rolland · Philip Amortila · Pierre-Luc Bacon · Prakash Panangaden · Qi Cai · Qiang Liu · Quanquan Gu · Raihan Seraj · Richard Sutton · Rick Valenzano · Robert Dadashi · Rodrigo Toro Icarte · Roshan Shariff · Roy Fox · Ruosong Wang · Saeed Ghadimi · Samuel Sokota · Sean Sinclair · Sepp Hochreiter · Sergey Levine · Sergio Valcarcel Macua · Sham Kakade · Shangtong Zhang · Sheila McIlraith · Shie Mannor · Shimon Whiteson · Shuai Li · Shuang Qiu · Wai Lok Li · Siddhartha Banerjee · Sitao Luan · Tamer Basar · Thinh Doan · Tianhe Yu · Tianyi Liu · Tom Zahavy · Toryn Klassen · Tuo Zhao · Vicenç Gómez · Vincent Liu · Volkan Cevher · Wesley Suttle · Xiao-Wen Chang · Xiaohan Wei · Xiaotong Liu · Xingguo Li · Xinyi Chen · Xingyou Song · Yao Liu · YiDing Jiang · Yihao Feng · Yilun Du · Yinlam Chow · Yinyu Ye · Yishay Mansour · · Yonathan Efroni · Yongxin Chen · Yuanhao Wang · Bo Dai · Chen-Yu Wei · Harsh Shrivastava · Hongyang Zhang · Qinqing Zheng · SIDDHARTHA SATPATHI · Xueqing Liu · Andreu Vall -
2019 Workshop: Safety and Robustness in Decision-making »
Mohammad Ghavamzadeh · Shie Mannor · Yisong Yue · Marek Petrik · Yinlam Chow -
2019 Poster: A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation »
Runzhe Yang · Xingyuan Sun · Karthik Narasimhan -
2019 Poster: DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections »
Ofir Nachum · Yinlam Chow · Bo Dai · Lihong Li -
2019 Spotlight: DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections »
Ofir Nachum · Yinlam Chow · Bo Dai · Lihong Li -
2018 : Harnessing the synergy between natural language and interactive learning »
Karthik Narasimhan -
2018 Poster: A Lyapunov-based Approach to Safe Reinforcement Learning »
Yinlam Chow · Ofir Nachum · Edgar Duenez-Guzman · Mohammad Ghavamzadeh -
2018 Poster: A Block Coordinate Ascent Algorithm for Mean-Variance Optimization »
Tengyang Xie · Bo Liu · Yangyang Xu · Mohammad Ghavamzadeh · Yinlam Chow · Daoming Lyu · Daesub Yoon -
2015 Poster: A Reduced-Dimension fMRI Shared Response Model »
Cameron Po-Hsuan Chen · Janice Chen · Yaara Yeshurun · Uri Hasson · James Haxby · Peter J. Ramadge -
2015 Oral: A Reduced-Dimension fMRI Shared Response Model »
Cameron Po-Hsuan Chen · Janice Chen · Yaara Yeshurun · Uri Hasson · James Haxby · Peter J. Ramadge -
2012 Poster: Kernel Hyperalignment »
Alexander Lorbert · Peter J. Ramadge -
2012 Spotlight: Kernel Hyperalignment »
Alexander Lorbert · Peter J. Ramadge -
2011 Poster: Learning Sparse Representations of High Dimensional Data on Large Scale Dictionaries »
Zhen James Xiang · Hao Xu · Peter J. Ramadge -
2011 Oral: Learning Sparse Representations of High Dimensional Data on Large Scale Dictionaries »
Zhen James Xiang · Hao Xu · Peter J. Ramadge -
2009 Poster: Boosting with Spatial Regularization »
Zhen James Xiang · Yongxin Xi · Uri Hasson · Peter J. Ramadge -
2009 Spotlight: Boosting with Spatial Regularization »
Zhen James Xiang · Yongxin Xi · Uri Hasson · Peter J. Ramadge -
2009 Poster: fMRI-Based Inter-Subject Cortical Alignment Using Functional Connectivity »
Bryan Conroy · Ben Singer · James Haxby · Peter J. Ramadge