Timezone: »
Dealing with non-stationarity in environments (e.g., in the transition dynamics) and objectives (e.g., in the reward functions) is a challenging problem that is crucial in real-world applications of reinforcement learning (RL). While most current approaches model the changes as a single shared embedding vector, we leverage insights from the recent causality literature to model non-stationarity in terms of individual latent change factors, and causal graphs across different environments. In particular, we propose Factored Adaptation for Non-Stationary RL (FANS-RL), a factored adaption approach that learns jointly both the causal structure in terms of a factored MDP, and a factored representation of the individual time-varying change factors. We prove that under standard assumptions, we can completely recover the causal graph representing the factored transition and reward function, as well as a partial structure between the individual change factors and the state components. Through our general framework, we can consider general non-stationary scenarios with different function types and changing frequency, including changes across episodes and within episodes. Experimental results demonstrate that FANS-RL outperforms existing approaches in terms of return, compactness of the latent state representation, and robustness to varying degrees of non-stationarity.
Author Information
Fan Feng (City University of Hong Kong)
Biwei Huang (University of California, San Diego)
Kun Zhang (CMU & MBZUAI)
Sara Magliacane (University of Amsterdam, MIT-IBM Watson AI Lab)
More from the Same Authors
-
2022 : Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors »
Zeyu Tang · Yatong Chen · Yang Liu · Kun Zhang -
2022 : Structural Causal Model for Molecular Dynamics Simulation »
Qi Liu · Yuanqi Du · Fan Feng · Qiwei Ye · Jie Fu -
2022 : Scalable Causal Discovery with Score Matching »
Francesco Montagna · Nicoletta Noceti · Lorenzo Rosasco · Kun Zhang · Francesco Locatello -
2023 Poster: Generator Identification for Linear SDEs with Additive and Multiplicative Noise »
Yuanyuan Wang · Xi Geng · Wei Huang · Biwei Huang · Mingming Gong -
2023 Poster: GRD: A Generative Approach for Interpretable Reward Redistribution in Reinforcement Learning »
Yudi Zhang · Yali Du · Biwei Huang · Ziyan Wang · Jun Wang · Meng Fang · Mykola Pechenizkiy -
2023 Poster: On the Identifiability of Sparse ICA without Assuming Non-Gaussianity »
Ignavier Ng · Yujia Zheng · Xinshuai Dong · Kun Zhang -
2023 Poster: Generalizing Nonlinear ICA Beyond Structural Sparsity »
Yujia Zheng · Kun Zhang -
2023 Poster: Counterfactual Generation with Identifiability Guarantee »
hanqi yan · Lingjing Kong · Lin Gui · Yuejie Chi · Eric Xing · Yulan He · Kun Zhang -
2023 Poster: Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning »
Fan Feng · Sara Magliacane -
2023 Poster: Temporally Disentangled Representation Learning under Unknown Nonstationarity »
Xiangchen Song · Weiran Yao · Yewen Fan · Xinshuai Dong · Guangyi Chen · Juan Carlos Niebles · Eric Xing · Kun Zhang -
2023 Poster: Identification of Nonlinear Latent Hierarchical Models »
Lingjing Kong · Biwei Huang · Feng Xie · Eric Xing · Yuejie Chi · Kun Zhang -
2023 Poster: Modulated Neural ODEs »
Ilze Amanda Auzina · Çağatay Yıldız · Sara Magliacane · Matthias Bethge · Efstratios Gavves -
2023 Poster: Subspace Identification for Multi-Source Domain Adaptation »
Zijian Li · Ruichu Cai · Guangyi Chen · Boyang Sun · Zhifeng Hao · Kun Zhang -
2023 Poster: Learning World Models with Identifiable Factorization »
Yuren Liu · Biwei Huang · Zhengmao Zhu · Honglong Tian · Mingming Gong · Yang Yu · Kun Zhang -
2023 Oral: Generalizing Nonlinear ICA Beyond Structural Sparsity »
Yujia Zheng · Kun Zhang -
2023 Workshop: I Can’t Believe It’s Not Better (ICBINB): Failure Modes in the Age of Foundation Models »
Estefany Kelly Buchanan · Fan Feng · Andreas Kriegler · Ian Mason · Tobias Uelwer · Yubin Xie · Rui Yang -
2023 Workshop: Causal Representation Learning »
Sara Magliacane · Atalanti Mastakouri · Yuki Asano · Claudia Shi · Cian Eastwood · Sébastien Lachapelle · Bernhard Schölkopf · Caroline Uhler -
2022 Spotlight: Latent Hierarchical Causal Structure Discovery with Rank Constraints »
Biwei Huang · Charles Jia Han Low · Feng Xie · Clark Glymour · Kun Zhang -
2022 : Kun Zhang: Causal Principles Meet Deep Learning: Successes and Challenges. »
Kun Zhang -
2022 : Kun Zhang: Causal Principles Meet Deep Learning: Successes and Challenges. »
Kun Zhang -
2022 Workshop: A causal view on dynamical systems »
Sören Becker · Alexis Bellot · Cecilia Casolo · Niki Kilbertus · Sara Magliacane · Yuyang (Bernie) Wang -
2022 Workshop: I Can’t Believe It’s Not Better: Understanding Deep Learning Through Empirical Falsification »
Arno Blaas · Sahra Ghalebikesabi · Javier Antorán · Fan Feng · Melanie F. Pradier · Ian Mason · David Rohde -
2022 Workshop: Causal Machine Learning for Real-World Impact »
Nick Pawlowski · Jeroen Berrevoets · Caroline Uhler · Kun Zhang · Mihaela van der Schaar · Cheng Zhang -
2022 Poster: On the Identifiability of Nonlinear ICA: Sparsity and Beyond »
Yujia Zheng · Ignavier Ng · Kun Zhang -
2022 Poster: Independence Testing-Based Approach to Causal Discovery under Measurement Error and Linear Non-Gaussian Models »
Haoyue Dai · Peter Spirtes · Kun Zhang -
2022 Poster: Latent Hierarchical Causal Structure Discovery with Rank Constraints »
Biwei Huang · Charles Jia Han Low · Feng Xie · Clark Glymour · Kun Zhang -
2022 Poster: MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models »
Erdun Gao · Ignavier Ng · Mingming Gong · Li Shen · Wei Huang · Tongliang Liu · Kun Zhang · Howard Bondell -
2022 Poster: Causal Discovery in Linear Latent Variable Models Subject to Measurement Error »
Yuqin Yang · AmirEmad Ghassami · Mohamed Nafea · Negar Kiyavash · Kun Zhang · Ilya Shpitser -
2022 Poster: Unsupervised Image-to-Image Translation with Density Changing Regularization »
Shaoan Xie · Qirong Ho · Kun Zhang -
2022 Poster: Counterfactual Fairness with Partially Known Causal Graph »
Aoqi Zuo · Susan Wei · Tongliang Liu · Bo Han · Kun Zhang · Mingming Gong -
2022 Poster: Temporally Disentangled Representation Learning »
Weiran Yao · Guangyi Chen · Kun Zhang -
2022 Poster: Truncated Matrix Power Iteration for Differentiable DAG Learning »
Zhen Zhang · Ignavier Ng · Dong Gong · Yuhang Liu · Ehsan Abbasnejad · Mingming Gong · Kun Zhang · Javen Qinfeng Shi -
2020 Workshop: Causal Discovery and Causality-Inspired Machine Learning »
Biwei Huang · Sara Magliacane · Kun Zhang · Danielle Belgrave · Elias Bareinboim · Daniel Malinsky · Thomas Richardson · Christopher Meek · Peter Spirtes · Bernhard Schölkopf -
2017 Poster: Learning Causal Structures Using Regression Invariance »
AmirEmad Ghassami · Saber Salehkaleybar · Negar Kiyavash · Kun Zhang