Timezone: »
We propose an energy-based model (EBM) in the latent space of a generator model, so that the EBM serves as a prior model that stands on the top-down network of the generator model. Both the latent space EBM and the top-down network can be learned jointly by maximum likelihood, which involves short-run MCMC sampling from both the prior and posterior distributions of the latent vector. Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well. We show that the learned model exhibits strong performances in terms of image and text generation and anomaly detection. The one-page code can be found in supplementary materials.
Author Information
Bo Pang (UCLA)
Tian Han (Stevens Institute of Technology)
Erik Nijkamp (UCLA)
Song-Chun Zhu (UCLA)
Ying Nian Wu (University of California, Los Angeles)
More from the Same Authors
-
2020 : Paper 2: Energy-Based Continuous Inverse Optimal Control »
Yifei Xu · Jianwen Xie · Chris Baker · Yibiao Zhao · Ying Nian Wu -
2021 : Theorem-Aware Geometry Problem Solving with Symbolic Reasoning and Theorem Prediction »
Pan Lu · Ran Gong · Shibiao Jiang · Liang Qiu · Siyuan Huang · Xiaodan Liang · Song-Chun Zhu · Ran Gong -
2021 : Towards Diagram Understanding and Cognitive Reasoning in Icon Question Answering »
Pan Lu · Liang Qiu · Jiaqi Chen · Tanglin Xia · Yizhou Zhao · Wei Zhang · Zhou Yu · Xiaodan Liang · Song-Chun Zhu -
2021 : Deep Generative model with Hierarchical Latent Factors for Timeseries Anomaly Detection »
Cristian Challu · Peihong Jiang · Ying Nian Wu · Laurent Callot -
2021 : Unsupervised Meta-Learning via Latent Space Energy-based Model of Symbol Vector Coupling »
Bo Pang · Deqian Kong · Ying Nian Wu -
2021 : Deep Generative model with Hierarchical Latent Factors for Timeseries Anomaly Detection »
Cristian Challu · Peihong Jiang · Ying Nian Wu · Laurent Callot -
2022 Poster: Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning »
Yuanpei Chen · Tianhao Wu · Shengjie Wang · Xidong Feng · Jiechuan Jiang · Zongqing Lu · Stephen McAleer · Hao Dong · Song-Chun Zhu · Yaodong Yang -
2022 : Learn to Select Good Examples with Reinforcement Learning for Semi-structured Mathematical Reasoning »
Pan Lu · Liang Qiu · Kai-Wei Chang · Ying Nian Wu · Song-Chun Zhu · Tanmay Rajpurohit · Peter Clark · Ashwin Kalyan -
2022 : Conformal Isometry of Lie Group Representation in Recurrent Network of Grid Cells »
Dehong Xu · Ruiqi Gao · Wenhao Zhang · Xue-Xin Wei · Ying Nian Wu -
2022 : Neural-Symbolic Recursive Machine for Systematic Generalization »
Qing Li · Yixin Zhu · Yitao Liang · Ying Nian Wu · Song-Chun Zhu · Siyuan Huang -
2023 Poster: Learning Energy-Based Prior Model with Diffusion-Amortized MCMC »
Peiyu Yu · Yaxuan Zhu · Sirui Xie · Xiaojian (Shawn) Ma · Ruiqi Gao · Song-Chun Zhu · Ying Nian Wu -
2023 Poster: Learning non-Markovian Decision-Making from State-only Sequences »
Aoyang Qin · Feng Gao · Qing Li · Song-Chun Zhu · Sirui Xie -
2023 Poster: A Recurrent Neural Circuit Mechanism of Temporal-scaling Equivariant Representation »
Junfeng Zuo · Xiao Liu · Ying Nian Wu · Si Wu · Wenhao Zhang -
2023 Poster: Evaluating and Inducing Personality in Pre-trained Language Models »
Guangyuan Jiang · Manjie Xu · Song-Chun Zhu · Wenjuan Han · Chi Zhang · Yixin Zhu -
2023 Poster: Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models »
Pan Lu · Baolin Peng · Hao Cheng · Michel Galley · Kai-Wei Chang · Ying Nian Wu · Song-Chun Zhu · Jianfeng Gao -
2023 Poster: Learning Energy-based Model via Dual-MCMC Teaching »
Jiali Cui · Tian Han -
2023 Poster: Diplomat: A Dialogue Dataset for Situated PragMATic Reasoning »
Hengli Li · Song-Chun Zhu · Zilong Zheng -
2022 Spotlight: Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning »
Yuanpei Chen · Tianhao Wu · Shengjie Wang · Xidong Feng · Jiechuan Jiang · Zongqing Lu · Stephen McAleer · Hao Dong · Song-Chun Zhu · Yaodong Yang -
2022 : Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization »
Omead Pooladzandi · Pasha Khosravi · Erik Nijkamp · Baharan Mirzasoleiman -
2022 Poster: EgoTaskQA: Understanding Human Tasks in Egocentric Videos »
Baoxiong Jia · Ting Lei · Song-Chun Zhu · Siyuan Huang -
2022 Poster: Emergent Graphical Conventions in a Visual Communication Game »
Shuwen Qiu · Sirui Xie · Lifeng Fan · Tao Gao · Jungseock Joo · Song-Chun Zhu · Yixin Zhu -
2022 Poster: Translation-equivariant Representation in Recurrent Networks with a Continuous Manifold of Attractors »
Wenhao Zhang · Ying Nian Wu · Si Wu -
2022 Poster: MATE: Benchmarking Multi-Agent Reinforcement Learning in Distributed Target Coverage Control »
Xuehai Pan · Mickel Liu · Fangwei Zhong · Yaodong Yang · Song-Chun Zhu · Yizhou Wang -
2022 Poster: Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model »
Zhisheng Xiao · Tian Han -
2022 Poster: Learning Probabilistic Models from Generator Latent Spaces with Hat EBM »
Mitch Hill · Erik Nijkamp · Jonathan Mitchell · Bo Pang · Song-Chun Zhu -
2022 Poster: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering »
Pan Lu · Swaroop Mishra · Tanglin Xia · Liang Qiu · Kai-Wei Chang · Song-Chun Zhu · Oyvind Tafjord · Peter Clark · Ashwin Kalyan -
2021 Poster: On Path Integration of Grid Cells: Group Representation and Isotropic Scaling »
Ruiqi Gao · Jianwen Xie · Xue-Xin Wei · Song-Chun Zhu · Ying Nian Wu -
2021 Poster: Iterative Teacher-Aware Learning »
Luyao Yuan · Dongruo Zhou · Junhong Shen · Jingdong Gao · Jeffrey L Chen · Quanquan Gu · Ying Nian Wu · Song-Chun Zhu -
2021 Poster: Unsupervised Foreground Extraction via Deep Region Competition »
Peiyu Yu · Sirui Xie · Xiaojian (Shawn) Ma · Yixin Zhu · Ying Nian Wu · Song-Chun Zhu -
2020 : Contributed Talk 2: From em-Projections to Variational Auto-Encoder »
Tian Han -
2019 : Extended Poster Session »
Travis LaCroix · Marie Ossenkopf · Mina Lee · Nicole Fitzgerald · Daniela Mihai · Jonathon Hare · Ali Zaidi · Alexander Cowen-Rivers · Alana Marzoev · Eugene Kharitonov · Luyao Yuan · Tomasz Korbak · Paul Pu Liang · Yi Ren · Roberto Dessì · Peter Potash · Shangmin Guo · Tatsunori Hashimoto · Percy Liang · Julian Zubek · Zipeng Fu · Song-Chun Zhu · Adam Lerer -
2019 Poster: Learning Perceptual Inference by Contrasting »
Chi Zhang · Baoxiong Jia · Feng Gao · Yixin Zhu · HongJing Lu · Song-Chun Zhu -
2019 Spotlight: Learning Perceptual Inference by Contrasting »
Chi Zhang · Baoxiong Jia · Feng Gao · Yixin Zhu · HongJing Lu · Song-Chun Zhu -
2019 Poster: PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points »
Siyuan Huang · Yixin Chen · Tao Yuan · Siyuan Qi · Yixin Zhu · Song-Chun Zhu -
2019 Poster: Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model »
Erik Nijkamp · Mitch Hill · Song-Chun Zhu · Ying Nian Wu -
2018 Poster: Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation »
Siyuan Huang · Siyuan Qi · Yinxue Xiao · Yixin Zhu · Ying Nian Wu · Song-Chun Zhu -
2013 Poster: Unsupervised Structure Learning of Stochastic And-Or Grammars »
Kewei Tu · Maria Pavlovskaia · Song-Chun Zhu -
2011 Poster: Image Parsing with Stochastic Scene Grammar »
Yibiao Zhao · Song-Chun Zhu