Timezone: »
We study the problem of conditional generative modeling based on designated semantics or structures. Existing models that build conditional generators either require massive labeled instances as supervision or are unable to accurately control the semantics of generated samples. We propose structured generative adversarial networks (SGANs) for semi-supervised conditional generative modeling. SGAN assumes the data x is generated conditioned on two independent latent variables: y that encodes the designated semantics, and z that contains other factors of variation. To ensure disentangled semantics in y and z, SGAN builds two collaborative games in the hidden space to minimize the reconstruction error of y and z, respectively. Training SGAN also involves solving two adversarial games that have their equilibrium concentrating at the true joint data distributions p(x, z) and p(x, y), avoiding distributing the probability mass diffusely over data space that MLE-based methods may suffer. We assess SGAN by evaluating its trained networks, and its performance on downstream tasks. We show that SGAN delivers a highly controllable generator, and disentangled representations; it also establishes start-of-the-art results across multiple datasets when applied for semi-supervised image classification (1.27%, 5.73%, 17.26% error rates on MNIST, SVHN and CIFAR-10 using 50, 1000 and 4000 labels, respectively). Benefiting from the separate modeling of y and z, SGAN can generate images with high visual quality and strictly following the designated semantic, and can be extended to a wide spectrum of applications, such as style transfer.
Author Information
Zhijie Deng (Tsinghua University)
Hao Zhang (Carnegie Mellon University)
Xiaodan Liang (Sun Yat-sen University)
Luona Yang (Carnegie Mellon University)
Shizhen Xu (Tsinghua University)
Jun Zhu (Tsinghua University)
Eric Xing (Petuum Inc. / Carnegie Mellon University)
More from the Same Authors
-
2021 : One Million Scenes for Autonomous Driving: ONCE Dataset »
Jiageng Mao · Niu Minzhe · ChenHan Jiang · hanxue liang · Jingheng Chen · Xiaodan Liang · Yamin Li · Chaoqiang Ye · Wei Zhang · Zhenguo Li · Jie Yu · Hang Xu · Chunjing XU -
2021 : FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark »
Mingjie Li · Wenjia Cai · Rui Liu · Yuetian Weng · Xiaoyun Zhao · Cong Wang · Xin Chen · Zhong Liu · Caineng Pan · Mengke Li · yingfeng zheng · Yizhi Liu · Flora Salim · Karin Verspoor · Xiaodan Liang · Xiaojun Chang -
2021 : IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning »
Pan Lu · Liang Qiu · Jiaqi Chen · Tanglin Xia · Yizhou Zhao · Wei Zhang · Zhou Yu · Xiaodan Liang · Song-Chun Zhu -
2021 : SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving »
Jianhua Han · Xiwen Liang · Hang Xu · Kai Chen · Lanqing Hong · Jiageng Mao · Chaoqiang Ye · Wei Zhang · Zhenguo Li · Xiaodan Liang · Chunjing XU -
2021 : Theorem-Aware Geometry Problem Solving with Symbolic Reasoning and Theorem Prediction »
Pan Lu · Ran Gong · Shibiao Jiang · Liang Qiu · Siyuan Huang · Xiaodan Liang · Song-Chun Zhu · Ran Gong -
2021 : Towards Diagram Understanding and Cognitive Reasoning in Icon Question Answering »
Pan Lu · Liang Qiu · Jiaqi Chen · Tanglin Xia · Yizhou Zhao · Wei Zhang · Zhou Yu · Xiaodan Liang · Song-Chun Zhu -
2021 : Geometric Question Answering Towards Multimodal Numerical Reasoning »
Jiaqi Chen · Jianheng Tang · Jinghui Qin · Xiaodan Liang · Lingbo Liu · Eric Xing · Liang Lin -
2021 : Multi-modal Self-supervised Pre-training for Large-scale Genome Data »
Shentong Mo · Xi Fu · Chenyang Hong · Yizhen Chen · Yuxuan Zheng · Xiangru Tang · Yanyan Lan · Zhiqiang Shen · Eric Xing -
2022 Poster: Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning »
Zaiyu Huang · Hanhui Li · Zhenyu Xie · Michael Kampffmeyer · qingling Cai · Xiaodan Liang -
2022 : Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation »
yifan zhang · Hanlin Zhang · Zachary Lipton · Li Erran Li · Eric Xing -
2022 : Sample-Specific Contextualized Graphical Models Using Clinical and Molecular Data Reveal Transcriptional Network Heterogeneity Across 7000 Tumors »
Caleb Ellington · Ben Lengerich · Thomas Watkins · Jiekun Yang · Manolis Kellis · Eric Xing -
2022 Spotlight: Lightning Talks 4B-3 »
Zicheng Zhang · Mancheng Meng · Antoine Guedon · Yue Wu · Wei Mao · Zaiyu Huang · Peihao Chen · Shizhe Chen · Yongwei Chen · Keqiang Sun · Yi Zhu · chen rui · Hanhui Li · Dongyu Ji · Ziyan Wu · miaomiao Liu · Pascal Monasse · Yu Deng · Shangzhe Wu · Pierre-Louis Guhur · Jiaolong Yang · Kunyang Lin · Makarand Tapaswi · Zhaoyang Huang · Terrence Chen · Jiabao Lei · Jianzhuang Liu · Vincent Lepetit · Zhenyu Xie · Richard I Hartley · Dinggang Shen · Xiaodan Liang · Runhao Zeng · Cordelia Schmid · Michael Kampffmeyer · Mathieu Salzmann · Ning Zhang · Fangyun Wei · Yabin Zhang · Fan Yang · Qifeng Chen · Wei Ke · Quan Wang · Thomas Li · qingling Cai · Kui Jia · Ivan Laptev · Mingkui Tan · Xin Tong · Hongsheng Li · Xiaodan Liang · Chuang Gan -
2022 Spotlight: Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning »
Zaiyu Huang · Hanhui Li · Zhenyu Xie · Michael Kampffmeyer · qingling Cai · Xiaodan Liang -
2022 Spotlight: CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation »
Zicheng Zhang · Yi Zhu · Jianzhuang Liu · Xiaodan Liang · Wei Ke -
2022 Poster: Structure-Preserving 3D Garment Modeling with Neural Sewing Machines »
Xipeng Chen · Guangrun Wang · Dizhong Zhu · Xiaodan Liang · Philip Torr · Liang Lin -
2022 Poster: Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark »
Jiaxi Gu · Xiaojun Meng · Guansong Lu · Lu Hou · Niu Minzhe · Xiaodan Liang · Lewei Yao · Runhui Huang · Wei Zhang · Xin Jiang · Chunjing XU · Hang Xu -
2022 Poster: DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection »
Lewei Yao · Jianhua Han · Youpeng Wen · Xiaodan Liang · Dan Xu · Wei Zhang · Zhenguo Li · Chunjing XU · Hang Xu -
2022 Poster: Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving »
Xiwen Liang · Yangxin Wu · Jianhua Han · Hang Xu · Chunjing XU · Xiaodan Liang -
2022 Poster: CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation »
Zicheng Zhang · Yi Zhu · Jianzhuang Liu · Xiaodan Liang · Wei Ke -
2021 Workshop: Math AI for Education (MATHAI4ED): Bridging the Gap Between Research and Smart Education »
Pan Lu · Yuhuai Wu · Sean Welleck · Xiaodan Liang · Eric Xing · James McClelland -
2021 Poster: Multi-task Learning of Order-Consistent Causal Graphs »
Xinshi Chen · Haoran Sun · Caleb Ellington · Eric Xing · Le Song -
2021 Poster: Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN »
Zhenyu Xie · Zaiyu Huang · Fuwei Zhao · Haoye Dong · Michael Kampffmeyer · Xiaodan Liang -
2020 : Panel Discussion & Closing »
Yejin Choi · Alexei Efros · Chelsea Finn · Kristen Grauman · Quoc V Le · Yann LeCun · Ruslan Salakhutdinov · Eric Xing -
2020 Workshop: Self-Supervised Learning -- Theory and Practice »
Pengtao Xie · Shanghang Zhang · Pulkit Agrawal · Ishan Misra · Cynthia Rudin · Abdelrahman Mohamed · Wenzhen Yuan · Barret Zoph · Laurens van der Maaten · Xingyi Yang · Eric Xing -
2020 Poster: Regularizing Black-box Models for Improved Interpretability »
Gregory Plumb · Maruan Al-Shedivat · Ángel Alexander Cabrera · Adam Perer · Eric Xing · Ameet Talwalkar -
2020 Poster: AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning »
Hao Zhang · Yuan Li · Zhijie Deng · Xiaodan Liang · Lawrence Carin · Eric Xing -
2020 Poster: Improving GAN Training with Probability Ratio Clipping and Sample Reweighting »
Yue Wu · Pan Zhou · Andrew Wilson · Eric Xing · Zhiting Hu -
2020 Poster: Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation »
Yangxin Wu · Gengwei Zhang · Hang Xu · Xiaodan Liang · Liang Lin -
2020 Poster: Towards Interpretable Natural Language Understanding with Explanations as Latent Variables »
Wangchunshu Zhou · Jinyi Hu · Hanlin Zhang · Xiaodan Liang · Maosong Sun · Chenyan Xiong · Jian Tang -
2019 : Poster Presentations »
Rahul Mehta · Andrew Lampinen · Binghong Chen · Sergio Pascual-Diaz · Jordi Grau-Moya · Aldo Faisal · Jonathan Tompson · Yiren Lu · Khimya Khetarpal · Martin Klissarov · Pierre-Luc Bacon · Doina Precup · Thanard Kurutach · Aviv Tamar · Pieter Abbeel · Jinke He · Maximilian Igl · Shimon Whiteson · Wendelin Boehmer · Raphaël Marinier · Olivier Pietquin · Karol Hausman · Sergey Levine · Chelsea Finn · Tianhe Yu · Lisa Lee · Benjamin Eysenbach · Emilio Parisotto · Eric Xing · Ruslan Salakhutdinov · Hongyu Ren · Anima Anandkumar · Deepak Pathak · Christopher Lu · Trevor Darrell · Alexei Efros · Phillip Isola · Feng Liu · Bo Han · Gang Niu · Masashi Sugiyama · Saurabh Kumar · Janith Petangoda · Johan Ferret · James McClelland · Kara Liu · Animesh Garg · Robert Lange -
2019 Workshop: Learning with Rich Experience: Integration of Learning Paradigms »
Zhiting Hu · Andrew Wilson · Chelsea Finn · Lisa Lee · Taylor Berg-Kirkpatrick · Ruslan Salakhutdinov · Eric Xing -
2019 Poster: Learning Robust Global Representations by Penalizing Local Predictive Power »
Haohan Wang · Songwei Ge · Zachary Lipton · Eric Xing -
2019 Poster: Learning Data Manipulation for Augmentation and Weighting »
Zhiting Hu · Bowen Tan · Russ Salakhutdinov · Tom Mitchell · Eric Xing -
2019 Poster: Learning Sample-Specific Models with Low-Rank Personalized Regression »
Ben Lengerich · Bryon Aragam · Eric Xing -
2019 Poster: Heterogeneous Graph Learning for Visual Commonsense Reasoning »
Weijiang Yu · Jingwen Zhou · Weihao Yu · Xiaodan Liang · Nong Xiao -
2019 Spotlight: Heterogeneous Graph Learning for Visual Commonsense Reasoning »
Weijiang Yu · Jingwen Zhou · Weihao Yu · Xiaodan Liang · Nong Xiao -
2018 Poster: The Sample Complexity of Semi-Supervised Learning with Nonparametric Mixture Models »
Chen Dan · Liu Leqi · Bryon Aragam · Pradeep Ravikumar · Eric Xing -
2018 Poster: Symbolic Graph Reasoning Meets Convolutions »
Xiaodan Liang · Zhiting Hu · Hao Zhang · Liang Lin · Eric Xing -
2018 Poster: DAGs with NO TEARS: Continuous Optimization for Structure Learning »
Xun Zheng · Bryon Aragam · Pradeep Ravikumar · Eric Xing -
2018 Spotlight: DAGs with NO TEARS: Continuous Optimization for Structure Learning »
Xun Zheng · Bryon Aragam · Pradeep Ravikumar · Eric Xing -
2018 Poster: Learning Pipelines with Limited Data and Domain Knowledge: A Study in Parsing Physics Problems »
Mrinmaya Sachan · Kumar Avinava Dubey · Tom Mitchell · Dan Roth · Eric Xing -
2018 Poster: Deep Generative Models with Learnable Knowledge Constraints »
Zhiting Hu · Zichao Yang · Russ Salakhutdinov · LIANHUI Qin · Xiaodan Liang · Haoye Dong · Eric Xing -
2018 Poster: Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation »
Yuan Li · Xiaodan Liang · Zhiting Hu · Eric Xing -
2018 Poster: Hybrid Knowledge Routed Modules for Large-scale Object Detection »
ChenHan Jiang · Hang Xu · Xiaodan Liang · Liang Lin -
2018 Poster: Neural Architecture Search with Bayesian Optimisation and Optimal Transport »
Kirthevasan Kandasamy · Willie Neiswanger · Jeff Schneider · Barnabas Poczos · Eric Xing -
2018 Spotlight: Neural Architecture Search with Bayesian Optimisation and Optimal Transport »
Kirthevasan Kandasamy · Willie Neiswanger · Jeff Schneider · Barnabas Poczos · Eric Xing -
2018 Poster: Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis »
Haoye Dong · Xiaodan Liang · Ke Gong · Hanjiang Lai · Jia Zhu · Jian Yin -
2018 Poster: Unsupervised Text Style Transfer using Language Models as Discriminators »
Zichao Yang · Zhiting Hu · Chris Dyer · Eric Xing · Taylor Berg-Kirkpatrick -
2017 : Posters and Coffee »
Jean-Baptiste Tristan · Yunseong Lee · Anna Veronika Dorogush · Shohei Hido · Michael Terry · Mennatullah Siam · Hidemoto Nakada · Cody Coleman · Jung-Woo Ha · Hao Zhang · Adam Stooke · Chen Meng · Christopher Kappler · Lane Schwartz · Christopher Olston · Sebastian Schelter · Minmin Sun · Daniel Kang · Waldemar Hummer · Jichan Chung · Tim Kraska · Kannan Ramchandran · Nick Hynes · Christoph Boden · Donghyun Kwak -
2016 : Eric Xing »
Eric Xing -
2016 Poster: Variance Reduction in Stochastic Gradient Langevin Dynamics »
Kumar Avinava Dubey · Sashank J. Reddi · Sinead Williamson · Barnabas Poczos · Alexander Smola · Eric Xing -
2016 Poster: Learning HMMs with Nonparametric Emissions via Spectral Decompositions of Continuous Matrices »
Kirthevasan Kandasamy · Maruan Al-Shedivat · Eric Xing -
2016 Poster: Stochastic Variational Deep Kernel Learning »
Andrew Wilson · Zhiting Hu · Russ Salakhutdinov · Eric Xing -
2016 Poster: Tree-Structured Reinforcement Learning for Sequential Object Localization »
Zequn Jie · Xiaodan Liang · Jiashi Feng · Xiaojie Jin · Wen Lu · Shuicheng Yan -
2015 Workshop: Nonparametric Methods for Large Scale Representation Learning »
Andrew G Wilson · Alexander Smola · Eric Xing -
2015 Poster: The Human Kernel »
Andrew Wilson · Christoph Dann · Chris Lucas · Eric Xing -
2015 Spotlight: The Human Kernel »
Andrew Wilson · Christoph Dann · Chris Lucas · Eric Xing -
2014 Workshop: Modern Nonparametrics 3: Automating the Learning Pipeline »
Eric Xing · Mladen Kolar · Arthur Gretton · Samory Kpotufe · Han Liu · Zoltán Szabó · Alan Yuille · Andrew G Wilson · Ryan Tibshirani · Sasha Rakhlin · Damian Kozbur · Bharath Sriperumbudur · David Lopez-Paz · Kirthevasan Kandasamy · Francesco Orabona · Andreas Damianou · Wacha Bounliphone · Yanshuai Cao · Arijit Das · Yingzhen Yang · Giulia DeSalvo · Dmitry Storcheus · Roberto Valerio -
2014 Workshop: Modern Machine Learning and Natural Language Processing »
Ankur P Parikh · Avneesh Saluja · Chris Dyer · Eric Xing -
2014 Poster: On Model Parallelization and Scheduling Strategies for Distributed Machine Learning »
Seunghak Lee · Jin Kyu Kim · Xun Zheng · Qirong Ho · Garth Gibson · Eric Xing -
2014 Poster: Dependent nonparametric trees for dynamic hierarchical clustering »
Kumar Avinava Dubey · Qirong Ho · Sinead Williamson · Eric Xing -
2013 Poster: More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server »
Qirong Ho · James Cipar · Henggang Cui · Seunghak Lee · Jin Kyu Kim · Phillip B. Gibbons · Garth Gibson · Greg Ganger · Eric Xing -
2013 Oral: More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server »
Qirong Ho · James Cipar · Henggang Cui · Seunghak Lee · Jin Kyu Kim · Phillip B. Gibbons · Garth Gibson · Greg Ganger · Eric Xing -
2013 Poster: Variance Reduction for Stochastic Gradient Optimization »
Chong Wang · Xi Chen · Alexander Smola · Eric Xing -
2013 Poster: Restricting exchangeable nonparametric distributions »
Sinead Williamson · Steven MacEachern · Eric Xing -
2013 Spotlight: Restricting exchangeable nonparametric distributions »
Sinead Williamson · Steven MacEachern · Eric Xing -
2013 Poster: A Scalable Approach to Probabilistic Latent Space Inference of Large-Scale Networks »
Junming Yin · Qirong Ho · Eric Xing -
2012 Workshop: Spectral Algorithms for Latent Variable Models »
Ankur P Parikh · Le Song · Eric Xing -
2012 Poster: Monte Carlo Methods for Maximum Margin Supervised Topic Models »
Qixia Jiang · Jun Zhu · Maosong Sun · Eric Xing -
2012 Poster: On Triangular versus Edge Representations --- Towards Scalable Modeling of Networks »
Qirong Ho · Junming Yin · Eric Xing -
2012 Poster: Symmetric Correspondence Topic Models for Multilingual Text Analysis »
Kosuke Fukumasu · Koji Eguchi · Eric Xing -
2012 Spotlight: Symmetric Correspondence Topic Models for Multilingual Text Analysis »
Kosuke Fukumasu · Koji Eguchi · Eric Xing -
2011 Poster: Infinite Latent SVM for Classification and Multi-task Learning »
Jun Zhu · Ning Chen · Eric Xing -
2011 Poster: Kernel Embeddings of Latent Tree Graphical Models »
Le Song · Ankur P Parikh · Eric Xing -
2011 Poster: Large-Scale Category Structure Aware Image Categorization »
Bin Zhao · Li Fei-Fei · Eric Xing -
2010 Poster: Large Margin Learning of Upstream Scene Understanding Models »
Jun Zhu · Li-Jia Li · Li Fei-Fei · Eric Xing -
2010 Poster: Predictive Subspace Learning for Multi-view Data: a Large Margin Approach »
Ning Chen · Jun Zhu · Eric Xing -
2010 Poster: Object Bank: A High-Level Image Representation for Scene Classification & Semantic Feature Sparsification »
Li-Jia Li · Hao Su · Eric Xing · Li Fei-Fei -
2010 Poster: Adaptive Multi-Task Lasso: with Application to eQTL Detection »
Seunghak Lee · Jun Zhu · Eric Xing -
2009 Poster: Heterogeneous multitask learning with joint sparsity constraints »
Xiaolin Yang · Seyoung Kim · Eric Xing -
2009 Poster: Time-Varying Dynamic Bayesian Networks »
Le Song · Mladen Kolar · Eric Xing -
2009 Spotlight: Time-Varying Dynamic Bayesian Networks »
Le Song · Mladen Kolar · Eric Xing -
2009 Poster: Sparsistent Learning of Varying-coefficient Models with Structural Changes »
Mladen Kolar · Le Song · Eric Xing -
2009 Spotlight: Sparsistent Learning of Varying-coefficient Models with Structural Changes »
Mladen Kolar · Le Song · Eric Xing -
2008 Workshop: Analyzing Graphs: Theory and Applications »
Edo M Airoldi · David Blei · Jake M Hofman · Tony Jebara · Eric Xing -
2008 Poster: Mixed Membership Stochastic Blockmodels »
Edo M Airoldi · David Blei · Stephen E Fienberg · Eric Xing -
2008 Spotlight: Mixed Membership Stochastic Blockmodels »
Edo M Airoldi · David Blei · Stephen E Fienberg · Eric Xing -
2008 Poster: Partially Observed Maximum Entropy Discrimination Markov Networks »
Jun Zhu · Eric Xing · Bo Zhang -
2007 Workshop: Statistical Network Models »
Kevin Murphy · Lise Getoor · Eric Xing · Raphael Gottardo -
2007 Poster: HM-BiTAM: Bilingual Topic Exploration, Word Alignment, and Translation »
Bing Zhao · Eric Xing -
2006 Poster: A Hidden Markov Dirichlet Process Model for Genetic Recombination in Open Ancestral Space »
KyungAh Sohn · Eric Xing -
2006 Talk: A Hidden Markov Dirichlet Process Model for Genetic Recombination in Open Ancestral Space »
KyungAh Sohn · Eric Xing