Timezone: »
Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias. Existing work focuses on a handful of curated generative processes which require expert knowledge to design, making it hard to scale up. To overcome this, we propose training with a large dataset of twenty-one thousand programs, each one generating a diverse set of synthetic images. These programs are short code snippets, which are easy to modify and fast to execute using OpenGL. The proposed dataset can be used for both supervised and unsupervised representation learning, and reduces the gap between pre-training with real and procedurally generated images by 38%.
Author Information
Manel Baradad (Massachusetts Institute of Technology)
Richard Chen (JPMorgan Chase)
Jonas Wulff (CSAIL MIT / Xyla)
Tongzhou Wang (MIT)
Rogerio Feris (MIT-IBM Watson AI Lab, IBM Research)
Antonio Torralba (MIT)
Phillip Isola (Massachusetts Institute of Technology)
More from the Same Authors
-
2021 : The Neural MMO Platform for Massively Multiagent Research »
Joseph Suarez · Yilun Du · Clare Zhu · Igor Mordatch · Phillip Isola -
2021 Spotlight: Learning to See by Looking at Noise »
Manel Baradad Jurjo · Jonas Wulff · Tongzhou Wang · Phillip Isola · Antonio Torralba -
2021 : Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation »
Aadarsh Sahoo · Rameswar Panda · Rogerio Feris · Kate Saenko · Abir Das -
2022 : Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings »
Tongzhou Wang · Phillip Isola -
2022 : Real world relevance of generative counterfactual explanations »
Swami Sankaranarayanan · Thomas Hartvigsen · Lauren Oakden-Rayner · Marzyeh Ghassemi · Phillip Isola -
2023 Poster: StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners »
Yonglong Tian · Lijie Fan · Phillip Isola · Huiwen Chang · Dilip Krishnan -
2023 Poster: LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections »
Muhammad Jehanzeb Mirza · Leonid Karlinsky · Wei Lin · Horst Possegger · Mateusz Kozinski · Rogerio Feris · Horst Bischof -
2023 Poster: Improving CLIP Training with Language Rewrites »
Lijie Fan · Dilip Krishnan · Phillip Isola · Dina Katabi · Yonglong Tian -
2023 Poster: 3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes »
Haotian Xue · Antonio Torralba · Josh Tenenbaum · Dan Yamins · Yunzhu Li · Hsiao-Yu Tung -
2023 Poster: Learning New Dimensions of Human Visual Similarity using Synthetic Data »
Stephanie Fu · Netanel Tamir · Shobhita Sundaram · Lucy Chai · Richard Zhang · Tali Dekel · Phillip Isola -
2023 Poster: Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models »
Sivan Doveh · Assaf Arbelle · Sivan Harary · Roei Herzig · Donghyun Kim · Paola Cascante-Bonilla · Amit Alfassy · Rameswar Panda · Raja Giryes · Rogerio Feris · Shimon Ullman · Leonid Karlinsky -
2023 Poster: Structure from Duplicates: Neural Inverse Graphics from a Single Image »
Tianhang Cheng · Wei-Chiu Ma · Kaiyu Guan · Antonio Torralba · Shenlong Wang -
2023 Poster: A Function Interpretation Benchmark for Evaluating Interpretability Methods »
Sarah Schwettmann · Tamar Shaham · Joanna Materzynska · Neil Chowdhury · Shuang Li · Jacob Andreas · David Bau · Antonio Torralba -
2023 Poster: Learning Human Action Recognition Representations Without Real Humans »
Howard Zhong · Samarth Mishra · Donghyun Kim · SouYoung Jin · Rameswar Panda · Hilde Kuehne · Leonid Karlinsky · Venkatesh Saligrama · Aude Oliva · Rogerio Feris -
2023 Poster: Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning »
Joseph Suarez · David Bloomin · Kyoung Choe · Hao Xiang Li · Ryan Sullivan · Nishaanth Kanna · Daniel Scott · Rose Shuman · Herbie Bradley · Louis Castricato · Phillip Isola · Chenghui Yu · Yuhao Jiang · Qimai Li · Jiaxin Chen · Xiaolong Zhu -
2023 Workshop: Goal-Conditioned Reinforcement Learning »
Benjamin Eysenbach · Ishan Durugkar · Jason Yecheng Ma · Andi Peng · Tongzhou Wang · Amy Zhang -
2023 Competition: The NeurIPS 2023 Neural MMO Challenge: Multi-Task Reinforcement Learning and Curriculum Generation »
Joseph Suarez · Phillip Isola · David Bloomin · Kyoung Choe · Hao Li · Ryan Sullivan · Nishaanth Kanna · Daniel Scott · Rose Shuman · Herbie Bradley · Louis Castricato · Chenghui Yu · Yuhao Jiang · Qimai Li · Jiaxin Chen · Xiaolong Zhu · Dipam Chakrabroty · Sharada Mohanty -
2022 Competition: The Third Neural MMO Challenge: Learning to Specialize in Massively Multiagent Open Worlds »
Joseph Suarez · Hanmo Chen · Arbin Chen · Bo Wu · Xiaolong Zhu · enhong liu · JUN HU · Chenghui Yu · Phillip Isola -
2022 : Line Drawings as Communication »
Phillip Isola -
2022 Poster: Semantic uncertainty intervals for disentangled latent spaces »
Swami Sankaranarayanan · Anastasios Angelopoulos · Stephen Bates · Yaniv Romano · Phillip Isola -
2022 Poster: Learning Neural Acoustic Fields »
Andrew Luo · Yilun Du · Michael Tarr · Josh Tenenbaum · Antonio Torralba · Chuang Gan -
2022 Poster: Offline Multi-Agent Reinforcement Learning with Knowledge Distillation »
Wei-Cheng Tseng · Tsun-Hsuan Johnson Wang · Yen-Chen Lin · Phillip Isola -
2022 Poster: Pre-Trained Language Models for Interactive Decision-Making »
Shuang Li · Xavier Puig · Chris Paxton · Yilun Du · Clinton Wang · Linxi Fan · Tao Chen · De-An Huang · Ekin Akyürek · Anima Anandkumar · Jacob Andreas · Igor Mordatch · Antonio Torralba · Yuke Zhu -
2022 Poster: How Transferable are Video Representations Based on Synthetic Data? »
Yo-whan Kim · Samarth Mishra · SouYoung Jin · Rameswar Panda · Hilde Kuehne · Leonid Karlinsky · Venkatesh Saligrama · Kate Saenko · Aude Oliva · Rogerio Feris -
2022 Poster: FETA: Towards Specializing Foundational Models for Expert Task Applications »
Amit Alfassy · Assaf Arbelle · Oshri Halimi · Sivan Harary · Roei Herzig · Eli Schwartz · Rameswar Panda · Michele Dolfi · Christoph Auer · Peter Staar · Kate Saenko · Rogerio Feris · Leonid Karlinsky -
2022 Poster: ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment »
Joseph DelPreto · Chao Liu · Yiyue Luo · Michael Foshey · Yunzhu Li · Antonio Torralba · Wojciech Matusik · Daniela Rus -
2021 Poster: Learning to Ground Multi-Agent Communication with Autoencoders »
Toru Lin · Jacob Huh · Christopher Stauffer · Ser Nam Lim · Phillip Isola -
2021 Poster: Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data »
Ashraful Islam · Chun-Fu (Richard) Chen · Rameswar Panda · Leonid Karlinsky · Rogerio Feris · Richard J. Radke -
2021 Poster: Learning to See by Looking at Noise »
Manel Baradad Jurjo · Jonas Wulff · Tongzhou Wang · Phillip Isola · Antonio Torralba -
2021 Poster: IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers »
Bowen Pan · Rameswar Panda · Yifan Jiang · Zhangyang Wang · Rogerio Feris · Aude Oliva -
2021 Poster: Adaptable Agent Populations via a Generative Model of Policies »
Kenneth Derek · Phillip Isola -
2020 Poster: Supervised Contrastive Learning »
Prannay Khosla · Piotr Teterwak · Chen Wang · Aaron Sarna · Yonglong Tian · Phillip Isola · Aaron Maschinot · Ce Liu · Dilip Krishnan -
2020 Poster: What Makes for Good Views for Contrastive Learning? »
Yonglong Tian · Chen Sun · Ben Poole · Dilip Krishnan · Cordelia Schmid · Phillip Isola -
2020 Poster: AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning »
Ximeng Sun · Rameswar Panda · Rogerio Feris · Kate Saenko -
2020 Poster: Debiased Contrastive Learning »
Ching-Yao Chuang · Joshua Robinson · Yen-Chen Lin · Antonio Torralba · Stefanie Jegelka -
2020 Spotlight: Debiased Contrastive Learning »
Ching-Yao Chuang · Joshua Robinson · Yen-Chen Lin · Antonio Torralba · Stefanie Jegelka -
2020 : Discussion Panel: Hugo Larochelle, Finale Doshi-Velez, Devi Parikh, Marc Deisenroth, Julien Mairal, Katja Hofmann, Phillip Isola, and Michael Bowling »
Hugo Larochelle · Finale Doshi-Velez · Marc Deisenroth · Devi Parikh · Julien Mairal · Katja Hofmann · Phillip Isola · Michael Bowling -
2019 : Poster Presentations »
Rahul Mehta · Andrew Lampinen · Binghong Chen · Sergio Pascual-Diaz · Jordi Grau-Moya · Aldo Faisal · Jonathan Tompson · Yiren Lu · Khimya Khetarpal · Martin Klissarov · Pierre-Luc Bacon · Doina Precup · Thanard Kurutach · Aviv Tamar · Pieter Abbeel · Jinke He · Maximilian Igl · Shimon Whiteson · Wendelin Boehmer · Raphaël Marinier · Olivier Pietquin · Karol Hausman · Sergey Levine · Chelsea Finn · Tianhe Yu · Lisa Lee · Benjamin Eysenbach · Emilio Parisotto · Eric Xing · Ruslan Salakhutdinov · Hongyu Ren · Anima Anandkumar · Deepak Pathak · Christopher Lu · Trevor Darrell · Alexei Efros · Phillip Isola · Feng Liu · Bo Han · Gang Niu · Masashi Sugiyama · Saurabh Kumar · Janith Petangoda · Johan Ferret · James McClelland · Kara Liu · Animesh Garg · Robert Lange -
2019 : Oral Presentations »
Janith Petangoda · Sergio Pascual-Diaz · Jordi Grau-Moya · Raphaël Marinier · Olivier Pietquin · Alexei Efros · Phillip Isola · Trevor Darrell · Christopher Lu · Deepak Pathak · Johan Ferret -
2019 : Adaptive Multi-Task Neural Networks for Efficient Inference »
Rogerio Feris -
2019 Poster: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2019 Spotlight: Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity »
Deepak Pathak · Christopher Lu · Trevor Darrell · Phillip Isola · Alexei Efros -
2018 : Panel Discussion »
Antonio Torralba · Douwe Kiela · Barbara Landau · Angeliki Lazaridou · Joyce Chai · Christopher Manning · Stevan Harnad · Roozbeh Mottaghi -
2018 : Antonio Torralba - Learning to See and Hear »
Antonio Torralba -
2018 Poster: Meta-Learning MCMC Proposals »
Tongzhou Wang · YI WU · Dave Moore · Stuart Russell -
2018 Poster: Delta-encoder: an effective sample synthesis method for few-shot object recognition »
Eli Schwartz · Leonid Karlinsky · Joseph Shtok · Sivan Harary · Mattias Marder · Abhishek Kumar · Rogerio Feris · Raja Giryes · Alex Bronstein -
2018 Spotlight: Delta-encoder: an effective sample synthesis method for few-shot object recognition »
Eli Schwartz · Leonid Karlinsky · Joseph Shtok · Sivan Harary · Mattias Marder · Abhishek Kumar · Rogerio Feris · Raja Giryes · Alex Bronstein -
2018 Poster: Dialog-based Interactive Image Retrieval »
Xiaoxiao Guo · Hui Wu · Yu Cheng · Steven Rennie · Gerald Tesauro · Rogerio Feris -
2018 Poster: Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding »
Kexin Yi · Jiajun Wu · Chuang Gan · Antonio Torralba · Pushmeet Kohli · Josh Tenenbaum -
2018 Poster: 3D-Aware Scene Manipulation via Inverse Graphics »
Shunyu Yao · Tzu Ming Hsu · Jun-Yan Zhu · Jiajun Wu · Antonio Torralba · Bill Freeman · Josh Tenenbaum -
2018 Spotlight: Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding »
Kexin Yi · Jiajun Wu · Chuang Gan · Antonio Torralba · Pushmeet Kohli · Josh Tenenbaum -
2018 Poster: Co-regularized Alignment for Unsupervised Domain Adaptation »
Abhishek Kumar · Prasanna Sattigeri · Kahini Wadhawan · Leonid Karlinsky · Rogerio Feris · Bill Freeman · Gregory Wornell -
2016 Poster: Generating Videos with Scene Dynamics »
Carl Vondrick · Hamed Pirsiavash · Antonio Torralba -
2016 Poster: SoundNet: Learning Sound Representations from Unlabeled Video »
Yusuf Aytar · Carl Vondrick · Antonio Torralba -
2015 Poster: Skip-Thought Vectors »
Jamie Kiros · Yukun Zhu · Russ Salakhutdinov · Richard Zemel · Raquel Urtasun · Antonio Torralba · Sanja Fidler -
2015 Poster: Where are they looking? »
Adria Recasens · Aditya Khosla · Carl Vondrick · Antonio Torralba -
2015 Spotlight: Where are they looking? »
Adria Recasens · Aditya Khosla · Carl Vondrick · Antonio Torralba -
2015 Poster: Learning visual biases from human imagination »
Carl Vondrick · Hamed Pirsiavash · Aude Oliva · Antonio Torralba -
2011 Poster: Understanding the Intrinsic Memorability of Images »
Phillip Isola · Devi Parikh · Antonio Torralba · Aude Oliva