Timezone: »
We consider the problem of distilling an image into an ordered set of maximally informative patches, given prior data from the same domain. We cast this problem as one of maximizing a pointwise mutual information (PMI) objective between a subset of an image's patches and the perceptual content of the entire image. We take an image synthesis-based approach, reasoning that the patches that are most informative would also be most useful for predicting other pixel values. We capture this idea with an image completion CNN trained to model the PMI between an image's perceptual content and any of its subregions. Because our PMI objective is a submodular, monotonic function, we can greedily construct patch sets using the CNN to obtain a provably close approximation to the intractable optimal solution. We evaluate our approach on datasets of faces, common objects, and line drawings. For all datasets, we find that a surprisingly few number of patches are needed to reconstruct most images, demonstrating a particular type of redundancy of information in images, and new potentials in their sparse representations. We also show that these minimal patch sets may be used effectively for downstream tasks such as image classification.
Author Information
Howard Zhong (MIT)
Guha Balakrishnan (MIT)
Richard Bowen (Cornell University)
Ramin Zabih (Google Research)
Bill Freeman (MIT/Google)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 : Finding Maximally Informative Patches in Images »
Dates n/a. Room
More from the Same Authors
-
2021 Spotlight: Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering »
Vincent Sitzmann · Semon Rezchikov · Bill Freeman · Josh Tenenbaum · Fredo Durand -
2023 Poster: Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision »
Ayush Tewari · Tianwei Yin · George Cazenavette · Semon Rezchikov · Josh Tenenbaum · Fredo Durand · Bill Freeman · Vincent Sitzmann -
2023 Poster: Test-Time Distribution Normalization for Contrastively Learned Visual-language Models »
Yifei Zhou · Juntao Ren · Fengyu Li · Ramin Zabih · Ser Nam Lim -
2023 Poster: Learning Human Action Recognition Representations Without Real Humans »
Howard Zhong · Samarth Mishra · Donghyun Kim · SouYoung Jin · Rameswar Panda · Hilde Kuehne · Leonid Karlinsky · Venkatesh Saligrama · Aude Oliva · Rogerio Feris -
2022 Poster: Associating Objects and Their Effects in Video through Coordination Games »
Erika Lu · Forrester Cole · Weidi Xie · Tali Dekel · Bill Freeman · Andrew Zisserman · Michael Rubinstein -
2021 Poster: Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering »
Vincent Sitzmann · Semon Rezchikov · Bill Freeman · Josh Tenenbaum · Fredo Durand -
2020 Poster: Multi-Plane Program Induction with 3D Box Priors »
Yikai Li · Jiayuan Mao · Xiuming Zhang · Bill Freeman · Josh Tenenbaum · Noah Snavely · Jiajun Wu -
2020 Demonstration: MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval »
Mark Hamilton · Stephanie Fu · Mindren Lu · Johnny Bui · Margaret Wang · Felix Tran · Marina Rogers · Darius Bopp · Christopher Hoder · Lei Zhang · Bill Freeman -
2019 : Feathers, wings and the future of computer vision research »
Bill Freeman -
2019 Poster: Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization »
Miika Aittala · Prafull Sharma · Lukas Murmann · Adam Yedidia · Gregory Wornell · Bill Freeman · Fredo Durand -
2018 Poster: Learning to Reconstruct Shapes from Unseen Classes »
Xiuming Zhang · Zhoutong Zhang · Chengkai Zhang · Josh Tenenbaum · Bill Freeman · Jiajun Wu -
2018 Oral: Learning to Reconstruct Shapes from Unseen Classes »
Xiuming Zhang · Zhoutong Zhang · Chengkai Zhang · Josh Tenenbaum · Bill Freeman · Jiajun Wu -
2018 Poster: Visual Object Networks: Image Generation with Disentangled 3D Representations »
Jun-Yan Zhu · Zhoutong Zhang · Chengkai Zhang · Jiajun Wu · Antonio Torralba · Josh Tenenbaum · Bill Freeman -
2018 Poster: Learning to Exploit Stability for 3D Scene Parsing »
Yilun Du · Zhijian Liu · Hector Basevi · Ales Leonardis · Bill Freeman · Josh Tenenbaum · Jiajun Wu -
2018 Poster: 3D-Aware Scene Manipulation via Inverse Graphics »
Shunyu Yao · Tzu Ming Hsu · Jun-Yan Zhu · Jiajun Wu · Antonio Torralba · Bill Freeman · Josh Tenenbaum -
2018 Poster: Co-regularized Alignment for Unsupervised Domain Adaptation »
Abhishek Kumar · Prasanna Sattigeri · Kahini Wadhawan · Leonid Karlinsky · Rogerio Feris · Bill Freeman · Gregory Wornell -
2017 : Sight and sound »
Bill Freeman -
2017 Spotlight: Shape and Material from Sound »
Zhoutong Zhang · Qiujia Li · Zhengjia Huang · Jiajun Wu · Josh Tenenbaum · Bill Freeman -
2017 Spotlight: Scene Physics Acquisition via Visual De-animation »
Jiajun Wu · Erika Lu · Pushmeet Kohli · Bill Freeman · Josh Tenenbaum -
2017 Poster: Learning to See Physics via Visual De-animation »
Jiajun Wu · Erika Lu · Pushmeet Kohli · Bill Freeman · Josh Tenenbaum -
2017 Poster: Shape and Material from Sound »
Zhoutong Zhang · Qiujia Li · Zhengjia Huang · Jiajun Wu · Josh Tenenbaum · Bill Freeman -
2017 Poster: MarrNet: 3D Shape Reconstruction via 2.5D Sketches »
Jiajun Wu · Yifan Wang · Tianfan Xue · Xingyuan Sun · Bill Freeman · Josh Tenenbaum -
2016 : Bill Freeman »
Bill Freeman -
2016 Poster: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling »
Jiajun Wu · Chengkai Zhang · Tianfan Xue · Bill Freeman · Josh Tenenbaum -
2016 Poster: Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks »
Tianfan Xue · Jiajun Wu · Katherine Bouman · Bill Freeman -
2016 Oral: Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks »
Tianfan Xue · Jiajun Wu · Katherine Bouman · Bill Freeman -
2015 Poster: Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning »
Jiajun Wu · Ilker Yildirim · Joseph Lim · Bill Freeman · Josh Tenenbaum -
2014 Poster: Shape and Illumination from Shading using the Generic Viewpoint Assumption »
Daniel Zoran · Dilip Krishnan · José Bento · Bill Freeman -
2010 Workshop: Machine Learning meets Computational Photography »
Stefan Harmeling · Michael Hirsch · Bill Freeman · Peyman Milanfar -
2009 Poster: Segmenting Scenes by Matching Image Composites »
Bryan C Russell · Alexei A Efros · Josef Sivic · Bill Freeman · Andrew Zisserman -
2009 Poster: Nonparametric Bayesian Texture Learning and Synthesis »
Leo Zhu · Yuanhao Chen · Bill Freeman · Antonio Torralba -
2008 Mini Symposium: Computational Photography »
Bill Freeman · Bernhard Schölkopf