Timezone: »
Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs).We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.
Author Information
Oren Rippel (MIT)
Jasper Snoek (Twitter Cortex)
Ryan Adams (Harvard)
More from the Same Authors
-
2021 Spotlight: Slice Sampling Reparameterization Gradients »
David Zoltowski · Diana Cai · Ryan Adams -
2021 Spotlight: Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate »
Xingyuan Sun · Tianju Xue · Szymon Rusinkiewicz · Ryan Adams -
2021 : ProBF: Probabilistic Safety Certificates with Barrier Functions »
Sulin Liu · Athindran Ramesh Kumar · Jaime Fisac · Ryan Adams · Peter J. Ramadge -
2021 : Reading the Road: Leveraging Meta-Learning to Learn Other Driver Behavior »
Anat Kleiman · Ryan Adams -
2021 : Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning »
Zachary Nado · Neil Band · Mark Collier · Josip Djolonga · Mike Dusenberry · Sebastian Farquhar · Qixuan Feng · Angelos Filos · Marton Havasi · Rodolphe Jenatton · Ghassen Jerfel · Jeremiah Liu · Zelda Mariet · Jeremy Nixon · Shreyas Padhy · Jie Ren · Tim G. J. Rudner · Yeming Wen · Florian Wenzel · Kevin Murphy · D. Sculley · Balaji Lakshminarayanan · Jasper Snoek · Yarin Gal · Dustin Tran -
2022 : A code superoptimizer through neural Monte-Carlo tree search »
Wenda Zhou · Olga Solodova · Ryan Adams -
2022 : A code superoptimizer through neural Monte-Carlo tree search »
Wenda Zhou · Olga Solodova · Ryan Adams -
2022 Poster: Multi-fidelity Monte Carlo: a pseudo-marginal approach »
Diana Cai · Ryan Adams -
2021 : Randomized Automatic Differentiation - Ryan Adams - Princeton University »
Ryan Adams -
2021 Poster: Slice Sampling Reparameterization Gradients »
David Zoltowski · Diana Cai · Ryan Adams -
2021 Poster: Amortized Synthesis of Constrained Configurations Using a Differentiable Surrogate »
Xingyuan Sun · Tianju Xue · Szymon Rusinkiewicz · Ryan Adams -
2021 Poster: Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability »
Dibya Ghosh · Jad Rahme · Aviral Kumar · Amy Zhang · Ryan Adams · Sergey Levine -
2020 : Orals 1.1: Randomized Automatic Differentiation »
Deniz Oktay · Nick McGreivy · Alex Beatson · Ryan Adams -
2020 Workshop: Machine Learning for Engineering Modeling, Simulation and Design »
Alex Beatson · Priya Donti · Amira Abdel-Rahman · Stephan Hoyer · Rose Yu · J. Zico Kolter · Ryan Adams -
2020 Poster: On Warm-Starting Neural Network Training »
Jordan Ash · Ryan Adams -
2020 Poster: Task-Agnostic Amortized Inference of Gaussian Process Hyperparameters »
Sulin Liu · Xingyuan Sun · Peter J. Ramadge · Ryan Adams -
2020 Poster: Learning Composable Energy Surrogates for PDE Order Reduction »
Alex Beatson · Jordan Ash · Geoffrey Roeder · Tianju Xue · Ryan Adams -
2020 Oral: Learning Composable Energy Surrogates for PDE Order Reduction »
Alex Beatson · Jordan Ash · Geoffrey Roeder · Tianju Xue · Ryan Adams -
2019 Workshop: Learning Meaningful Representations of Life »
Elizabeth Wood · Yakir Reshef · Jonathan Bloom · Jasper Snoek · Barbara Engelhardt · Scott Linderman · Suchi Saria · Alexander Wiltschko · Casey Greene · Chang Liu · Kresten Lindorff-Larsen · Debora Marks -
2019 Poster: SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers »
Igor Fedorov · Ryan Adams · Matthew Mattina · Paul Whatmough -
2019 Poster: Discrete Object Generation with Reversible Inductive Construction »
Ari Seff · Wenda Zhou · Farhan Damani · Abigail Doyle · Ryan Adams -
2019 Poster: Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift »
Jasper Snoek · Yaniv Ovadia · Emily Fertig · Balaji Lakshminarayanan · Sebastian Nowozin · D. Sculley · Joshua Dillon · Jie Ren · Zachary Nado -
2018 : Discussion Panel: Ryan Adams, Nicolas Heess, Leslie Kaelbling, Shie Mannor, Emo Todorov (moderator: Roy Fox) »
Ryan Adams · Nicolas Heess · Leslie Kaelbling · Shie Mannor · Emo Todorov · Roy Fox -
2018 : Inference and Control of Learning Behavior in Rodents (Ryan Adams) »
Ryan Adams -
2018 Poster: A Bayesian Nonparametric View on Count-Min Sketch »
Diana Cai · Michael Mitzenmacher · Ryan Adams -
2017 Workshop: Machine Learning for Health (ML4H) - What Parts of Healthcare are Ripe for Disruption by Machine Learning Right Now? »
Jason Fries · Alex Wiltschko · Andrew Beam · Isaac S Kohane · Jasper Snoek · Peter Schulam · Madalina Fiterau · David Kale · Rajesh Ranganath · Bruno Jedynak · Michael Hughes · Tristan Naumann · Natalia Antropova · Adrian Dalca · SHUBHI ASTHANA · Prateek Tandon · Jaz Kandola · Uri Shalit · Marzyeh Ghassemi · Tim Althoff · Alexander Ratner · Jumana Dakka -
2017 Poster: PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference »
Jonathan Huggins · Ryan Adams · Tamara Broderick -
2017 Spotlight: PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference »
Jonathan Huggins · Ryan Adams · Tamara Broderick -
2017 Poster: Reducing Reparameterization Gradient Variance »
Andrew Miller · Nick Foti · Alexander D'Amour · Ryan Adams -
2016 : Panel Discussion »
Shakir Mohamed · David Blei · Ryan Adams · José Miguel Hernández-Lobato · Ian Goodfellow · Yarin Gal -
2016 : A Tribute to David MacKay »
Ryan Adams -
2016 Workshop: Bayesian Optimization: Black-box Optimization and Beyond »
Roberto Calandra · Bobak Shahriari · Javier Gonzalez · Frank Hutter · Ryan Adams -
2016 : Leveraging Structure in Bayesian Optimization »
Ryan Adams -
2016 Poster: Bayesian latent structure discovery from multi-neuron recordings »
Scott Linderman · Ryan Adams · Jonathan Pillow -
2016 Poster: Composing graphical models with neural networks for structured representations and fast inference »
Matthew Johnson · David Duvenaud · Alex Wiltschko · Ryan Adams · Sandeep R Datta -
2015 Workshop: Bayesian Optimization: Scalability and Flexibility »
Bobak Shahriari · Ryan Adams · Nando de Freitas · Amar Shah · Roberto Calandra -
2015 Workshop: Statistical Methods for Understanding Neural Systems »
Alyson Fletcher · Jakob H Macke · Ryan Adams · Jascha Sohl-Dickstein -
2015 Poster: Convolutional Networks on Graphs for Learning Molecular Fingerprints »
David Duvenaud · Dougal Maclaurin · Jorge Iparraguirre · Rafael Bombarell · Timothy Hirzel · Alan Aspuru-Guzik · Ryan Adams -
2015 Poster: A Gaussian Process Model of Quasar Spectral Energy Distributions »
Andrew Miller · Albert Wu · Jeffrey Regier · Jon McAuliffe · Dustin Lang · Mr. Prabhat · David Schlegel · Ryan Adams -
2015 Poster: Dependent Multinomial Models Made Easy: Stick-Breaking with the Polya-gamma Augmentation »
Scott Linderman · Matthew Johnson · Ryan Adams -
2014 Workshop: Bayesian Optimization in Academia and Industry »
Zoubin Ghahramani · Ryan Adams · Matthew Hoffman · Kevin Swersky · Jasper Snoek -
2014 Poster: A framework for studying synaptic plasticity with neural spike train data »
Scott Linderman · Christopher H Stock · Ryan Adams -
2013 Workshop: Bayesian Optimization in Theory and Practice »
Matthew Hoffman · Jasper Snoek · Nando de Freitas · Michael A Osborne · Ryan Adams · Sebastien Bubeck · Philipp Hennig · Remi Munos · Andreas Krause -
2013 Poster: Multi-Task Bayesian Optimization »
Kevin Swersky · Jasper Snoek · Ryan Adams -
2013 Poster: Message Passing Inference with Chemical Reaction Networks »
Nils E Napp · Ryan Adams -
2013 Oral: Message Passing Inference with Chemical Reaction Networks »
Nils E Napp · Ryan Adams -
2013 Poster: A Determinantal Point Process Latent Variable Model for Inhibition in Neural Spiking Data »
Jasper Snoek · Richard Zemel · Ryan Adams -
2013 Poster: Contrastive Learning Using Spectral Methods »
James Y Zou · Daniel Hsu · David Parkes · Ryan Adams -
2012 Poster: Bayesian n-Choose-k Models for Classification and Ranking »
Kevin Swersky · Danny Tarlow · Richard Zemel · Ryan Adams · Brendan J Frey -
2012 Poster: Priors for Diversity in Generative Latent Variable Models »
James Y Zou · Ryan Adams -
2012 Poster: Cardinality Restricted Boltzmann Machines »
Kevin Swersky · Danny Tarlow · Ilya Sutskever · Richard Zemel · Russ Salakhutdinov · Ryan Adams -
2012 Poster: Practical Bayesian Optimization of Machine Learning Algorithms »
Jasper Snoek · Hugo Larochelle · Ryan Adams -
2011 Workshop: Bayesian Nonparametric Methods: Hope or Hype? »
Emily Fox · Ryan Adams -
2010 Workshop: Transfer Learning Via Rich Generative Models. »
Russ Salakhutdinov · Ryan Adams · Josh Tenenbaum · Zoubin Ghahramani · Tom Griffiths -
2010 Workshop: Monte Carlo Methods for Bayesian Inference in Modern Day Applications »
Ryan Adams · Mark A Girolami · Iain Murray -
2010 Oral: Tree-Structured Stick Breaking for Hierarchical Data »
Ryan Adams · Zoubin Ghahramani · Michael Jordan -
2010 Oral: Slice sampling covariance hyperparameters of latent Gaussian models »
Iain Murray · Ryan Adams -
2010 Poster: Tree-Structured Stick Breaking for Hierarchical Data »
Ryan Adams · Zoubin Ghahramani · Michael Jordan -
2010 Poster: Slice sampling covariance hyperparameters of latent Gaussian models »
Iain Murray · Ryan Adams -
2008 Poster: The Gaussian Process Density Sampler »
Ryan Adams · Iain Murray · David MacKay -
2008 Spotlight: The Gaussian Process Density Sampler »
Ryan Adams · Iain Murray · David MacKay