Timezone: »
We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques developed in chemical physics and exploit our continuous time framework to derive high performance samplers that we show can outperform discrete time methods for discrete data. The continuous time treatment also enables us to derive a novel theoretical result bounding the error between the generated sample distribution and the true data distribution.
Author Information
Andrew Campbell (University of Oxford)
Joe Benton (University of Oxford)

PhD student in the Department of Statistics at the University of Oxford, supervised by Arnaud Doucet and George Deligiannidis. Working on developing a theoretical understanding of generative modelling techniques, with a particular focus on diffusion models and variational autoencoders.
Valentin De Bortoli (ENS Ulm, CNRS)
Thomas Rainforth (University of Oxford)
George Deligiannidis (Oxford)
Arnaud Doucet (Oxford)
More from the Same Authors
-
2021 Spotlight: Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling »
Valentin De Bortoli · James Thornton · Jeremy Heng · Arnaud Doucet -
2021 Spotlight: Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms »
Alexander Camuto · George Deligiannidis · Murat Erdogdu · Mert Gurbuzbalaban · Umut Simsekli · Lingjiong Zhu -
2021 : Certifiably Robust Variational Autoencoders »
Ben Barrett · Alexander Camuto · Matthew Willetts · Thomas Rainforth -
2021 : Certifiably Robust Variational Autoencoders »
Ben Barrett · Alexander Camuto · Matthew Willetts · Thomas Rainforth -
2021 : Certifiably Robust Variational Autoencoders »
Ben Barrett · Alexander Camuto · Matthew Willetts · Thomas Rainforth -
2022 : Spectral Diffusion Processes »
Angus Phillips · Thomas Seror · Michael Hutchinson · Valentin De Bortoli · Arnaud Doucet · Emile Mathieu -
2023 : Fast protein backbone generation with SE(3) flow matching »
Jason Yim · Andrew Campbell · Yue Kwang Foong · Sarah Lewis · Victor Satorras · Michael Gastegger · Bas Veeling · Jose Jimenez-Luna · Regina Barzilay · Tommi Jaakkola · Frank Noe -
2023 : Measuring Feature Sparsity in Language Models »
Mingyang Deng · Lucas Tao · Joe Benton -
2023 Workshop: NeurIPS 2023 Workshop on Diffusion Models »
Bahjat Kawar · Valentin De Bortoli · Charlotte Bunne · James Thornton · Jiaming Song · Jong Chul Ye · Chenlin Meng -
2023 Poster: Trans-Dimensional Generative Modeling via Jump Diffusion Models »
Andrew Campbell · William Harvey · Christian Weilbach · Valentin De Bortoli · Thomas Rainforth · Arnaud Doucet -
2023 Poster: Diffusion Schrödinger Bridge Matching »
Yuyang Shi · Valentin De Bortoli · Andrew Campbell · Arnaud Doucet -
2023 Poster: Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits »
Muhammad Faaiz Taufiq · Arnaud Doucet · Rob Cornish · Jean-Francois Ton -
2023 Poster: Deep Stochastic Processes via Functional Markov Transition Operators »
Jin Xu · Emilien Dupont · Kaspar Märtens · Thomas Rainforth · Yee Whye Teh -
2023 Poster: Tree-Based Diffusion Schrödinger Bridge with Applications to Wasserstein Barycenters »
Maxence Noble · Valentin De Bortoli · Arnaud Doucet · Alain Durmus -
2023 Poster: A Unified Framework for U-Net Design and Analysis »
Christopher Williams · Fabian Falck · George Deligiannidis · Chris C Holmes · Arnaud Doucet · Saifuddin Syed -
2023 Poster: Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics »
Kamélia Daudel · Joe Benton · Yuyang Shi · Arnaud Doucet -
2022 Spotlight: Lightning Talks 1A-4 »
Siwei Wang · Jing Liu · Nianqiao Ju · Shiqian Li · Eloïse Berthier · Muhammad Faaiz Taufiq · Arsene Fansi Tchango · Chen Liang · Chulin Xie · Jordan Awan · Jean-Francois Ton · Ziad Kobeissi · Wenguan Wang · Xinwang Liu · Kewen Wu · Rishab Goel · Jiaxu Miao · Suyuan Liu · Julien Martel · Ruobin Gong · Francis Bach · Chi Zhang · Rob Cornish · Sanmi Koyejo · Zhi Wen · Yee Whye Teh · Yi Yang · Jiaqi Jin · Bo Li · Yixin Zhu · Vinayak Rao · Wenxuan Tu · Gaetan Marceau Caron · Arnaud Doucet · Xinzhong Zhu · Joumana Ghosn · En Zhu -
2022 Panel: Panel 1B-2: Multi-Game Decision Transformers… & A Continuous Time… »
Andrew Campbell · Kuang-Huei Lee -
2022 Spotlight: Conformal Off-Policy Prediction in Contextual Bandits »
Muhammad Faaiz Taufiq · Jean-Francois Ton · Rob Cornish · Yee Whye Teh · Arnaud Doucet -
2022 Workshop: NeurIPS 2022 Workshop on Score-Based Methods »
Yingzhen Li · Yang Song · Valentin De Bortoli · Francois-Xavier Briol · Wenbo Gong · Alexia Jolicoeur-Martineau · Arash Vahdat -
2022 Poster: Conformal Off-Policy Prediction in Contextual Bandits »
Muhammad Faaiz Taufiq · Jean-Francois Ton · Rob Cornish · Yee Whye Teh · Arnaud Doucet -
2022 Poster: Can Push-forward Generative Models Fit Multimodal Distributions? »
Antoine Salmona · Valentin De Bortoli · Julie Delon · Agnes Desolneux -
2022 Poster: Score-Based Diffusion meets Annealed Importance Sampling »
Arnaud Doucet · Will Grathwohl · Alexander Matthews · Heiko Strathmann -
2022 Poster: Rethinking Variational Inference for Probabilistic Programs with Stochastic Support »
Tim Reichelt · Luke Ong · Thomas Rainforth -
2022 Poster: A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs »
Fabian Falck · Christopher Williams · Dominic Danks · George Deligiannidis · Christopher Yau · Chris C Holmes · Arnaud Doucet · Matthew Willetts -
2022 Poster: Riemannian Score-Based Generative Modelling »
Valentin De Bortoli · Emile Mathieu · Michael Hutchinson · James Thornton · Yee Whye Teh · Arnaud Doucet -
2022 Poster: Towards Learning Universal Hyperparameter Optimizers with Transformers »
Yutian Chen · Xingyou Song · Chansoo Lee · Zi Wang · Richard Zhang · David Dohan · Kazuya Kawakami · Greg Kochanski · Arnaud Doucet · Marc'Aurelio Ranzato · Sagi Perel · Nando de Freitas -
2022 Poster: Wavelet Score-Based Generative Modeling »
Florentin Guth · Simon Coste · Valentin De Bortoli · Stephane Mallat -
2022 Poster: Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation »
Jannik Kossen · Sebastian Farquhar · Yarin Gal · Thomas Rainforth -
2021 Poster: Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling »
Valentin De Bortoli · James Thornton · Jeremy Heng · Arnaud Doucet -
2021 Poster: Group Equivariant Subsampling »
Jin Xu · Hyunjik Kim · Thomas Rainforth · Yee Teh -
2021 Poster: Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods »
Desi R Ivanova · Adam Foster · Steven Kleinegesse · Michael Gutmann · Thomas Rainforth -
2021 Poster: Online Variational Filtering and Parameter Learning »
Andrew Campbell · Yuyang Shi · Thomas Rainforth · Arnaud Doucet -
2021 Poster: Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning »
Jannik Kossen · Neil Band · Clare Lyle · Aidan Gomez · Thomas Rainforth · Yarin Gal -
2021 Poster: Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms »
Alexander Camuto · George Deligiannidis · Murat Erdogdu · Mert Gurbuzbalaban · Umut Simsekli · Lingjiong Zhu -
2021 Oral: Online Variational Filtering and Parameter Learning »
Andrew Campbell · Yuyang Shi · Thomas Rainforth · Arnaud Doucet -
2020 Poster: Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks »
Umut Simsekli · Ozan Sener · George Deligiannidis · Murat Erdogdu -
2020 Poster: Quantitative Propagation of Chaos for SGD in Wide Neural Networks »
Valentin De Bortoli · Alain Durmus · Xavier Fontaine · Umut Simsekli -
2020 Spotlight: Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks »
Umut Simsekli · Ozan Sener · George Deligiannidis · Murat Erdogdu -
2019 Poster: On the Fairness of Disentangled Representations »
Francesco Locatello · Gabriele Abbati · Thomas Rainforth · Stefan Bauer · Bernhard Schölkopf · Olivier Bachem -
2019 Poster: Variational Bayesian Optimal Experimental Design »
Adam Foster · Martin Jankowiak · Elias Bingham · Paul Horsfall · Yee Whye Teh · Thomas Rainforth · Noah Goodman -
2019 Spotlight: Variational Bayesian Optimal Experimental Design »
Adam Foster · Martin Jankowiak · Elias Bingham · Paul Horsfall · Yee Whye Teh · Thomas Rainforth · Noah Goodman -
2019 Poster: Augmented Neural ODEs »
Emilien Dupont · Arnaud Doucet · Yee Whye Teh -
2018 : Panel on research process »
Zachary Lipton · Charles Sutton · Finale Doshi-Velez · Hanna Wallach · Suchi Saria · Rich Caruana · Thomas Rainforth -
2018 Workshop: Critiquing and Correcting Trends in Machine Learning »
Thomas Rainforth · Matt Kusner · Benjamin Bloem-Reddy · Brooks Paige · Rich Caruana · Yee Whye Teh -
2018 Poster: Faithful Inversion of Generative Models for Effective Amortized Inference »
Stefan Webb · Adam Golinski · Rob Zinkov · Siddharth N · Thomas Rainforth · Yee Whye Teh · Frank Wood -
2018 Poster: Hamiltonian Variational Auto-Encoder »
Anthony Caterini · Arnaud Doucet · Dino Sejdinovic -
2017 : Poster Spotlights »
Francesco Locatello · Ari Pakman · Da Tang · Thomas Rainforth · Zalan Borsos · Marko Järvenpää · Eric Nalisnick · Gabriele Abbati · XIAOYU LU · Jonathan Huggins · Rachit Singh · Rui Luo -
2017 Poster: Filtering Variational Objectives »
Chris Maddison · John Lawson · George Tucker · Nicolas Heess · Mohammad Norouzi · Andriy Mnih · Arnaud Doucet · Yee Teh -
2017 Poster: Clone MCMC: Parallel High-Dimensional Gaussian Gibbs Sampling »
Andrei-Cristian Barbos · Francois Caron · Jean-François Giovannelli · Arnaud Doucet -
2016 : Probabilistic structure discovery in time series data »
David Janz · Brooks Paige · Thomas Rainforth · Jan-Willem van de Meent -
2016 Poster: Bayesian Optimization for Probabilistic Programs »
Thomas Rainforth · Tuan Anh Le · Jan-Willem van de Meent · Michael A Osborne · Frank Wood -
2015 Workshop: Scalable Monte Carlo Methods for Bayesian Analysis of Big Data »
Babak Shahbaba · Yee Whye Teh · Max Welling · Arnaud Doucet · Christophe Andrieu · Sebastian J. Vollmer · Pierre Jacob -
2015 Poster: Expectation Particle Belief Propagation »
Thibaut Lienart · Yee Whye Teh · Arnaud Doucet -
2014 Poster: Asynchronous Anytime Sequential Monte Carlo »
Brooks Paige · Frank Wood · Arnaud Doucet · Yee Whye Teh -
2014 Oral: Asynchronous Anytime Sequential Monte Carlo »
Brooks Paige · Frank Wood · Arnaud Doucet · Yee Whye Teh -
2009 Poster: Bayesian Nonparametric Models on Decomposable Graphs »
Francois Caron · Arnaud Doucet -
2009 Tutorial: Sequential Monte-Carlo Methods »
Arnaud Doucet · Nando de Freitas -
2007 Spotlight: Bayesian Policy Learning with Trans-Dimensional MCMC »
Matthew Hoffman · Arnaud Doucet · Nando de Freitas · Ajay Jasra -
2007 Poster: Bayesian Policy Learning with Trans-Dimensional MCMC »
Matthew Hoffman · Arnaud Doucet · Nando de Freitas · Ajay Jasra