Understanding Denoising Diffusion Probabilistic Models and their Noise Schedules via the Ornstein--Uhlenbeck Process

The aim of this short note is to show that Denoising Diffusion Probabilistic Model (DDPM), a non-homogeneous discrete-time Markov process, can be represented by a time-homogeneous continuous-time Markov process observed at non-uniformly sampled discrete times. Surprisingly, this continuous-time Markov process is the well-known and well-studied Ornstein--Ohlenbeck (O--U) process, which was developed in 1930's for studying Brownian particles in Harmonic potentials. We establish the formal equivalence between DDPM and the O--U process using its analytical solution. We further demonstrate that the design problem of the noise scheduler for non-homogeneous DDPM is equivalent to designing observation times for the O--U process. We present several heuristic designs for observation times based on principled quantities such as auto-variance and Fisher Information and connect them to \emph{ad hoc} noise schedules for DDPM. Interestingly, we show that the Fisher-Information-motivated schedule corresponds exactly the \emph{cosine schedule}, which was developed without any theoretical foundation but is the current state-of-the-art noise schedule. Our numerical experiments further show its superior performance.