Marginals by Flows, Dependence by Energy: Learning Multimodal Joint Distributions

Energy-Based Models (EBMs) offer a flexible way to represent complex multimodal joint distributions, but their practical use is limited by the difficulty of jointly learning modality-specific realism and cross-modal dependence. We propose to separate these roles. Each modality is modeled in a continuous latent space by an autoencoder equipped with a Normalizing Flow (NF), and the product of the learned marginals defines a tractable reference distribution. A residual EBM then learns the density correction required to couple these marginals into a coherent multimodal joint. This construction gives the energy a direct probabilistic interpretation: with exact marginals, the optimal residual energy is the negative pointwise dependency score---PMI in the two-modality case and the pointwise analogue of multi-information for more modalities. We train the compatibility energy with dependency-ratio contrastive pretraining followed by persistent maximum-likelihood refinement, and sample the model with modality-wise MALA updates in the flow base space. On PolyMNIST and MNIST--SVHN, the resulting model achieves a strong quality--coherence trade-off. Controlled experiments further show that the reference distribution is already marginally plausible, that the residual energy captures cross-modal dependence rather than merely improving initialization, and that better marginal models improve the final joint model. These results support a simple design principle for multimodal energy-based modeling: model the marginals explicitly, and spend the energy on dependence.