Compositional Generation with Pareto-aware Joint Energy Based Models

Compositional generation with energy-based models aims to synthesize samples that satisfy multiple attribute constraints, but remains brittle in the presence of strong attribute entanglement and conflicting objectives. To address this challenge, we introduce Pareto-JEM, a unified Pareto-aware framework that reformulates hybrid generative–discriminative training and compositional inference as multi objective optimization. By leveraging conflict-aware gradient aggregation, our method constructs update directions that provably avoid increasing any objective to first order, mitigating gradient interference between generative modeling and attribute prediction. At inference time, we extend the same Pareto geometry to the data space, enabling robust resolution of conflicting attribute guidance while preserving unconditional realism. Experiments on image generation and protein sequence generation demonstrate that this unified perspective improves hybrid modeling ability in images and consistently enhances compositional controllability under conflicting constraints across both continuous and discrete domains.