Improving Conditional Score-Based Generation with Calibrated Classification and Joint Training

Score-based Generative Model (SGM) is a popular family of deep generative models that can achieve leading image generation quality. Earlier works have extended SGMs to tackle class-conditional generation with the guidance of well-trained classifiers. Nevertheless, we find that the classifier-guided SGMs actually do not achieve accurate conditional generation when evaluated with class-conditional measures. We argue that the lack of control roots from inaccurate gradients within the classifiers. We then propose to improve classifier-guided SGMs by calibrating classifiers using principles from energy-based models. In addition, we design a joint-training architecture to further enhance the conditional generation performance. Empirical results on CIFAR-10 demonstrate that the proposed model improves the conditional generation accuracy significantly while maintaining similar generation quality. The results support the potential of memory-efficient SGMs for conditional generation based on classifier guidance.