Effective Text-to-Image Alignment with Quality Aware Pair Ranking

Fine-tuning techniques such as Reinforcement Learning with Human Feedback(RLHF) and Direct Preference Optimization (DPO) allow us to steer Large Language Models (LLMs) to align better with human preferences. Alignment isequally important in text-to-image generation. Recent adoption of DPO, specifically Diffusion-DPO, for Text-to-Image (T2I) diffusion models has proven towork effectively in improving visual appeal and prompt-image alignment. Thementioned works fine-tune on Pick-a-Pic dataset, consisting of approximately onemillion image preference pairs, collected via crowdsourcing at scale. However, doall preference pairs contribute equally to alignment fine-tuning? Preferences can besubjective at times and may not always translate into effectively aligning the model.In this work, we investigate the above-mentioned question. We develop a qualitymetric to rank image preference pairs and achieve effective Diffusion-DPO-basedalignment fine-tuning.We show that the SD-1.5 and SDXL models fine-tuned usingthe top 5.33% of the data perform better both quantitatively and qualitatively thanthe models fine-tuned on the full dataset.