Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules

Humans excel at discovering regular structures from limited samples and applying inferred rules to novel settings. We investigate whether modern generative AI systems can similarly learn underlying rules from finite samples and perform reasoning through conditional sampling. Inspired by Raven’s Progressive Matrices task, we designed GenRAVEN dataset, where each sample consists of three rows and one of the 40 relational rules governing the objects position, number or attributes applies to all three rows. We trained generative models to learn the data distribution, where samples are encoded as 3×9×9 integer arrays to focus on rule learning. We compare two major families of generative models: diffusion models (EDM, DiT, SiT) and autoregressive models (GPT2, Mamba). We evaluated their ability to generate structurally consistent samples and perform panel completion via unconditional and conditional sampling. We found that diffusion models excel at unconditional generation, producing novel and more consistent samples from scratch and memorize less, but perform less well in panel completion, even with advanced conditional sampling methods like Twisted Diffusion Sampler. Conversely, autoregressive models excel at completing missing panels in a rule-consistent manner but generate less consistent samples unconditionally. We observe diverse data scaling behaviors: for both types of models, rule learning emerges at a certain dataset size, around thousand examples per rule. With more training data, diffusion models improve both their unconditional and conditional generation capabilities. However, for autoregressive models, while panel completion improves with more training data, unconditional generation consistency declines. Our findings highlight complementary capabilities and limitations of diffusion and autoregressive models in rule learning and reasoning tasks, suggesting avenues for further research into their mechanisms and potential for human-like reasoning.