Autoregressive Models Beyond Language

Autoregressive modeling is no longer confined to language. Recent work shows that the same next-element prediction principle can achieve state-of-the-art performance in generative modeling, representation learning, and multi-modal tasks across images, video, audio, robotics, and scientific data. Yet, extending autoregressive methods to these data is far from straightforward. Many inductive biases used in autoregressive language models no longer hold for other data modalities, and thus, many new techniques have been proposed in recent years to adapt autoregressive models to data beyond language.

This tutorial will review the core theory of autoregressive models, present practical design choices for generative modeling, representation learning, and multi-modal learning, and spotlight open challenges in this area. We hope our tutorial can provide the attendees with a clear conceptual roadmap and hands-on resources to apply and extend autoregressive techniques across diverse data domains.