Transformer-based Causal Language Models from a Meta-Learning Perspective

The Transformer architecture has become prominent for developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process that may happen within the Transformer. Further, from within the inner optimization, we discover a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments conducted on pre-trained large language models and real-world data.