Large Language Models (LLMs) have achieved remarkable success in recent years, yet their monolithic architecture presents challenges for scalability, adaptability, and efficiency. This talk explores modular deep learning as a promising solution to these challenges, enabling the development of more flexible and robust AI systems. We will begin by motivating the need for modularity in the age of LLMs, focusing on key benefits such as compartmentalization, distributed training, continual learning, and compositionality. Next, we provide a comprehensive introduction to modular deep learning, covering various module architectures (ranging from parameter-efficient methods to full models), routing algorithms, and composition functions. Finally, we offer an outlook on the future of modular deep learning, highlighting current limitations and potential research directions. This talk aims to provide attendees with a deeper understanding of modular deep learning and its potential to shape the next generation of LLMs.