The success of language models, especially transformers in natural language processing, has trickled into scientific domains, giving rise to the concept of "scientific language models" that operate on small molecules, proteins or polymers. In chemistry, language models contribute to accelerating the molecule discovery cycle, as evidenced by promising recent findings in early-stage drug discovery. In this perspective, we review the role of language models in molecular discovery, underlining their strengths and examining their weaknesses in de novo drug design, property prediction and reaction chemistry. We highlight valuable open-source software assets to lower the entry barrier to the field of scientific language modeling. Furthermore, as a solution to some of the weaknesses we identify, we outline a vision for future molecular design that integrates a chat-bot interface with available computational chemistry tools. Our contribution serves as a valuable resource for researchers, chemists, and AI enthusiasts interested in understanding how language models can and will be used to accelerate chemical discovery.