Deep learning has made great progress in a variety of language and vision tasks. However, there are still many practical and theoretical problems and limitations. In this talk I will introduce solutions to the following questions: How to have a single input and output encoding for words. How to predict previously unseen words during test time encounters. How to grow a single deep learning model for many increasingly complex language tasks. Can an end-to-end trainable architecture solve both visual and textual question answering?