Meta-learning is a powerful set of approaches that promises to replace many components of the deep learning toolbox by learned alternatives, such as learned architectures, optimizers, hyperparameters, and weight initializations. While typical approaches focus on only one of these components at a time, in this talk, I will discuss various efficient approaches for tackling two of them simultaneously. I will also highlight the advantages of not learning complete algorithms from scratch but of rather exploiting the inductive bias of existing algorithms by learning to improve existing algorithms. Finally, I will briefly discuss the connection of meta-learning and benchmarks.