Modern supervised learning methods have shown that utilizing large amounts of diverse data can lead to remarkable results especially when it comes to generalization. On the other hand, reinforcement learning approaches, while powerful, remain fairly limited in terms of diversity of data they can effectively utilize, making it difficult to apply them to diverse, open-ended settings such as real-world robotics. In this talk, I'll talk about the steps we are taking towards making reinforcement learning a better "data sponge" and how they allow us to apply such methods to real robots at scale.