Remote sensing is a crucial tool for monitoring events affecting climate change, such as tracking forest loss, identifying pollution sources, and monitoring the deployment of renewable energy infrastructure. However, applying state-of-the-art deep learning models to monitor the entire Earth is expensive. In this paper, we propose a cascade framework to reduce this cost: we apply a small MLP on precomputed embeddings of each image patch to serve as a preliminary filter, identifying key patches that warrant further examination by more resource-intensive deep models. Our approach reduces per-task inference runtime by 5x with a <1% impact on accuracy. By reducing inference cost, our method enables nonprofits and other organizations with limited resources to scale monitoring efforts to more environmental and conservation applications.