DRL-EPANET: Deep reinforcement learning for optimal control at scale in Water Distribution Systems

Deep Reinforcement learning has known a revolution in recent years, it has allowed researchers to tackle a wide range of sequential decision problems that were inaccessible to previous methods. However, the use of this technique in Water Distribution Systems is still very shy. In this paper, we show that DRL can be coupled with the widely popular hydraulic simulator Epanet, and that DRL-Epanet can be used on a number of WDS problems that represent a challenge to current techniques. We take as a concrete example the problem of pressure control in WDS. We show that DRL-Epanet can scale to huge action spaces, and we demonstrate its effectiveness on a problem with more than 1 million possible actions at each time step. We also show that it can deal with uncertainty such as stochastic demands, contamination, or other risks, as an example, we take on the problem of pressure control in the presence of random pipe bursts. We show that the BDQ algorithm is able to learn in this setting and we improve it with an algorithmic modification BDQF (BDQ with Fixed actions) which achieves better rewards especially when allowed actions are sparse in the action space. Finally, we argue that DRL-Epanet can be used for real-time control in smart WDS, another advantage over current methods.