The growing energy demands of machine learning workloads require sustainable data centers with lower carbon footprints and reduced energy consumption. Supercomputing and many high-performance computing (HPC) data centers, which use liquid cooling for greater efficiency than traditional air cooling systems, can significantly benefit from advanced optimization techniques to control liquid cooling. We present RL-LC, a novel Reinforcement Learning (RL) based approach designed to enhance the efficiency of liquid cooling in these environments. RL-LC integrates a customizable analytical liquid cooling model suitable for simulations or digital twins of data centers, focusing on minimizing energy consumption and carbon emissions. Our method achieves an average reduction of approximately 4% compared to industry-standard ASHRAE guidelines, contributing to more sustainable data center management and offering valuable insights for reducing the environmental impact of HPC operations.