Sepsis is a leading cause of mortality and its treatment is very expensive. Sepsis treatment is also very challenging because there is no consensus on what interventions work best and different patients respond very differently to the same treatment. Deep Reinforcement Learning methods can be used to come up with optimal policies for treatment strategies mirroring physician actions. In the health care scenario, the available data is mostly collected offline with no interaction with the environment, which necessitates the use of offline RL techniques. However, offline RL paradigm suffers from action distribution shifts which in turn negatively affect learning an optimal policy for the treatment. In this work, we propose to use the Conservative-Q Learning (CQL) algorithm to mitigate this shift. Experimental results on MIMIC-III dataset demonstrate that the learned policy is more similar to the physicians’ policy as compared to the policies learned from conventional deep Q Learning algorithms. The policy learned from the proposed CQL approach could help clinicians in Intensive Care Units to make better decisions while treating septic patients and improve the survival rate.