As predictive models are increasingly being employed to make consequential decisions in various real-world applications, it becomes important to ensure that relevant stakeholders and decision makers correctly understand the functionality of these models so that they can diagnose errors and potential biases in them, and decide when and how to employ these models. To this end, recent research in AI/ML has focused on developing techniques which aim to explain complex models to relevant stakeholders. In this talk, I will give a brief overview of the field of explainable AI while highlighting our research in this area. More specifically, I will discuss our work on: (a) developing inherently interpretable models and post hoc explanation methods, (b) identifying the vulnerabilities and shortcomings of these methods, and addressing them, (c) evaluating the reliability (correctness, robustness, fairness) and human understanding of the explanations output by these methods, and (d) theoretical results on unifying these methods. I will conclude this talk by shedding light on some exciting future research directions – e.g., rethinking model explainability as a (natural language) dialogue between humans and AI, and redesigning explainable AI tools to cater to large pretrained models.