Ensuring machines remain beneficial to humans requires that machine learning systems are still able to communicate their inner workings such that another observer can infer its reasoning and intent/s. This process, known as explainability, is crucial in helping shape our relationship with machine learning systems. Despite the advantages of existing approaches to implement explainability in machine learning systems and learn through more natural interactions with humans and other agents, current algorithms generally (1) are not evaluated in teamwork and human decision-making scenarios and (2) often require large numbers of examples on how to solve a task. These are both crucial aspects for humans to operate alongside machine learning systems, especially in interactive settings. To address the above-mentioned limitations, in our work we conducted three studies centered around first, understanding the role of explanations in human-machine teamwork, second, exploring human learning from intelligent systems using machine-generated explanations, and thirdly, incorporating human explanations into machine learning. In presenting our computational models around these aspects, we hope to advance our knowledge and understanding of different facets of explainable agency in machine learning and enable successful human-AI partnership and knowledge transfer.