Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have worked on the computational modeling of how humans incorporate AI recommendations into their final decisions, and how to utilize these models to improve human-AI team performance. Currently, due to the ``black-box'' nature of AI models, the incorporation of explanations in AI-assisted decision making is becoming more common to help humans better rely on AI recommendations. In this work, we explore whether we can quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign. Furthermore, individuals often fail to detect any anomalies in these explanations, despite their behavior being affected by them.