Estimating the homography between two images is a crucial process for various tasks, such as image stitching, reconstruction. However, applying supervised learning methods is often challenging or expensive due to the difficulty of collecting ground truth data. Consequently, methods based on unsupervised learning have been developed. Most of these methods assume that the given image pairs are from the same camera or have slight differences in lighting. While these studies perform effectively under these assumptions, they generally fail when the input pair images come from different domains, referred to as multimodal image pairs. To overcome these limitations, we propose a learning framework for estimating homography in multimodal image pairs using unsupervised learning, which does not require ground truth. Our method employs a two-phase optimization framework similar to Expectation-Maximization (EM), featuring phases to reduce the geometry gap and to capture the modality gap. To handle these gaps, we also propose to use Barlow Twins loss for modality gap and its extended version, named `Geometry Barlow Twins, for geometry gap. As a result, we have demonstrated that it is possible to train a various architectures of networks to outperform than other unsupervised based methods on multimodal datasets.