In spite of many proposals to solve the neural machine translation problem of low-resource languages, it continues to be difficult for languages with few resources. The issue becomes even more complicated when few resources cover only a single domain. In our attempt to combat this issue, we propose a new approach to improve NMT for low-resource languages. The proposed approach using the transformer model shows 5.3, 5.0, and 3.7 BLEU score improvement for Gamo-English, Gofa-English, and Dawuro-English language pairs, respectively. We discuss our contributions and envisage future steps in this challenging research area.