In this paper, we seek to reduce the communication barrier between the hearing-impaired community and the larger society who are usually not familiar with sign language in the region with the largest occurrences of hearing disability cases, i.e. sub-Saharan Africa while using Nigeria as a case study. The dataset is a pioneer dataset for the Nigerian Sign Language and it was created via collaboration with relevant stakeholders. We preprocessed the data in readiness for two different object detection models and a classification model and employed diverse evaluation metrics. We convert the sign texts to speech and deploy the best performing model in a lightweight sign-to-speech machine learning application that works in real-time and achieves impressive results converting sign words/phrases to text and speech.