Unsupervised Domain Adaptation in the Real World: A Case Study

In real world applications of machine learning, adaptation to new domains (e.g. new regions, new populations, new sensors, or new points in time) has been shown to be an ongoing challenge. In unsupervised domain adaptation, the assumption is that the user has access to a large labeled set of source domain data, and the goal is to adapt to a new target domain without the use of any labeled target data. The open question is how unlabeled samples from the target domain should be incorporated into the model training process. In this work we document our experiences applying recently proposed unsupervised domain adaption techniques for object detection to a novel application domain: counting fish in sonar video. We find that: (i) prior works that show progress on standard domain adaptation benchmark datasets do not necessarily translate to our domain, (ii) validation methods are often unrealistic in these prior works, and (iii) higher complexity (in terms of implementation and parameters) techniques work better. We aim for this work to be a useful guide for other practitioners looking to use unsupervised domain adaptation techniques in real world applications.