Timezone: »

Learning to Discover and Detect Objects
Vladimir Fomenko · Ismail Elezi · Deva Ramanan · Laura Leal-Taixé · Aljosa Osep

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #230

We tackle the problem of novel class discovery, detection, and localization (NCDL). In this setting, we assume a source dataset with labels for objects of commonly observed classes. Instances of other classes need to be discovered, classified, and localized automatically based on visual similarity, without human supervision. To this end, we propose a two-stage object detection network Region-based NCDL (RNCDL), that uses a region proposal network to localize object candidates and is trained to classify each candidate, either as one of the known classes, seen in the source dataset, or one of the extended set of novel classes, with a long-tail distribution constraint on the class assignments, reflecting the natural frequency of classes in the real world. By training our detection network with this objective in an end-to-end manner, it learns to classify all region proposals for a large variety of classes, including those that are not part of the labeled object class vocabulary. Our experiments conducted using COCO and LVIS datasets reveal that our method is significantly more effective compared to multi-stage pipelines that rely on traditional clustering algorithms or use pre-extracted crops. Furthermore, we demonstrate the generality of our approach by applying our method to a large-scale Visual Genome dataset, where our network successfully learns to detect various semantic classes without explicit supervision. Code is available at: https://github.com/vlfom/RNCDL.

Author Information

Vladimir Fomenko (Technical University of Munich)
Vladimir Fomenko

I currently conduct applied research and machine learning engineering for large language models as an applied scientist in Azure AI. Prior to this, I completed my master's degree in Artificial Intelligence from Technical University of Munich. My research interests are in natural language processing, computer vision, and representation learning, using advanced deep learning methods.

Ismail Elezi (Technical University of Munich)
Deva Ramanan (Carnegie Mellon University)
Laura Leal-Taixé (TUM)
Aljosa Osep (TU Munich)

More from the Same Authors