Timezone: »

 
Poster
OpenAUC: Towards AUC-Oriented Open-Set Recognition
Zitai Wang · Qianqian Xu · Zhiyong Yang · Yuan He · Xiaochun Cao · Qingming Huang

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #919
in Poster Session 3 »

Traditional machine learning follows a close-set assumption that the training and test set share the same label space. While in many practical scenarios, it is inevitable that some test samples belong to unknown classes (open-set). To fix this issue, Open-Set Recognition (OSR), whose goal is to make correct predictions on both close-set samples and open-set samples, has attracted rising attention. In this direction, the vast majority of literature focuses on the pattern of open-set samples. However, how to evaluate model performance in this challenging task is still unsolved. In this paper, a systematic analysis reveals that most existing metrics are essentially inconsistent with the aforementioned goal of OSR: (1) For metrics extended from close-set classification, such as Open-set F-score, Youden's index, and Normalized Accuracy, a poor open-set prediction can escape from a low performance score with a superior close-set prediction. (2) Novelty detection AUC, which measures the ranking performance between close-set and open-set samples, ignores the close-set performance. To fix these issues, we propose a novel metric named OpenAUC. Compared with existing metrics, OpenAUC enjoys a concise pairwise formulation that evaluates open-set performance and close-set performance in a coupling manner. Further analysis shows that OpenAUC is free from the aforementioned inconsistency properties. Finally, an end-to-end learning method is proposed to minimize the OpenAUC risk, and the experimental results on popular benchmark datasets speak to its effectiveness.

[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Zitai Wang (Institute of Computing Technology, Chinese Academy of Sciences)
Qianqian Xu (Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences)
Zhiyong Yang (Chinese Academy of Sciences)
Yuan He (Alibaba Group)
Xiaochun Cao (SUN YAT-SEN UNIVERSITY)
Qingming Huang (University of Chinese Academy of Sciences)

More from the Same Authors