Timezone: »
Traditional machine learning follows a close-set assumption that the training and test set share the same label space. While in many practical scenarios, it is inevitable that some test samples belong to unknown classes (open-set). To fix this issue, Open-Set Recognition (OSR), whose goal is to make correct predictions on both close-set samples and open-set samples, has attracted rising attention. In this direction, the vast majority of literature focuses on the pattern of open-set samples. However, how to evaluate model performance in this challenging task is still unsolved. In this paper, a systematic analysis reveals that most existing metrics are essentially inconsistent with the aforementioned goal of OSR: (1) For metrics extended from close-set classification, such as Open-set F-score, Youden's index, and Normalized Accuracy, a poor open-set prediction can escape from a low performance score with a superior close-set prediction. (2) Novelty detection AUC, which measures the ranking performance between close-set and open-set samples, ignores the close-set performance. To fix these issues, we propose a novel metric named OpenAUC. Compared with existing metrics, OpenAUC enjoys a concise pairwise formulation that evaluates open-set performance and close-set performance in a coupling manner. Further analysis shows that OpenAUC is free from the aforementioned inconsistency properties. Finally, an end-to-end learning method is proposed to minimize the OpenAUC risk, and the experimental results on popular benchmark datasets speak to its effectiveness.
Author Information
Zitai Wang (Institute of Computing Technology, Chinese Academy of Sciences)
Qianqian Xu (Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences)
Zhiyong Yang (Chinese Academy of Sciences)
Yuan He (Alibaba Group)
Xiaochun Cao (SUN YAT-SEN UNIVERSITY)
Qingming Huang (University of Chinese Academy of Sciences)
More from the Same Authors
-
2022 Poster: OPEN: Orthogonal Propagation with Ego-Network Modeling »
Liang Yang · Lina Kang · Qiuliang Zhang · Mengzhe Li · bingxin niu · Dongxiao He · Zhen Wang · Chuan Wang · Xiaochun Cao · Yuanfang Guo -
2022 Poster: Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm »
HuiYang Shao · Qianqian Xu · Zhiyong Yang · Shilong Bao · Qingming Huang -
2022 Poster: Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability »
Peisong Wen · Qianqian Xu · Zhiyong Yang · Yuan He · Qingming Huang -
2022 Spotlight: OpenAUC: Towards AUC-Oriented Open-Set Recognition »
Zitai Wang · Qianqian Xu · Zhiyong Yang · Yuan He · Xiaochun Cao · Qingming Huang -
2022 Poster: Rethinking Image Restoration for Object Detection »
Shangquan Sun · Wenqi Ren · Tao Wang · Xiaochun Cao -
2022 Poster: OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport »
Zongsheng Cao · Qianqian Xu · Zhiyong Yang · Yuan He · Xiaochun Cao · Qingming Huang -
2022 Poster: The Minority Matters: A Diversity-Promoting Collaborative Metric Learning Algorithm »
Shilong Bao · Qianqian Xu · Zhiyong Yang · Yuan He · Xiaochun Cao · Qingming Huang -
2021 Poster: When False Positive is Intolerant: End-to-End Optimization with Low FPR for Multipartite Ranking »
Peisong Wen · Qianqian Xu · Zhiyong Yang · Yuan He · Qingming Huang -
2020 Poster: Heuristic Domain Adaptation »
Shuhao Cui · Xuan Jin · Shuhui Wang · Yuan He · Qingming Huang -
2019 Poster: Generalized Block-Diagonal Structure Pursuit: Learning Soft Latent Task Assignment against Negative Transfer »
Zhiyong Yang · Qianqian Xu · Yangbangyan Jiang · Xiaochun Cao · Qingming Huang -
2019 Poster: DM2C: Deep Mixed-Modal Clustering »
Yangbangyan Jiang · Qianqian Xu · Zhiyong Yang · Xiaochun Cao · Qingming Huang -
2019 Spotlight: DM2C: Deep Mixed-Modal Clustering »
Yangbangyan Jiang · Qianqian Xu · Zhiyong Yang · Xiaochun Cao · Qingming Huang -
2019 Poster: iSplit LBI: Individualized Partial Ranking with Ties via Split LBI »
Qianqian Xu · Xinwei Sun · Zhiyong Yang · Xiaochun Cao · Qingming Huang · Yuan Yao