Timezone: »
Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.
Author Information
Jingkang Yang (Nanyang Technological University)
Pengyun Wang (Beijing University of Posts and Telecommunications)
Dejian Zou (Beijing University of Posts and Telecommunications)
Zitang Zhou (Beijing University of Posts and Telecommunications)
Kunyuan Ding (Beijing University of Posts and Telecommunications)
WENXUAN PENG (Nanyang Technological University)
Haoqi Wang (SenseTime)
Guangyao Chen (Peking University)
Bo Li (Nanyang Technological University)
Yiyou Sun (University of Wisconsin, Madison)
Xuefeng Du (UW-Madison)
Kaiyang Zhou (Nanyang Technological University)
Wayne Zhang (SenseTime Research)
Dan Hendrycks (Center for AI Safety)
Yixuan Li (University of Wisconsin-Madison)
Ziwei Liu (Nanyang Technological University)
More from the Same Authors
-
2021 : CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review »
Dan Hendrycks · Collin Burns · Anya Chen · Spencer Ball -
2021 : Measuring Coding Challenge Competence With APPS »
Dan Hendrycks · Steven Basart · Saurav Kadavath · Mantas Mazeika · Akul Arora · Ethan Guo · Collin Burns · Samir Puranik · Horace He · Dawn Song · Jacob Steinhardt -
2021 : PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures »
Dan Hendrycks · Andy Zou · Mantas Mazeika · Leonard Tang · Dawn Song · Jacob Steinhardt -
2021 : Semi-Supervised Domain Generalization with Stochastic StyleMatch »
Kaiyang Zhou · Chen Change Loy · Ziwei Liu -
2021 : What Would Jiminy Cricket Do? Towards Agents That Behave Morally »
Dan Hendrycks · Mantas Mazeika · Andy Zou · Sahil Patel · Christine Zhu · Jesus Navarro · Dawn Song · Bo Li · Jacob Steinhardt -
2021 : Measuring Mathematical Problem Solving With the MATH Dataset »
Dan Hendrycks · Collin Burns · Saurav Kadavath · Akul Arora · Steven Basart · Eric Tang · Dawn Song · Jacob Steinhardt -
2022 Poster: SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning »
Haobo Wang · Mingxuan Xia · Yixuan Li · Yuren Mao · Lei Feng · Gang Chen · Junbo Zhao -
2022 Poster: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning »
Yangru Huang · Peixi Peng · Yifan Zhao · Guangyao Chen · Yonghong Tian -
2022 Poster: Audio-Driven Co-Speech Gesture Video Generation »
Xian Liu · Qianyi Wu · Hang Zhou · Yuanqi Du · Wayne Wu · Dahua Lin · Ziwei Liu -
2022 Poster: Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Ziwei Liu -
2022 : Sparse Mixture-of-Experts are Domain Generalizable Learners »
Bo Li · Yifei Shen · Jingkang Yang · Yezhen Wang · Jiawei Ren · Tong Che · Jun Zhang · Ziwei Liu -
2022 : GIST: Distributed Training for Large-Scale Graph Convolutional Networks »
Cameron Wolfe · Jingkang Yang · Fangshuo Liao · Arindam Chowdhury · Chen Dun · Artun Bayer · Santiago Segarra · Anastasios Kyrillidis -
2023 Poster: Dream the Impossible: Outlier Imagination with Diffusion Models »
Xuefeng Du · Yiyou Sun · Jerry Zhu · Yixuan Li -
2023 Poster: What Makes Good Examples for Visual In-Context Learning? »
Yuanhan Zhang · Kaiyang Zhou · Ziwei Liu -
2023 Poster: Segment Any Point Cloud Sequences by Distilling Vision Foundation Models »
Youquan Liu · Lingdong Kong · Jun CEN · Runnan Chen · Wenwei Zhang · Liang Pan · Kai Chen · Ziwei Liu -
2023 Poster: InsActor: Instruction-driven Physics-based Characters »
Jiawei Ren · Mingyuan Zhang · Cunjun Yu · Xiao Ma · Liang Pan · Ziwei Liu -
2023 Poster: SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation »
Mengcheng Lan · Xinjiang Wang · Yiping Ke · Jiaxing Xu · Litong Feng · Wayne Zhang -
2023 Poster: A Graph-Theoretic Framework for Understanding Open-World Representation Learning »
Yiyou Sun · Zhenmei Shi · Yixuan Li -
2023 Poster: Towards Robust and Expressive Whole-body Human Pose and Shape Estimation »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Qingyi Tao · Zhonghua Wu · Ziwei Liu -
2023 Poster: Learning to Augment Distributions for Out-of-distribution Detection »
Qizhou Wang · Zhen Fang · Yonggang Zhang · Feng Liu · Yixuan Li · Bo Han -
2023 Poster: FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing »
Mingyuan Zhang · Huirong Li · Zhongang Cai · Jiawei Ren · Lei Yang · Ziwei Liu -
2023 Poster: 4D Panoptic Scene Graph Generation »
Jingkang Yang · Jun CEN · WENXUAN PENG · Shuai Liu · Fangzhou Hong · Xiangtai Li · Kaiyang Zhou · Qifeng Chen · Ziwei Liu -
2023 Poster: PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation »
Zhaoxi Chen · Fangzhou Hong · Haiyi Mei · Guangcong Wang · Lei Yang · Ziwei Liu -
2023 Poster: Language Models are Visual Reasoning Coordinators »
Liangyu Chen · Bo Li · Sheng Shen · Jingkang Yang · Chunyuan Li · Kurt Keutzer · Trevor Darrell · Ziwei Liu -
2023 Poster: SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation »
Zhongang Cai · Wanqi Yin · Ailing Zeng · CHEN WEI · Qingping SUN · Yanjun Wang · Hui En Pang · Haiyi Mei · Mingyuan Zhang · Lei Zhang · Chen Change Loy · Lei Yang · Ziwei Liu -
2023 Poster: DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models »
Boxin Wang · Weixin Chen · Hengzhi Pei · Chulin Xie · Mintong Kang · Chenhui Zhang · Chejian Xu · Zidi Xiong · Ritik Dutta · Rylan Schaeffer · Sang Truong · Simran Arora · Mantas Mazeika · Dan Hendrycks · Zinan Lin · Yu Cheng · Sanmi Koyejo · Dawn Song · Bo Li -
2023 Poster: RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars »
Dongwei Pan · Long Zhuo · Jingtan Piao · Huiwen Luo · Wei Cheng · Yuxin WANG · Siming Fan · Shengqi Liu · Lei Yang · Bo Dai · Ziwei Liu · Chen Change Loy · Chen Qian · Wayne Wu · Dahua Lin · Kwan-Yee Lin -
2023 Oral: DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models »
Boxin Wang · Weixin Chen · Hengzhi Pei · Chulin Xie · Mintong Kang · Chenhui Zhang · Chejian Xu · Zidi Xiong · Ritik Dutta · Rylan Schaeffer · Sang Truong · Simran Arora · Mantas Mazeika · Dan Hendrycks · Zinan Lin · Yu Cheng · Sanmi Koyejo · Dawn Song · Bo Li -
2023 Competition: TDC 2023 (LLM Edition): The Trojan Detection Challenge »
Mantas Mazeika · Andy Zou · Norman Mu · Long Phan · Zifan Wang · Chunru Yu · Adam Khoja · Fengqing Jiang · Aidan O'Gara · Zhen Xiang · Arezoo Rajabi · Dan Hendrycks · Radha Poovendran · Bo Li · David Forsyth -
2022 Workshop: Workshop on Machine Learning Safety »
Dan Hendrycks · Victoria Krakovna · Dawn Song · Jacob Steinhardt · Nicholas Carlini -
2022 Competition: The Trojan Detection Challenge »
Mantas Mazeika · Dan Hendrycks · Huichen Li · Xiaojun Xu · Andy Zou · Sidney Hough · Arezoo Rajabi · Dawn Song · Radha Poovendran · Bo Li · David Forsyth -
2022 Spotlight: Lightning Talks 5B-3 »
Yanze Wu · Jie Xiao · Nianzu Yang · Jieyi Bi · Jian Yao · Yiting Chen · Qizhou Wang · Yangru Huang · Yongqiang Chen · Peixi Peng · Yuxin Hong · Xintao Wang · Feng Liu · Yining Ma · Qibing Ren · Xueyang Fu · Yonggang Zhang · Kaipeng Zeng · Jiahai Wang · GEN LI · Yonggang Zhang · Qitian Wu · Yifan Zhao · Chiyu Wang · Junchi Yan · Feng Wu · Yatao Bian · Xiaosong Jia · Ying Shan · Zhiguang Cao · Zheng-Jun Zha · Guangyao Chen · Tianjun Xiao · Han Yang · Jing Zhang · Jinbiao Chen · MA Kaili · Yonghong Tian · Junchi Yan · Chen Gong · Tong He · Binghui Xie · Yuan Sun · Francesco Locatello · Tongliang Liu · Yeow Meng Chee · David P Wipf · Tongliang Liu · Bo Han · Bo Han · Yanwei Fu · James Cheng · Zheng Zhang -
2022 Spotlight: Spectrum Random Masking for Generalization in Image-based Reinforcement Learning »
Yangru Huang · Peixi Peng · Yifan Zhao · Guangyao Chen · Yonghong Tian -
2022 Spotlight: Lightning Talks 4B-4 »
Ziyue Jiang · Zeeshan Khan · Yuxiang Yang · Chenze Shao · Yichong Leng · Zehao Yu · Wenguan Wang · Xian Liu · Zehua Chen · Yang Feng · Qianyi Wu · James Liang · C.V. Jawahar · Junjie Yang · Zhe Su · Songyou Peng · Yufei Xu · Junliang Guo · Michael Niemeyer · Hang Zhou · Zhou Zhao · Makarand Tapaswi · Dongfang Liu · Qian Yang · Torsten Sattler · Yuanqi Du · Haohe Liu · Jing Zhang · Andreas Geiger · Yi Ren · Long Lan · Jiawei Chen · Wayne Wu · Dahua Lin · Dacheng Tao · Xu Tan · Jinglin Liu · Ziwei Liu · 振辉 叶 · Danilo Mandic · Lei He · Xiangyang Li · Tao Qin · sheng zhao · Tie-Yan Liu -
2022 Spotlight: Audio-Driven Co-Speech Gesture Video Generation »
Xian Liu · Qianyi Wu · Hang Zhou · Yuanqi Du · Wayne Wu · Dahua Lin · Ziwei Liu -
2022 Spotlight: Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Ziwei Liu -
2022 Workshop: Robustness in Sequence Modeling »
Nathan Ng · Haoran Zhang · Vinith Suriyakumar · Chantal Shaib · Kyunghyun Cho · Yixuan Li · Alice Oh · Marzyeh Ghassemi -
2022 Poster: How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios »
Mantas Mazeika · Eric Tang · Andy Zou · Steven Basart · Jun Shern Chan · Dawn Song · David Forsyth · Jacob Steinhardt · Dan Hendrycks -
2022 Poster: Forecasting Future World Events With Neural Networks »
Andy Zou · Tristan Xiao · Ryan Jia · Joe Kwon · Mantas Mazeika · Richard Li · Dawn Song · Jacob Steinhardt · Owain Evans · Dan Hendrycks -
2022 Poster: AnimeRun: 2D Animation Visual Correspondence from Open Source 3D Movies »
Li Siyao · Yuhang Li · Bo Li · Chao Dong · Ziwei Liu · Chen Change Loy -
2022 Poster: SIREN: Shaping Representations for Detecting Out-of-Distribution Objects »
Xuefeng Du · Gabriel Gozum · Yifei Ming · Yixuan Li -
2022 Poster: Delving into Out-of-Distribution Detection with Vision-Language Representations »
Yifei Ming · Ziyang Cai · Jiuxiang Gu · Yiyou Sun · Wei Li · Yixuan Li -
2022 Poster: Is Out-of-Distribution Detection Learnable? »
Zhen Fang · Yixuan Li · Jie Lu · Jiahua Dong · Bo Han · Feng Liu -
2021 : Live panel: Perspectives on ImageNet. »
Dawn Song · Ross Wightman · Dan Hendrycks -
2021 : Using ImageNet to Measure Robustness and Uncertainty »
Dawn Song · Dan Hendrycks -
2021 : Uncovering the Deep Unknowns of ImageNet Model: Challenges and Opportunties »
Yixuan Li -
2021 Poster: On the Importance of Gradients for Detecting Distributional Shifts in the Wild »
Rui Huang · Andrew Geng · Yixuan Li -
2021 Poster: Can multi-label classification networks know what they don’t know? »
Haoran Wang · Weitang Liu · Alex Bocchieri · Yixuan Li -
2021 Poster: Balanced Chamfer Distance as a Comprehensive Metric for Point Cloud Completion »
Tong Wu · Liang Pan · Junzhe Zhang · Tai WANG · Ziwei Liu · Dahua Lin -
2021 Poster: Garment4D: Garment Reconstruction from Point Cloud Sequences »
Fangzhou Hong · Liang Pan · Zhongang Cai · Ziwei Liu -
2021 Poster: Few-Shot Object Detection via Association and DIscrimination »
Yuhang Cao · Jiaqi Wang · Ying Jin · Tong Wu · Kai Chen · Ziwei Liu · Dahua Lin -
2021 Poster: Unsupervised Object-Level Representation Learning from Scene Images »
Jiahao Xie · Xiaohang Zhan · Ziwei Liu · Yew Soon Ong · Chen Change Loy -
2021 : VisDA21: Visual Domain Adaptation + Q&A »
Kate Saenko · Kuniaki Saito · Donghyun Kim · Samarth Mishra · Ben Usman · Piotr Teterwak · Dina Bashkirova · Dan Hendrycks -
2021 Poster: ReAct: Out-of-distribution Detection With Rectified Activations »
Yiyou Sun · Chuan Guo · Yixuan Li -
2020 Poster: Energy-based Out-of-distribution Detection »
Weitang Liu · Xiaoyun Wang · John Owens · Yixuan Li -
2019 Poster: Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty »
Dan Hendrycks · Mantas Mazeika · Saurav Kadavath · Dawn Song -
2019 Poster: Multi-source Domain Adaptation for Semantic Segmentation »
Sicheng Zhao · Bo Li · Xiangyu Yue · Yang Gu · Pengfei Xu · Runbo Hu · Hua Chai · Kurt Keutzer -
2018 Poster: Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise »
Dan Hendrycks · Mantas Mazeika · Duncan Wilson · Kevin Gimpel