Timezone: »
We propose a novel confidence metric, namely, attribution-based confidence (ABC) for deep neural networks (DNNs). ABC metric characterizes whether the output of a DNN on an input can be trusted. DNNs are known to be brittle on inputs outside the training distribution and are, hence, susceptible to adversarial attacks. This fragility is compounded by a lack of effectively computable measures of model confidence that correlate well with the accuracy of DNNs. These factors have impeded the adoption of DNNs in high-assurance systems. The proposed ABC metric addresses these challenges. It does not require access to the training data, the use of ensembles, or the need to train a calibration model on a held-out validation set. Hence, the new metric is usable even when only a trained model is available for inference. We mathematically motivate the proposed metric and evaluate its effectiveness with two sets of experiments. First, we study the change in accuracy and the associated confidence over out-of-distribution inputs. Second, we consider several digital and physically realizable attacks such as FGSM, CW, DeepFool, PGD, and adversarial patch generation methods. The ABC metric is low on out-of-distribution data and adversarial examples, where the accuracy of the model is also low. These experiments demonstrate the effectiveness of the ABC metric to make DNNs more trustworthy and resilient.
Author Information
Susmit Jha (SRI)
Susmit Jha is a Principal Computer Scientist in the Computer Science Laboratory at SRI International where he is the principal investigator for DARPA Assured Autonomy, DARPA Symbiotic Design of CPS, DARPA Intent-driven Design of Adaptive Systems, IARPA TrojAI, US ARL’s Principles of Robust Learning in IoBT CRA, and NSF Self-improving Cyberphysical Systems. Prior to joining SRI, Dr. Jha was a Staff Scientist at UTRC (Raytheon Technologies), Berkeley, and a Research Scientist at Intel. Dr. Jha received his Ph.D. in Electrical Engineering and Computer Science from the University of California, Berkeley in 2011.
Sunny Raj (University of Central Florida)
Steven Fernandes (University of Central Florida)
Sumit K Jha (University of Central Florida)
Dr. Sumit K. Jha is an Associate Professor of Computer Science at the University of Central Florida (UCF), Orlando. Dr. Jha joined the University of Central Florida in 2010 after receiving his Ph.D. in Computer Science at Carnegie Mellon University. Before joining Carnegie Mellon, he graduated with B.Tech (Honors) in Computer Science and Engineering from the Indian Institute of Technology Kharagpur in 2004. Dr. Jha has worked on R&D problems at Microsoft Research India, General Motors, INRIA France and the Air Force Research Lab Information Directorate. His research has been supported by the National Science Foundation, the Air Force Office of Scientific Research, the Oak Ridge National Laboratory, the Royal Bank of Canada, the Florida Center for Cybersecurity, and the Air Force Research Laboratory. He is a full member of the Sigma Xi and is a recipient of the IEEE Orlando Engineering Educator Excellence Award. Dr. Jha was awarded the prestigious Air Force Young Investigator Award in 2016 and his research has led to three Best Paper awards.
Somesh Jha (University of Wisconsin, Madison)
Brian Jalaian (U.S. Army Research Laboratory)
Gunjan Verma (U.S. Army Research Laboratory)
Ananthram Swami (Army Research Laboratory, Adelphi)
More from the Same Authors
-
2022 : Socially Responsible Reasoning with Large Language Models and The Impact of Proper Nouns »
Sumit Jha · Rickard Ewetz · Alvaro Velasquez · Susmit Jha -
2022 : Best of Both Worlds: Towards Adversarial Robustness with Transduction and Rejection »
Nils Palumbo · Yang Guo · Xi Wu · Jiefeng Chen · Yingyu Liang · Somesh Jha -
2022 Spotlight: Lightning Talks 2A-2 »
Harikrishnan N B · Jianhao Ding · Juha Harviainen · Yizhen Wang · Lue Tao · Oren Mangoubi · Tong Bu · Nisheeth Vishnoi · Mohannad Alhanahnah · Mikko Koivisto · Aditi Kathpalia · Lei Feng · Nithin Nagaraj · Hongxin Wei · Xiaozhu Meng · Petteri Kaski · Zhaofei Yu · Tiejun Huang · Ke Wang · Jinfeng Yi · Jian Liu · Sheng-Jun Huang · Mihai Christodorescu · Songcan Chen · Somesh Jha -
2022 Spotlight: Robust Learning against Relational Adversaries »
Yizhen Wang · Mohannad Alhanahnah · Xiaozhu Meng · Ke Wang · Mihai Christodorescu · Somesh Jha -
2022 Poster: Physics-Informed Implicit Representations of Equilibrium Network Flows »
Kevin D. Smith · Francesco Seccamonte · Ananthram Swami · Francesco Bullo -
2022 Poster: Overparameterization from Computational Constraints »
Sanjam Garg · Somesh Jha · Saeed Mahloujifar · Mohammad Mahmoody · Mingyuan Wang -
2022 Poster: Robust Learning against Relational Adversaries »
Yizhen Wang · Mohannad Alhanahnah · Xiaozhu Meng · Ke Wang · Mihai Christodorescu · Somesh Jha -
2022 Poster: A Quantitative Geometric Approach to Neural-Network Smoothness »
Zi Wang · Gautam Prakriya · Somesh Jha -
2021 Poster: Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles »
Jiefeng Chen · Frederick Liu · Besim Avci · Xi Wu · Yingyu Liang · Somesh Jha -
2021 Poster: A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks »
Samuel Deng · Sanjam Garg · Somesh Jha · Saeed Mahloujifar · Mohammad Mahmoody · Abhradeep Guha Thakurta -
2020 Poster: Unsupervised Joint k-node Graph Representations with Compositional Energy-Based Models »
Leonardo Cotta · Carlos H. C. Teixeira · Ananthram Swami · Bruno Ribeiro -
2019 : Posters »
Colin Graber · Yuan-Ting Hu · Tiantian Fang · Jessica Hamrick · Giorgio Giannone · John Co-Reyes · Boyang Deng · Eric Crawford · Andrea Dittadi · Peter Karkus · Matthew Dirks · Rakshit Trivedi · Sunny Raj · Javier Felip Leon · Harris Chan · Jan Chorowski · Jeff Orchard · Aleksandar Stanić · Adam Kortylewski · Ben Zinberg · Chenghui Zhou · Wei Sun · Vikash Mansinghka · Chun-Liang Li · Marco Cusumano-Towner -
2019 Poster: Robust Attribution Regularization »
Jiefeng Chen · Xi Wu · Vaibhav Rastogi · Yingyu Liang · Somesh Jha -
2019 Poster: Error Correcting Output Codes Improve Probability Estimation and Adversarial Robustness of Deep Neural Networks »
Gunjan Verma · Ananthram Swami -
2018 : Semantic Adversarial Examples by Somesh Jha »
Somesh Jha