Timezone: »

 
Exploring the Limits of Epistemic Uncertainty Quantification in Low-Shot Settings
Matias Valdenegro-Toro

Tue Dec 07 11:45 AM -- 12:00 PM (PST) @

Uncertainty quantification in neural network promises to increase safety of AI systems, but it is not clear how performance might vary with the training set size. In this paper we evaluate seven uncertainty methods on Fashion MNIST and CIFAR10, as we sub-sample and produce varied training set sizes. We find that calibration error and out of distribution detection performance strongly depend on the training set size, with most methods being miscalibrated on the test set with small training sets. Gradient-based methods seem to poorly estimate epistemicuncertainty and are the most affected by training set size. We expect our results can guide future research into uncertainty quantification and help practitioners select methods based on their particular available data.

Author Information

Matias Valdenegro-Toro (German Research Center for Artificial Intelligence)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors

  • 2020 : Automatic Detection and Classification of Tick-borne Skin Lesions using Deep Learning »
    Matias Valdenegro-Toro
  • 2021 : Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning »
    Aaqib Parvez Mohammed · Matias Valdenegro-Toro
  • 2021 : Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning »
    Aaqib Parvez Mohammed · Matias Valdenegro-Toro
  • 2021 : Exploring the Limits of Epistemic Uncertainty Quantification in Low-Shot Settings »
    Matias Valdenegro-Toro
  • 2021 : Q&A Oral presentations »
    Matias Valdenegro-Toro · Andres Munoz · Johan Obando Ceron · Anil Batra
  • 2020 : QA Long Presentation II »
    Matias Valdenegro-Toro · Gefersom Lima · Nicolas Araque · Matías Molina
  • 2020 : Unsupervised Difficulty Estimation »
    Octavio Arriaga · Matias Valdenegro-Toro
  • 2019 : Poster session »
    Sebastian Farquhar · Erik Daxberger · Andreas Look · Matt Benatan · Ruiyi Zhang · Marton Havasi · Fredrik Gustafsson · James A Brofos · Nabeel Seedat · Micha Livne · Ivan Ustyuzhaninov · Adam Cobb · Felix D McGregor · Patrick McClure · Tim R. Davidson · Gaurush Hiranandani · Sanjeev Arora · Masha Itkina · Didrik Nielsen · William Harvey · Matias Valdenegro-Toro · Stefano Peluchetti · Riccardo Moriconi · Tianyu Cui · Vaclav Smidl · Taylan Cemgil · Jack Fitzsimons · He Zhao · · mariana vargas vieyra · Apratim Bhattacharyya · Rahul Sharma · Geoffroy Dubourg-Felonneau · Jonathan Warrell · Slava Voloshynovskiy · Mihaela Rosca · Jiaming Song · Andrew Ross · Homa Fashandi · Ruiqi Gao · Hooshmand Shokri Razaghi · Joshua Chang · Zhenzhong Xiao · Vanessa Boehm · Giorgio Giannone · Ranganath Krishnan · Joe Davison · Arsenii Ashukha · Jeremiah Liu · Sicong (Sheldon) Huang · Evgenii Nikishin · Sunho Park · Nilesh Ahuja · Mahesh Subedar · · Artyom Gadetsky · Jhosimar Arias Figueroa · Tim G. J. Rudner · Waseem Aslam · Adrián Csiszárik · John Moberg · Ali Hebbal · Kathrin Grosse · Pekka Marttinen · Bang An · Hlynur Jónsson · Samuel Kessler · Abhishek Kumar · Mikhail Figurnov · Omesh Tickoo · Steindor Saemundsson · Ari Heljakka · Dániel Varga · Niklas Heim · Simone Rossi · Max Laves · Waseem Gharbieh · Nicholas Roberts · Luis Armando Pérez Rey · Matthew Willetts · Prithvijit Chakrabarty · Sumedh Ghaisas · Carl Shneider · Wray Buntine · Kamil Adamczewski · Xavier Gitiaux · Suwen Lin · Hao Fu · Gunnar Rätsch · Aidan Gomez · Erik Bodin · Dinh Phung · Lennart Svensson · Juliano Tusi Amaral Laganá Pinto · Milad Alizadeh · Jianzhun Du · Kevin Murphy · Beatrix Benkő · Shashaank Vattikuti · Jonathan Gordon · Christopher Kanan · Sontje Ihler · Darin Graham · Michael Teng · Louis Kirsch · Tomas Pevny · Taras Holotyak