Timezone: »
Influence estimation attempts to estimate the effect of removing a training example on downstream predictions. Prior work has shown that a first-order approximation to estimate influence does not agree with the ground-truth of re-training or fine-tuning without a training example. Recently, Feldman and Zhang [2020] created an influence estimator that provides meaningful influence estimates but requires training thousands of models on large subsets of a dataset. In this work, we explore how the method in Feldman and Zhang [2020] scales with the number of trained models. We also show empirical and analytical results in the standard influence estimation setting that provide intuitions about the role of nondeterminism in neural network training and how the accuracy of test predictions affects the number of models needed to detect an influential training example. We ultimately find that a large amount of models are needed for influence estimation, though the exact number is hard to quantify due to training nondeterminism and depends on test example difficulty, which varies between tasks.
Author Information
Sasha (Alexandre) Doubov (University of Toronto)
Tianshi Cao (University of Toronto)
David Acuna (University of Toronto, Nvidia, Vector Institute)
Sanja Fidler (TTI at Chicago)
More from the Same Authors
-
2021 : Studying BatchNorm Learning Rate Decay on Meta-Learning Inner-Loop Adaptation »
Alexander Wang · Sasha (Alexandre) Doubov · Gary Leung -
2022 Poster: Optimizing Data Collection for Machine Learning »
Rafid Mahmood · James Lucas · Jose M. Alvarez · Sanja Fidler · Marc Law -
2023 : Trajeglish: Learning the Language of Driving Scenarios »
Jonah Philion · Xue Bin Peng · Sanja Fidler -
2022 Spotlight: GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images »
Jun Gao · Tianchang Shen · Zian Wang · Wenzheng Chen · Kangxue Yin · Daiqing Li · Or Litany · Zan Gojcic · Sanja Fidler -
2022 Poster: EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations »
Ahmad Darkhalil · Dandan Shan · Bin Zhu · Jian Ma · Amlan Kar · Richard Higgins · Sanja Fidler · David Fouhey · Dima Damen -
2022 Poster: LION: Latent Point Diffusion Models for 3D Shape Generation »
xiaohui zeng · Arash Vahdat · Francis Williams · Zan Gojcic · Or Litany · Sanja Fidler · Karsten Kreis -
2022 Poster: GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images »
Jun Gao · Tianchang Shen · Zian Wang · Wenzheng Chen · Kangxue Yin · Daiqing Li · Or Litany · Zan Gojcic · Sanja Fidler -
2021 Poster: Scalable Neural Data Server: A Data Recommender for Transfer Learning »
Tianshi Cao · Sasha (Alexandre) Doubov · David Acuna · Sanja Fidler -
2021 Poster: Don’t Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence »
Tianshi Cao · Alex Bie · Arash Vahdat · Sanja Fidler · Karsten Kreis -
2021 Poster: Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation »
David Acuna · Jonah Philion · Sanja Fidler -
2020 Poster: Variational Amodal Object Completion »
Huan Ling · David Acuna · Karsten Kreis · Seung Wook Kim · Sanja Fidler