Probing Classifiers are Unreliable for Concept Removal and Detection

Abhinav Kumar · Chenhao Tan · Amit Sharma

Keywords: [ Fairness ] [ spurious correlation ] [ Adversarial Removal ] [ Null-Space Removal ] [ Probing ]

[ Abstract ]
[ Poster [ OpenReview
Spotlight presentation: Lightning Talks 1B-1
Tue 6 Dec 9 a.m. PST — 9:15 a.m. PST


Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model's representation. Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the concepts entirely, and in the worst case may end up destroying all task-relevant features. The reason is the methods' reliance on a probing classifier as a proxy for the concept. Even under the most favorable conditions for learning a probing classifier when a concept's relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features and thus post-hoc or adversarial methods will fail to remove the concept correctly. These theoretical implications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets. For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose a spuriousness metric to gauge the quality of the final classifier.

Chat is not available.