Causation Does Not Imply Correlation: A Study of Circuit Mechanisms and Model Behaviors
Jenny Kaufmann · Victoria R. Li · Martin Wattenberg · David Alvarez-Melis · Naomi Saphra
2024 Poster Session
in
Workshop: Scientific Methods for Understanding Neural Networks
in
Workshop: Scientific Methods for Understanding Neural Networks
Abstract
Using a toy balanced parenthesis classification task with an ambiguous rule, we investigate the correspondence between attention patterns and out-of-distribution generalization behavior of small transformer models. We find that observational tools can predict OOD behavior, challenging the common notion among interpretability researchers that causal intervention is the only basis for explaining model behavior.
Chat is not available.
Successful Page Load