Sometimes I am a Tree: Data Drives Fragile Hierarchical Generalization

When training deep neural networks, models can adopt various heuristics, leading to different out-of-distribution (OOD) behaviors. Previous works have attributed these preferences to choices of model architecture or training objective, but the role of training data is less explored. Using the question formation task as a case study, we investigate how data composition drives language models to favor a hierarchical rule heuristic and its inconsistent outcomes across random seeds. We show that LMs consistently apply a simple rule OOD when trained on a mix of declarative and question sentences and we identify syntactic features that guide models toward either hierarchical or linear rules. Additionally, we show that models stabilize their OOD behavior in training only when committing to a simple rule. Our findings highlight how training data shapes generalization patterns and how competition between data subsets can lead to inconsistent training results.