Exact Predictions Do Not Determine Continuous Margins in Differentiable Decision Trees

Exact differentiable formulations of hard decision trees enable gradient-based training while preserving hard-tree predictions. The central question is whether the resulting continuous class margin also reflects robustness to input perturbations. For binary decision trees with linear splits, the analysis characterizes cases in which the class margin equals the minimum perturbation required to change the hard prediction. In dimension two or higher, the class margin can instead be arbitrarily small relative to this perturbation for every full binary topology with at least three leaves and both classes in each internal subtree. Moreover, two trees with the same topology and leaf labels can fit the same sample while cross-entropy is lower for one tree at every training observation even though its predictions can be arbitrarily easier to change. Experiments on five binary datasets show the same disagreement among trained trees, including pairs with identical test predictions and training checkpoints with unchanged hard predictions.