Unreasonable Confidence from Simulator Misspecification: Directional Robustness on the SPEED+ Sim-to-Real Benchmark

Vision-based navigation systems for orbit must be trained almost entirely on synthetic imagery, because real labelled images of a given spacecraft in flight rarely exist. Benchmarks like SPEED+ are built for exactly that situation: the test sets are real images of a satellite mockup under two lighting conditions, but the training set comes from a simulator. That simulator approximates how light reflects off mylar and aluminium, and like every scientific simulator it is misspecified. The error is radiometric: it does not reproduce the specular highlights, hard shadows, and sensor saturation that are ordinary in real images. The resulting synthetic-to-real gap is the central risk for such systems. We probe it with a deliberately simple monocular pose estimator — a $2 \times 2$ study of backbone adaptation and photometric augmentation, trained on synthetic images and tested on both real hardware-in-the-loop domains. We report three findings. (1) The model does not know it is failing. Regions conformally calibrated to 90% on synthetic data cover 9–15% of rotation errors $E_R = 2\arccos|\langle \hat{q}, q \rangle|$ on sunlamp and 21–38% on lightbox, so a wrong pose carries the confidence of a correct one. (2) Transfer is asymmetric. Scaling data and features improved synthetic rotation error from $1.67$ to $1.05$ rad but moved the harshly lit sunlamp domain by only $0.03$ rad, whereas adapting the backbone's last stage carried 70–80% of its marginal synthetic gain across to both real domains. (3) Augmentation only helps in the direction it models. Our photometric stack improved the diffusely lit lightbox domain from $1.17$ to $0.99$ rad on every seed, yet moved sunlamp no further than seed-to-seed variation. The question to ask of a robustness intervention is therefore not how much it helps, but which way it pushes the training distribution.