HR-Merge: Model Merging with Headroom-Weighted Residual Refinement

We study fine-tuned models, task vectors, and post-merge residuals as neural network artifacts, asking which task-specific capabilities survive model merging. When experts fine-tuned from a shared backbone are combined into one model, their updates can interfere, and a small subset of tasks accounts for most of the accuracy loss. We measure this uneven preservation by a task's validation headroom, the fraction of that task's expert accuracy the merged model fails to retain on held-out validation data. Guided by this finding, we propose HR-Merge, a training-free two-stage refinement. The first stage is the base merge, which leaves each task a residual, the part of its update the merge failed to capture. The second stage strengthens the tasks the merge under-served: it weights each residual by its task's validation headroom, then re-merges the weighted residuals with a merging rule matched to the base rather than summing them. The refinement's strength is tuned to maximize validation accuracy and set to zero when no strength helps. Because the second stage reuses the base operator and adds no training, the method is merge-method-agnostic. Across three CLIP ViT backbones and 8/14/20-task suites under full fine-tuning, and on LoRA merging benchmarks in both language (LLaMA-3 8B) and vision, with every baseline re-run under one protocol, HR-Merge consistently improves or preserves each setting's average within evaluation noise. Across all evaluated settings, the best-performing model is an HR-Merge variant.