MDM-OC: Orthogonal Delta Merging for Scalable and Reversible Model Composition

In real-world machine learning deployments, models must be continually updated, composed, and, when required, selectively undone. Existing approaches to model merging and continual learning often suffer from task interference, catastrophic forgetting, or a lack of reversibility. We propose Modular Delta Merging with Orthogonal Constraints (MDM-OC), a framework for scalable, interference-free, and reversible composition of fine-tuned models. Each task-specific model is encoded as a parameter delta from a shared base model and orthogonalized against previously integrated deltas, either jointly, via a single SVD/QR factorization of the stacked delta matrix that treats all tasks symmetrically, or sequentially, via numerically stable Modified Gram-Schmidt (MGS) updates for continual integration. Orthogonalized deltas are combined into a unified model whose per-task contribution can be exactly algebraically subtracted, giving closed-form, data-free unmerging in the full-rank case and a provably bounded reconstruction error under low-rank PCA/SVD compression. An optional data-aware refinement stage tunes merge coefficients against a small validation set when one is available, without altering the zero-data reversibility guarantees of the base method. Experiments on vision (CIFAR-100, ImageNet-100) and language (AG News, DBpedia, Yahoo Answers) continual-learning benchmarks show that MDM-OC improves average accuracy over strong merging baselines (Task Arithmetic, TIES-Merging, AdapterFusion) by up to 6.3 percentage points, while reducing unmerge accuracy drop to 1.8% (vision) and 2.3% (language) and cutting unmerge recovery time by more than 3x. These results support MDM-OC as a practical building block for modular, compliant, and continually updatable AI systems, e.g., for GDPR-style selective data/model removal.