SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation

ArXi:2412.19098v4 Announce Type: replace Model merging combines independently trained models into a single multi-task model. However, most existing approaches focus primarily on avoiding task interference. We argue that its greater potential lies in enabling task synergy, where tasks actively improve one another. We identify cross-task performance, defined by compatibility between encoders and predictors across tasks, as a key indicator of merge quality. We nstrate that adapting only a single task-specific layer is sufficient to induce such synergy.