Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs

ArXi:2506.10054v4 Announce Type: replace-cross Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which leads to inefficient data utilization and suboptimal performance.