AI RESEARCH
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
arXiv CS.LG
•
ArXi:2605.28440v1 Announce Type: cross DPO has become a widely adopted alternative to RLHF for aligning LLMs with human preferences, eliminating the need for a separate reward model or RL loop. Recent theoretical analysis uncovers an asymmetric gradient behavior in DPO: the loss suppresses dispreferred responses substantially faster than it promotes preferred ones, causing the model to learn to avoid bad answers rather than to generate good ones. We propose AdaDPO, a Self-Adaptive variant of the DPO algorithm that.