AI RESEARCH
Token-weighted Direct Preference Optimization with Attention
arXiv CS.CL
•
ArXi:2605.21883v1 Announce Type: new Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra