AI RESEARCH
When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
arXiv CS.LG
•
ArXi:2605.25629v1 Announce Type: cross Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train--test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets.