Reward Learning from Best-of-$N$ Preference Data: Targets, Tradeoffs, and Design Principles

ArXi:2605.30619v1 Announce Type: cross Best-of-$N$ sampling is widely used to construct pairwise preference data: $N$ candidates are drawn from a base distribution, and the best is paired with a rejected response. Despite its widespread use, what Bradley--Terry (BT) reward learning extracts from such data, and how to choose $N$ and the base distribution, remain unclear. We specialize a recent analysis of preference data via its induced conditional distribution to Best-of-$N