AI RESEARCH
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
arXiv CS.AI
•
ArXi:2605.27140v1 Announce Type: new Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.