AI RESEARCH
The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
arXiv CS.LG
•
ArXi:2605.26246v1 Announce Type: new Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teacher (hard labels) or the teacher's full next-token distribution (soft labels). Despite soft labels appear strictly richer, we find that mixing hard and soft labels consistently yields better results. Crucially, we show that this gain cannot be explained by closer teacher matching during.