AI RESEARCH
Rethinking the Role of Temperature in Large Language Model Distillation
arXiv CS.AI
•
ArXi:2606.00306v1 Announce Type: cross Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature $\tau$, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and