AI RESEARCH
Understanding and Improving Noisy Embedding Techniques in Instruction Finetuning
arXiv CS.AI
•
ArXi:2605.23171v1 Announce Type: cross Recent advancements in instructional fine-tuning have injected noise into embeddings, with NEFTune (Jain, 2024) setting benchmarks using uniform noise. Despite NEFTune's empirical findings that uniform noise outperforms Gaussian noise, the reasons for this remain unclear. This paper aims to clarify this by offering a thorough analysis, both theoretical and empirical, indicating comparable performance among these noise types. Additionally, we