AI RESEARCH
SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning
arXiv CS.LG
•
ArXi:2605.21147v1 Announce Type: new As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity. Theory suggests that LoRA fine-tuning with rank r converges toward the top r singular values of the pre-trained weight matrix.