AI RESEARCH
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
arXiv CS.LG
•
ArXi:2605.31484v1 Announce Type: new Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we.