AI RESEARCH
AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
arXiv CS.LG
•
ArXi:2601.17261v4 Announce Type: replace Zeroth-Order (ZO) optimization has emerged as a promising solution for fine-tuning LLMs under strict memory constraints, as it avoids the prohibitive memory cost of storing activations for backpropagation. However, existing ZO methods typically employ isotropic perturbations, neglecting the rich structural information available during the forward pass. In this paper, we identify a crucial link between gradient formation and activation structure: the gradient of a linear layer is confined to the subspace spanned by its input activations.