AI RESEARCH
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
arXiv CS.LG
•
ArXi:2606.02857v1 Announce Type: new Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory.