AI RESEARCH
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
arXiv CS.LG
•
ArXi:2605.31371v1 Announce Type: new Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magnitudes and fail to account for parameter heterogeneity, often leading to oscillation rather than convergence.