AI RESEARCH
Momentum Streams for Optimizer-Inspired Transformers
arXiv CS.AI
•
ArXi:2605.24425v1 Announce Type: cross The residual update of a pre-norm Transformer layer admits an interpretation as one step of a first-order optimizer acting on a surrogate token energy, wherein the attention and MLP sublayers function as gradient oracles. Based on this observation, we build a family of optimizer-inspired Transformers (triple-momentum, Adam/AdamW, Muon, SOAP) and compare them under matched compute. In our main pre