The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

ArXi:2605.20749v1 Announce Type: new Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a compact eigenvalue distribution. Building on this finding, we further analyze the resulting.