Towards the Connection between Activation Sparsity and Flat Minima

ArXi:2605.25612v1 Announce Type: cross The observation that activation sparsity emerges in MLP blocks of standardly trained Transformers offers an opportunity to drastically reduce computation costs without sacrificing performance. To theoretically explain this phenomenon, existing works have shown that activation sparsity does not result from the data properties or data fitting but from the implicit bias of the