Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables

ArXi:2605.30229v1 Announce Type: new We use a mean-field-based transformer model to theoretically investigate how auxiliary variables, such as positional encoding, prevent mode collapse of self-attention mechanisms. The use of mean-field transformers to analyze the properties of self-attention mechanisms has garnered significant attention in recent years due to their ability to comprehensively analyze token interactions.