AI RESEARCH
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
arXiv CS.CL
•
ArXi:2511.20102v3 Announce Type: replace Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a.