AI RESEARCH
Approaching I/O-optimality for Approximate Attention
arXiv CS.LG
•
ArXi:2605.23751v1 Announce Type: new We revisit the I/O complexity of attention in large language models. Given query-key-value matrices $Q,K,V\in\mathbb{R}^{n\times d}$, and a machine with fast memory size $M$, the goal is to compute the "attention matrix" $A=\text{softmax}(Q K ^{\top}/\sqrt{d}) V$ with the minimal number of data transfers between fast and slow memory.