Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures

ArXi:2605.27458v1 Announce Type: cross Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well. We categorize attention structures of Transformer into two types based on the source of the input information: homogenous and heterogenous attention structures. Heterogenous attention structures, with co-attention as a typical example, process information from different sources. Heterogenous attention structure is the foundation for Transformer models to achieve complex functions and integrate modal information.