Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

ArXi:2605.30022v1 Announce Type: cross Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and d remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval \cite{chen-etal-2025-hope}. Hence, a better understanding of the internal positional mechanism could help design better PE.