Singular Vectors of Attention Heads Align with Features

ArXi:2602.13524v2 Announce Type: replace-cross Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we nstrate that singular vectors robustly align with features in a model where features can be directly observed.