AI RESEARCH
Task Structure Reverses Layerwise State Encoding in Sequence Models
arXiv CS.LG
•
ArXi:2606.00926v1 Announce Type: new Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it. We find that the same architecture reverses this profile when the task changes. Across Transformers, Mamba, Mamba-2, LSTMs, and GRUs, Parity is concentrated late in Mamba and the recurrent baselines and built gradually by Transformer; on bounded-depth Dyck-k the pattern flips.