Where does a language model maintain word order, grammar, and meaning — and is this determined entirely by the chosen network architecture?
Layer-by-layer testing demonstrates that at a comparable loss level, syntactic structures and semantics are encoded in a remarkably similar manner. The primary divergence between architectural solutions reduces to the mechanism used for token position fixing. As part of this research, the author trained 12 compact models differing solely in architecture and evaluated every layer of each checkpoint using linear probes.
Anatomy of Layer-by-Layer Probes
Three network construction variants were deployed for comparative testing: the recurrent architecture RWKV-7, a classical Transformer with Rotary Position Embedding (RoPE), and a Transformer completely devoid of positional encoding (NoPE). Model parameters were strictly constrained to exactly two sizes (56.6M and 115M parameters), utilizing two languages (Russian and English), with a compute budget allocated per model ranging from 1 to 2 billion tokens.
An evaluation probe represents a simple model operating on frozen token hidden states, trained independently for each of the 13 layers. Linear ridge regression and classifiers served as the instruments to measure linearly extractable features, with computations executed on a server provided by the RWKV author.
«A probe is a simple model (linear ridge regression or classifier) on frozen token hidden states, trained separately for each of the 13 layers»
This approach isolates internal representations of hidden states and tracks data evolution from embeddings through to the final network blocks. The training corpus comprised around 2.1 billion tokens per language: roughly 76% of the volume originated from Wikipedia, with the remainder composed of news articles, literature, scientific texts, and conversational transcripts.
How precisely do these architectures handle the tracking of input data sequences?
Mechanics of the Positional Signal
Different approaches display fundamental divergence when extracting ordinal word numbers. A linear probe extracts token position from NoPE models almost perfectly, from RoPE models noticeably worse, and from RWKV models with even lower fidelity. Within the NoPE architecture, the Transformer utilizes no designated markers, leaving the causal mask as the sole orientation reference.
«A linear probe extracts token position from NoPE models almost perfectly, from RoPE models noticeably worse, and from RWKV models with even lower fidelity»
At the final checkpoint, the R² score of the optimal layer for position reaches 0.99 for NoPE across all four configurations.
At the 56.6M–115M parameter scale, architectural differences remain minimal, and extrapolating these conclusions to industrial-grade models remains unproven due to modest test corpus sizes and compact network dimensions. Alternative architectures like RWKV-7 draw interest thanks to linear computational complexity compared to the quadratic cost growth of classical Transformers as context lengths expand. At the same time, the model scale (56.6M and 115M parameters) and compute volume in this study were modest. Transferring such results to large models requires separate verification, since feature behavior on small datasets does not guarantee that the effect will hold at larger scale.
For comparison
An R² score of 0.99 for token position in the NoPE transformer proves that a network can track sequence order purely through a causal mask without dedicated positional markers. This stark divergence in how architectures encode sequence order does not stop both approaches from organizing syntax and semantics in nearly identical internal patterns. A val-loss gap of 0.10–0.13 leaves open how both architectures will perform once scaled beyond a modest 115M parameters.
Our stories: RWKV vs Transformer: Efficient AI Architecture and Economics (June 18); KAIST and Naver Decode Hidden AI Reasoning Steps in LLMs (September 12); Recurrent Looped Transformers: AI Compute Economics (September 14); StateBench Limits Transformer Long-Context Tracking (November 4).