Which State Should Prediction Read?

Sole-author paper, under review at ICLR 2027.

Built with: LLM Architecture & Interpretability

GitHub: RyanKim17920/mechanistic-sps

Models: ryankim17920/mechanistic-sps

• Trained State–Prediction Separation (SPS) LMs, which split a Transformer into separate state and prediction streams; ablations showed the two divide the work: state tracks context, prediction retrieves from it.
• Reached a FLOP-matched Transformer’s loss on 2.1× fewer tokens and ended 0.064 nats/token lower after 20B tokens by designing Sequential, a final-state-read architecture, from these ablations (92M non-embedding params).

August – September 2026