Models & training Paper
Fixed-point training improves looped models and speeds prefill and RL updates
The work uses learned depth priors and orthogonal input injection to improve perplexity across 100M–1.6B models, and reports 1.79× faster student prefill and 2× faster RL updates. Its fixed-point analysis connects model architecture choices to training and inference costs.
- WhatFixed-point training tunes looped language models to reduce recurrence costs across training, decoding, prefill, and reinforcement learning.
- MethodA learned depth prior and orthogonal input injection lower perplexity across tested scales; fixed points enable truncated backpropagation and terminal KV sharing.
- Why it mattersFor model builders, the approach enables KV sharing with almost no accuracy loss, prefill up to 1.79× faster, and RL gradient updates 2× faster.
