← Sunday, October 11, 2026Models & trainingPaper
Recurrent ViT matches full-depth accuracy with 70% fewer stored parameters
reViT programs a shared bank of FFN experts by recurrent depth; its from-scratch model matches DeiT III accuracy with about 70% fewer stored parameters. The distilled model also retains nearly all teacher linear-probe accuracy across several tasks.
- What
- reViT applies one Transformer block recurrently, matching a full-depth vision encoder’s accuracy at comparable inference FLOPs.
- Method
- At each recurrent depth, a normalized-depth coordinate programs a convex mixture of shared experts for the FFN transformation.
- Why it matters
- It achieves DeiT III accuracy with about 70% fewer stored parameters, and elastic-depth training supports multiple tested depths from one checkpoint.
Discussion
Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.
Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.
