← Sunday, October 11, 2026Models & trainingPaper

Byte language models beat subword models at matched parameter counts

arxiv.org 71 min read Editor score 8/10

The paper reports lower optimal loss for raw-byte Transformers using token-superposition training and hash embeddings, and says they develop local text abstractions. The result challenges tokenizer assumptions with potential implications for model design and compute allocation.

What

Byte language models process text directly as bytes, avoiding fixed external tokenizers but handling longer sequences.

Method

With token-superposition training and hash embeddings, flat byte Transformers outperform subword models as model size scales and learn local text abstractions.

Why it matters

Learned boundary structure concentrates uncertainty, enabling speculative decoding to accept 3.4× more tokens than in subword Transformers.

Read on arxiv.org → Lobsters

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.