← Friday, October 9, 2026Models & trainingRelease

ttok 1.0 switches its default tokenizer to the GPT-5 family

simonwillison.net Editor score 6/10

An experiment found seven GPT-5 and GPT-6 models matching on all 31 tokenizer fixtures, with each reporting 44,794 tokens. Reliable token counts matter when estimating and managing LLM input sizes.

What

- ttok 0.4 is a CLI tool for counting tokens using OpenAI’s open-source tiktoken library.

How

- The release fixes a Click warning, updates CI, adds `--list-models`, and supports counting text piped into `uvx ttok`.

Why it matters

- Token counting and truncation help engineers work with text in token-based LLM workflows.

Read on simonwillison.net →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.