← Thursday, October 1, 2026Models & trainingTool

TurboGPT trains a tiny byte-level transformer in CUDA C++

github.com Editor score 7/10

The project trains on 1.5 billion tokens and reports 2.5295 bits per byte, with checkpoints and batch-level logs. Its compact CUDA training implementation and measured result make it worth examining for engineers interested in low-level training systems.

Point

TurboGPT is a tiny byte-level GPT training project written in CUDA C++; its title describes a 22KiB transformer trained in 13 seconds.

Point

The excerpt reports 2.5295 BPB after training on 1.5G tokens.

Read on github.com → Hacker News · 10 comments

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.