← Tuesday, October 6, 2026Models & trainingPaper

Dust uses token-level activation perturbations to train transformers without backpropagation

qlabs.sh 27 min read Editor score 8/10

Dust perturbs activations independently at each token; the authors report competitive results against backprop and estimate 1,000–10,000× greater efficiency than EGGROLL from one million tokens onward. The method and scaling results make it worth watching for teams exploring alternatives to standard training.

What

Dust is a zeroth-order method for pretraining transformer language models without backpropagation.

Method

It independently perturbs activations at every token, treating each token as a virtual population member evaluated in parallel in one forward pass.

Why it matters

For AI training systems, Dust offers a backprop alternative that is orders of magnitude more efficient than weight-space ES in the reported comparison.

Read on qlabs.sh → Hacker News · 40 comments

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.