← Tuesday, October 13, 2026InferenceBenchmark

Janas streams model experts from SSD and reports 24 tok/s on a 48 GB model

github.com 14 min read Editor score 8/10

On a 32 GB laptop, Janas reports 24 tok/s generating with a 48 GB Qwen3-Next-80B model, versus 2.8 tok/s for llama.cpp, and also reports faster prompt processing. The results could inform trade-offs for running models larger than available RAM.

What: Janas-LLM runs large mixture-of-experts models by streaming needed experts from an NVMe SSD while keeping recurring ones in memory.
How: It reports 24 tok/s for a 48.4 GB Qwen3-Next-80B-A3B model on a 32 GB RAM laptop, compared with 2.8 tok/s for llama.cpp.
Why it matters: Streaming weights from storage lets models larger than RAM run on ordinary computers, making storage and memory traffic central to inference design.

Janas streams needed experts from NVMe storage and keeps recurring experts in memory. Sketched from the article's own text.

Read on github.com → Hacker News · 37 comments

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.