← Friday, October 2, 2026Models & trainingRelease

Olmo-core 3 opens a redesigned training system for large MoE models

huggingface.co 6 min read Editor score 8/10

Olmo-core 3 is an open Mixture-of-Experts training framework designed to scale toward trillion-parameter models while preserving computational efficiency; the release includes a tech report and code. Its training infrastructure could inform labs building large models.

What: Olmo-core 3 is a redesigned open training system intended to scale mixture-of-experts models into the trillion-parameter range.
How: It replaces an earlier FSDP-based implementation with DDP, keeping experts on GPUs and routing relevant data to them.
Why it matters: In one benchmark, capacity grew from 4.6B to 47B parameters while throughput fell by less than 5%, addressing MoE scaling costs.

DDP routes data to experts that remain resident on GPUs. Sketched from the article's own text.

Read on huggingface.co →

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.