← Friday, October 9, 2026Models & trainingDeep dive

Megatron Core cuts deterministic pretraining overhead to about 2% at 2,432 GPUs

developer.nvidia.com by Ashwath Aithal 10 min read Editor score 8/10

Across three Nemotron workloads, optimization reduced the determinism tax to about 2% at 2,432 GPUs while maintaining bitwise determinism for 800 steps. Per-rank fingerprints and deterministic kernel changes support reproducible failure replay and training debugging at scale.

What

Bitwise determinism makes independent and checkpoint-resumed pretraining runs follow identical numerical trajectories under fixed conditions.

How

Across three Nemotron workloads, optimizations brought determinism overhead to about 2% at 2,432 GPUs while maintaining determinism over 800 steps.

Why it matters

Reproducible replay and validation help engineers debug failures and compare system changes in costly, trillion-parameter training.

Read on developer.nvidia.com →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.