← Monday, September 28, 2026InferenceBenchmark

TensorRT multi-GPU serving cuts Cosmos generation latency from 156.6 to 34.2 seconds

developer.nvidia.com by Tanya Lenz 8 min read Editor score 8/10

A single TensorRT model instance distributed across eight GPUs reduced Cosmos 3 Nano generation latency from 156.6 to 34.2 seconds. The integration shows how multi-device execution can simplify serving while retaining a single gRPC endpoint.

What

TensorRT multi-device inference lets one TensorRT network run across multiple GPUs, integrated with Dynamo-Triton as one gRPC-served model.

How

For Cosmos 3 Nano video generation, latency fell from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs.

Why it matters

Engineers can scale inference across GPUs behind a single model endpoint, without coordinating GPU ranks in the client.

Read on developer.nvidia.com →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.