← Thursday, October 1, 2026InferenceGuide
FlexKV caching cuts HSTU recommender latency by up to 5.93×
An HSTU inference workflow combines AOTI-compiled models with FlexKV caching; at batch size 8, caching cut latency by up to 5.93× on an RTX PRO 6000 Blackwell. The measured gain and deployment workflow are useful for engineers serving sequence-based recommenders.
- What
- An end-to-end workflow serves HSTU generative recommender models through NVIDIA Dynamo-Triton, with PyTorch AOTI compilation and FlexKV-backed KV caching.
- How
- FlexKV caching avoids recomputing reusable attention state; at batch size 8, an eight-layer model achieved up to 5.93× lower latency with a 100% GPU cache hit rate.
- Why it matters
- Compiling and validating models for deployment while caching repeated attention state can reduce serving latency for long user histories.
Read on developer.nvidia.com →
Discussion
Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.
Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.
