Release
llm-d 1.0 coordinates distributed inference for bursty agent workloads
The Kubernetes-native stack pools model servers to address duplicated prefix computation, repeated accelerator state and bursts from multi-step agents. Its design targets serving patterns that generic round-robin load balancing and single-machine deployments struggle with.
- Whatllm-d 1.0 is an open-source, Kubernetes-native stack that coordinates distributed LLM inference across model servers.
- HowCache-aware routing keeps agent steps on replicas holding their context, tiered KV cache reuses state, and flow control queues bursts.
- Why it mattersSession-aware routing and priority controls help agent workloads share accelerators with latency-sensitive requests.
