Wednesday, October 21, 2026

Inference

7 inference stories from the Oct 21 edition.

PulseOne beat per story, in page order. Taller beats scored higher.

DeepInfer releases a production inference framework for DeepSeek modelsScore 8/10 Google open-sources TorchTPU, a PyTorch backend for TPUsScore 8/10 SolarRenderer generates structured video in four steps, 12.3 times faster than 50-step samplingScore 8/10 VLA-Rolling reuses discarded action tails to reduce denoising costScore 8/10 TorchTPU runs Hugging Face models natively on Google TPUsScore 7/10 PyTorch 2.15 adds FP8 and faster decoding paths to MPSScore 7/10 TorchNeuron makes AWS Trainium a native PyTorch deviceScore 7/10

Release

DeepInfer releases a production inference framework for DeepSeek models

DeepInfer lists optimized kernels, parallelism strategies, asynchronous scheduling, deterministic sampling and confidence-scheduled draft verification. Its production-serving focus and concrete inference features make it relevant to engineers building high-throughput model services.

What• DeepInfer is an open-source offline inference framework that powers DeepSeek's large-scale production services.
How• It includes optimized kernels, parallelism, confidence-scheduled verification, asynchronous scheduling and structured outputs.
Why it matters• A complete service still needs components such as a KV cache, prefill/decode disaggregation and load balancing, which this release omits.

Original · github.com HN 2.2K pts · 849 comments

Notes & discussion

Tool

Google open-sources TorchTPU, a PyTorch backend for TPUs

TorchTPU provides ATen kernels, a compilation cache, distributed training support and examples for model inference, with integration for torch.compile(). It gives engineers a new route to running PyTorch workloads on Google TPUs.

Original · github.com 4 min read HN 312 pts · 68 comments

Notes & discussion

Paper

SolarRenderer generates structured video in four steps, 12.3 times faster than 50-step sampling

Two lightweight LoRAs adapt a frozen video backbone to structural inputs, enabling four-step rendering; a block-causal variant renders 768p video in real time. The design shows how to add controllable generation while reducing sampling steps.

Original · huggingface.co by Guian Fang et al.33 upvotes Code · 153 stars

Notes & discussion

Paper

VLA-Rolling reuses discarded action tails to reduce denoising cost

The framework carries unexecuted action tails into the next planning round and allocates refinement steps using per-action confidence; it lowers inference cost across all 16 tested model–benchmark pairs. Adaptive reuse could reduce the latency and compute cost of iterative action generation.

Original · huggingface.co by Sicheng He et al.25 upvotes Code · 6 stars

Notes & discussion

Tool

TorchTPU runs Hugging Face models natively on Google TPUs

TorchTPU makes TPU a native PyTorch device for Transformers, Diffusers and Kernels, avoiding a port to torch_xla or JAX. This could simplify moving existing model workloads onto TPU hardware.

Original · huggingface.co 5 min read

Notes & discussion

Release

PyTorch 2.15 adds FP8 and faster decoding paths to MPS

The MPS update adds FP8, faster language-model decoding paths through linear layers and softmax, and dedicated Metal kernels for audio, video and training. These backend changes matter to engineers running models on Apple hardware.

Original · huggingface.co 10 min read

Notes & discussion

Tool

TorchNeuron makes AWS Trainium a native PyTorch device

TorchNeuron is an open-source backend that lets developers program Trainium as a standard PyTorch device, using PyTorch APIs for training and serving libraries. This could simplify adapting AI workloads to Trainium without replacing their PyTorch-based stack.

Original · pytorch.org 16 min read

Notes & discussion