Daily inference stories worth an engineer's time.https://diffbrief.tatsatpandey.com/en-ushttps://github.com/deepseek-ai/DeepInferhttps://github.com/deepseek-ai/DeepInferDeepInfer lists optimized kernels, parallelism strategies, asynchronous scheduling, deterministic sampling and confidence-scheduled draft verification. Its production-serving focus and concrete inference features make it relevant to engineers building high-throughput model services.Wed, 21 Oct 2026 06:38:00 GMTInferencereleasehttps://github.com/google-pytorch/torch_tpuhttps://github.com/google-pytorch/torch_tpuTorchTPU provides ATen kernels, a compilation cache, distributed training support and examples for model inference, with integration for torch.compile(). It gives engineers a new route to running PyTorch workloads on Google TPUs.Wed, 21 Oct 2026 06:38:00 GMTInferencetoolhttps://huggingface.co/papers/2610.28630https://huggingface.co/papers/2610.28630Two lightweight LoRAs adapt a frozen video backbone to structural inputs, enabling four-step rendering; a block-causal variant renders 768p video in real time. The design shows how to add controllable generation while reducing sampling steps.Wed, 21 Oct 2026 06:38:00 GMTInferencepaperhttps://huggingface.co/papers/2610.25043https://huggingface.co/papers/2610.25043The framework carries unexecuted action tails into the next planning round and allocates refinement steps using per-action confidence; it lowers inference cost across all 16 tested model–benchmark pairs. Adaptive reuse could reduce the latency and compute cost of iterative action generation.Wed, 21 Oct 2026 06:38:00 GMTInferencepaperhttps://huggingface.co/blog/hf-torchtpuhttps://huggingface.co/blog/hf-torchtpuTorchTPU makes TPU a native PyTorch device for Transformers, Diffusers and Kernels, avoiding a port to torch_xla or JAX. This could simplify moving existing model workloads onto TPU hardware.Wed, 21 Oct 2026 06:38:00 GMTInferencetoolhttps://huggingface.co/blog/mps-pytorch-2-15https://huggingface.co/blog/mps-pytorch-2-15The MPS update adds FP8, faster language-model decoding paths through linear layers and softmax, and dedicated Metal kernels for audio, video and training. These backend changes matter to engineers running models on Apple hardware.Wed, 21 Oct 2026 06:38:00 GMTInferencereleasehttps://pytorch.org/blog/introducing-torchneuron-native-pytorch-backend-for-aws-trainium/https://pytorch.org/blog/introducing-torchneuron-native-pytorch-backend-for-aws-trainium/TorchNeuron is an open-source backend that lets developers program Trainium as a standard PyTorch device, using PyTorch APIs for training and serving libraries. This could simplify adapting AI workloads to Trainium without replacing their PyTorch-based stack.Wed, 21 Oct 2026 06:38:00 GMTInferencetoolhttps://huggingface.co/blog/hf-torchneuronhttps://huggingface.co/blog/hf-torchneuronTorchNeuron adds eager mode, torch.compile, and distributed APIs including FSDP and tensor parallelism, with support in Transformers, Diffusers, and Kernels. This could make Trainium easier to adopt across model workflows built on PyTorch.Tue, 20 Oct 2026 06:31:51 GMTInferencereleasehttps://developer.nvidia.com/blog/accelerating-mamba-2-with-high-performance-forward-pass-kernels-in-nvidia-cuda-tile/https://developer.nvidia.com/blog/accelerating-mamba-2-with-high-performance-forward-pass-kernels-in-nvidia-cuda-tile/The implementation uses cuTile for three Mamba-2 kernels, including Tensor Core matrix multiplication to advance hidden state across sequence chunks. The kernel design offers practical material for engineers optimizing GPU execution of state-space models.Tue, 20 Oct 2026 06:31:51 GMTInferencedeep-divehttps://jrdsgl.com/if-you-dont-run-your-own-llm-proxy-how-the-fuck-do-you-even-know-whats-going-on/https://jrdsgl.com/if-you-dont-run-your-own-llm-proxy-how-the-fuck-do-you-even-know-whats-going-on/It argues that provider interfaces show distracting activity rather than the time-series token statistics engineers need. That visibility matters when diagnosing usage and making decisions about LLM-backed systems.Tue, 20 Oct 2026 06:31:51 GMTInferenceessayhttps://www.inceptionlabs.ai/blog/introducing-mercury-3https://www.inceptionlabs.ai/blog/introducing-mercury-3Inception says its diffusion-based LLM reaches 1,068 tokens per second on NVIDIA GPUs, supports a one-million-token context, and costs $0.20 per million input tokens. The release is a concrete claim to evaluate against serving latency, quality, and cost requirements.Tue, 20 Oct 2026 06:31:51 GMTInferencereleasehttps://github.com/vllm-project/vllm/releases/tag/v0.32.0https://github.com/vllm-project/vllm/releases/tag/v0.32.0The release adds NVFP4 and 4-bit KV cache options across GPU architectures; one GLM-5.x configuration halves KV memory per rank with 2.3–5.5% more throughput but 7–11% higher TTFT. These trade-offs help serving teams tune memory, throughput and latency.Mon, 19 Oct 2026 06:43:22 GMTInferencereleasehttps://huggingface.co/papers/2610.23787https://huggingface.co/papers/2610.23787DLite uses target-model hidden states for local speculative prediction instead of maintaining historical context; existing draft models lose up to 80% of acceptance length on agentic workloads. The approach targets both acceptance degradation and KV-cache overhead in multi-turn serving.Mon, 19 Oct 2026 06:43:22 GMTInferencepaperhttps://huggingface.co/papers/2610.20556https://huggingface.co/papers/2610.20556The method pads larger bidirectional windows with noise latents, caches predictions while conditions stay unchanged and delivers results incrementally with replaceable future frames. It reframes the latency–quality trade-off for interactive video generation, but the excerpt reports no measured gains.Mon, 19 Oct 2026 06:43:22 GMTInferencepaperhttps://aws.amazon.com/blogs/machine-learning/metas-muse-spark-open-now-available-in-amazon-sagemaker-jumpstart/https://aws.amazon.com/blogs/machine-learning/metas-muse-spark-open-now-available-in-amazon-sagemaker-jumpstart/The coding-optimized model is available as BF16 and NVFP4 variants, with a one-million-token context, and can be deployed to a SageMaker endpoint. The variants and managed deployment path are useful details when evaluating model-serving options.Mon, 19 Oct 2026 06:43:22 GMTInferencereleasehttps://eunomia.dev/blog/2026/10/17/bpf-gpu-server/https://eunomia.dev/blog/2026/10/17/bpf-gpu-server/The bpf-gpu-server moves TCP handling into eBPF and io_uring so no user thread reads, schedules or sends requests, yet CPU use and latency matched an efficient epoll server. The comparison helps identify when kernel-bypass complexity is unlikely to improve a GPU serving path.Sun, 18 Oct 2026 06:29:34 GMTInferencebenchmarkhttps://llmspeed.dev/https://llmspeed.dev/Llmspeed lets users browse local LLM speed estimates by hardware and model, drawing on community benchmark measurements. It can help engineers compare candidate setups before buying hardware, though the excerpt gives no specific performance results.Sun, 18 Oct 2026 06:29:34 GMTInferencetoolhttps://developer.nvidia.com/blog/scale-agentic-coding-workflows-with-metas-muse-spark-open-on-nvidia-gb300-nvl72/https://developer.nvidia.com/blog/scale-agentic-coding-workflows-with-metas-muse-spark-open-on-nvidia-gb300-nvl72/Muse Spark Open activates 42B of 816B parameters per token, uses a 1M-token context and NVFP4 checkpoints, and is reported to reach 50K tokens/s per GPU on GB300 NVL72. These figures matter when evaluating large MoE models and hardware for agent serving.Sun, 18 Oct 2026 06:29:34 GMTInferencereleasehttps://machinelearning.apple.com/research/mlx-distributed-inferencehttps://machinelearning.apple.com/research/mlx-distributed-inferenceApple reports over 36 tokens per second for the nearly trillion-parameter Muse Spark Open on four Mac Studios, using MLX and RDMA over Thunderbolt at an 8,192-token context. The setup offers a concrete reference for distributed inference on unified-memory hardware.Sat, 17 Oct 2026 06:01:58 GMTInferencedeep-divehttps://github.com/cobanov/PS5LMhttps://github.com/cobanov/PS5LMPS5LM runs llama.cpp's Vulkan backend on the PS5 GPU, uses 11.4 GB of shared memory, and exposes OpenAI- and Anthropic-compatible APIs to devices on the network. It is a concrete edge-inference deployment, though tied to jailbroken console hardware.Sat, 17 Oct 2026 06:01:58 GMTInferencetoolhttps://huggingface.co/papers/2610.22276https://huggingface.co/papers/2610.22276MJev-Spatial scores candidate answers using pretrained query and key projections, handling choices, numeric values, and points without generating answer tokens. A single forward pass for each decision could reduce latency in spatial tasks that require frequent interaction.Sat, 17 Oct 2026 06:01:58 GMTInferencepaperhttps://cloud.google.com/blog/topics/ai-infrastructure/day-0-support-with-5x-perf-for-reflection-beam-on-google-cloud/https://cloud.google.com/blog/topics/ai-infrastructure/day-0-support-with-5x-perf-for-reflection-beam-on-google-cloud/Google Cloud and Reflection report up to 5× higher performance and 8× lower latency for Beam using llm-d with vLLM or NVIDIA Dynamo with SGLang. The serving recipes and reported gains are relevant to teams running large models for agent workloads.Sat, 17 Oct 2026 06:01:58 GMTInferencebenchmarkhttps://developer.nvidia.com/blog/code-test-and-deploy-on-one-machine-with-nvidia-rtx-spark/https://developer.nvidia.com/blog/code-test-and-deploy-on-one-machine-with-nvidia-rtx-spark/RTX Spark pairs a GB10 system-on-chip with up to 128GB of unified memory for running coding agents, fine-tuning models, and deploying locally. A single-machine setup could simplify local agent development and testing.Sat, 17 Oct 2026 06:01:58 GMTInferencereleasehttps://huggingface.co/papers/2610.22347https://huggingface.co/papers/2610.22347Speculative Forcing accepts tokens early when a verifier judges their intermediate estimates sufficiently close to final outputs, while retaining their cached keys and values. It reports matching the four-step LongLive generator with about 1.7 denoising steps, a promising route to cheaper video generation.Fri, 16 Oct 2026 06:22:18 GMTInferencepaperhttps://huggingface.co/papers/2610.22311https://huggingface.co/papers/2610.22311Velox uses camera-frustum overlap to select historical context, reporting 4.36× faster attention and 2.43× faster denoising at chunk 48. The approach targets growing KV-cache costs in long interactive video rollouts.Fri, 16 Oct 2026 06:22:18 GMTInferencepaperhttps://pytorch.org/blog/bringing-executorch-to-arm-cortex-m-cpus/https://pytorch.org/blog/bringing-executorch-to-arm-cortex-m-cpus/The Cortex-M backend runs four MLPerf Tiny models on an Alif Ensemble E8 in 21.5 ms or less, using under 90 KiB of RAM; the core runtime is under 20 KiB of flash. This shows a single PyTorch workflow can reach tightly constrained devices without a framework conversion step.Fri, 16 Oct 2026 06:22:18 GMTInferencedeep-divehttps://huggingface.co/papers/2610.14448https://huggingface.co/papers/2610.14448KV4-RL keeps attention sinks and recent context in INT8 while quantizing the rest to INT4, achieving 1.80× cumulative rollout speedup and 1.27× full-iteration speedup with final accuracy within 0.5 percentage points of BF16. The results show how selective KV-cache protection can improve RL training throughput while limiting quality loss.Thu, 15 Oct 2026 06:22:59 GMTInferencepaperhttps://aws.amazon.com/blogs/machine-learning/deploy-and-evaluate-models-with-amazon-sagemaker-jumpstart/https://aws.amazon.com/blogs/machine-learning/deploy-and-evaluate-models-with-amazon-sagemaker-jumpstart/The guide deploys a pretrained model, runs a lightweight performance test with the SageMaker SDK and removes the endpoint afterward. Named endpoint configurations target low cost, low latency or general-purpose workloads.Thu, 15 Oct 2026 06:22:59 GMTInferenceguidehttps://huggingface.co/papers/2610.13990https://huggingface.co/papers/2610.13990DoSA uses score bounds to avoid fully scoring candidates that cannot enter Top-K, reporting up to 2.74× Indexer speedup and up to 1.94× time-to-first-token speedup. Its exact-selection approach offers a concrete path to faster sparse-attention serving.Wed, 14 Oct 2026 06:38:54 GMTInferencepaperhttps://pytorch.org/blog/extending-inductor-for-the-ibm-spyre-accelerator/https://pytorch.org/blog/extending-inductor-for-the-ibm-spyre-accelerator/IBM moved Spyre’s torch.compile backend from a separately built stack to an Inductor backend, leaving most of Inductor unchanged and shrinking the compiler about 7×. It’s a concrete example of adapting a shared compiler to a non-GPU inference accelerator.Wed, 14 Oct 2026 06:38:54 GMTInferencecase-studyhttps://pulseos.financehttps://pulseos.financeA tuned Qwen3 0.6B model turns spending questions into structured queries, code handles the arithmetic, and a 4B model checks answers, with disagreement prompting uncertainty. This is a concrete pattern for keeping sensitive data local while limiting model errors.Wed, 14 Oct 2026 06:38:54 GMTInferencetoolhttps://www.datalab.to/blog/indra-ocrhttps://www.datalab.to/blog/indra-ocrIndra OCR pairs a 1M-parameter recognizer with layout boxes, labels and reading order, reporting 49 pages per second on 32 CPU cores and higher OmniParseBench scores than Tesseract and PP-OCR. Its CPU throughput and structured output may make document ingestion cheaper and easier to integrate.Wed, 14 Oct 2026 06:38:54 GMTInferencereleasehttps://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai/https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai/The post describes fused MoE operations that combine token routing, data shuffling, and expert computation, and discusses hardware-specific kernel optimizations. Efficient MoE kernels can directly affect inference latency and cost, though the excerpt gives no performance results.Wed, 14 Oct 2026 06:38:54 GMTInferencedeep-divehttps://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/The demo combines in-browser semantic search with a small language model that summarizes the four most relevant passages, and measures answer faithfulness. Local execution keeps typed questions on the user's machine; the excerpt does not give the measurement results.Wed, 14 Oct 2026 06:38:54 GMTInferencedeep-divehttps://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/d-series.htmlhttps://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/d-series.htmlIntel describes a 350 W, air-cooled PCIe accelerator with up to 480 GB of LPDDR5X, aimed at long contexts and concurrent agent sessions. The specifications are relevant to inference capacity planning, though no independent performance results are provided.Wed, 14 Oct 2026 06:38:54 GMTInferencereleasehttps://developer.nvidia.com/blog/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding/https://developer.nvidia.com/blog/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding/ReplaySSM stores a checkpoint plus recent inputs, cutting state-memory traffic roughly in half and speeding decoding kernels 1.46–1.80x; speculative throughput rises 1.85–2.10x. The design targets memory traffic, rollback, and precision trade-offs in hybrid-model serving.Tue, 13 Oct 2026 08:14:15 GMTInferencedeep-divehttps://github.com/prabanta-dev/janashttps://github.com/prabanta-dev/janasOn a 32 GB laptop, Janas reports 24 tok/s generating with a 48 GB Qwen3-Next-80B model, versus 2.8 tok/s for llama.cpp, and also reports faster prompt processing. The results could inform trade-offs for running models larger than available RAM.Tue, 13 Oct 2026 08:14:15 GMTInferencebenchmarkhttps://github.com/Geramy/LemonSeed-Studiohttps://github.com/Geramy/LemonSeed-StudioThe app bundles the upstream Linux amdgpu and amdkfd drivers in a PCIDriverKit extension, then runs its inference engine on an AMD GPU in a Thunderbolt enclosure. The design is a notable example of bringing GPU inference to an iPad.Tue, 13 Oct 2026 08:14:15 GMTInferencetoolhttps://huggingface.co/papers/2610.12570https://huggingface.co/papers/2610.12570DCOCO keeps multiple draft-token candidates and checks their compatibility across positions in parallel, restoring context among draft tokens at low additional cost. Its throughput results across four verifier backbones and ten benchmarks make it relevant to speculative-decoding design.Tue, 13 Oct 2026 08:14:15 GMTInferencepaperhttps://aws.amazon.com/blogs/machine-learning/accessing-openai-gpt-models-on-amazon-bedrock-with-global-cross-region-inference-in-thailand-malaysia-singapore-indonesia-and-taiwan/https://aws.amazon.com/blogs/machine-learning/accessing-openai-gpt-models-on-amazon-bedrock-with-global-cross-region-inference-in-thailand-malaysia-singapore-indonesia-and-taiwan/The guide configures global cross-Region inference for Bedrock-hosted GPT models and covers API calls, permissions, and monitoring. This offers serving teams a path around capacity limits in a single Region.Tue, 13 Oct 2026 08:14:15 GMTInferenceguidehttps://aws.amazon.com/blogs/machine-learning/deploy-decision-models-with-llama-cpp-on-amazon-sagemaker-ai/https://aws.amazon.com/blogs/machine-learning/deploy-decision-models-with-llama-cpp-on-amazon-sagemaker-ai/The guide uses llama.cpp on SageMaker for text generation and extends the runtime with SemIf to return probabilities over declared options without decoding an answer token. That pattern suits agent routing and other systems that need scores over actions rather than generated text.Tue, 13 Oct 2026 08:14:15 GMTInferenceguidehttps://huggingface.co/papers/2610.09788https://huggingface.co/papers/2610.09788LatentGRM compresses evaluation trajectories by 8.9–9.2x and reduces judge inference time by 6.1–7.0x at vote@5, with competitive preference accuracy against explicit SFT judges. It shows how rubric-guided latent reasoning can lower the cost of model evaluation.Mon, 12 Oct 2026 07:53:06 GMTInferencepaperhttps://huggingface.co/papers/2610.13171https://huggingface.co/papers/2610.13171PreSTO batches prefetched Metropolis–Hastings proposals while preserving PowerMH's transition kernel, reporting median speedups of 1.71×, 1.91× and 2.33× across seven datasets and five base LLMs. The approach targets sequential sampling overhead without changing the sampler's behavior.Mon, 12 Oct 2026 07:53:06 GMTInferencepaperhttps://huggingface.co/papers/2610.07121https://huggingface.co/papers/2610.07121SoloQ rotates weights and activations into a predictable distribution for data-independent low-bit quantization, and applies commit-time KV-cache quantization to block-diffusion models. Calibration-free deployment could simplify quantization where activations shift across denoising steps.Mon, 12 Oct 2026 07:53:06 GMTInferencepaperhttps://betterwispr.com/https://betterwispr.com/The open-source dictation tool uses built-in models on the Mac, works offline after model downloads, and has no cloud fallback. It is a concrete example of local inference for a user-facing application, though the listing gives no performance details.Mon, 12 Oct 2026 07:53:06 GMTInferencetoolhttps://nishtahir.com/build-your-own-decision-model/https://nishtahir.com/build-your-own-decision-model/The approach masks all but a fixed set of answer tokens and picks the highest-probability option in one pass, instead of generating a constrained response token by token. This can make structured, fixed-option answers faster to produce.Sun, 11 Oct 2026 07:44:51 GMTInferenceguidehttps://huggingface.co/papers/2610.11216https://huggingface.co/papers/2610.11216The paper models decoding schedules as paths on a corruption lattice and defines their cost by discarded dependence; for Markov data on a graph, zero-cost schedules require steps equal to the graph's treedepth. This offers a way to assess decoding schedules before running them.Sun, 11 Oct 2026 07:44:51 GMTInferencepaperhttps://huggingface.co/papers/2610.04875https://huggingface.co/papers/2610.04875SpecFold uses token-level residual gating to reuse parent attention and FFN computation across speculative branches, with a Triton kernel for sparse execution and reported throughput gains. The algorithm-system co-design targets a distinct source of inference cost in diffusion language models.Sun, 11 Oct 2026 07:44:51 GMTInferencepaperhttps://huggingface.co/papers/2610.06801https://huggingface.co/papers/2610.06801MC-Sparse groups similar queries for GPU execution and caches selected KV tokens and dense-to-sparse residuals across denoising steps; experiments report better dense-attention fidelity and larger speedups than sparse baselines. Its training-free approach targets the latency and quality trade-off in long-sequence generation.Sat, 10 Oct 2026 07:39:10 GMTInferencepaperhttps://github.com/darshi1337/apogeehttps://github.com/darshi1337/apogeeApogee summarizes web pages and other content without sending or storing data, using Ollama, WebGPU or Transformers.js for local inference. It offers a concrete example of bringing AI summarization onto users' devices.Sat, 10 Oct 2026 07:39:10 GMTInferencetoolhttps://github.com/SamsungLabs/LittleBithttps://github.com/SamsungLabs/LittleBitLittleBit factorizes weight matrices into binary latent factors with learned scales; LittleBit-2 adds latent rotation and iterative quantization before quantization-aware training. The released implementation gives inference engineers a concrete extreme-compression approach to evaluate.Fri, 09 Oct 2026 07:53:56 GMTInferencepaperhttps://www.anthropic.com/claude-haiku-5-5https://www.anthropic.com/claude-haiku-5-5Anthropic says Haiku 5.5 costs about 75% less to run than Haiku 4.5 and is designed for fast, repetitive tasks, including use as a coding subagent. The price and speed claims matter for teams balancing model capability against serving costs.Fri, 09 Oct 2026 07:53:56 GMTInferencereleasehttps://cactuscompute.com/blog/whistlehttps://cactuscompute.com/blog/whistleWhistle is a 16.9 MB speech model that runs on CPU without dependencies, with transcription, word timestamps and speech embeddings across seven languages. Its compact local runtime is relevant to teams deploying speech features on constrained devices.Fri, 09 Oct 2026 07:53:56 GMTInferencetoolhttps://github.com/ymcrcat/rgpuhttps://github.com/ymcrcat/rgpurGPU offers a PyTorch device interface and CUDA shims that let client applications run GPU work on a remote NVIDIA machine. The two integration paths could help teams use remote accelerators without rewriting existing workloads.Fri, 09 Oct 2026 07:53:56 GMTInferencetoolhttps://huggingface.co/papers/2609.38169https://huggingface.co/papers/2609.38169STEPQuant assigns precision by error magnitude and memory lifetime, and fits scales to key-row and value-column effects; tests on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct report accuracy close to FP32 at a nominal 6-bit budget. This targets recurrent-state memory costs in concurrent serving.Thu, 08 Oct 2026 07:59:03 GMTInferencepaperhttps://huggingface.co/papers/2610.10524https://huggingface.co/papers/2610.10524GRACE retains a frozen base latent and learns a residual for information lost under stronger compression, while aligning compressed latents to the pretrained distribution. This targets faster video diffusion without requiring a DiT retrain.Thu, 08 Oct 2026 07:59:03 GMTInferencepaperhttps://simonwillison.net/2026/Oct/7/claude-haiku-5-5/https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/Haiku 5.5 costs $0.10/$0.50 per million input/output tokens up to 100,000 tokens, but uses about 1.25 times as many tokens for the same long prompt as Haiku 4.5. Above 100,000 tokens its price rises fivefold, making workload length important to serving cost.Thu, 08 Oct 2026 07:59:03 GMTInferencereleasehttps://aws.amazon.com/blogs/machine-learning/introducing-claude-haiku-5-5-on-aws/https://aws.amazon.com/blogs/machine-learning/introducing-claude-haiku-5-5-on-aws/AWS says Haiku 5.5 is available on Bedrock and costs around 75% less than Haiku 4.5 for most tasks, with AWS IAM, audit and monitoring controls. The release is relevant to teams comparing model cost and deployment options.Thu, 08 Oct 2026 07:59:03 GMTInferencerelease