On October 20, 2026, DeepSeek released DeepInfer, the inference engine it says powers its production services, on GitHub under Apache 2.0. Inference is where a model’s running cost and reliability are decided. DeepInfer gives a rare look at that layer, but the release is narrower than the headlines suggest.
What Is in the Release
DeepInfer supports one model, DeepSeek-V4.1-Flash, and loads the 1017 version by default. Its feature list includes custom kernels, parallelism for attention and for mixture-of-experts layers, asynchronous scheduling that overlaps CPU planning with GPU work, and structured output that forces answers into a regex, JSON schema, or EBNF grammar.
It also includes DSpark, a speculative decoding system. A small draft model proposes up to five tokens at a time, and a confidence estimate decides which ones the main model should check. Batch-invariant inference and deterministic sampling make results identical no matter what else is in the batch, which matters for reproducible tests.
What Is Missing
The README says this release provides offline inference. A complete service would also need a KV cache system, prefill and decode disaggregation, and load balancing across replicas. All three are listed as not included. So this is the engine core, not the full production stack. SiliconANGLE also reports that Hopper and older Nvidia chips are not supported.
Nvidia and Huawei
DeepInfer runs on Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 chips, with CANN 9.2 required on Ascend. But batch-invariant inference and deterministic sampling are marked Nvidia only. Ascend support exists, with fewer guarantees.
SiliconANGLE reports that DeepSeek brought its TileLang kernel language to Ascend on September 30 with Huawei’s help. It also cites a September Bloomberg report that DeepSeek plans to deploy at least 160,000 Ascend 950DT accelerators in Inner Mongolia. That is a plan, not a confirmed deployment.
What Cannot Be Verified
The README shows throughput charts for prefill, measured on 4 SM100 GPUs, and for decode, measured on 64. It gives no figures in text, and the results are DeepSeek’s own. A sweep script lets others reproduce them. That DeepInfer powers DeepSeek’s production services is also DeepSeek’s statement.
The README says DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. The vLLM project said it would bring ideas from DeepInfer back into its own support. The repository had 17.4k stars when checked, but popularity is not validation.
What Could Come Next
In the near term, expect outside teams to run the sweep script and compare results. DeepSeek said inference code for future models will go into the same repository. The long-term picture is speculation. If open inference stacks keep improving, serving cost may depend more on chip access than on software.
