DeepSeek has open-sourced DeepInfer, the inference framework that serves its production models. This is the layer between the model weights and the API, and it decides how much each answer costs. The code is on GitHub under Apache 2.0, alongside a companion library, deepseek-recipe. It comes three days after DeepSeek opened its own search engine to developers.
IT Home reported the 20 October announcement. DeepSeek’s framing: a model’s capability is set by the model, but whether you can serve it at reasonable cost depends on “a middle layer the public has long not understood”. The inference framework schedules the compute cluster, and its efficiency “directly determines the compute cost and power consumption of AI services”. DeepSeek says it is sharing this so that companies can run private and public-cloud deployments more cheaply, in pursuit of what it calls 普惠, AI that’s affordable for everyone.
What’s in the box
According to the README, DeepInfer currently supports one model, DeepSeek-V4.1-Flash, using the 1017 checkpoint that shipped last week. The feature list is aimed at serving engineers:
- DeepSeek’s own kernels (FlashMLA, DeepGEMM, DeepEP and others), plus custom TileLang kernels;
- attention data parallelism, MoE expert parallelism and load balancing;
- “DSpark” speculative decoding with confidence-scheduled verification;
- bounded replay for sliding-window attention, to cut prefill work;
- zero-overhead asynchronous batch scheduling;
- batch-invariant inference and deterministic sampling: identical input and sampling settings give identical output, whatever else is in the batch (NVIDIA only for now);
- structured output constrained to regex, JSON schema or grammar.
Supported hardware is NVIDIA’s SM100 (Blackwell-class) GPUs and Huawei’s Ascend 950 NPUs. That second line is the geopolitical one. DeepSeek is publishing a production serving stack that runs on domestic Chinese silicon.
The release also leaves things out. The README says a complete service still needs a KV cache system, prefill/decode disaggregation and load balancing across replicas, and none of those are included. DeepSeek also acknowledges that DeepInfer “was originally built primarily on vLLM”, with ideas from SGLang and TensorRT-LLM. So this isn’t DeepSeek’s API in a box. It’s the engine without the gearbox. Developers clearly want it anyway: the repository had passed 15,000 GitHub stars by Wednesday morning.
deepseek-recipe handles the less glamorous part. It’s a set of Rust libraries with Python bindings that convert Messages, Chat Completions and Responses-style requests into DeepSeek prompts and parse the output back. DeepSeek says it is “fully aligned” with the official API’s handling. One detail that’s useful to anyone benchmarking: default output limits are 8,192 tokens in normal mode, 65,536 in thinking mode and 131,072 at maximum reasoning effort.
Why search people should care
Inference cost decides how many queries get an AI answer. Every search product with a generative layer, from Baidu and Yandex to Naver and Cốc Cốc, is rationing answers by cost per query. When one of China’s most widely used open-weight model families publishes how it serves answers cheaply, everyone building on DeepSeek gets cheaper too. That includes the many Chinese apps and search features that plug it in. Expect more AI answers on more query types, sooner.
Determinism matters too. One of the hardest problems in measuring AI visibility is that the same prompt gives different answers. Batch-invariant inference with fixed seeds doesn’t fix that for consumer chat apps, which still vary system prompts, retrieval and temperature. But it does mean a DeepSeek-based deployment can be made reproducible. That opens the door to auditable answer testing, if the operator chooses to switch it on. If you track citations in DeepSeek-powered tools, ask the vendor whether they do.
There’s a sting for Western buyers. The US Department of Homeland Security has just ordered federal agencies to remove DeepSeek, including third-party products built on DeepSeek implementations (our report). An Apache-licensed inference engine isn’t a model, but procurement teams won’t parse that distinction kindly. Expect “do you use any DeepSeek code?” to join vendor questionnaires.
The pattern
Within a week, DeepSeek has refunded a full week of API usage after a five-hour outage, opened a search API and now released its serving engine. It’s building a stack: model, retrieval, serving. And it’s giving the serving layer away, which pushes the price of AI answers towards zero. That’s bad news for rivals who charge for those answers.