DeepInfer Reads Like an Answer to DeepSeek’s Own 2025 Objections
- * *
On October 20, 2026, DeepSeek announced DeepInfer, the inference framework it says serves DeepSeek-V4.1-Flash in production, and published the code under Apache 2.0. Eighteen months earlier, the same company had explained in writing why it would not release its engine. As The New Stack quotes that post, the software was “impractical for public deployment without significant modifications.”
Both statements can be true at once, and the space between them is where this release gets interesting. DeepInfer is a real production engine: kernels, a scheduler, a sampler, and multi-chip parallelism. It is also an offline inference release that leaves out the KV cache system, prefill and decode disaggregation, and load balancing across replicas. Those missing layers are the ones that bind most tightly to a specific cluster.
This post reads the repository, its examples, and the first round of reporting, and it marks which claims are DeepSeek’s and which readings are mine.
> Quick take: DeepInfer is most useful as a reference implementation and as a day-one path to V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is not a drop-in serving stack, DeepSeek’s speed figures have no independent verification yet, and the determinism features work on NVIDIA only.
- * *
What DeepSeek said in 2025
DeepSeek’s earlier open-source push went component by component. In late February 2025, its Open Source Week shipped one building block a day: FlashMLA, an MLA decoding kernel for Hopper GPUs; DeepEP, an expert-parallel communication library; and DeepGEMM, an FP8 GEMM library. The week also covered the EPLB expert-parallel load balancer, the DualPipe training schedule, and the 3FS file system.
Inference code was the thin spot. For V3, the model card offered the DeepSeek-Infer demo, described as a “simple and lightweight demo” for FP8 and BF16, alongside community engines such as SGLang and LMDeploy for serving it.
In April 2025 the same index gained a post on open-sourcing the inference engine. According to The New Stack’s account, DeepSeek had considered releasing the whole system and chose to contribute pieces to existing projects instead. Its reasons, in short: the engine had grown from an early vLLM version into a heavily customized system, a small research team would struggle to maintain another major open-source project, and the code was entangled with internal infrastructure.
The entanglement point matters most for what follows. The post tied it specifically to cluster management tooling.
DeepSeek also made a promise about hardware. It said it would synchronize inference engineering with partners ahead of future model launches, aiming for day-one availability across chip architectures. Keep that in mind for the Huawei section below.
- * *
What shipped on October 20
The repository supports one model, DeepSeek-V4.1-Flash, on two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017, the updated weights DeepSeek released days earlier.
Two companion facts shape the picture. DeepSeek credits vLLM as the base: DeepInfer was originally built primarily on it, with inspiration from SGLang and TensorRT-LLM. And a second repository, deepseek-recipe, ships alongside. It handles how API requests become model inputs and how outputs become responses, in Rust and Python, so self-hosters can match the behavior of DeepSeek’s official API.
The upstream loop DeepSeek described in 2025 is already visible. TNW reports that the vLLM project thanked DeepSeek and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. The repository had passed 8,000 stars by Tuesday afternoon, per the same report, and showed about 12,400 stars and 1,100 forks at the time of writing.
!Mermaid Diagram
- * *
The scope is the argument
Here is the boundary, drawn from the README’s own list of what a complete inference service still needs.
!Mermaid Diagram
The announcement called DeepInfer “fully open source.” The README is more careful. It describes offline inference with the core features needed for serving at scale, then lists three components it does not include: a KV cache system, VE/prefill/decode disaggregation (with the note that most of the required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode.
Set that against the entanglement objection. What the README leaves out is the layer that routes requests to replicas, decides where cached prefixes live, and splits phases across machines. Those parts depend on your network, your scheduler, and your traffic. The release boundary sits roughly where the coupling to infrastructure would be tightest.
That is my reading, and DeepSeek has not said so. The New Stack makes a similar observation and points out that the company has not explained how it overcame the earlier difficulties.
For planning, the line is simple. Offline batch inference, evals, and research on a cluster you control: DeepInfer is close to complete. A multi-tenant chat API with prefix caching and replica routing: budget significant engineering time for the layer above it.
- * *
Inside the engine
The README’s feature list is easier to judge with the switches next to it.
| Feature | What it does | How to turn it on |
| --- | --- | --- |
| Parallelism | Attention DP and balanced CP, plus MoE expert parallelism with EPLB | --ep, --dp, and --cp flags in the offline example |
| Asynchronous scheduling | Overlaps CPU batch scheduling with device execution | On by default |
| DSpark with confidence-scheduled verification | Decodes with draft tokens and uses confidence estimates to schedule verification | DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length (default 5) |
| Decoder SWA bounded replay | Cuts decoder prefill work by replaying only a bounded suffix | DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 |
| Batch-invariant inference | Keeps results identical across batch compositions | DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) |
| Deterministic sampling | Identical outputs for identical input tokens and sampling parameters | DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed (NVIDIA only) |
| Structured outputs | Constrains generation to a regex, JSON schema, or EBNF grammar | so_args passed to SamplingParams |
Read the last column twice. The DSpark recipe sets DEEPINFER_ENABLE_MTP_VARLEN=1, and the deterministic-sampling recipe sets the same variable to 0. If you need reproducible outputs for evals or regression tests, the two documented recipes pull in opposite directions. The README does not quantify the cost, so test before assuming they compose.
Bounded replay exposes a model-specific constant. The README says the retained suffix must cover the sliding-window-attention window plus the maximum backtracking length, and it gives 140 tokens as the default for V4.1. Treat 140 as a V4.1 setting, not a universal one. Anyone adapting the engine to another model has a number to derive rather than copy.
The kernel list is the 2025 strategy, assembled. FlashMLA, DeepGEMM, and DeepEP are the Open Source Week repositories. TileKernels, DeepSelect, and custom TileLang kernels in the tree join them. Read together, DeepInfer is where the formerly separate components meet a scheduler and a model runtime.
- * *
The vision encoder rides along
V4.1-Flash accepts images, and the engine handles them in-process. With --prompt_mode image, the examples feed raw images to the engine directly, with the ViT and the LLM in the same engine. CPU preprocessing runs in asynchronous MultimodalProcessor worker processes, off the step critical path.
That explains the odd “VE” in the list of missing components. VE is the vision encoder, and the examples benchmark it on its own. A sweep script loads only the VE-side weights, then times the GPU-side pad, ViT, downsample, and aligner path across image sizes and batch counts. Throughput is scaled by the number of LLM tokens each image produces, the same unit as the prefill curve, so the two are comparable.
So “VE/prefill/decode disaggregation” is a three-stage split with the encoder as a separable stage. What the release leaves to you is the orchestration around it.
The vision regression suite is refreshingly small: five inputs, made of two text-only samples, a single-image sample, a three-image sample, and a five-image four-shot sample. Every image comes from just two pictures, carrots and ribs.
- * *
Reading the benchmark charts without fooling yourself
The README shows two Pareto plots: prefill (non-cached input throughput against latency) and decode (output throughput against per-user speed). DeepSeek measured them on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. TNW relays Bloomberg’s note that no one has independently benchmarked DeepInfer’s gains yet.
The examples README documents the harness in unusual detail, and one line changes how you should read the decode curve. Decode throughput is computed like this:
decode tokens/s/GPU = avg_SPS x decode_num_requests x (1 + nextn) / cp
The docs call this full-acceptance accounting: every request is credited with 1 + nextn tokens per step, regardless of the true acceptance rate. With the defaults of 64 requests and a draft length of 5, one step counts as 384 tokens.
Real speculative decoding rarely accepts every draft token, so a full-acceptance figure is a ceiling. The sweep script takes an acceptance-rate argument, and the docs’ own example sweeps a draft length of 3 and scales the decode plot by 0.75. Which scaling, if any, the published figure uses is not stated. Before comparing against vLLM or SGLang, check that setting.
The methodology has strengths worth copying. The harness separates prefill from decode, drops intervals that collide with MoE load-balance statistics flushes, and runs sweeps with the profiler off because it perturbs timings. It also warns that profiler-traced step times can be inflated by nearly 50% at batch size 1 and must not be used to derive steps per second.
Numerics are held to an exact standard. Perplexity baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them sample by sample. A refresh_offline_decode_ppl.py script updates them once a numeric change is confirmed as expected, and the docs say to re-measure the short and long baselines on NVIDIA and Ascend separately after changing the vision samples. A --check_structured_output flag extends the same run to regex, JSON, and EBNF checks.
If you plan to compare DeepInfer with another engine, hold these constant. The list is mine, built from what the harness documents:
- Hardware and scale-up domain. Match the layout behind the published figures, or state how yours differs.
- Workload shape. Harness defaults are 32,768 context tokens with 4,096 non-cached. The documented nightly profile configuration uses 128k context with YoCo clipping.
- Draft length and token accounting. State your draft length, and whether you count full acceptance or measured acceptance.
- Raw quantities. --result_jsonl appends the configuration plus window step count, token count, elapsed time, and avg_SPS, so you can recompute throughput under your own accounting.
- One tag per curve. The sweep tooling blocks plotting when model, ep_size, --cp, context length, draft length, or redundant expert count differ within a tag. Borrow that discipline.
- * *
Nvidia and Huawei, side by side
The hardware list is the most strategic line in the README. Next to NVIDIA’s SM100 parts, which The New Stack identifies as Blackwell data-center GPUs, sits Huawei’s Ascend 950 series, with CANN 9.2 as its prerequisite. Older NVIDIA architectures such as Hopper are absent.
The New Stack places this inside a run of reporting. Bloomberg said in September that DeepSeek planned to deploy at least 160,000 Ascend 950DT chips at a new Inner Mongolia data center. Reuters later that month reported DeepSeek and Huawei collaborating on tools to make Ascend chips easier to program. TNW adds that DeepSeek published tooling for Huawei chips earlier in October as an alternative to CUDA.
Parity is incomplete. Batch-invariant inference and deterministic sampling are NVIDIA only, and the published performance numbers all come from NVIDIA GPUs. The README gives no Ascend numbers.
The examples do show Ascend taken seriously. They document debugging variables such as ASCEND_LAUNCH_BLOCKING=1, which runs operators synchronously so stack traces are readable, and, with the separate numeric baselines described earlier, that reads as a first-class target rather than an afterthought.
This matches the 2025 promise of day-one cross-chip support, delivered for one model on two chip families. It is a narrow delivery of a broad commitment. TNW reports that DeepSeek says future models and improvements will land in the same repository.
- * *
Running it today
The prerequisites are Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment.
export DEEPINFER_ACCELERATOR=nvidia # use "ascend" on Ascend
./build.sh
# first run downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache
python examples/offline_inference.py --ep 8
The feature switches in the table above are environment variables, so export them in the same shell before launching.
The offline example sets its layout with flags. The documented variants are --ep 8, --ep 2 --dp 8, and --ep 2 --cp 8. Set DEEPINFER_HF_MODEL_DIR to run from a local copy of the weights.
The benchmark harness works differently, and the difference trips people up. There, ep_size equals WORLD_SIZE (the number of nodes) times the number of visible devices, and --cp must divide it. Redundant experts default to 32 through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, but that value is only a lower bound: the engine rounds the total up so the expert count divides evenly by ep_size.
The hardware requirement is the main gate: the README supports SM100 and Ascend 950 only.
- * *
Who should adopt it
| If you are | Verdict | Why |
| --- | --- | --- |
| An inference-engine engineer | Read the code this week | A major lab’s account of what matters at scale, under Apache 2.0 |
| A hosting provider on Blackwell or Ascend 950 | Pilot it | You get the kernels and parallelism, but you supply KV cache, disaggregation, and routing, and you must verify the speed claims yourself |
| A team running offline batch jobs or evals on SM100 | Strong fit | Offline inference is what shipped, and batch invariance helps reproducibility on NVIDIA |
| An app team calling a hosted API | Skip for now | Nothing in the release changes how you call a model |
| A team on Hopper or older GPUs | Wait | The hardware is unsupported as released, so keep using vLLM or SGLang for now |
- * *
What to watch next
Four signals will show how far DeepInfer travels beyond the announcement.
- Independent benchmarks on identical hardware and identical token accounting, against vLLM and SGLang.
- The missing layers. The README calls them required for a complete service. Whether DeepSeek will ship them is unannounced.
- What lands upstream. vLLM said it would use ideas from DeepInfer. Watch for merged changes rather than statements.
- Ascend parity for batch-invariant inference and deterministic sampling.
The 2025 objections did not disappear. They moved to the edge of the release, where your own cluster begins. For anyone who builds or tunes inference engines, DeepInfer is a look at what a major lab says it runs in production. For teams that need a serving stack today, it is a release to watch rather than deploy.
---
Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
Mentions
DeepInfer Reads Like an Answer to DeepSeek’s Own 2025 Objections - * * On October 20, 2026, DeepSeek announced DeepInfer, the inference framework it says serves DeepSeek-V4.1-Flash in production, and published the code under Apache 2.0. Eighteen months earlier, the same company had explained in writing why it would not release its engine. As The New Stack quotes that post, the software was “impractical for public deployment without significant modifications.” Both statements can be true at once, and the space between them is where this release gets interesting. DeepInfer is a real production engine: kernels, a scheduler, a sampler, and multi-chip parallelism. It is also an offline inference release that leaves out the KV cache system, prefill and decode disaggregation, and load balancing across replicas. Those missing layers are the ones that bind most tightly to a specific cluster. This post reads the repository, its examples, and the first round of reporting, and it marks which claims are DeepSeek’s and which readings are mine. > Quick take: DeepInfer is most useful as a reference implementation and as a day-one path to V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is not a drop-in serving stack, DeepSeek’s speed figures have no independent verification yet, and the determinism features work on NVIDIA only. - * * What DeepSeek said in 2025 DeepSeek’s earlier open-source push went component by component. In late February 2025, its Open Source Week shipped one building block a day: FlashMLA, an MLA decoding kernel for Hopper GPUs; DeepEP, an expert-parallel communication library; and DeepGEMM, an FP8 GEMM library. The week also covered the EPLB expert-parallel load balancer, the DualPipe training schedule, and the 3FS file system. Inference code was the thin spot. For V3, the model card offered the DeepSeek-Infer demo, described as a “simple and lightweight demo” for FP8 and BF16, alongside community engines such as SGLang and LMDeploy for serving it. In April 2025 the same index gained a post on open-sourcing the inference engine. According to The New Stack’s account, DeepSeek had considered releasing the whole system and chose to contribute pieces to existing projects instead. Its reasons, in short: the engine had grown from an early vLLM version into a heavily customized system, a small research team would struggle to maintain another major open-source project, and the code was entangled with internal infrastructure. The entanglement point matters most for what follows. The post tied it specifically to cluster management tooling. DeepSeek also made a promise about hardware. It said it would synchronize inference engineering with partners ahead of future model launches, aiming for day-one availability across chip architectures. Keep that in mind for the Huawei section below. - * * What shipped on October 20 The repository supports one model, DeepSeek-V4.1-Flash, on two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017, the updated weights DeepSeek released days earlier. Two companion facts shape the picture. DeepSeek credits vLLM as the base: DeepInfer was originally built primarily on it, with inspiration from SGLang and TensorRT-LLM. And a second repository, deepseek-recipe, ships alongside. It handles how API requests become model inputs and how outputs become responses, in Rust and Python, so self-hosters can match the behavior of DeepSeek’s official API. The upstream loop DeepSeek described in 2025 is already visible. TNW reports that the vLLM project thanked DeepSeek and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. The repository had passed 8,000 stars by Tuesday afternoon, per the same report, and showed about 12,400 stars and 1,100 forks at the time of writing. !Mermaid Diagram - * * The scope is the argument Here is the boundary, drawn from the README’s own list of what a complete inference service still needs. !Mermaid Diagram The announcement called DeepInfer “fully open source.” The README is more careful. It describes offline inference with the core features needed for serving at scale, then lists three components it does not include: a KV cache system, VE/prefill/decode disaggregation (with the note that most of the required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode. Set that against the entanglement objection. What the README leaves out is the layer that routes requests to replicas, decides where cached prefixes live, and splits phases across machines. Those parts depend on your network, your scheduler, and your traffic. The release boundary sits roughly where the coupling to infrastructure would be tightest. That is my reading, and DeepSeek has not said so. The New Stack makes a similar observation and points out that the company has not explained how it overcame the earlier difficulties. For planning, the line is simple. Offline batch inference, evals, and research on a cluster you control: DeepInfer is close to complete. A multi-tenant chat API with prefix caching and replica routing: budget significant engineering time for the layer above it. - * * Inside the engine The README’s feature list is easier to judge with the switches next to it. | Feature | What it does | How to turn it on | | --- | --- | --- | | Parallelism | Attention DP and balanced CP, plus MoE expert parallelism with EPLB | --ep, --dp, and --cp flags in the offline example | | Asynchronous scheduling | Overlaps CPU batch scheduling with device execution | On by default | | DSpark with confidence-scheduled verification | Decodes with draft tokens and uses confidence estimates to schedule verification | DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length (default 5) | | Decoder SWA bounded replay | Cuts decoder prefill work by replaying only a bounded suffix | DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 | | Batch-invariant inference | Keeps results identical across batch compositions | DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) | | Deterministic sampling | Identical outputs for identical input tokens and sampling parameters | DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed (NVIDIA only) | | Structured outputs | Constrains generation to a regex, JSON schema, or EBNF grammar | so_args passed to SamplingParams | Read the last column twice. The DSpark recipe sets DEEPINFER_ENABLE_MTP_VARLEN=1, and the deterministic-sampling recipe sets the same variable to 0. If you need reproducible outputs for evals or regression tests, the two documented recipes pull in opposite directions. The README does not quantify the cost, so test before assuming they compose. Bounded replay exposes a model-specific constant. The README says the retained suffix must cover the sliding-window-attention window plus the maximum backtracking length, and it gives 140 tokens as the default for V4.1. Treat 140 as a V4.1 setting, not a universal one. Anyone adapting the engine to another model has a number to derive rather than copy. The kernel list is the 2025 strategy, assembled. FlashMLA, DeepGEMM, and DeepEP are the Open Source Week repositories. TileKernels, DeepSelect, and custom TileLang kernels in the tree join them. Read together, DeepInfer is where the formerly separate components meet a scheduler and a model runtime. - * * The vision encoder rides along V4.1-Flash accepts images, and the engine handles them in-process. With --prompt_mode image, the examples feed raw images to the engine directly, with the ViT and the LLM in the same engine. CPU preprocessing runs in asynchronous MultimodalProcessor worker processes, off the step critical path. That explains the odd “VE” in the list of missing components. VE is the vision encoder, and the examples benchmark it on its own. A sweep script loads only the VE-side weights, then times the GPU-side pad, ViT, downsample, and aligner path across image sizes and batch counts. Throughput is scaled by the number of LLM tokens each image produces, the same unit as the prefill curve, so the two are comparable. So “VE/prefill/decode disaggregation” is a three-stage split with the encoder as a separable stage. What the release leaves to you is the orchestration around it. The vision regression suite is refreshingly small: five inputs, made of two text-only samples, a single-image sample, a three-image sample, and a five-image four-shot sample. Every image comes from just two pictures, carrots and ribs. - * * Reading the benchmark charts without fooling yourself The README shows two Pareto plots: prefill (non-cached input throughput against latency) and decode (output throughput against per-user speed). DeepSeek measured them on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. TNW relays Bloomberg’s note that no one has independently benchmarked DeepInfer’s gains yet. The examples README documents the harness in unusual detail, and one line changes how you should read the decode curve. Decode throughput is computed like this: decode tokens/s/GPU = avg_SPS x decode_num_requests x (1 + nextn) / cp The docs call this full-acceptance accounting: every request is credited with 1 + nextn tokens per step, regardless of the true acceptance rate. With the defaults of 64 requests and a draft length of 5, one step counts as 384 tokens. Real speculative decoding rarely accepts every draft token, so a full-acceptance figure is a ceiling. The sweep script takes an acceptance-rate argument, and the docs’ own example sweeps a draft length of 3 and scales the decode plot by 0.75. Which scaling, if any, the published figure uses is not stated. Before comparing against vLLM or SGLang, check that setting. The methodology has strengths worth copying. The harness separates prefill from decode, drops intervals that collide with MoE load-balance statistics flushes, and runs sweeps with the profiler off because it perturbs timings. It also warns that profiler-traced step times can be inflated by nearly 50% at batch size 1 and must not be used to derive steps per second. Numerics are held to an exact standard. Perplexity baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them sample by sample. A refresh_offline_decode_ppl.py script updates them once a numeric change is confirmed as expected, and the docs say to re-measure the short and long baselines on NVIDIA and Ascend separately after changing the vision samples. A --check_structured_output flag extends the same run to regex, JSON, and EBNF checks. If you plan to compare DeepInfer with another engine, hold these constant. The list is mine, built from what the harness documents: - Hardware and scale-up domain. Match the layout behind the published figures, or state how yours differs. - Workload shape. Harness defaults are 32,768 context tokens with 4,096 non-cached. The documented nightly profile configuration uses 128k context with YoCo clipping. - Draft length and token accounting. State your draft length, and whether you count full acceptance or measured acceptance. - Raw quantities. --result_jsonl appends the configuration plus window step count, token count, elapsed time, and avg_SPS, so you can recompute throughput under your own accounting. - One tag per curve. The sweep tooling blocks plotting when model, ep_size, --cp, context length, draft length, or redundant expert count differ within a tag. Borrow that discipline. - * * Nvidia and Huawei, side by side The hardware list is the most strategic line in the README. Next to NVIDIA’s SM100 parts, which The New Stack identifies as Blackwell data-center GPUs, sits Huawei’s Ascend 950 series, with CANN 9.2 as its prerequisite. Older NVIDIA architectures such as Hopper are absent. The New Stack places this inside a run of reporting. Bloomberg said in September that DeepSeek planned to deploy at least 160,000 Ascend 950DT chips at a new Inner Mongolia data center. Reuters later that month reported DeepSeek and Huawei collaborating on tools to make Ascend chips easier to program. TNW adds that DeepSeek published tooling for Huawei chips earlier in October as an alternative to CUDA. Parity is incomplete. Batch-invariant inference and deterministic sampling are NVIDIA only, and the published performance numbers all come from NVIDIA GPUs. The README gives no Ascend numbers. The examples do show Ascend taken seriously. They document debugging variables such as ASCEND_LAUNCH_BLOCKING=1, which runs operators synchronously so stack traces are readable, and, with the separate numeric baselines described earlier, that reads as a first-class target rather than an afterthought. This matches the 2025 promise of day-one cross-chip support, delivered for one model on two chip families. It is a narrow delivery of a broad commitment. TNW reports that DeepSeek says future models and improvements will land in the same repository. - * * Running it today The prerequisites are Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment. export DEEPINFER_ACCELERATOR=nvidia # use "ascend" on Ascend ./build.sh # first run downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache python examples/offline_inference.py --ep 8 The feature switches in the table above are environment variables, so export them in the same shell before launching. The offline example sets its layout with flags. The documented variants are --ep 8, --ep 2 --dp 8, and --ep 2 --cp 8. Set DEEPINFER_HF_MODEL_DIR to run from a local copy of the weights. The benchmark harness works differently, and the difference trips people up. There, ep_size equals WORLD_SIZE (the number of nodes) times the number of visible devices, and --cp must divide it. Redundant experts default to 32 through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, but that value is only a lower bound: the engine rounds the total up so the expert count divides evenly by ep_size. The hardware requirement is the main gate: the README supports SM100 and Ascend 950 only. - * * Who should adopt it | If you are | Verdict | Why | | --- | --- | --- | | An inference-engine engineer | Read the code this week | A major lab’s account of what matters at scale, under Apache 2.0 | | A hosting provider on Blackwell or Ascend 950 | Pilot it | You get the kernels and parallelism, but you supply KV cache, disaggregation, and routing, and you must verify the speed claims yourself | | A team running offline batch jobs or evals on SM100 | Strong fit | Offline inference is what shipped, and batch invariance helps reproducibility on NVIDIA | | An app team calling a hosted API | Skip for now | Nothing in the release changes how you call a model | | A team on Hopper or older GPUs | Wait | The hardware is unsupported as released, so keep using vLLM or SGLang for now | - * * What to watch next Four signals will show how far DeepInfer travels beyond the announcement. - Independent benchmarks on identical hardware and identical token accounting, against vLLM and SGLang. - The missing layers. The README calls them required for a complete service. Whether DeepSeek will ship them is unannounced. - What lands upstream. vLLM said it would use ideas from DeepInfer. Watch for merged changes rather than statements. - Ascend parity for batch-invariant inference and deterministic sampling. The 2025 objections did not disappear. They moved to the edge of the release, where your own cluster begins. For anyone who builds or tunes inference engines, DeepInfer is a look at what a major lab says it runs in production. For teams that need a serving stack today, it is a release to watch rather than deploy. --- Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
FlashMLA ⭐ 13k An open-source MLA decoding kernel for developers running models on Hopper GPUs. 🔗 deepseek-ai/FlashMLA #opensource t.co
aaaaand it's upstreamed! FlashMLA#230
attention計算に特化した、DeepSeekのkernel library。 FlashMLAはDeepSeek-V4.1向けにNVIDIA GPUとAscend NPUのsparse attention等を提供。9/30版はHopper・旧model対応を外し、KV cache形式も変更しています。 deepseek-ai/FlashMLA
FlashMLA: Efficient Multi-head Latent Attention Kernels (13k stars) What it does All text below is quoted directly from the project README. >FlashMLA is DeepSeek's library of optimized attention kernels, powering inference of the DeepSeek-V4.1 model on NVIDIA GPUs and Huawei Ascend NPUs. This repo contains token-level sparse attention kernels for prefill and for decoding with an FP8 / FP4 KV cache, a fused kernel that combines Q-norm, Q-RoPE, attention, O-RoPE (conjugate) and the cast to FP8, and dense attention kernels for prefill and backward. The sparse kernels power DeepSeek Sparse Attention (DSA), as introduced in this paper. News: - 2026.09.30 Release of Ascend Attention Kernels: We've released sparse attention prefill and decoding kernels for the Huawei Ascend 950 NPU, which achieve up to 410 TFlops (95% hardware peak) and 360 TFlops (83% of hardware peak) during prefill and decoding, respectively. - 2026.09.10 Release of DeepSeek v4.1's Attention Kernels: We've released attention kernels for DeepSeek-V4.1, including both prefill and decoding (with FP8 or FP4 KV cache). - 2025.09.29 Release of Sparse Attention Kernels: With the launch of DeepSeek-V3.2, we are releasing the corresponding token-level sparse attention kernels. These kernels power the model's DeepSeek Sparse Attention (DSA) and achieve up to 640 TFlops during prefilling and 410 TFlops during decoding. - 2025.08.01 Kernels for MHA on SM100: Thanks to NVIDIA's PR for MHA forward / backward kernels on SM100! >Breaking change notice (2026.09.30): In the 2026.09.30 release, we removed support for the Hopper architecture and for earlier models (including DeepSeek V3 / V3.2 / V4.0), and we changed the FP8 / FP4 KV cache format. This version is therefore not compatible with previous ones. Repository deepseek-ai/FlashMLA Repository stats |Metric|Value| |:-|:-| |Stars|13,049| |Forks|1,183| |Last release|None| |Languages|C++ 55.6%, Cuda 28.6%, Python 15.5%| |Contributors|19| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|None| Main contributor |Field|Value| |:-|:-| |Name|Shengyu Liu| |GitHub|@interestingLSY| |Commits to this repo|14 (top contributor)| |Company / Location|@deepseek-ai, 中国，浙江省，杭州市| |Website|interestinglsy.github.io| |Followers|1,082| |Public repos|31| |On GitHub since|2017-08-05|