DeepInfer: High-performance inference framework for DeepSeek models

A nuclear bomb

"Based on this framework, we have achieved high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash. Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (at off-peak pricing)"

Source: https://zhuanlan.zhihu.com/p/2095560295930508953

Interesting that the DHS "Exclusion and Removal Order #2026-01" that was made public yesterday was actually released July 24th, 2026 and appears to have taken effect then. Agencies would have started pulling deepseek at that time. Which means it's likely DS became aware of the memo around then.

The release of this repo hours after the DHS order is a tactical move they started preparing when the DHS order was issued almost 3 months ago, and they timed publishing the repo right as the order hit the federal register.

DHS: "China bad and scary. Don't send them data."

DS: "You don't have to. Happy xmass kids."

We will pick this up, benchmark and publish on https://tokenbytoken.org

Currently we have started benchmarking Qwen family and also publishing the configuration that we have been using

That's Kami Kaze for US frontier labs. There's no way from here for them towards trillion dollar valuations.

That's strategic sabotage to say the least.

Ban all open weight models! Cried the VCs.

Whatever the consequences, I have to admit China has understood that all those are raw materials and that open sourcing everything will help growing their economy (as a whole).

The worst idea was to build corporations providing services with AI. It was an extension of the latest 50 years model, which is about to crumble. Now, we have our revolution. (Ok, ok, I am a little bit too enthusiastic, I now need a fallout shelter).

Supporting Huawei Ascend 950 NPUs with open source inference code has huge implications in the global AI hardware market. Imagine if all smaller labs now can build on top of this stack and add increasing support for more compatible hardware , software and models. This could seriously challenge NVidias death grip. Although inference is still only a part of all AI workloads and open sourcing training on Huawei chips would be even bigger.

Relevant: Deepseek reports its Revenue per GPU per hour at off peak prices (38$) [reddit link] https://www.reddit.com/r/DeepSeek/comments/1xanexq/deepseek_...

Also cool (from the translated screenshot since original is in Chineese https://zhuanlan.zhihu.com/p/2095560295930508953)

> the DeepSeek V4.1 Flash inference system has established three goals

> 1-..... 2-..... 3. Supports retaining KVCache for more than 72 hours

This is awesome

DeepSeek is truly making a difference on many levels, but the one most refreshing is how they just deliver once again with no fanfare or drama. Not even a website. Just hand us over the goods for free the DeepSeek way.

This month I changed my Claude Max Subscription to DeepSeek API. V4.1 Flash is fast and perfect for everything and refuses to almost nothing.

Hope DeepSeek starts month plans soon.

Let’s hope this is just the beginning of bringing the drama queens (Dario and Sam) closer to planet Earth.

Lot of comments proclaiming it’s over for Anthropic and OpenAI, but none really explaining why. Many replies asking why and being wholesale ignored. Love to see it.

Interesting how a comment asking about the reasoning behind releasing this as open source is getting attacked, as if doing a good thing should not ever be questioned. I do wonder of China's motives behind this as well.

The government would not allow this open work if it did not benefit them somehow, or perhaps they feel that keeping it closed is not very important right now (maybe because they are somewhat behind the frontier).

Comparing a university student Torvalds and state actors open sourcing code is pretty naive...

20 merge requests in the last two hours. And all look somewhat reasonable on first glance.

I am happy for not maintaining a relevant open-source repo.

Back in September HN had a great discussion on how different DeepSeek providers have very different performance:

https://mmoustafa.com/blog/so-you-want-to-use-openrouter/

And I was speculating that it was likely due to hosting differences like this. It would be fascinating to see if this would improve, for example, tool calling, something that third-party DeepSeek providers never do as well as DeepSeek themselves.

For everyone wants to deeper understanding of the repo here is an audio overview: https://notebooklm.link.google/vxayNvBVmi5V

I wonder how the US AI business model feels about all this. Seeing all the openness in the space and inference costs going down I'm not sure how they'll justify multitrillion dollar valuations. It's gonna be hard to do so based on frontier models alone.

And I don't think they're worth that and even less ethically. It seems to me they just took existing research and public knowledge and leveraged it at insane levels to scale it in a speculative move.

For comparison, rental prices seem to be ~$2.75/ per H-100-hour [1]. Meanwhile, Deepseek says "38 USD per hour per card”. They haven’t specified which card as far as I can tell.

[1] https://www.ft.com/content/a1815bc5-b5d5-47ff-bbbc-e1a689929...

I'm hoping that hosting providers such as Hetzner will soon offer inference at reasonable prices and hosted in the EU with this kind of framework.

I didn't expect to find myself aligned with the CCP on open source, but in 2026, here we are. Strange times (but good).

Curious how the performance compares to vLLM or SGLang!

They say it was inspired by both so I am wondering what the performance gain is.

Maybe OpenAI can use it so I stop getting this 'Model temporarily unavailable' error

Notably missing their kv cache system

For those saying this creates a threat to Nvidia's deathgrip on the market, it really doesn't

The main reason people don't use Huawei hardware is due to regulation. If you're in the US you'd straight up get your hardware seized by customs because they're illegal to buy or bring into the country. EU has similar guidelines I believe.

Nvidia will continue to have its monopoly as long as the powers that be mandate it.

I must say this is the sort of war I can get behind, where one side seems to be offering us ever fluffier socks and back rubs and the other shouts into the wind that only their flavor of wholesale intellectual theft is moral.

I was under the impression mixture of experts models allow, during the processing layer 37 for 1 token for 1 user, only a subset of layer 37's weights need be read from ram - dramatically reducing memory bandwidth, since weights are far bigger than activations.

However, if that machine is processing requests for 10k users right now, they'll all need different experts, and the machine will end up reading every weight anyway.

I was expecting big providers to split the work between machines, such that machine 1 processes experts 1-100, machine 2 does experts 101-200 etc. Each request would be sent to the specific machines for the experts needed, and results collected again ready for the next layer (on another set of machines).

However I don't see code for any of that.

Missed optimization, or am I wrong about the best way to slice the math into the hardware?

Fun wrinkle with today's release: a US host that adopts DeepInfer to serve anything is now running a DeepSeek product.

I don't see any hosting loophole in that. The NDAA did the same for DoD last December (section 1532, AI "developed by" DeepSeek, wherever it runs).

And people wonder why I support and wish the Chinese models/companies succeed?

This is really cool. But, from what I can tell, its not a huge game changer. Anyone was able to run the open weights model locally anyway. This is just higher performance.

Unless someone disagrees.

One of the fascinating dynamics around Chinese open-weights models is the competition among American (and some European) inference providers to deliver the best inference performance on them. They've increasingly beaten the Chinese model makers themselves on inference throughput and latency.

Together.ai, Baseten, and others have already achieved higher throughput, lower latency, and lower E2E latency for DeepSeek V4.1 Flash 1017 than DeepSeek's own inference service.

https://openrouter.ai/deepseek/deepseek-v4.1-flash-1017

The Real OpenAI

> Required for a complete inference service (not included in this release)

Those items are pretty important…

I can see Deepseek becoming the standard and other providers will build on Deepseek. Like the Linux kernel and its distros.

The smaller providers on OpenRouter are probably popping champagne rn. With a framework like this in hand, the barrier to entry for LLM hosting drops even lower, so expect another wave of api price cuts for all no-frontier models

Supported hardware: Huawei Ascend.

Deepseek moment.

Does anyone have rough numbers on how this compares to vLLM in terms of price performance? Maybe with something like Dynamo for disaggregated serving and KV-aware routing.

I'm curious if it's a modest increase in performance or a significant one.

AND, presumably, this now and more and more on Chinese chips, mind: "We are profitable, here are the spoils: We are sharing the results of our research and the actual mechanics. And, we do not need you".-

This is fantastic. I feel the KV cache system is basically a must for any serious inference hosting, both for the provider as well as the consumer. I guess they have this but just haven’t open sourced it yet.

If anyone is interested in getting this running on sm80s, please let me know. :)

In response, the US government has just banned DeepSeek from parts of the federal government. Here goes the duopoly capture.

Notice of Order Regarding Exclusion and Removal of DeepSeek Products From Federal Civilian Executive Branch Information Technology Systems and Procurements

https://www.federalregister.gov/documents/2026/10/20/2026-21...

the banning of sending advanced chips, EUV machines to slow down the chinese was an own goal by the USA and the west.

zAI served their latest model for free on chinese chips, now deep seek is saying run inference for cheap on Huawei chips.

even the RAM madness is about to end once the chinese ram providers unleash Ram onto the market in 2027.

lastly remember the chinese have the largest electricity production and the cheapest.

Can somebody explain like I'm five what this exactly is? How are providers like Fireworks or Novita providing inference today? Do they use some other inference engine software? If yes, what is the advantage of the DeepSeek inference engine?

Is it in Debian yet?

It would be cool if we could get a rough idea of how much human effort/time/labour went into this.

so many new small AI labs will be born because of this. what a time to be alive

Even though their API price is cheap compared to OpenAI/Anthropic, I still can’t “afford” to use their API because the subscriptions from OAI/Anthropic provide far higher value and peace of mind. I know for sure how much I need to pay each month. I believe most developers fall into this category as well, where $20/$100/$200 from Anthropic/OAI provides more value.

This raises the question: why doesn’t DeepSeek provide a subscription service as well? Is it a better move to have a single pricing model like they currently do?

If only we can buy Huawei Ascends!

What's the difference between this and a typical inference engine like vLLM?

Kinda funny to see hackernews clamouring to ban the Chinese models. Truly in the thrall of the frontier labs

I think the reason DeepSeek does this is partly to neutralize competitors' technology, but ultimately the core reason might be that data is what matters.

they must know something is coming from the american labs for some time, you don't release this just like that!

If that's the thing that makes $0.00X cached inputs possible, then this is huge!

The quality of technical discussions here has clearly nosedived over the past few years. This release basically confirms it.

Open source is different from open weights

Well played!

I don't understand what is in it for them. I understand that the government wants openness but who's paying them to give away their secrets?

DeepSeek doesn't really care about models, they're aiming to become a platform: https://try.works/a-follow-up-on-china's-open-ai-race

They open source it because they don't need it anymore. Probably they are doing something better with Huawei

the only advantage i see long run is they can get free contributions world wide to lower their engineering costs. In the short run, frontier labs gets to use some of the ideas.

Need to see a Strata style take on this.

hi

Just another tour de force from DeepSeek.

I'm not saying that the US labs (Anthropic, OpenAI, and Meta) aren't pulling their weight, they are all doing amazing work as well, but Deepseek may belong in that tier.

So what? We already have multiple open source inference engines for DeepSeek models - that's how there are over 30 providers already running DeepSeek around the world. There's no secret sauce here, they've been publishing everything they do in papers from day 1.

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

[dead]

[flagged]

[flagged]

Deepseek released just enough to stir the pot but not enough to cook your own from scratch.

Wake me up when we can clone, point it at a dozen H100s we own, and get a working production service.

Anthropic and OpenAI will do just fine.