FreeLLMDeepSeek open‑sources DeepInfer inference stack, claims 155 trillion tokens per day
DeepSeek releases DeepInfer, an open‑source framework for large‑scale inference that it says handles 155 T tokens daily with high cache hit rates and strong margins.
DeepSeek announced the open‑source release of DeepInfer, the inference engine used for its V4.1 Flash model.
The company reports that a single cluster processes 155 trillion tokens per day, achieves 96 % input‑cache hits, and delivers an average first‑token latency of 400 ms.
The repository includes optimized kernels such as FlashMLA and DeepGEMM, and requires Linux, GCC 13, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend) to build.
What it means for you
For developers, DeepInfer offers a production‑grade, open‑source stack that can be adapted for offline serving, though key KV‑cache and load‑balancing components remain closed. Users of free or locally hosted models can experiment with the framework on standard GPU or NPU hardware, but must supply their own orchestration for full‑scale deployment.
Sources
- [AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback, Latent Space, 21 Oct 2026
- deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models, GitHub trending (official), 16 Oct 2026
Written automatically from the sources above, then checked by our code: every number and name must appear in them, and no sentence may copy a source. Spotted a mistake? Report it.
Which AI can you use for free today?
Every free AI model, ranked by blind human votes, with the memory and speed it needs on your own GPU.