deepseek-recipe

I highly recommend checking out the deepseek-recipe repo if you're building a recipe that involves any deepseek model. It allows for your recipe to handle requests better and behave more closely with official API. deepseek-ai/deepseek-recipe

DeepSeek Did It Again (And It’s Free on GitHub) DeepSeek did it again, and this time the engine that runs its cheap AI is free on GitHub. I go through what it released and how it gets so much out of each chip, using DeepSeek's own numbers. Then I look at what that means if you use AI. Chapters: 00:00 Intro 00:38 2025: the last time 01:35 April 2025: the engine stays closed 01:54 DeepInfer and deepseek-recipe on GitHub 02:23 The numbers from DeepSeek's servers 03:11 Trick 1: splitting the work 04:22 Trick 2: keeping every GPU busy 05:11 Trick 3: DSpark 06:11 Trick 4: the KV cache 07:40 The chips it runs on 08:03 The DHS order 08:30 What it changes for you 09:10 Recap Resources: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek open infra (2025 cost and revenue post, April 2025 engine post): deepseek-ai/open-infra-index DeepSeek API pricing: api-docs.deepseek.com DHS order (Federal Register 2026-21447): federalregister.gov Subscribe for more videos like this. Facebook: facebook.com X: x.com [ar] DeepSeek فعلتها مجدداً (وأصبحت متاحة مجاناً على GitHub) أبدعت شركة DeepSeek مجددًا، وهذه المرة، المحرك الذي يُشغّل ذكاءها الاصطناعي منخفض التكلفة متاح مجانًا على GitHub. سأستعرض ما أصدرته وكيف تستغل كل شريحة بكفاءة عالية، مستخدمًا بيانات DeepSeek نفسها. ثم سأتناول دلالات ذلك عند استخدام الذكاء الاصطناعي. الفصول: 00:00 مقدمة 00:38 2025: آخر مرة 01:35 أبريل 2025: يبقى المحرك مغلقًا 01:54 DeepInfer و deepseek-recipe على GitHub 02:23 الأرقام من خوادم DeepSeek 03:11 الحيلة 1: تقسيم العمل 04:22 الحيلة 2: إبقاء جميع وحدات معالجة الرسومات مشغولة 05:11 الحيلة 3: DSpark 06:11 الحيلة 4: ذاكرة التخزين المؤقت KV 07:40 الرقائق التي يعمل عليها 08:03 طلب DHS 08:30 ما الذي يغيره بالنسبة لك 09:10 ملخص الموارد: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe البنية التحتية المفتوحة لـ DeepSeek (بيانات التكلفة والإيرادات لعام 2025، وبيانات المحرك لشهر أبريل 2025): deepseek-ai/open-infra-index أسعار واجهة برمجة تطبيقات DeepSeek: api-docs.deepseek.com أمر وزارة الأمن الداخلي (السجل الفيدرالي 2026-21447): federalregister.gov اشترك للمزيد من الفيديوهات المشابهة. فيسبوك: facebook.com X: x.com [bn] DeepSeek আবারও করে দেখাল (এবং এটি গিটহাবে বিনামূল্যে পাওয়া যাচ্ছে) DeepSeek আবারও চমক দেখিয়েছে, এবং এবার তাদের সাশ্রয়ী AI চালনাকারী ইঞ্জিনটি GitHub-এ বিনামূল্যে পাওয়া যাচ্ছে। আমি DeepSeek-এর নিজস্ব পরিসংখ্যান ব্যবহার করে আলোচনা করব যে তারা কী প্রকাশ করেছে এবং কীভাবে এটি প্রতিটি চিপ থেকে এত বেশি কার্যকারিতা অর্জন করে। তারপর আমি দেখব, আপনি যদি AI ব্যবহার করেন তবে এর অর্থ কী দাঁড়ায়। অধ্যায়সমূহ: ০০:০০ ভূমিকা ০০:৩৮ ২০২৫: শেষবার ০১:৩৫ এপ্রিল ২০২৫: ইঞ্জিনটি বন্ধ থাকবে ০১:৫৪ গিটহাবে DeepInfer এবং deepseek-recipe ০২:২৩ DeepSeek-এর সার্ভার থেকে প্রাপ্ত সংখ্যাসমূহ ০৩:১১ কৌশল ১: কাজ ভাগ করে নেওয়া ০৪:২২ কৌশল ২: প্রতিটি GPU-কে ব্যস্ত রাখা ০৫:১১ কৌশল ৩: DSpark ০৬:১১ কৌশল ৪: KV ক্যাশে ০৭:৪০ যে চিপগুলোতে এটি চলে ০৮:০৩ DHS অর্ডার ০৮:৩০ এটি আপনার জন্য কী পরিবর্তন আনে ০৯:১০ সারসংক্ষেপ সম্পদ: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek ওপেন ইনফ্রা (২০২৫ সালের খরচ ও আয় সংক্রান্ত পোস্ট, এপ্রিল ২০২৫-এর ইঞ্জিন পোস্ট): deepseek-ai/open-infra-index ডিপসিক এপিআই-এর মূল্য: api-docs.deepseek.com ডিএইচএস-এর আদেশ (ফেডারেল রেজিস্টার ২০২৬-২১৪৪৭): federalregister.gov এই ধরনের আরও ভিডিওর জন্য সাবস্ক্রাইব করুন। ফেসবুক: facebook.com এক্স: x.com [de-DE] DeepSeek hat es wieder getan (und es ist kostenlos auf GitHub verfügbar) DeepSeek hat es wieder getan, und dieses Mal ist die Engine, die die kostengünstige KI antreibt, kostenlos auf GitHub verfügbar. Ich erkläre anhand von DeepSeeks eigenen Angaben, was genau veröffentlicht wurde und wie die Engine so viel Leistung aus jedem Chip herausholt. Anschließend betrachte ich die Auswirkungen auf die KI-Nutzung. ... Kapitel: 00:00 Einleitung 00:38 2025: Das letzte Mal 01:35 April 2025: Die Engine bleibt geschlossen 01:54 DeepInfer und deepseek-recipe auf GitHub 02:23 Die Zahlen von den DeepSeek-Servern 03:11 Trick 1: Arbeitsteilung 04:22 Trick 2: Alle GPUs ausgelastet halten 05:11 Trick 3: DSpark 06:11 Trick 4: Der KV-Cache 07:40 Die verwendeten Chips 08:03 Die DHS-Bestellung 08:30 Was sich für Sie ändert 09:10 Zusammenfassung Ressourcen: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek Open Infrastructure (Kosten- und Umsatzprognose 2025, Engine-Prognose April 2025): deepseek-ai/open-infra-index Preise der DeepSeek-API: api-docs.deepseek.com Anordnung des US-Heimatschutzministeriums (Federal Register 2026-21447): federalregister.gov Abonnieren Sie unseren Kanal für weitere Videos dieser Art. Facebook: facebook.com X: x.com [es-US] DeepSeek lo volvió a hacer (y está gratis en GitHub) DeepSeek lo ha vuelto a hacer, y esta vez el motor que impulsa su IA de bajo coste está disponible gratuitamente en GitHub. Analizo lo que ha publicado y cómo consigue sacar tanto partido a cada chip, utilizando datos de DeepSeek. Después, examino qué implicaciones tiene esto para el uso de la IA. Capítulos: 00:00 Introducción 00:38 2025: la última vez 01:35 Abril de 2025: el motor permanece cerrado 01:54 DeepInfer y deepseek-recipe en GitHub 02:23 Datos de los servidores de DeepSeek 03:11 Truco 1: dividir el trabajo 04:22 Truco 2: mantener todas las GPU ocupadas 05:11 Truco 3: DSpark 06:11 Truco 4: la caché KV 07:40 Los chips en los que se ejecuta 08:03 La orden DHS 08:30 Qué cambia para ti 09:10 Resumen Recursos: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe Infraestructura abierta de DeepSeek (Publicación sobre costos e ingresos de 2025, publicación sobre el motor de abril de 2025): deepseek-ai/open-infra-index Precios de la API de DeepSeek: api-docs.deepseek.com Orden del DHS (Registro Federal 2026-21447): federalregister.gov Suscríbete para ver más videos como este. Facebook: facebook.com X: x.com [fr-FR] DeepSeek a récidivé (et c'est gratuit sur GitHub) DeepSeek a encore frappé fort, et cette fois-ci, le moteur qui fait tourner son IA à bas coût est disponible gratuitement sur GitHub. J'analyse les fonctionnalités publiées et comment le moteur tire le meilleur parti de chaque puce, en m'appuyant sur les chiffres fournis par DeepSeek. J'examine ensuite les implications pour l'utilisation de l'IA. Chapitres : 00:00 Introduction 00:38 2025 : la dernière fois 01:35 Avril 2025 : le moteur reste fermé 01:54 DeepInfer et deepseek-recipe sur GitHub 02:23 Les chiffres des serveurs de DeepSeek 03:11 Astuce 1 : répartition du travail 04:22 Astuce 2 : utilisation optimale de chaque GPU 05:11 Astuce 3 : DSpark 06:11 Astuce 4 : le cache KV 07:40 Les puces utilisées 08:03 L’ordre du DHS 08:30 Conséquences 09:10 Récapitulatif Ressources : DeepInfer : deepseek-ai/DeepInfer deepseek-recipe : deepseek-ai/deepseek-recipe Infrastructure ouverte de DeepSeek (Article sur les coûts et les revenus de 2025, article sur le moteur d'avril 2025) : deepseek-ai/open-infra-index Tarifs de l'API DeepSeek : api-docs.deepseek.com Arrêté du DHS (Federal Register 2026-21447) : federalregister.gov Abonnez-vous pour plus de vidéos comme celle-ci. Facebook : facebook.com X : x.com [hi] DeepSeek ने फिर कमाल कर दिया (और यह GitHub पर मुफ्त उपलब्ध है) DeepSeek ने एक बार फिर कमाल कर दिखाया है, और इस बार इसका सस्ता AI इंजन GitHub पर मुफ़्त में उपलब्ध है। मैं DeepSeek के खुद के आंकड़ों का इस्तेमाल करते हुए, इसके द्वारा जारी की गई सुविधाओं और प्रत्येक चिप से इतना अधिक लाभ कैसे प्राप्त किया जाता है, इस पर चर्चा करूँगा। फिर मैं देखूंगा कि AI का उपयोग करने पर इसका क्या अर्थ है। अध्याय: 00:00 परिचय 00:38 2025: आखिरी बार 01:35 अप्रैल 2025: इंजन बंद रहेगा 01:54 GitHub पर DeepInfer और deepseek-recipe 02:23 DeepSeek के सर्वरों से प्राप्त आंकड़े 03:11 तरकीब 1: काम को बांटना 04:22 तरकीब 2: हर GPU को व्यस्त रखना 05:11 तरकीब 3: DSpark 06:11 तरकीब 4: KV कैश 07:40 जिन चिप्स पर यह चलता है 08:03 DHS ऑर्डर 08:30 इससे आपके लिए क्या बदलाव आते हैं 09:10 सारांश संसाधन: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek ओपन इंफ्रास्ट्रक्चर (2025 की लागत और राजस्व संबंधी जानकारी, अप्रैल 2025 में इंजन संबंधी जानकारी): deepseek-ai/open-infra-index DeepSeek API की कीमत: api-docs.deepseek.com DHS आदेश (फेडरल रजिस्टर 2026-21447): federalregister.gov इस तरह के और वीडियो के लिए सब्सक्राइब करें। फेसबुक: facebook.com X: x.com [id] DeepSeek Berulah Lagi (Dan Tersedia Gratis di GitHub) DeepSeek melakukannya lagi, dan kali ini mesin yang menjalankan AI murahnya tersedia gratis di GitHub. Saya akan membahas apa yang telah dirilis dan bagaimana DeepSeek mampu memaksimalkan setiap chip, menggunakan angka-angka dari DeepSeek sendiri. Kemudian saya akan melihat apa artinya jika Anda menggunakan AI. Bab: 00:00 Pendahuluan 00:38 2025: terakhir kali 01:35 April 2025: mesin tetap tertutup 01:54 DeepInfer dan deepseek-recipe di GitHub 02:23 Angka-angka dari server DeepSeek 03:11 Trik 1: membagi pekerjaan 04:22 Trik 2: menjaga setiap GPU tetap sibuk 05:11 Trik 3: DSpark 06:11 Trik 4: cache KV 07:40 Chip yang digunakan 08:03 Perintah DHS 08:30 Apa yang berubah untuk Anda 09:10 Rekap Sumber Daya: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek open infra (2025) (Postingan biaya dan pendapatan, April 2025: deepseek-ai/open-infra-index) Harga API DeepSeek: api-docs.deepseek.com Perintah DHS (Federal Register 2026-21447): federalregister.gov Berlangganan untuk video lainnya seperti ini. Facebook: facebook.com X: x.com [it] DeepSeek l'ha fatto di nuovo (ed è gratis su GitHub) DeepSeek l'ha fatto di nuovo, e questa volta il motore che alimenta la sua IA a basso costo è disponibile gratuitamente su GitHub. Analizzo i risultati del rilascio e come riesce a sfruttare al massimo ogni singolo chip, utilizzando i dati forniti dalla stessa DeepSeek. Infine, esamino le implicazioni di tutto ciò per chi utilizza l'IA. Capitoli: 00:00 Introduzione 00:38 2025: l'ultima volta 01:35 Aprile 2025: il motore rimane chiuso 01:54 DeepInfer e deepseek-recipe su GitHub 02:23 I numeri dai server di DeepSeek 03:11 Trucco 1: suddividere il lavoro 04:22 Trucco 2: tenere occupate tutte le GPU 05:11 Trucco 3: DSpark 06:11 Trucco 4: la cache KV 07:40 I chip su cui gira 08:03 L'ordine DHS 08:30 Cosa cambia per te 09:10 Riepilogo Risorse: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: Italiano: deepseek-ai/deepseek-recipe Infrastruttura aperta di DeepSeek (post su costi e ricavi del 2025, post sul motore di aprile 2025): deepseek-ai/open-infra-index Prezzi dell'API di DeepSeek: api-docs.deepseek.com Ordinanza DHS (Federal Register 2026-21447): federalregister.gov Iscriviti per altri video come questo. Facebook: facebook.com X: x.com [iw] DeepSeek עשו זאת שוב (וזה זמין בחינם ב-GitHub) DeepSeek עשתה זאת שוב, והפעם המנוע שמפעיל את הבינה המלאכותית הזולה שלה זמין בחינם ב-GitHub. אני עובר על מה שהיא הוציאה ואיך היא מוציאה כל כך הרבה מכל שבב, תוך שימוש במספרים של DeepSeek עצמה. לאחר מכן אני בוחן מה זה אומר אם משתמשים בבינה מלאכותית. פרקים: 00:00 מבוא 00:38 2025: הפעם האחרונה 01:35 אפריל 2025: המנוע נשאר סגור 01:54 DeepInfer ו-deepseek-recipe ב-GitHub 02:23 המספרים משרתי DeepSeek 03:11 טריק 1: חלוקת העבודה 04:22 טריק 2: שמירה על כל GPU עסוק 05:11 טריק 3: DSpark 06:11 טריק 4: מטמון KV 07:40 השבבים שעליהם הוא פועל 08:03 פקודת DHS 08:30 מה זה משנה עבורך 09:10 סיכום משאבים: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek פתוחה (פוסט על עלויות והכנסות 2025, פוסט מנוע אפריל 2025): deepseek-ai/open-infra-index תמחור API של DeepSeek: api-docs.deepseek.com צו DHS (Federal Register 2026-21447): federalregister.gov הירשמו לסרטונים נוספים כמו זה. פייסבוק: facebook.com X: x.com [ja] DeepSeekがまたやった（しかもGitHubで無料公開） DeepSeekがまたやってくれました。今回は、同社の低価格AIを支えるエンジンをGitHubで無料公開しています。公開された内容と、DeepSeek自身が公表した数値を用いて、各チップからいかに高い性能を引き出しているのかを解説します。そして、AIを活用する上で、それがどのような意味を持つのかを考察します。 目次： 00:00 はじめに 00:38 2025年：最後のチャンス 01:35 2025年4月：エンジンは閉鎖されたまま 01:54 GitHub上のDeepInferとdeepseek-recipe 02:23 DeepSeekサーバーのデータ 03:11 テクニック1：処理の分割 04:22 テクニック2：すべてのGPUを常に稼働させる 05:11 テクニック3：DSpark 06:11 テクニック4：KVキャッシュ 07:40 動作チップ 08:03 DHS命令 08:30 あなたにとっての変化 09:10 まとめ リソース： DeepInfer：deepseek-ai/DeepInfer deepseek-recipe：deepseek-ai/deepseek-recipe DeepSeek公開インフラ（2025年のコストと収益に関する記事、2025年4月のエンジンに関する記事）：deepseek-ai/open-infra-index DeepSeek API料金：api-docs.deepseek.com 国土安全保障省命令（連邦官報2026-21447）：federalregister.gov このような動画をもっとご覧になりたい方は、チャンネル登録をお願いします。 Facebook：facebook.com X：x.com [ko] DeepSeek이 또 해냈다 (심지어 GitHub에 무료 공개) DeepSeek이 또 ​​한 번 일을 냈습니다. 이번에는 저렴한 AI를 구동하는 엔진을 GitHub에서 무료로 공개했습니다. DeepSeek이 공개한 내용과 각 칩에서 어떻게 그렇게 많은 성능을 끌어내는지 DeepSeek 자체 데이터를 바탕으로 살펴보겠습니다. 그리고 이것이 AI를 사용할 때 어떤 의미를 갖는지도 알아보겠습니다. 목차: 00:00 소개 00:38 2025년: 마지막 시도 01:35 2025년 4월: 엔진은 계속 닫혀 있음 01:54 GitHub의 DeepInfer 및 deepseek-recipe 02:23 DeepSeek 서버 데이터 03:11 첫 번째 비결: 작업 분산 04:22 두 번째 비결: 모든 GPU를 항상 가동 05:11 세 번째 비결: DSpark 06:11 네 번째 비결: KV 캐시 07:40 DeepSeek이 실행되는 칩 08:03 DHS 주문 08:30 사용자에게 미치는 영향 09:10 요약 참고 자료: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek 오픈 인프라(2025년 비용 및 수익 보고서, 2025년 4월 엔진 출시 보고서): deepseek-ai/open-infra-index DeepSeek API 가격: api-docs.deepseek.com 국토안보부 명령(연방 관보 2026-21447): federalregister.gov 더 많은 영상을 보려면 구독하세요. 페이스북: facebook.com X: x.com [ml] DeepSeek വീണ്ടും അത് ആവർത്തിച്ചു (ഇത് GitHub-ൽ സൗജന്യവുമാണ്) DeepSeek വീണ്ടും അത് ചെയ്തു, ഇത്തവണ വിലകുറഞ്ഞ AI പ്രവർത്തിപ്പിക്കുന്ന എഞ്ചിൻ GitHub-ൽ സൗജന്യമാണ്. DeepSeek-ന്റെ സ്വന്തം നമ്പറുകൾ ഉപയോഗിച്ച്, അത് എന്താണ് പുറത്തിറക്കിയതെന്നും ഓരോ ചിപ്പിൽ നിന്നും അത് എങ്ങനെ ഇത്രയധികം പ്രയോജനപ്പെടുത്തുന്നുവെന്നും ഞാൻ പരിശോധിക്കുന്നു. തുടർന്ന് നിങ്ങൾ AI ഉപയോഗിക്കുകയാണെങ്കിൽ അതിന്റെ അർത്ഥമെന്താണെന്ന് ഞാൻ നോക്കുന്നു. അധ്യായങ്ങൾ: 00:00 ആമുഖം 00:38 2025: അവസാന സമയം 01:35 ഏപ്രിൽ 2025: എഞ്ചിൻ അടച്ചിട്ടിരിക്കുന്നു 01:54 GitHub-ൽ DeepInfer ഉം deepseek-recipe ഉം 02:23 DeepSeek-ന്റെ സെർവറുകളിൽ നിന്നുള്ള നമ്പറുകൾ 03:11 തന്ത്രം 1: ജോലി വിഭജിക്കൽ 04:22 തന്ത്രം 2: ഓരോ GPU-യും തിരക്കിലാക്കി നിർത്തൽ 05:11 തന്ത്രം 3: DSpark 06:11 തന്ത്രം 4: KV കാഷെ 07:40 അത് പ്രവർത്തിക്കുന്ന ചിപ്പുകൾ 08:03 DHS ഓർഡർ 08:30 നിങ്ങൾക്കായി ഇത് എന്താണ് മാറ്റുന്നത് 09:10 സംഗ്രഹം ഉറവിടങ്ങൾ: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek ഓപ്പൺ ഇൻഫ്രാ (2025 ചെലവ്, വരുമാന പോസ്റ്റ്, ഏപ്രിൽ 2025 എഞ്ചിൻ പോസ്റ്റ്): deepseek-ai/open-infra-index DeepSeek API വിലനിർണ്ണയം: api-docs.deepseek.com DHS ഓർഡർ (ഫെഡറൽ രജിസ്റ്റർ 2026-21447): federalregister.gov ഇതുപോലുള്ള കൂടുതൽ വീഡിയോകൾക്കായി സബ്‌സ്‌ക്രൈബുചെയ്യുക. ഫേസ്ബുക്ക്: facebook.com X: x.com [nl-NL] DeepSeek heeft het weer geflikt (en het staat gratis op GitHub) DeepSeek heeft het weer voor elkaar gekregen, en dit keer is de engine die hun goedkope AI aandrijft gratis beschikbaar op GitHub. Ik bespreek wat ze hebben uitgebracht en hoe ze zoveel uit elke chip halen, aan de hand van DeepSeek's eigen cijfers. Vervolgens bekijk ik wat dat betekent voor het gebruik van AI. Hoofdstukken: 00:00 Intro 00:38 2025: de laatste keer 01:35 April 2025: de engine blijft gesloten 01:54 DeepInfer en deepseek-recipe op GitHub 02:23 De cijfers van de DeepSeek-servers 03:11 Truc 1: het werk verdelen 04:22 Truc 2: elke GPU bezig houden 05:11 Truc 3: DSpark 06:11 Truc 4: de KV-cache 07:40 De chips waarop het draait 08:03 De DHS-volgorde 08:30 Wat het voor jou verandert 09:10 Samenvatting Bronnen: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek open infrastructuur (Kosten- en inkomstenbericht 2025, enginebericht april 2025): deepseek-ai/open-infra-index Prijzen van de DeepSeek API: api-docs.deepseek.com Besluit van het Department of Homeland Security (Federal Register 2026-21447): federalregister.gov Abonneer je voor meer video's zoals deze. Facebook: facebook.com X: x.com [pa] DeepSeek ਨੇ ਫਿਰ ਕਮਾਲ ਕਰ ਦਿੱਤਾ (ਅਤੇ ਇਹ GitHub 'ਤੇ ਮੁਫ਼ਤ ਹੈ) ਡੀਪਸੀਕ ਨੇ ਇਹ ਫਿਰ ਕੀਤਾ, ਅਤੇ ਇਸ ਵਾਰ ਇਸਦਾ ਸਸਤਾ ਏਆਈ ਚਲਾਉਣ ਵਾਲਾ ਇੰਜਣ ਗਿਟਹਬ 'ਤੇ ਮੁਫਤ ਹੈ। ਮੈਂ ਡੀਪਸੀਕ ਦੇ ਆਪਣੇ ਨੰਬਰਾਂ ਦੀ ਵਰਤੋਂ ਕਰਦੇ ਹੋਏ ਦੇਖਦਾ ਹਾਂ ਕਿ ਇਸਨੇ ਕੀ ਜਾਰੀ ਕੀਤਾ ਹੈ ਅਤੇ ਇਹ ਹਰੇਕ ਚਿੱਪ ਤੋਂ ਇੰਨਾ ਕੁਝ ਕਿਵੇਂ ਪ੍ਰਾਪਤ ਕਰਦਾ ਹੈ। ਫਿਰ ਮੈਂ ਦੇਖਦਾ ਹਾਂ ਕਿ ਜੇਕਰ ਤੁਸੀਂ ਏਆਈ ਦੀ ਵਰਤੋਂ ਕਰਦੇ ਹੋ ਤਾਂ ਇਸਦਾ ਕੀ ਅਰਥ ਹੈ। ਅਧਿਆਏ: 00:00 ਜਾਣ-ਪਛਾਣ 00:38 2025: ਆਖਰੀ ਵਾਰ 01:35 ਅਪ੍ਰੈਲ 2025: ਇੰਜਣ ਬੰਦ ਰਹਿੰਦਾ ਹੈ 01:54 GitHub 'ਤੇ DeepInfer ਅਤੇ deepseek-recipe 02:23 DeepSeek ਦੇ ਸਰਵਰਾਂ ਤੋਂ ਨੰਬਰ 03:11 ਚਾਲ 1: ਕੰਮ ਨੂੰ ਵੰਡਣਾ 04:22 ਚਾਲ 2: ਹਰੇਕ GPU ਨੂੰ ਵਿਅਸਤ ਰੱਖਣਾ 05:11 ਚਾਲ 3: DSpark 06:11 ਚਾਲ 4: KV ਕੈਸ਼ 07:40 ਉਹ ਚਿਪਸ ਜਿਨ੍ਹਾਂ 'ਤੇ ਇਹ ਚੱਲਦਾ ਹੈ 08:03 DHS ਆਰਡਰ 08:30 ਇਹ ਤੁਹਾਡੇ ਲਈ ਕੀ ਬਦਲਦਾ ਹੈ 09:10 ਸੰਖੇਪ ਸਰੋਤ: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek ਹੇਠਾਂ ਖੋਲ੍ਹੋ (2025 ਦੀ ਲਾਗਤ ਅਤੇ ਆਮਦਨੀ ਪੋਸਟ, ਅਪ੍ਰੈਲ 2025 ਇੰਜਣ ਪੋਸਟ): deepseek-ai/open-infra-index DeepSeek API ਕੀਮਤ: api-docs.deepseek.com DHS ਆਰਡਰ (ਫੈਡਰਲ ਰਜਿਸਟਰ 2026-21447): federalregister.gov ਇਸ ਤਰ੍ਹਾਂ ਦੇ ਹੋਰ ਵੀਡੀਓਜ਼ ਲਈ ਸਬਸਕ੍ਰਾਈਬ ਕਰੋ। ਫੇਸਬੁੱਕ: facebook.com X: x.com [pl] DeepSeek znów to zrobiło (i udostępniło za darmo na GitHubie) DeepSeek znów to zrobił i tym razem silnik, który napędza jego tanią sztuczną inteligencję, jest dostępny za darmo na GitHubie. Przeanalizuję, co firma wydała i jak tak wiele z każdego układu scalonego wyciąga, korzystając z własnych danych DeepSeek. Następnie przyjrzę się, co to oznacza w przypadku użycia sztucznej inteligencji. Rozdziały: 00:00 Wprowadzenie 00:38 2025: ostatni raz 01:35 Kwiecień 2025: silnik pozostaje zamknięty 01:54 DeepInfer i deepseek-recipe na GitHubie 02:23 Liczby z serwerów DeepSeek 03:11 Sztuczka 1: podział pracy 04:22 Sztuczka 2: zajęcie wszystkich procesorów graficznych 05:11 Sztuczka 3: DSpark 06:11 Sztuczka 4: pamięć podręczna KV 07:40 Chipy, na których działa 08:03 Kolejność DHS 08:30 Co to dla Ciebie zmienia 09:10 Podsumowanie Zasoby: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek open infra (dane o kosztach i przychodach z 2025 r., dane o silniku z kwietnia 2025 r.): deepseek-ai/open-infra-index Cennik API DeepSeek: api-docs.deepseek.com Zarządzenie DHS (Federal Register 2026-21447): federalregister.gov Subskrybuj, aby otrzymywać więcej takich filmów. Facebook: facebook.com X: x.com [pt-BR] DeepSeek Fez Novamente (E Está Disponível Gratuitamente no GitHub) A DeepSeek fez isso de novo, e desta vez o mecanismo que executa sua IA de baixo custo está disponível gratuitamente no GitHub. Analiso o que foi lançado e como a empresa consegue extrair tanto de cada chip, usando os próprios números da DeepSeek. Em seguida, examino o que isso significa para você, caso utilize IA. Capítulos: 00:00 Introdução 00:38 2025: a última vez 01:35 Abril de 2025: o mecanismo permanece fechado 01:54 DeepInfer e deepseek-recipe no GitHub 02:23 Os números dos servidores da DeepSeek 03:11 Truque 1: dividindo o trabalho 04:22 Truque 2: mantendo todas as GPUs ocupadas 05:11 Truque 3: DSpark 06:11 Truque 4: o cache KV 07:40 Os chips em que roda 08:03 A ordem DHS 08:30 O que isso muda para você 09:10 Resumo Recursos: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe Infraestrutura aberta da DeepSeek (Postagem sobre custos e receitas de 2025, postagem sobre o mecanismo de abril de 2025): deepseek-ai/open-infra-index Preços da API do DeepSeek: api-docs.deepseek.com Ordem do DHS (Federal Register 2026-21447): federalregister.gov Inscreva-se para mais vídeos como este. Facebook: facebook.com X: x.com [ru] DeepSeek снова это сделали (и теперь это бесплатно на GitHub) DeepSeek снова отличилась, и на этот раз движок, запускающий её недорогой ИИ, доступен бесплатно на GitHub. Я расскажу о том, что они выпустили и как им удаётся извлекать максимум пользы из каждого чипа, используя собственные данные DeepSeek. Затем я покажу, что это значит для использования ИИ. Разделы: 00:00 Введение 00:38 2025: последний раз 01:35 Апрель 2025: движок остаётся закрытым 01:54 DeepInfer и deepseek-recipe на GitHub 02:23 Данные с серверов DeepSeek 03:11 Трюк 1: разделение работы 04:22 Трюк 2: загрузка всех графических процессоров 05:11 Трюк 3: DSpark 06:11 Трюк 4: кэш KV 07:40 Чипы, на которых он работает 08:03 Порядок DHS 08:30 Что это меняет для вас 09:10 Итоги Ресурсы: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe Открытая инфраструктура DeepSeek (публикация о затратах и ​​доходах за 2025 год, публикация о движке за апрель 2025 года): deepseek-ai/open-infra-index Цены на API DeepSeek: api-docs.deepseek.com Распоряжение Министерства внутренней безопасности США (Федеральный реестр 2026-21447): federalregister.gov Подписывайтесь на канал, чтобы получать больше подобных видео. Facebook: facebook.com X: x.com [ta] DeepSeek மீண்டும் மிரட்டுகிறது (மற்றும் இது GitHub-இல் இலவசம்) டீப்ஸீக் மீண்டும் ஒருமுறை அசத்தியுள்ளது, இம்முறை அதன் மலிவான செயற்கை நுண்ணறிவை இயக்கும் எஞ்சின் கிட்ஹப்பில் இலவசமாகக் கிடைக்கிறது. டீப்ஸீக்கின் சொந்த எண்களைப் பயன்படுத்தி, அது வெளியிட்டவை மற்றும் ஒவ்வொரு சிப்பிலிருந்தும் அது எவ்வாறு இவ்வளவு செயல்திறனைப் பெறுகிறது என்பதை நான் விளக்குகிறேன். பின்னர், நீங்கள் செயற்கை நுண்ணறிவைப் பயன்படுத்தினால் அதன் அர்த்தம் என்ன என்பதையும் பார்க்கிறேன். அத்தியாயங்கள்: 00:00 அறிமுகம் 00:38 2025: கடைசி முறை 01:35 ஏப்ரல் 2025: இயந்திரம் மூடப்பட்டே இருக்கும் 01:54 GitHub-இல் DeepInfer மற்றும் deepseek-recipe 02:23 DeepSeek-இன் சர்வர்களிலிருந்து கிடைக்கும் எண்கள் 03:11 தந்திரம் 1: வேலையைப் பிரித்தல் 04:22 தந்திரம் 2: ஒவ்வொரு GPU-வையும் சுறுசுறுப்பாக வைத்திருத்தல் 05:11 தந்திரம் 3: DSpark 06:11 தந்திரம் 4: KV கேச் 07:40 அது இயங்கும் சிப்கள் 08:03 DHS ஆணை 08:30 அது உங்களுக்கு என்ன மாற்றங்களை ஏற்படுத்துகிறது 09:10 மீள்பார்வை வளங்கள்: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe DeepSeek திறந்துள்ளது உள்கட்டமைப்பு (2025 செலவு மற்றும் வருவாய் பதிவு, ஏப்ரல் 2025 இன்ஜின் பதிவு): deepseek-ai/open-infra-index டீப்ஸீக் ஏபிஐ விலை நிர்ணயம்: api-docs.deepseek.com டிஎச்எஸ் ஆணை (ஃபெடரல் ரெஜிஸ்டர் 2026-21447): federalregister.gov இதுபோன்ற மேலும் பல வீடியோக்களுக்கு சப்ஸ்கிரைப் செய்யுங்கள். ஃபேஸ்புக்: facebook.com எக்ஸ்: x.com [te] DeepSeek మళ్ళీ అదరగొట్టింది (మరియు ఇది GitHubలో ఉచితం) డీప్‌సీక్ మళ్ళీ అద్భుతం చేసింది, ఈసారి దాని చవకైన AIని నడిపే ఇంజిన్ గిట్‌హబ్‌లో ఉచితంగా లభిస్తోంది. డీప్‌సీక్ విడుదల చేసిన దాని గురించి, మరియు అది ప్రతి చిప్ నుండి ఎంత సామర్థ్యాన్ని రాబట్టుకుంటుందో, వారి సొంత గణాంకాలను ఉపయోగించి నేను వివరిస్తాను. ఆ తర్వాత, మీరు AIని ఉపయోగిస్తే దాని అర్థం ఏమిటో చూస్తాను. అధ్యాయాలు: 00:00 పరిచయం 00:38 2025: చివరిసారి 01:35 ఏప్రిల్ 2025: ఇంజిన్ మూసివేయబడుతుంది 01:54 గిట్‌హబ్‌లో డీప్‌ఇన్‌ఫర్ మరియు డీప్‌సీక్-రెసిపీ 02:23 డీప్‌సీక్ సర్వర్‌ల నుండి సంఖ్యలు 03:11 ట్రిక్ 1: పనిని విభజించడం 04:22 ట్రిక్ 2: ప్రతి GPUని బిజీగా ఉంచడం 05:11 ట్రిక్ 3: DSpark 06:11 ట్రిక్ 4: KV కాష్ 07:40 ఇది పనిచేసే చిప్‌లు 08:03 DHS ఆర్డర్ 08:30 ఇది మీ కోసం ఏమి మారుస్తుంది 09:10 పునశ్చరణ వనరులు: డీప్‌ఇన్‌ఫర్: deepseek-ai/DeepInfer డీప్‌సీక్-రెసిపీ: deepseek-ai/deepseek-recipe డీప్‌సీక్ ఓపెన్ ఇన్‌ఫ్రా (2025 ఖర్చు మరియు రాబడి పోస్ట్, ఏప్రిల్ 2025 ఇంజిన్ పోస్ట్): deepseek-ai/open-infra-index డీప్‌సీక్ API ధరలు: api-docs.deepseek.com DHS ఆర్డర్ (ఫెడరల్ రిజిస్టర్ 2026-21447): federalregister.gov ఇలాంటి మరిన్ని వీడియోల కోసం సబ్‌స్క్రైబ్ చేసుకోండి. ఫేస్‌బుక్: facebook.com X: x.com [uk] DeepSeek зробили це знову (і це безкоштовно на GitHub) DeepSeek знову це зробив, і цього разу движок, на якому працює його дешевий штучний інтелект, безкоштовний на GitHub. Я розглядаю, що вони випустили та як вони отримують так багато від кожного чіпа, використовуючи власні цифри DeepSeek. Потім я розглядаю, що це означає, якщо використовувати штучний інтелект. Розділи: 00:00 Вступ 00:38 2025: останній раз 01:35 Квітень 2025: двигун залишається закритим 01:54 DeepInfer та deepseek-recipe на GitHub 02:23 Цифри з серверів DeepSeek 03:11 Хитрощі 1: розподіл роботи 04:22 Хитрощі 2: завантаження кожного графічного процесора 05:11 Хитрощі 3: DSpark 06:11 Хитрощі 4: кеш KV 07:40 Чіпи, на яких він працює 08:03 Наказ DHS 08:30 Що це змінює для вас 09:10 Підсумки Ресурси: DeepInfer: deepseek-ai/DeepInfer deepseek-recipe: deepseek-ai/deepseek-recipe Відкрита інфраструктура DeepSeek (вартість та дохід за 2025 рік) публікація, публікація двигуна за квітень 2025 року): deepseek-ai/open-infra-index Ціни на DeepSeek API: api-docs.deepseek.com Наказ DHS (Федеральний реєстр 2026-21447): federalregister.gov Підпишіться, щоб переглядати більше подібних відео. Facebook: facebook.com X: x.com

DeepSeek Just Showed Everyone How to Make 80% Margins on AI (DeepInfer) Sources & further reading - DeepSeek — DeepInfer code, supported models, and release scope: deepseek-ai/DeepInfer - DeepSeek (@deepseek_ai) — launch thread and future-model commitments: x.com - DeepSeek — April 2025 explanation for not releasing the engine: github.com/deepseek-ai/open-infra-index/blob/main… - DeepSeek — deployment Q&A: cache retention, storage, routing, and revenue: github.com/deepseek-ai/DeepInfer/blob/main/doc/in… - DeepSeek — Chinese engineering write-up on Zhihu: zhuanlan.zhihu.com - DeepSeek — production statistics table used in our calculations: pic2.zhimg.com - DeepSeek — API pricing used in the revenue calculation: api-docs.deepseek.com - DeepSeek — V4.1-Flash technical report and 890-byte KV footprint: arxiv.org - Xiaoyin Qu (@quxiaoyin) — Claude Code session costs and the 13× lunch break: x.com - @om_patel5 — session audit and gaps longer than five minutes: x.com - Anthropic — prompt-cache lifetimes and write/read pricing: platform.claude.com - Anthropic — model-specific cache-read prices: platform.claude.com - OpenAI — prompt-cache lifetimes, write fees, and refresh rules: developers.openai.com - Google — Gemini context caching and storage charges: ai.google.dev - Hacker News — release discussion, including the 70,000-GPU estimate: news.ycombinator.com - Yuchen Jin (@Yuchenj_UW) — 82% margin estimate assuming $7 per GPU-hour: x.com - Alexey Fateev (@superalesha) — “cheap AI isn't charity” commentary: x.com - AINews / swyx — the daily-revenue estimate examined in the video: swyx.substack.com - vLLM (@vllm_project) — decode-performance comparison and revenue benchmark: x.com - SemiAnalysis — AgentX trace-replay methodology and measurement window: inferencex.semianalysis.com - Wu et al. (2026) — DualPath paper and adaptive request scheduling: arxiv.org - Jiashi Li (@beginlner), DeepSeek — reply declining to merge the community port: DeepInfer#92 - DeepSeek — deepseek-recipe and API-behavior compatibility: deepseek-ai/deepseek-recipe - @FuckAnthropic — purported Liang Wenfeng investor-call transcript; authenticity unconfirmed: x.com - DeepSeek — DeepEP-Ascend and Huawei hardware/firmware caveats: deepseek-ai/DeepEP-Ascend - Nexus Compute — enterprise SSD shortage and reported KV-cache demand in China: nexus-compute.com Chapters 00:00 What DeepSeek released—and kept 02:47 155 trillion tokens, mostly cache 04:00 The 13× lunch break 05:22 Checking the $38 per GPU-hour claim 06:34 How a three-day memory works 08:54 Is any of this a moat? 11:53 Why give the stack away? 14:44 Cheap tokens, expensive SSDs 15:21 What changes for you

DeepInfer Reads Like an Answer to DeepSeek’s Own 2025 Objections - * * On October 20, 2026, DeepSeek announced DeepInfer, the inference framework it says serves DeepSeek-V4.1-Flash in production, and published the code under Apache 2.0. Eighteen months earlier, the same company had explained in writing why it would not release its engine. As The New Stack quotes that post, the software was “impractical for public deployment without significant modifications.” Both statements can be true at once, and the space between them is where this release gets interesting. DeepInfer is a real production engine: kernels, a scheduler, a sampler, and multi-chip parallelism. It is also an offline inference release that leaves out the KV cache system, prefill and decode disaggregation, and load balancing across replicas. Those missing layers are the ones that bind most tightly to a specific cluster. This post reads the repository, its examples, and the first round of reporting, and it marks which claims are DeepSeek’s and which readings are mine. > Quick take: DeepInfer is most useful as a reference implementation and as a day-one path to V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is not a drop-in serving stack, DeepSeek’s speed figures have no independent verification yet, and the determinism features work on NVIDIA only. - * * What DeepSeek said in 2025 DeepSeek’s earlier open-source push went component by component. In late February 2025, its Open Source Week shipped one building block a day: FlashMLA, an MLA decoding kernel for Hopper GPUs; DeepEP, an expert-parallel communication library; and DeepGEMM, an FP8 GEMM library. The week also covered the EPLB expert-parallel load balancer, the DualPipe training schedule, and the 3FS file system. Inference code was the thin spot. For V3, the model card offered the DeepSeek-Infer demo, described as a “simple and lightweight demo” for FP8 and BF16, alongside community engines such as SGLang and LMDeploy for serving it. In April 2025 the same index gained a post on open-sourcing the inference engine. According to The New Stack’s account, DeepSeek had considered releasing the whole system and chose to contribute pieces to existing projects instead. Its reasons, in short: the engine had grown from an early vLLM version into a heavily customized system, a small research team would struggle to maintain another major open-source project, and the code was entangled with internal infrastructure. The entanglement point matters most for what follows. The post tied it specifically to cluster management tooling. DeepSeek also made a promise about hardware. It said it would synchronize inference engineering with partners ahead of future model launches, aiming for day-one availability across chip architectures. Keep that in mind for the Huawei section below. - * * What shipped on October 20 The repository supports one model, DeepSeek-V4.1-Flash, on two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017, the updated weights DeepSeek released days earlier. Two companion facts shape the picture. DeepSeek credits vLLM as the base: DeepInfer was originally built primarily on it, with inspiration from SGLang and TensorRT-LLM. And a second repository, deepseek-recipe, ships alongside. It handles how API requests become model inputs and how outputs become responses, in Rust and Python, so self-hosters can match the behavior of DeepSeek’s official API. The upstream loop DeepSeek described in 2025 is already visible. TNW reports that the vLLM project thanked DeepSeek and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. The repository had passed 8,000 stars by Tuesday afternoon, per the same report, and showed about 12,400 stars and 1,100 forks at the time of writing. !Mermaid Diagram - * * The scope is the argument Here is the boundary, drawn from the README’s own list of what a complete inference service still needs. !Mermaid Diagram The announcement called DeepInfer “fully open source.” The README is more careful. It describes offline inference with the core features needed for serving at scale, then lists three components it does not include: a KV cache system, VE/prefill/decode disaggregation (with the note that most of the required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode. Set that against the entanglement objection. What the README leaves out is the layer that routes requests to replicas, decides where cached prefixes live, and splits phases across machines. Those parts depend on your network, your scheduler, and your traffic. The release boundary sits roughly where the coupling to infrastructure would be tightest. That is my reading, and DeepSeek has not said so. The New Stack makes a similar observation and points out that the company has not explained how it overcame the earlier difficulties. For planning, the line is simple. Offline batch inference, evals, and research on a cluster you control: DeepInfer is close to complete. A multi-tenant chat API with prefix caching and replica routing: budget significant engineering time for the layer above it. - * * Inside the engine The README’s feature list is easier to judge with the switches next to it. | Feature | What it does | How to turn it on | | --- | --- | --- | | Parallelism | Attention DP and balanced CP, plus MoE expert parallelism with EPLB | --ep, --dp, and --cp flags in the offline example | | Asynchronous scheduling | Overlaps CPU batch scheduling with device execution | On by default | | DSpark with confidence-scheduled verification | Decodes with draft tokens and uses confidence estimates to schedule verification | DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length (default 5) | | Decoder SWA bounded replay | Cuts decoder prefill work by replaying only a bounded suffix | DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 | | Batch-invariant inference | Keeps results identical across batch compositions | DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) | | Deterministic sampling | Identical outputs for identical input tokens and sampling parameters | DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed (NVIDIA only) | | Structured outputs | Constrains generation to a regex, JSON schema, or EBNF grammar | so_args passed to SamplingParams | Read the last column twice. The DSpark recipe sets DEEPINFER_ENABLE_MTP_VARLEN=1, and the deterministic-sampling recipe sets the same variable to 0. If you need reproducible outputs for evals or regression tests, the two documented recipes pull in opposite directions. The README does not quantify the cost, so test before assuming they compose. Bounded replay exposes a model-specific constant. The README says the retained suffix must cover the sliding-window-attention window plus the maximum backtracking length, and it gives 140 tokens as the default for V4.1. Treat 140 as a V4.1 setting, not a universal one. Anyone adapting the engine to another model has a number to derive rather than copy. The kernel list is the 2025 strategy, assembled. FlashMLA, DeepGEMM, and DeepEP are the Open Source Week repositories. TileKernels, DeepSelect, and custom TileLang kernels in the tree join them. Read together, DeepInfer is where the formerly separate components meet a scheduler and a model runtime. - * * The vision encoder rides along V4.1-Flash accepts images, and the engine handles them in-process. With --prompt_mode image, the examples feed raw images to the engine directly, with the ViT and the LLM in the same engine. CPU preprocessing runs in asynchronous MultimodalProcessor worker processes, off the step critical path. That explains the odd “VE” in the list of missing components. VE is the vision encoder, and the examples benchmark it on its own. A sweep script loads only the VE-side weights, then times the GPU-side pad, ViT, downsample, and aligner path across image sizes and batch counts. Throughput is scaled by the number of LLM tokens each image produces, the same unit as the prefill curve, so the two are comparable. So “VE/prefill/decode disaggregation” is a three-stage split with the encoder as a separable stage. What the release leaves to you is the orchestration around it. The vision regression suite is refreshingly small: five inputs, made of two text-only samples, a single-image sample, a three-image sample, and a five-image four-shot sample. Every image comes from just two pictures, carrots and ribs. - * * Reading the benchmark charts without fooling yourself The README shows two Pareto plots: prefill (non-cached input throughput against latency) and decode (output throughput against per-user speed). DeepSeek measured them on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. TNW relays Bloomberg’s note that no one has independently benchmarked DeepInfer’s gains yet. The examples README documents the harness in unusual detail, and one line changes how you should read the decode curve. Decode throughput is computed like this: decode tokens/s/GPU = avg_SPS x decode_num_requests x (1 + nextn) / cp The docs call this full-acceptance accounting: every request is credited with 1 + nextn tokens per step, regardless of the true acceptance rate. With the defaults of 64 requests and a draft length of 5, one step counts as 384 tokens. Real speculative decoding rarely accepts every draft token, so a full-acceptance figure is a ceiling. The sweep script takes an acceptance-rate argument, and the docs’ own example sweeps a draft length of 3 and scales the decode plot by 0.75. Which scaling, if any, the published figure uses is not stated. Before comparing against vLLM or SGLang, check that setting. The methodology has strengths worth copying. The harness separates prefill from decode, drops intervals that collide with MoE load-balance statistics flushes, and runs sweeps with the profiler off because it perturbs timings. It also warns that profiler-traced step times can be inflated by nearly 50% at batch size 1 and must not be used to derive steps per second. Numerics are held to an exact standard. Perplexity baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them sample by sample. A refresh_offline_decode_ppl.py script updates them once a numeric change is confirmed as expected, and the docs say to re-measure the short and long baselines on NVIDIA and Ascend separately after changing the vision samples. A --check_structured_output flag extends the same run to regex, JSON, and EBNF checks. If you plan to compare DeepInfer with another engine, hold these constant. The list is mine, built from what the harness documents: - Hardware and scale-up domain. Match the layout behind the published figures, or state how yours differs. - Workload shape. Harness defaults are 32,768 context tokens with 4,096 non-cached. The documented nightly profile configuration uses 128k context with YoCo clipping. - Draft length and token accounting. State your draft length, and whether you count full acceptance or measured acceptance. - Raw quantities. --result_jsonl appends the configuration plus window step count, token count, elapsed time, and avg_SPS, so you can recompute throughput under your own accounting. - One tag per curve. The sweep tooling blocks plotting when model, ep_size, --cp, context length, draft length, or redundant expert count differ within a tag. Borrow that discipline. - * * Nvidia and Huawei, side by side The hardware list is the most strategic line in the README. Next to NVIDIA’s SM100 parts, which The New Stack identifies as Blackwell data-center GPUs, sits Huawei’s Ascend 950 series, with CANN 9.2 as its prerequisite. Older NVIDIA architectures such as Hopper are absent. The New Stack places this inside a run of reporting. Bloomberg said in September that DeepSeek planned to deploy at least 160,000 Ascend 950DT chips at a new Inner Mongolia data center. Reuters later that month reported DeepSeek and Huawei collaborating on tools to make Ascend chips easier to program. TNW adds that DeepSeek published tooling for Huawei chips earlier in October as an alternative to CUDA. Parity is incomplete. Batch-invariant inference and deterministic sampling are NVIDIA only, and the published performance numbers all come from NVIDIA GPUs. The README gives no Ascend numbers. The examples do show Ascend taken seriously. They document debugging variables such as ASCEND_LAUNCH_BLOCKING=1, which runs operators synchronously so stack traces are readable, and, with the separate numeric baselines described earlier, that reads as a first-class target rather than an afterthought. This matches the 2025 promise of day-one cross-chip support, delivered for one model on two chip families. It is a narrow delivery of a broad commitment. TNW reports that DeepSeek says future models and improvements will land in the same repository. - * * Running it today The prerequisites are Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment. export DEEPINFER_ACCELERATOR=nvidia # use "ascend" on Ascend ./build.sh # first run downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache python examples/offline_inference.py --ep 8 The feature switches in the table above are environment variables, so export them in the same shell before launching. The offline example sets its layout with flags. The documented variants are --ep 8, --ep 2 --dp 8, and --ep 2 --cp 8. Set DEEPINFER_HF_MODEL_DIR to run from a local copy of the weights. The benchmark harness works differently, and the difference trips people up. There, ep_size equals WORLD_SIZE (the number of nodes) times the number of visible devices, and --cp must divide it. Redundant experts default to 32 through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, but that value is only a lower bound: the engine rounds the total up so the expert count divides evenly by ep_size. The hardware requirement is the main gate: the README supports SM100 and Ascend 950 only. - * * Who should adopt it | If you are | Verdict | Why | | --- | --- | --- | | An inference-engine engineer | Read the code this week | A major lab’s account of what matters at scale, under Apache 2.0 | | A hosting provider on Blackwell or Ascend 950 | Pilot it | You get the kernels and parallelism, but you supply KV cache, disaggregation, and routing, and you must verify the speed claims yourself | | A team running offline batch jobs or evals on SM100 | Strong fit | Offline inference is what shipped, and batch invariance helps reproducibility on NVIDIA | | An app team calling a hosted API | Skip for now | Nothing in the release changes how you call a model | | A team on Hopper or older GPUs | Wait | The hardware is unsupported as released, so keep using vLLM or SGLang for now | - * * What to watch next Four signals will show how far DeepInfer travels beyond the announcement. - Independent benchmarks on identical hardware and identical token accounting, against vLLM and SGLang. - The missing layers. The README calls them required for a complete service. Whether DeepSeek will ship them is unannounced. - What lands upstream. vLLM said it would use ideas from DeepInfer. Watch for merged changes rather than statements. - Ascend parity for batch-invariant inference and deterministic sampling. The 2025 objections did not disappear. They moved to the edge of the release, where your own cluster begins. For anyone who builds or tunes inference engines, DeepInfer is a look at what a major lab says it runs in production. For teams that need a serving stack today, it is a release to watch rather than deploy. --- Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

DeepInfer vs. vLLM on 8 B200: Which Is Faster? TL;DR: We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup. In this run, DeepInfer-server delivered higher reported output throughput and shorter inter-token latency, while vLLM achieved lower time to first token and better P95 request latency. Efficient inference technology creates business value only when it can be deployed in a real service. DeepSeek's open-source DeepInfer gives developers a way to explore its inference technology. Teams that want to run models on their own infrastructure still need to answer several engineering questions: How do we expose an API? How should the runtime be configured? How do we manage GPUs? And how does the engine perform under a real workload? To answer those questions, we built a lightweight serving project called DeepInfer-server and integrated it with GPUStack's custom inference backend. The result was a working text-generation service for DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs. This article explains the integration, walks through the deployment process, and describes how we compared it with vLLM on the same hardware. !DeepInfer and vLLM model deployments in GPUStack 1. What Does DeepInfer Bring? DeepInfer is DeepSeek's open-source high-performance inference framework. The current public release provides offline inference and the core engine capabilities required for large-scale serving. It supports DeepSeek-V4.1-Flash and integrates computation and communication components including FlashMLA, DeepGEMM, DeepEP, and DeepSelect. It also supports multiple parallelism strategies, speculative decoding, and asynchronous scheduling. See the DeepInfer repository. These components optimize different parts of the inference path: computation executes faster, data moves between GPUs more efficiently, and scheduling makes better use of the hardware. What users ultimately care about is how many requests the same resources can serve and how long users have to wait. Turning an engine into a production-facing service, however, also requires request handling, protocol conversion, an HTTP service, and deployment management. The components in this integration have the following responsibilities: | Component | Role in this integration | | --- | --- | | DeepInfer | Runs model inference and manages low-level computation, communication, and parallel execution | | deepseek-recipe | Converts chat requests into model input and parses generated text, reasoning content, and tool calls | | DeepInfer-server | Connects the components and provides an HTTP API, streaming responses, and service lifecycle management | | GPUStack | Manages model deployments, GPU allocation, runtime configuration, and model access and testing | deepseek-recipe is DeepSeek's protocol-conversion and codec component. It translates between API requests, conversation formats, model token sequences, and responses. Model inference and HTTP transport are supplied by the caller. See the deepseek-recipe repository. Why Build DeepInfer-server? DeepInfer's current public entry point is primarily designed for offline inference. To start it like vllm serve and make it callable by applications, the engine must be connected to an HTTP request-processing path. deepseek-recipe handles protocol conversion and output parsing, but the caller must still supply model inference and HTTP transport. Installing both components in the same container does not automatically create a complete online inference service. A GPUStack custom-backend definition tells the platform which image and command to run and which arguments to pass. It does not automatically generate an API around an offline engine. This integration therefore needs a connecting layer that organizes the components into a service that can start, report its state, and receive requests. DeepInfer-server provides that serving layer: - Connect requests to inference. It receives Chat Completions requests, uses deepseek-recipe to convert them into model input, and calls DeepInfer to generate output. - Return output to applications. It converts generated tokens into text, reasoning content, or tool calls; returns standard JSON or SSE streaming responses; and reports token usage. - Manage the service lifecycle. It initializes inference processes, handles request dispatch, cancellation, timeouts, and errors, and exposes model-list and health-check endpoints. - Integrate with the deployment platform. It provides a container image and common startup arguments so that GPUStack can pass the model path, port, and GPU configuration and then start and manage the service. Applications can therefore call the model through the familiar OpenAI-compatible Chat Completions API, while GPUStack can reuse its custom-backend deployment workflow. DeepInfer still performs the underlying inference, and deepseek-recipe still handles protocol conversion and parsing. The current wrapper focuses on single-node, multi-GPU text inference. It is an independent demonstration project for validating the integration. It does not reproduce DeepSeek's complete production architecture for KV-cache management, disaggregated prefill and decode, or load balancing, and it should not be treated as equivalent to DeepSeek's official online service. 2. Deploying DeepInfer Through a GPUStack Custom Backend GPUStack's custom inference backend mechanism lets users define a container image, startup command, default parameters, and environment variables so that a new inference service can participate in the normal deployment workflow. Template variables can supply values such as the model path and service port at deployment time. See the GPUStack custom backend documentation. This allows teams to evaluate new inference engines on the same platform while reusing existing model-management and deployment operations. The integration includes the following backend definition: gpustack-deepinfer-server-backend.yaml Step 1: Prepare the Runtime Environment Our demonstration used a single server with eight NVIDIA B200 GPUs and the following model: deepseek-ai/DeepSeek-V4.1-Flash-1017 !Eight NVIDIA B200 GPUs in a single server Before deployment, prepare the complete model weights, the DeepInfer-server image, a compatible GPU driver and container runtime, and sufficient host memory. The configuration must also expose the correct RDMA devices and set up the communication environment. The NIC and network parameters in the accompanying YAML come from our validation environment and must be adapted to other servers. Step 2: Import the Custom Backend On GPUStack's inference-backend management page, create a custom backend and import the DeepInfer backend YAML definition. The definition contains the service startup method, common arguments, health-check path, and JIT-cache directory. At deployment time, GPUStack supplies the model path, service port, model name, and number of allocated GPUs. !Importing the DeepInfer custom backend in GPUStack Step 3: Create the Model Deployment When creating the model deployment, select the model weights and choose the DeepInfer-server inference backend created in the previous step. !Selecting the DeepInfer-server backend for a model deployment To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters. These values override the DP=1 default in the supplied YAML. Verify the final startup configuration before launching the service. | Setting | Example value | | --- | --- | | GPU count | 8 GPUs in one server | | Parallel configuration | EP=8, DP=8, CP=1 | | Maximum context length | 32,768 tokens | | Maximum running sequences | 16 per DP rank | | Batched-token budget | 32,768 per DP rank | | GPU memory utilization | Default 0.85 | | Startup timeout | 3,600 seconds | EP, DP, and CP refer to expert, data, and context parallelism. The current wrapper requires DP × CP = EP. We use CP=1 here, so the context is not split across GPUs. max-num-seqs=16 is the limit for each DP rank, giving the DP8 service a total admission limit of 128 sequences. The first startup loads the weights and may compile and warm up kernels. Use GPUStack to monitor the startup logs and wait for model initialization to finish before running inference tests. A persistent JIT cache helps reuse compiled artifacts. !DeepInfer-server initialized and passing its health check Step 4: Verify Inference After the model is ready, send a text request from the GPUStack Playground to verify streaming output. The same service can then be integrated with existing applications through its API. At this point, the full path from model weights and inference engine to API access is working. Teams can continue testing real prompts, monitoring resource utilization, and evaluating the value of further optimization. !Verifying DeepInfer inference in the GPUStack Playground 3. How Does It Compare with vLLM on the Same Eight B200 GPUs? Before looking at the numbers, we need to define the scope of this comparison. > This test is a service-integration and preliminary performance validation. Neither deployment was fully tuned for the hardware, and both differ from DeepSeek's production inference architecture. Because the public DeepInfer release does not expose a TP path corresponding to vLLM's tensor parallelism, we configured DeepInfer with CP1, DP8, and EP8 and vLLM with TP1, DP8, and EP8. This makes the parallel strategies more comparable, but does not represent the best possible configuration for either engine. DeepInfer's documented parallelism includes Attention DP or balanced CP and MoE EP with EPLB. Our single-node combined service does not reproduce its production disaggregated prefill/decode architecture, complete KV-cache management, or load balancing. See the DeepInfer documentation. Even with DP8+EP8 on both sides, their kernels, communication paths, Engram access, and schedulers are implemented differently. DeepInfer-server also adds request coordination and protocol processing written for this demonstration, and that overhead is included in the end-to-end results. These numbers do not represent the performance of DeepSeek's official service and should not be used to infer the performance ceiling of either engine. First, the Interactive Chat Experience After deployment, we started with a single-request chat in the GPUStack Playground. In the screenshots, the vLLM deployment produced approximately 91.4 tokens/s, while DeepInfer produced approximately 106.2 tokens/s in that interaction. !vLLM single-request chat output speed A single chat result depends on the prompt, context, answer length, and the UI's measurement method. This screenshot illustrates the interactive experience; it is not a single-request performance ranking and should not be compared directly with the concurrent-output throughput below. For the benchmark, both engines used DP8+EP8, an 8K input, a 1K output, and 16 concurrent requests. Test Environment and Configuration We used the same eight-B200 server and the same model weights. GPUStack's Benchmark feature issued 100 requests with a target concurrency of 16. | Item | DeepInfer-server | vLLM | | --- | --- | --- | | Hardware | 8 × NVIDIA B200 on the same Worker | 8 × NVIDIA B200 on the same Worker | | Model | DeepSeek-V4.1-Flash-1017 | DeepSeek-V4.1-Flash-1017 | | Version | DeepInfer commit 9bd9cf2; service image tag latest | 0.32.0 | | Parallel strategy | DP8, CP1, EP8 | DP8, TP1, EP8 | | Maximum context length | 32,768 tokens | 32,768 tokens | | max-num-seqs | 16 per DP rank | 16 per DP rank | | max-num-batched-tokens | 32,768 per DP rank | 32,768 per DP rank | | GPU memory utilization | 0.85 | 0.85 | | Speculative decoding | Disabled | Disabled | | Dataset | Random | Random | | Target input/output length | 8,192 / 1,024 tokens | 8,192 / 1,024 tokens | | Target client concurrency | 16 | 16 | | Number of requests | 100 | 100 | Raw Results in GPUStack !DeepInfer-server benchmark results !vLLM benchmark results The following table places the main metrics side by side: | Metric | DeepInfer-server: DP8+EP8 | vLLM: DP8+EP8 | | --- | ---: | ---: | | Successful/total requests | 100 / 100 | 100 / 100 | | Mean output throughput (tokens/s) | 1,392.44 | 1,112.20 | | Mean time to first token, TTFT (s) | 4.28 | 2.04 | | P95 time to first token, TTFT (s) | 16.33 | 8.27 | | Mean inter-token latency, ITL (ms) | 8.74 | 11.98 | | P95 inter-token latency, ITL (ms) | 8.93 | 13.70 | | Mean end-to-end request latency (s) | 13.23 | 14.30 | | P95 end-to-end request latency (s) | 25.46 | 20.03 | | Actual mean concurrency | 14.72 | 14.58 | Output throughput, TTFT, ITL, and request latency come from the exported report's output_tokens_per_second_mean, time_to_first_token_*, inter_token_latency_*, and request_latency_* fields. The table converts TTFT from milliseconds to seconds. Output throughput retains the report's mean statistic and is not converted into whole-run duration. ITL is not mixed with the report's time_per_output_token or per-SSE-chunk intervals. 4. What Do These Results Tell Us? In these two DP8+EP8 runs, DeepInfer-server reported higher mean output throughput and shorter mean inter-token latency, while vLLM delivered a shorter time to first token and lower tail latency. DeepInfer-server's reported mean output throughput was approximately 25.2% higher than vLLM's corresponding run. Its mean ITL was approximately 27.0% lower, and its mean end-to-end request latency was approximately 7.5% lower. These are differences between two specific test runs; they are not yet proof of a stable performance advantage. At the same time, DeepInfer-server's mean TTFT was 4.28 seconds, about 2.09× vLLM's 2.04 seconds. Its P95 request latency was 25.46 seconds, approximately 27.1% higher than vLLM's 20.03 seconds. The wait before an answer begins and the experience of the slowest requests both leave room for improvement. The result demonstrates why an inference service should be evaluated across throughput, time to first token, sustained generation speed, and tail latency. Interactive applications benefit from beginning a response quickly, while batch-generation workloads also care about how much output can be completed per unit of time. No single metric fully describes the service experience. Both DP8+EP8 configurations completed all 100 requests, validating service availability under this test load. One hundred successful requests are not equivalent to production stability certification, and the Random dataset does not evaluate answer quality. For GPUStack, the experiment demonstrates a reusable workflow: integrate a new engine through a custom backend, deploy and test it on one platform, and use measured data to choose the runtime that fits the workload. We plan to continue testing fixed prompts and controlled conditions at different concurrency levels, and to separate the overhead of the serving wrapper from engine execution. 5. Continuing the Work to Make Efficient Inference Easier to Use DeepInfer's open-source release gives the community more inference-optimization techniques that can be studied and validated. In its deployment Q&A, DeepSeek explains that its KV-cache management and load-balancing components are tightly coupled to internal infrastructure and have not yet been fully open sourced. DeepSeek also states that inference code and performance optimizations for future DeepSeek models will continue to be released through DeepInfer. See the DeepSeek inference deployment Q&A. This creates room for future integration work. The timing and scope of any additional open-source components should still be based on DeepSeek's official announcements. GPUStack will continue following DeepInfer and the related computation, communication, and codec components. As components become available and verifiable, we will evaluate further integration through real deployments and tests. Our goal is to help new inference technology reach developers and enterprises faster: lower the barrier to experimentation, make performance gains reproducible, make infrastructure cost measurable, and enable more teams to run large models effectively and affordably. Making low-cost inference broadly accessible requires sustained technical optimization—and it also requires making those optimizations easier to deploy and use. GPUStack: gpustack/gpustack DeepInfer-server: gpustack/DeepInfer-server

The current DeepInfer release only supports offline, single-turn inference, making comparisons InferenceX impossible. deepseek-recipe provides API infrastructure but the key load bearing components are still missing (2/5) deepseek-ai/deepseek-recipe

Топ-10 GitHub за день · 20 октября 2026: DeepInfer, harness2, ai-job-search Топ-10 GitHub за день · 20 октября 2026 (без повторов прошлого выпуска) Самые быстрорастущие open-source проекты GitHub за день: deepseek-ai/DeepInfer, google-research/harness2, MadsLorentzen/ai-job-search и ещё 7. 1. deepseek-ai/DeepInfer — +12 161 ★ — Открытый фреймворк офлайн-инференса для DeepSeek с оптимизированными ядрами; сервисная часть в релиз не входит. deepseek-ai/DeepInfer 2. google-research/harness2 — +2 453 ★ — Фреймворк для итеративной настройки окружения ИИ-агентов под новые задачи. google-research/harness2 3. MadsLorentzen/ai-job-search — +2 307 ★ — Фреймворк поиска работы на Claude Code: оценка вакансий, адаптация резюме, письма и подготовка к интервью. MadsLorentzen/ai-job-search 4. simbasjungles/free-my-tv — +1 208 ★ — Экран «домой» без рекламы для Google TV, LG и Samsung: свои приложения, погода, тренды и откат одной кнопкой. simbasjungles/free-my-tv 5. stablyai/orca — +1 073 ★ — Оркестратор ИИ-агентов: запускает Codex, Claude Code и другие параллельно, каждый в своём git worktree. stablyai/orca 6. deepseek-ai/deepseek-recipe — +1 069 ★ — Rust-библиотеки и Python-биндинги: переводят запросы Messages, Chat Completions и Responses в промпты DeepSeek V4/V4.1. deepseek-ai/deepseek-recipe 7. debpalash/VoiceStudio — +1 062 ★ — Локальный открытый аналог ElevenLabs: клонирование голосов, дубляж видео, диктовка и транскрибация на 646 языках. debpalash/VoiceStudio 8. tywr/Nordwand-Mono — +1 027 ★ — Компактный моноширинный неогротеск для кода; шрифт генерируется Python-скриптом, метрики можно поменять. tywr/Nordwand-Mono 9. Graphify-Labs/graphify — +885 ★ — Превращает код вместе с документами, SQL-схемами, конфигами и PDF в граф знаний, по которому можно задавать вопросы. Graphify-Labs/graphify 10. mattpocock/skills — +867 ★ — Набор навыков для ИИ-агентов (Claude Code, Codex и др.) для инженерной работы без «вайб-кодинга». mattpocock/skills Сообщество вайбкодеров в Telegram - vibeecode.ru Запускай и дорабатывай такие проекты с ИИ в Приложении ВайбКод - vibeecode.ru Ведение по вайбкодингу - vibeecode.ru Новые звёзды за день по официальной истории звёзд GitHub, сутки по UTC. #github #opensource #программирование #нейросети #вайбкодинг

DeepSeek ha appena rilasciato DeepInfer, il framework che usa internamente per far girare i modelli DeepSeek ha deciso di rendere open-source il suo motore di inferenza proprietario, quello che usano in produzione per V4.1 Flash . Non è solo un wrapper, è il loro stack vero. La cosa interessante è che supporta sia le GPU Nvidia SM100 che le Huawei Ascend 950 . Considerando tutto il casino che c'è intorno alle restrizioni sui chip, mi sembra una mossa non banale. Il punto però è che non è "plug and play": manca tutta la parte di KV cache e load balancing, quindi se volete farci un servizio completo da zero dovrete lavorarci sopra . Però se già smanettate con vLLM (visto che DeepInfer nasce da lì potrebbe essere una buona base per spremere un po' di più l'hardware . Qualcuno l'ha già provato? Sono curioso di sapere se i benchmark che hanno pubblicato reggono fuori dal loro cluster. Il link della repository è: deepseek-ai/DeepInfer Inoltre, DeepSeek ha reso open source anche il componente di codifica/decodifica deepseek-recipe: deepseek-ai/deepseek-recipe

Inside DeepSeek's Trillion-Token Engine: DeepInfer and deepseek-recipe Architecture from Silicon to Protocol Layer > [!NOTE] Executive Summary (TL;DR) > - Silicon & Kernel Foundations: DeepInfer is co-optimized for NVIDIA SM100 (Blackwell with NVLink 5) and Huawei Ascend 950 NPUs, leveraging custom kernels—FlashMLA (latent attention decoding), DeepGEMM (FP8/FP4 block-scaled GEMM), DeepEP (zero-bubble All-to-All communication), and TileKernels (TileLang domain-specific JIT compilation)—to eliminate hardware abstraction overhead. > - Cluster Topology & 1:1 PD Disaggregation: Breaking past single-node time-sharing limitations, the system establishes a symmetrical 1:1 ratio between 8-GPU compact Prefill nodes (compute-bound, high FLOPs density) and 64–128 GPU Wide-EP Decode supernodes (memory-bound, ultra-wide aggregate HBM bandwidth), supplemented by an isolated asynchronous Vision Encoder pipeline. > - Triad Algorithmic Acceleration Suite: Introduces DSpark Confidence-Scheduled Speculative Decoding (Confidence Head predicting adaptive verification trees, unlocking 4%–7% net throughput), SWA Bounded Replay (eliminating redundant recomputation across sliding window boundaries in long contexts), and dynamic MoE load balancing (resident core experts + on-demand redundant replicas to eliminate cross-node stragglers). > - Protocol Revolution & Unit Economics: Checkpoints officially discard traditional Jinja templates in favor of the Rust-native deepseek-recipe engine for zero-copy OpenAI/Anthropic protocol translation, <think> isolation, and tool-call parsing. The production report reveals single-cluster daily throughput of 155 Trillion Tokens, average prompt lengths of 77K, a 96% KV Cache hit rate, and $38/GPU/hour (~254 RMB/GPU/h) in off-peak revenue. > - Deep-Dive Cross References: Inter-node RDMA transport is detailed in P/D Disaggregation & Distributed KV Cache; speculative decoding theory is explored in EAGLE Speculative Decoding; underlying attention mechanics are derived in MLA Matrix Absorption and MoE Expert Parallelism; high-performance networking is covered in InfiniBand Production Tuning. --- Introduction: Revealing the Economics of Production Inference In modern AI research, open source often stops at releasing weights and publishing papers. The underlying engineering that enables frontier labs to sustainably serve hundreds of millions of requests under viable unit economics has remained a guarded trade secret. In October 2026, DeepSeek published its landmark technical report, Production Deployment Practices of DeepSeek V4.1 Flash at Scale, accompanied by the open-source release of its core production stack: the high-performance inference framework DeepInfer and the protocol adaptation layer deepseek-recipe. The disclosures revealed staggering production numbers: - Single-Cluster Daily Throughput: Operating at 155 Trillion Tokens/day; - Monetization & Efficiency: Generating $38 per GPU per hour in revenue even at off-peak rates; - Extreme Long-Context Resilience: Processing average prompts of 77K Tokens while sustaining a 96% KV Cache hit rate, maintaining Time to First Token (TTFT) at ~400ms, and clocking generation speeds between 230 and 300 tokens/s. These achievements stem from a ground-up reconstruction of the entire inference stack—spanning semiconductor microarchitectures, collective communication fabrics, specialized operator compilers, disaggregated prefill-decode clustering, and unified protocol parsing. This guide provides an exhaustive engineering dissection across all layers. --- Architectural Blueprint: The Eight-Layer DeepSeek Stack Before exploring micro-level mechanics, review the macro-level architecture spanning DeepInfer and deepseek-recipe: flowchart TD subgraph Layer_Protocol["Layer 7: Protocol & API Intermediary (deepseek-recipe)"] REC_In["OpenAI Chat / Anthropic Messages / Tool Call Requests"] --> REC_Core["deepseek-recipe (Rust Native Engine)"] REC_Core --> REC_Parse["&lt;think&gt; Reasoning Isolation / Tool Calling Extraction / Prompt Formatting"] end subgraph Layer_Scheduling["Layer 6: Algorithmic Scheduling & Acceleration"] SCH_DSpark["DSpark Confidence-Scheduled Speculative Decoding<br/>(Confidence Head Adaptive Tree Search)"] SCH_SWA["SWA Bounded Replay<br/>(Sliding Window Boundary Anchor Replay)"] SCH_MoE["Dynamic MoE Load Balancing<br/>(Resident Core Experts + Dynamic Redundant Replicas)"] SCH_KV["72h+ Tiered KV Cache Governance<br/>(96% Hit Rate / DRAM-NVMe Tiered Eviction)"] end subgraph Layer_Disaggregation["Layer 5: 1:1 Physical PD Disaggregation"] PD_Prefill["Prefill Pool (8-GPU Compact Nodes)<br/>Compute-Bound / High FLOPs Density / Large Context Throughput"] PD_Decode["Decode Supernodes (64-128 GPU Wide-EP)<br/>Memory-Bound / Massive HBM Bandwidth / Breaking Single-Node Bandwidth Walls"] PD_Vision["Isolated Vision Encoder Module<br/>Asynchronous Patch-Level Pre-encoding"] PD_Prefill -- "GPUDirect RDMA One-Sided Zero-Copy KV Push" --> PD_Decode end subgraph Layer_Kernel["Layer 4: High-Performance Kernels & Compilers (The Deep* Matrix)"] K_FlashMLA["FlashMLA (Latent Space Attention Decoding)"] K_DeepGEMM["DeepGEMM (FP8 / MXFP4 Block-Scaled Matrix Multiplication)"] K_DeepEP["DeepEP (Zero-Bubble All-to-All MoE Communication)"] K_TileKernels["TileKernels (Cross-Platform JIT via TileLang)"] K_DeepSelect["DeepSelect (Fused Top-K Gating Router)"] end subgraph Layer_Hardware["Layer 3: Physical Silicon & Fabric Foundations"] HW_NV["NVIDIA SM100 (Blackwell)<br/>NVLink 5 (1.8TB/s) / FP4 Tensor Cores"] HW_Ascend["Huawei Ascend 950 NPU<br/>HCCS Mesh / DaVinci Cube+Vector Units"] HW_Net["8x 400G/800G NDR InfiniBand & Lossless RoCEv2 (Leaf-Spine)"] end Layer_Protocol --> Layer_Scheduling Layer_Scheduling --> Layer_Disaggregation Layer_Disaggregation --> Layer_Kernel Layer_Kernel --> Layer_Hardware --- Layer 1: Physical Silicon and Interconnect Foundations Inference efficiency is bound by semiconductor physics. DeepInfer's design philosophy rejects generic hardware abstractions in favor of direct alignment with accelerator microarchitectures. 1. Dual-Track Silicon: NVIDIA Blackwell SM100 vs. Huawei Ascend 950 DeepSeek's production deployments officially embrace both NVIDIA and Huawei compute platforms: | Hardware Dimension | NVIDIA Blackwell (SM100) | Huawei Ascend 950 NPU | | :--- | :--- | :--- | | Process & Packaging | TSMC 4NP Dual-Die CoWoS-L | 7nm Advanced System-in-Package (SiP) | | Die-to-Die Interconnect | 10 TB/s NV-HBI Fabric | Ultra-Wide On-Package Bus | | Intra-Node P2P Bandwidth | NVLink 5 (1.8 TB/s Bidirectional per GPU) | HCCS (1.2 TB/s Bidirectional per NPU) | | Memory Subsystem | 192GB–288GB HBM3e (8.0 TB/s Bandwidth) | 128GB–192GB HBM3 (3.8 TB/s Bandwidth) | | Compute Engines | 5th-Gen Tensor Cores (Native FP4 / FP8) | DaVinci Architecture (Cube Matrix + Vector Units) | | Driver & Toolchain | CUDA 13.x / PTX / TileLang JIT | CANN 8.x / Ascend C / TileLang JIT | graph LR subgraph Chip_Interconnect["Physical Bandwidth Hierarchy (Orders of Magnitude Disparity)"] L_SRAM["On-Chip SRAM<br/>~20-30 TB/s Ultra-Fast L1/L2"] --> L_HBM["On-Package HBM3e<br/>3.8 - 8.0 TB/s Memory Bandwidth"] L_HBM --> L_NVLINK["Intra-Node NVLink 5 / HCCS<br/>1.2 - 1.8 TB/s Bidirectional Link"] L_NVLINK --> L_NIC["Inter-Node 400G/800G RDMA<br/>50 - 100 GB/s Network Fabric"] end 2. Physical Interconnect Gaps and Distributed Parallelism Laws The bandwidth disparity between on-package HBM (8.0 TB/s) and inter-node networks (50–100 GB/s) represents an 80x–100x performance drop. This physical reality establishes two core distributed rules: 1. Tensor Parallelism (TP) Must Remain Intra-Node: TP requires two high-frequency All-Reduce collective operations per transformer layer for every single token. Extending TP across physical nodes stalls Tensor Cores in network wait states over 95% of the time. 2. Expert Parallelism (EP) Requires Non-Blocking Fabrics: While MoE All-to-All collective communication exchanges lower per-token data volume than TP, it is hypersensitive to network tail latency and buffer jitter. Inter-node topologies mandate 8x 400G/800G InfiniBand (NDR) or rail-optimized lossless RoCEv2 fabrics (see InfiniBand Production Tuning). --- Layer 2: High-Performance Operators and Custom Compilers (The Deep* Matrix) Generic inference runtimes decompose complex models into generic CUDA kernels, causing excessive global memory round-trips. DeepInfer addresses this through a dedicated matrix of custom kernels: flowchart TD subgraph Deep_Matrix["DeepInfer Custom Kernel Suite"] direction TB F_MLA["FlashMLA<br/>Latent Space Attention"] D_GEMM["DeepGEMM<br/>FP8/FP4 Micro-Scaling"] D_EP["DeepEP<br/>Zero-Bubble All-to-All"] T_LANG["TileKernels<br/>TileLang Cross-Platform JIT"] D_SEL["DeepSelect<br/>Fused Top-K Gating"] end F_MLA & D_GEMM & D_EP & D_SEL --> T_LANG T_LANG --> Target_NV["NVIDIA SASS Binary"] T_LANG --> Target_Ascend["Ascend Machine Code"] 1. FlashMLA: Dedicated Latent Space Attention Kernel As analyzed in MLA Matrix Absorption, Multi-Head Latent Attention compresses keys and values into a low-rank latent representation $c_t^{KV}$, reducing per-token KV Cache footprint to less than one-sixth of traditional MHA. FlashAttention-2/3 assumes standard multi-head formats and cannot absorb low-rank projection matrices in-flight. FlashMLA provides: - On-Chip Matrix Absorption: Executes the multiplication of query projections with KV decompression weights $W_U^{KV}$ directly inside GPU shared memory (SRAM), eliminating the need to materialize uncompressed key-value matrices in HBM. - TMA Hardware Alignment: Aligns tile sizes with NVIDIA Blackwell's Tensor Memory Accelerator (TMA) for asynchronous, register-bypassing data movement. 2. DeepGEMM: Fine-Grained Block-Scaled Matrix Multiplication Large MoE checkpoints require low-precision quantization. However, coarse per-tensor quantization destroys numerical precision in reasoning tasks. - Block-Level Scaling: DeepGEMM applies fine-grained 1x128 or 128x128 block quantization, maintaining independent FP8 E8M0 scale factors per block. - Micro-Architecture Specialization: On Blackwell, DeepGEMM compiles directly to native MXFP4 (Microscaling FP4) and FP8 Tensor Core assembly. On Ascend 950, it maps to DaVinci 3D Cube mixed-precision execution pipelines without precision conversion penalties. 3. DeepEP: Zero-Bubble Communication-Computation Overlap In distributed MoE execution, tokens must be dispatched to experts on disparate nodes via all_to_all_v collectives. - Micro-Batch Pipelining: DeepEP chunks incoming batches into micro-batches. While micro-batch $n$ executes expert forward passes on GPU Tensor Cores, micro-batch $n+1$ asynchronously exchanges tokens across the RDMA fabric, achieving near-perfect communication hiding. - Hierarchical Transport: DeepEP routes intra-node token transfers across NVLink while routing inter-node traffic directly over GPUDirect RDMA. 4. TileKernels and TileLang: Bridging NVIDIA and Domestic Silicon To avoid maintaining separate CUDA and Ascend C codebases, DeepInfer integrates TileLang: - Engineers define compute and memory movement patterns using Pythonic Tile-level abstractions. - The TileLang compiler handles register allocation, shared memory bank conflict resolution, and instruction scheduling, emitting native SASS binaries for NVIDIA SM100 and optimized machine code for Huawei Ascend 950. --- Layer 3: Physical Cluster Topologies & Disaggregated Serving (Disambiguating Macro 1:1 vs. Micro 8 and 64) Co-locating compute-bound prefill operations with memory-bound decode operations on the same GPU creates severe resource interference. DeepSeek's production deployments address this via physical Prefill-Decode (PD) Disaggregation. However, the juxtaposition of "Prefill and Decode maintain an exact 1:1 GPU card count ratio" alongside "Prefill runs on 8-GPU nodes while Decode runs on 64–128 GPU supernodes" frequently puzzles systems engineers: How can 8 and 64 equate to a 1:1 ratio? How does an 8-GPU node stream its KV Cache to a 64-GPU supernode without incurring catastrophic network broadcast storms? This section resolves the paradox across four technical dimensions: macro-cluster capacity, micro-serving topologies, queueing theory proofs, and tensor resharding mechanics. --- 1. Conceptual Disambiguation: Macro Cluster Allocation vs. Micro-Pod Granularity The confusion stems from conflating macro-cluster capacity allocation with micro-serving execution boundaries: | Architectural Dimension | Prefill Pool (Prompt Ingestion) | Decode Pool (Token Generation) | Architectural Relationship | | :--- | :--- | :--- | :--- | | Micro-Serving Unit (Pod) | Single 8-GPU Server (P-Node) | 64–128 GPU Distributed Supernode (D-Supernode) | Micro granularity disparity: 8-card standalone node vs 8–16 physical nodes grouped into a supernode | | Parallel Sharding | Intra-node Tensor Parallelism (TP=8) | Intra-node TP=8 + Inter-node Wide-EP=8/16 | NVLink saturates attention locally; RDMA routes MoE expert tokens across nodes | | Execution Bottleneck | Compute-Bound (High FLOPs Density) | Memory-Bound (HBM Bandwidth Constrained) | 8 GPUs saturate Tensor Cores; Decode requires 64 GPUs to aggregate HBM bandwidth | | Macro Cluster Scale (512-GPU Example) | 256 GPUs (32 Independent 8-GPU Nodes) | 256 GPUs (4 Independent 64-GPU Supernodes) | Macro total card ratio is exactly 1 : 1 ($32 \times 8 = 4 \times 64 = 256$) | flowchart TB subgraph Macro_Cluster["Macro 512-GPU Production Cluster (Exact 1:1 Card Ratio)"] direction TB subgraph P_Cluster["Prefill Compute Pool (256 GPUs = 32 Independent 8-GPU Servers)"] P0["P-Node 01 (8-GPU)"] P1["P-Node 02 (8-GPU)"] P_Dots["... (32 Standalone Servers) ..."] P31["P-Node 32 (8-GPU)"] end subgraph D_Cluster["Decode Memory Pool (256 GPUs = 4 Distributed 64-GPU Supernodes)"] D0["D-Supernode 1 (64-GPU Fabric<br/>8 Interconnected Servers)"] D1["D-Supernode 2 (64-GPU Fabric<br/>8 Interconnected Servers)"] D2["D-Supernode 3 (64-GPU Fabric<br/>8 Interconnected Servers)"] D3["D-Supernode 4 (64-GPU Fabric<br/>8 Interconnected Servers)"] end end GW["Global Session Gateway"] -->|"Dispatch Text Prompt"| P_Cluster P_Cluster -->|"400G/800G RDMA Mesh Fabric (M:N Dynamic KV Cache Ingestion)"| D_Cluster Key Takeaway: The architecture does not pair one 8-GPU node with one 64-GPU node. Instead, 32 independent 8-GPU Prefill nodes serve prompts in parallel, streaming completed KV Caches across an M:N RDMA fabric into 4 distributed 64-GPU Decode supernodes. The aggregate GPU count ($32 \times 8 = 256$ vs. $4 \times 64 = 256$) yields a symmetrical 1:1 capacity ratio. --- 2. Queueing Theory & Roofline Proof: Why Cluster Card Ratios Converge to 1:1 A deeper question emerges: Prefill requires only tens of milliseconds, whereas autoregressive Decode takes multiple seconds (a 64x duration difference). Intuition suggests long tasks should require 10x to 50x more hardware. Why does the optimal ratio mathematically converge to 1:1? The proof lies in the mathematical product of the 96% KV Cache hit rate and per-unit concurrency capacity. (1) Input Workload Specifications (Production Telemetry) - Average prompt length: $L_{\text{prompt}} = 77,000$ tokens; - Production KV Cache hit rate: $H = 96\%$; - Effective un-cached tokens requiring GPU forward passes: $$L_{\text{eff}} = L_{\text{prompt}} \times (1 - H) = 77,000 \times (1 - 0.96) = 3,080 \text{ tokens}$$ - Average generated output length: $L_{\text{gen}} \approx 800$ tokens; - Global request arrival rate: $\lambda$ (req/s). (2) Prefill Node Throughput Bounds In the Prefill stage (Compute-Bound): - An 8-GPU HGX node executes DeepSeek-V4.1-Flash across intra-node NVLink (TP=8), utilizing FP8/FP4 Tensor Cores for peak arithmetic intensity. - Processing $L_{\text{eff}} = 3,080$ tokens through full attention and FFN layers requires: $$T_P \approx 50 \text{ ms} = 0.05 \text{ s}$$ - Under extreme 77K context lengths, transient activations and KV allocation consume tens of gigabytes per request. Physical HBM constraints limit instantaneous concurrency per 8-GPU node to $B_P \approx 1$ (maximum 2). - Consequently, the service rate of a single 8-GPU Prefill node is: $$\mu_{P_node} = \frac{B_P}{T_P} = \frac{1 \text{ req}}{0.05 \text{ s}} = 20 \text{ req/s}$$ - Across the 32-node Prefill pool (256 GPUs), aggregate prefill capacity is: $$\mu_{P_pool} = 32 \times \mu_{P_node} = 32 \times 20 = 640 \text{ req/s}$$ (3) Decode Supernode Throughput Bounds In the Decode stage (Memory-Bound): - A 64-GPU supernode (8 physical servers) shards trillion-parameter MoE weights across all 64 GPUs, activating over $1,000 \text{ TB/s}$ of aggregate HBM bandwidth per decoding step. - With generation velocity at $V_D = 250 \text{ tokens/s}$, emitting 800 tokens requires: $$T_D = \frac{L_{\text{gen}}}{V_D} = \frac{800 \text{ tokens}}{250 \text{ tokens/s}} = 3.2 \text{ s}$$ - Although $T_D (3.2\text{s})$ is 64x longer than $T_P (0.05\text{s})$, the 64-GPU supernode possesses 64 GPUs of pooled HBM. Leveraging MLA latent vector compression, each supernode sustains high-concurrency continuous batching of $B_D = 64$ (up to 128) concurrent streams simultaneously! - Thus, the service rate of a single 64-GPU Decode supernode is: $$\mu_{D_node} = \frac{B_D}{T_D} = \frac{64 \text{ req}}{3.2 \text{ s}} = 20 \text{ req/s}$$ - Across 4 Decode supernodes (256 GPUs), aggregate baseline capacity is: $$\mu_{D_pool} = 4 \times \mu_{D_node} = 4 \times 20 = 80 \text{ req/s}$$ (With continuous batching saturation at $B_D = 128 \sim 256$, saturated throughput reaches $160 \sim 320 \text{ req/s}$, matching Poisson arrival tails). (4) Steady-State Equilibrium Formulation To prevent queue accumulation, the system balances arrival and processing capacities: $$\mu_{P_pool} \approx \mu_{D_pool} \iff \frac{N_P \text{ (GPUs)}}{8} \times \frac{B_P}{T_P} = \frac{N_D \text{ (GPUs)}}{64} \times \frac{B_D}{T_D}$$ Substituting physical values: $$\frac{N_P}{8} \times \frac{1}{0.05} = \frac{N_D}{64} \times \frac{64}{3.2} \implies N_P \times 2.5 = N_D \times 0.3125 \times 8 \implies N_P \approx N_D$$ > [!IMPORTANT] Physical Reality: 1:1 is Dictated by the 96% Cache Hit Rate > - At 0% Cache Hit Rate: $L_{\text{eff}} = 77,000$ tokens, ballooning Prefill time from 50ms to 1,200ms (a 24x increase). Maintaining a 1:1 ratio would trigger immediate prefill queue collapse, demanding 10x more Prefill cards than Decode cards. > - At 96% Cache Hit Rate: The compute workload is reduced by 96%, allowing a 50% GPU allocation on Prefill to feed a 50% GPU allocation on Decode. The 1:1 ratio is not arbitrary symmetry—it is the exact mathematical equilibrium between long-cache reuse and HBM bandwidth limits. --- 3. Micro-Level Tensor Resharding: Direct 1-to-1 Injection from 8 GPUs to 64 GPUs Engineers often ask: If an 8-GPU node shards tensors via TP=8, and a 64-GPU supernode spans 8 physical servers (TP8 × EP8), does transferring KV Caches require an 8-to-64 network broadcast storm (All-to-All Resharding)? The answer is no. Inter-node RDMA transfers execute as clean 1-to-1 point-to-point pushes between matching GPU ranks. flowchart LR subgraph P_Node["Prefill 8-GPU Node (TP=8)"] P_GPU0["GPU 0 (Head Group 0)"] P_GPU1["GPU 1 (Head Group 1)"] P_GPUDots["..."] P_GPU7["GPU 7 (Head Group 7)"] end subgraph D_SuperNode["Decode 64-GPU Supernode (8 Interconnected Servers, TP=8 x EP=8)"] subgraph Machine0["Physical Server 0 (Assigned Attention Node for Session)"] D_GPU0["GPU 0 (Head Group 0)"] D_GPU1["GPU 1 (Head Group 1)"] D_GPUDots["..."] D_GPU7["GPU 7 (Head Group 7)"] end subgraph Machine1_7["Physical Servers 1 ~ 7 (Participate only during subsequent MoE MLP steps)"] M_Other["GPUs 8 ~ 63<br/>(Store Zero KV Cache for this session)"] end end P_GPU0 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU0 P_GPU1 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU1 P_GPU7 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU7 Why No 64-Way Broadcast Is Needed (MLA First Principles): 1. KV Cache Is Consumed Exclusively by Attention: In Transformer architectures, KV Caches are accessed only during Multi-Head Attention computation and are decoupled from subsequent MoE FFN layers. 2. Supernode Hierarchical Decomposition: Within a 64-GPU supernode, inter-server communication handles Expert Parallelism (EP=8), while each physical server executes intra-node Tensor Parallelism (TP=8) locally. 3. Session Affinity Binding: Upon completing prefill, the global router pins the session's attention evaluation to a single physical server (e.g., Server 0) within the 64-GPU supernode. 4. One-Sided Direct DMA Injection: GPU $i \in [0, 7]$ on the Prefill node initiates a GPUDirect RDMA Write targeting GPU $i$ on the destination server. Network transfers operate strictly as point-to-point 1-to-1 pipelines, transferring megabytes of compressed MLA latent vectors in under 3ms with zero broadcast overhead. Subsequent MoE token dispatch occurs across the local network during the MLP phase and involves no KV Cache movement. --- 4. Isolated Vision Encoder Pipeline For multimodal workloads, DeepInfer extracts image processing into a standalone tier: - Incoming images are pre-processed by dedicated Vision Encoder nodes into patch-level embeddings. - Visual tokens merge with text tokens at the gateway before ingestion by the Prefill pool. - This prevents lightweight convolutional or ViT kernels from disrupting LLM execution graphs. --- Layer 4: Triad Algorithmic Acceleration Suite Beyond hardware and kernel co-design, DeepInfer integrates three algorithmic breakthroughs: DSpark Speculative Decoding, SWA Bounded Replay, and Dynamic MoE Balancing. 1. DSpark Confidence-Scheduled Speculative Decoding Standard speculative decoding pairs a lightweight draft model with a primary target model, verifying $K$ drafted tokens in a single target forward pass (see EAGLE Speculative Decoding). However, conventional schemes suffer from the Fixed-Stride Penalty: With a fixed stride ($K=5$), if the first drafted token is rejected, all four remaining tokens are discarded. During deep reasoning phases or complex syntax emission, rigid strides introduce significant waste. flowchart TD subgraph Traditional["Traditional Speculative Decoding (Fixed K=5)"] T1["Token 1 (High Conf)"] --> T2["Token 2 (High Conf)"] T2 --> T3["Token 3 (Low Conf - Rejected)"] T3 --> T4["Token 4 (Wasted Math)"] T4 --> T5["Token 5 (Wasted Math)"] T_Val["Verification: Accepted 2, Discarded 3 (60% Wasted Compute)"] end subgraph DSpark["DeepInfer DSpark (Adaptive Verification Tree)"] D_Head["Draft Model + Confidence Head"] D_Head --> D1["Token 1 (Confidence 0.95 -> Expand)"] D1 --> D2["Token 2 (Confidence 0.91 -> Expand)"] D2 --> D3{"Token 3 Confidence 0.42 < Threshold ?"} D3 -- "Yes: Immediate Dynamic Pruning" --> D_Commit["Submit only 2 tokens (Zero Wasted Compute)"] D3 -- "No" --> D_Next["Continue Expansion"] end DSpark Confidence Head Formulation DSpark attaches a lightweight regression head to the draft model's final layer, predicting target acceptance probability: $$\hat{P}_{\text{accept}}(x_{t+i}) = \sigma \left( W_c \cdot h_{t+i} + b_c \right)$$ where $h_{t+i}$ denotes the draft model's hidden representation at step $t+i$. The scheduler dynamically bounds verification tree depth: $$\text{Depth}(T) = \max \left{ m \in [1, K_{\max}] \;\middle|\; \prod_{j=1}^{m} \hat{P}_{\text{accept}}(x_{t+j}) \ge \tau_{\text{threshold}} \right}$$ - Structured Syntax (JSON, Code Boilerplate, Prose): When cumulative confidence remains high, DSpark expands validation trees up to 8–12 tokens. - Complex Logic & Symbolic Math: As confidence dips, the verification tree shrinks to 1–2 tokens, preventing rollback overhead. - Production Impact: DSpark provides an additional 4% to 7% net throughput gain over baseline generation. --- 2. SWA Bounded Replay (Sliding Window Attention Boundary Anchoring) DeepSeek-V4.1-Flash employs Sliding Window Attention (SWA) to sustain efficiency across long contexts. In conversational turns, SWA typically encounters Boundary Recomputation Penalties: When the context window slides past length $W$, tokens outside the window lose cross-attention visibility. If historical context must be revisited or verified, the system is conventionally forced to re-execute prefill across earlier tokens. SWA Bounded Replay Mechanics: 1. Boundary Anchoring: At fixed window intervals $W$, the kernel snapshots an Attention Sink State Fingerprint in SRAM. 2. Localized Replay: During context shifting, the engine recomputes a bounded slice (< 128 tokens) from the nearest anchor rather than reprocessing the entire 77K context. 3. Efficiency: SWA Bounded Replay eliminates over 85% of redundant prefill computation in multi-turn dialogues. --- 3. Dynamic MoE Expert Load Balancing In multi-node expert parallelism, unmitigated routing skew creates severe Straggler Bottlenecks: if all tokens in a batch route to a small subset of popular experts, those GPU ranks saturate while others idle, blocking the entire cluster at the subsequent All-to-All barrier. DeepInfer employs a three-part balancing strategy: 1. Resident Core Experts: Offline traffic analysis identifies top-frequency experts, which are replicated across every physical node. Requests hitting these experts execute locally, generating zero cross-node network traffic. 2. Dynamic Redundant Replicas: Telemetry monitors token arrival rates per expert in real time. When an uncommon expert experiences sudden traffic spikes, the cluster broadcasts weights via RDMA to instantiate temporary replicas on underutilized nodes within milliseconds. 3. Capacity Factor Truncation & Reranking: Enforces strict token processing caps per expert with residual bypass connections, bounding inter-GPU rank computation variance to within 5%. --- 4. Tiered 72h+ KV Cache Management: Sustaining a 96% Hit Rate The production report's most striking metric is an online 96% KV Cache hit rate across 77K-token average inputs. This efficiency relies on a structured three-tier storage hierarchy: graph TB subgraph Storage_Hierarchy["DeepInfer Tiered KV Cache Storage Architecture"] Tier1["Tier 1: GPU HBM Memory Pool<br/>Capacity: Tens of TBs | Latency: < 1 µs<br/>Active working set for high-concurrency sessions"] Tier2["Tier 2: Host DRAM Shared Memory Pool<br/>Capacity: Hundreds of TBs | Latency: ~10-20 µs<br/>Warm cache for sessions idle within 1-6 hours"] Tier3["Tier 3: Distributed NVMe-oF Local Storage<br/>Capacity: Multiple PBs | Latency: ~100-300 µs<br/>Cold archive for 7-day Chatbot / 3-day Agent TTLs"] end Tier1 <== "GPUDirect Swapping" ==> Tier2 Tier2 <== "Asynchronous Write-Through" ==> Tier3 - Policy-Based TTL Enforcement: - Conversational Traffic (Chatbot): Configured with a 7-day TTL, ensuring multi-turn user follow-ups hit cache near-instantaneously. - Agent Workflows: Configured with a 3-day TTL, preserving long tool execution traces and environment observations. - Radix Tree Prefix Indexing: Incoming token streams are matched against a global prefix radix tree. Requests route directly to the Decode node hosting the matched KV Cache blocks, executing prefix-aware placement. --- Layer 5: Protocol and Gateway Layer (deepseek-recipe Deep Dive) Production deployments require reliable API serialization. Alongside DeepInfer, DeepSeek open-sourced deepseek-recipe. 1. Why Jinja Templates Were Deprecated Traditional Hugging Face repositories rely on chat_template.jinja rendered via Python. In massive production clusters, Jinja presents three primary bottlenecks: 1. CPU Parsing Overhead: Interpreting complex Jinja templates across hundreds of thousands of concurrent requests saturates gateway CPU cores. 2. State Machine Fragility: Reasoning models produce interleaved reasoning tags (<think>...</think>), structured function calling parameters, and multimodal references. String-based Jinja rendering frequently mishandles token boundaries and escaping. 3. Protocol Divergence: Bridging external OpenAI (Chat Completions / Responses) or Anthropic (Messages) schemas into Jinja templates requires fragile boilerplate. Consequently, DeepSeek-V4.1-Flash removed Jinja templates, delegating all prompt preparation to deepseek-recipe. 2. Rust Native Core with Python Bindings deepseek-recipe is authored in Rust with PyO3 bindings: flowchart LR A["API Request (OpenAI / Anthropic / Custom)"] --> B["Python Gateway (FastAPI / vLLM Server)"] B --> C["deepseek-recipe Core (Rust Native Shared Library)"] C --> D["Zero-Copy Conversation AST"] D --> E["Token ID Serialization & &lt;think&gt; State Machine"] E --> F["Native Token Buffers Dispatched to Engine"] 3. Implementation Example: Protocol Bridging and Reasoning Extraction Installable via pip install deepseek-recipe, the following example demonstrates production API bridging: """ production_gateway_recipe.py Production gateway integration using deepseek-recipe for protocol unification, reasoning trace separation, and function calling extraction. """ from typing import Dict, Any, List import deepseek_recipe as ds_recipe def handle_chat_completion_request(payload: Dict[str, Any]) -> ds_recipe.RenderedInput: """ Transforms standard OpenAI API payloads into native token streams via deepseek-recipe. """ # 1. Initialize structured conversation instance conversation = ds_recipe.Conversation() # 2. Configure thinking and reasoning behaviors enable_thinking = payload.get("enable_thinking", True) conversation.set_reasoning_mode(enable_thinking) # 3. Bind client tool declarations if "tools" in payload: for tool in payload["tools"]: conversation.register_tool( name=tool["function"]["name"], description=tool["function"].get("description", ""), parameters=tool["function"].get("parameters", {}) ) # 4. Ingest multi-turn message history for msg in payload["messages"]: role = msg["role"] content = msg["content"] if role == "system": conversation.add_system_message(content) elif role == "user": conversation.add_user_message(content) elif role == "assistant": # Preserve historical reasoning chains and tool calls cleanly reasoning_content = msg.get("reasoning_content", None) tool_calls = msg.get("tool_calls", None) conversation.add_assistant_message( content=content, reasoning_content=reasoning_content, tool_calls=tool_calls ) elif role == "tool": conversation.add_tool_result( tool_call_id=msg["tool_call_id"], result=content ) # 5. Compile to native token arrays and attention masks via Rust engine rendered_input = conversation.render_for_engine( tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash", add_generation_prompt=True ) return rendered_input def parse_engine_stream_output(raw_token_stream: List[int]) -> Dict[str, Any]: """ Streams and demuxes raw token streams into <think> traces and content payloads. """ parser = ds_recipe.StreamOutputParser(tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash") parsed_result = { "reasoning_chunk": "", "content_chunk": "", "detected_tool_calls": [] } for token_id in raw_token_stream: event = parser.step(token_id) if event.is_reasoning(): # Encapsulated within <think> blocks parsed_result["reasoning_chunk"] += event.text() elif event.is_content(): # Primary user-facing generation parsed_result["content_chunk"] += event.text() elif event.is_tool_call_completed(): # Fully parsed structured function invocation parsed_result["detected_tool_calls"].append(event.get_tool_call()) return parsed_result - Robust Syntax Isolation: The Rust parser guarantees strict syntactic tokenization, ensuring synthetic prompt injections (e.g., forged <|fim_begin|> or <think> tags) are safely escaped. --- Layer 6: Cluster Financials and Production Unit Economics How does DeepSeek maintain profitability while offering competitive API pricing, achieving $38/GPU/hour in revenue? A formal financial model clarifies the dynamics. 1. Revenue and Throughput Modeling Consider a baseline production deployment of 512 accelerators (256 Prefill GPUs + 256 Decode GPUs): - Revenue Rate per GPU: $\$38 \text{ / (GPU} \cdot \text{Hour)}$ - Aggregate Daily Revenue: $$\text{Daily Cluster Gross} = 512 \text{ GPUs} \times 24 \text{ Hours} \times \$38 \approx \$466,944 \text{ / Day}$$ - Annualized Gross Revenue: $$\text{Annualized Gross} = \$466,944 \times 365 \approx \$170.4 \text{ Million / Year}$$ 2. Comprehensive Total Cost of Ownership (TCO) Breakdown Operating 512 enterprise-grade GPUs (e.g., 64 8-GPU Blackwell / H100 nodes) incurs the following monthly cost model: | Cost Category | Monthly Allocation (USD) | Modeling Methodology | | :--- | :--- | :--- | | Server Hardware Amortization (CapEx) | $550,000 / month | 64 nodes amortized over 36 months straight-line + enterprise support | | Network Fabric & Optics Amortization | $65,000 / month | 800G NDR InfiniBand leaf-spine switches and active optical cabling | | Datacenter Power & Cooling (OpEx) | $130,000 / month | High-density liquid-cooled suites, 12kW–15kW per rack, PUE = 1.25 | | High-Bandwidth Transit & BGP Ports | $45,000 / month | Dedicated redundant transit ports supporting high egress volume | | AI Infrastructure Engineering Allocation| $90,000 / month | Dedicated cluster operations and systems engineering personnel | | Total Monthly Operating Cost | ~$880,000 / month | Yields an average daily run-rate of $29,333 / day | 3. Financial Margin Analysis: The Role of Cache Hit Rates Comparing gross revenues against operating costs: - Daily Gross Revenue: $466,944 / day - Daily Operating Run-Rate: $29,333 / day - Daily Operating Margin: $437,611 / day (93.7% Gross Margin) pie title Daily Financial Composition ($466,944 / Day) "Net Operating Margin" : 437611 "Server Amortization (CapEx)" : 18333 "Datacenter Power & Bandwidth" : 5833 "Network Optics & Fabric" : 2167 "AI Systems Personnel" : 3000 The True Driver: 96% KV Cache Hit Rate While large language model serving is often treated as compute-bound selling of FLOPs, DeepSeek's operational model primarily monetizes memory bandwidth and cache reuse: - Without tiered 72-hour KV caching, a 77K prompt requires full quadratic forward passes across Tensor Cores, incurring massive compute cost. - With 96% of input tokens hitting cache, the prefill compute burden drops by 96%. The cluster reads stored states and passes execution directly to high-throughput Wide-EP Decode supernodes. - Takeaway: MLA's ultra-compact latent state combined with a 96% KV hit rate decouples serving revenue from raw compute expenses. --- Layer 7: Cross-Framework Architectural Matrix How does DeepInfer contrast with other prominent inference engines in production? | Architectural Dimension | DeepSeek DeepInfer | vLLM (v1 Engine) | SGLang (FlashInfer) | TensorRT-LLM | | :--- | :--- | :--- | :--- | :--- | | Primary Design Target | High-Scale Profitable Production | Developer accessibility and universal serving | Complex multi-turn Agent & long contexts | Peak single-node NVIDIA performance | | P/D Disaggregation | Native Physical Disaggregation (1:1 Ratio) | Experimental community support | Mature distributed PD implementations | Supported via complex manual manifests | | MoE Parallelism | Native Wide-EP (64–128 GPU Supernodes)| Intra-node focus, multi-node via Ray | Cross-node EP integrated with DeepEP | Static TP/EP groupings | | Speculative Decoding | DSpark (Confidence Adaptive Tree Search) | Standard Speculative / EAGLE | Native EAGLE-2 Tree Decoding | Fixed-stride speculative decoding | | Long-Context Cache | SWA Bounded Replay + 72h Tiering | PagedAttention blocks | RadixAttention prefix trees | Basic KV paging | | Silicon Ecosystem | SM100 (Blackwell) + Huawei Ascend 950 | Broad community coverage (NVIDIA-first)| Primarily NVIDIA, expanding ROCm | NVIDIA Platforms Only | | API Protocol Layer | Custom deepseek-recipe (Rust Native) | Python Jinja template evaluation | Python abstraction layers | C++ native runtime / Triton | --- Layer 8: Production Troubleshooting and Failure Mode Playbooks Based on multi-node supernode testing with DeepInfer, four critical failure modes and their mitigations are highlighted below: 1. Wide-EP Inter-Switch Congestion and Hash Polarization - Symptom: Scaling Decode supernodes to 128 GPUs across multiple leaf switches triggers a 70% collapse in All-to-All throughput, with escalating packet drop counters (rx_prio0_drop). - Root Cause: Dense All-to-All collective exchanges cause Equal-Cost Multi-Path (ECMP) routing to suffer from Hash Polarization, directing multiple elephant flows onto identical spine uplinks and causing localized buffer exhaustion. - Remediation: Enable Adaptive Routing (AR) on InfiniBand fabrics to dynamically steer packets around congested links. For RoCEv2 environments, enforce strict DSCP traffic class mappings and hardware-level PFC deadlock detection (PFC_DEADLOCK_DETECTION_EN=1). 2. DSpark Confidence Score Drift on Extended Reasoning Traces - Symptom: During complex, multi-step code synthesis, speculative acceleration collapses from 1.6x down to 0.9x (slower than unspeculated baselines). - Root Cause: On deep reasoning chains, hidden representations deviate semantically from the draft model's training distribution. The Confidence Head overestimates acceptance probabilities, issuing low-quality draft sequences that trigger costly rollbacks. - Remediation: Configure conservative dynamic rollback safety gates: --dspark-confidence-threshold 0.85 \ --dspark-max-draft-tokens 4 \ --dspark-abort-on-rollback-rate 0.30 If rollbacks exceed 30% over three consecutive execution steps, the scheduler temporarily falls back to pure autoregressive generation. 3. Asynchronous RDMA KV Ingestion Race Conditions - Symptom: Prefill nodes pushing KV Cache blocks via RDMA into Decode GPU memory occasionally result in corrupted generation or CUDA illegal memory access crashes. - Root Cause: RDMA Write operations transfer data directly into GPU memory without host CPU intervention. If the target node's attention kernel attempts to read a block before transmission completes, it reads partial state. - Remediation: Implement a Two-Phase Memory Gate Protocol: upon completing data transfer, the sending NIC writes an 8-byte completion flag to a designated synchronization mailbox. The target GPU kernel polls this memory flag via a lightweight spinlock before unblocking attention computation. 4. Cache Eviction Storms under Tiered Storage Saturation - Symptom: Periodic latency spikes where TTFT jumps from 400ms to 5 seconds across active sessions. - Root Cause: When Tier 1 (GPU HBM) and Tier 2 (Host DRAM) reach 95% capacity thresholds simultaneously, synchronous eviction triggers thousands of session offloads to local NVMe-oF arrays, exhausting random write IOPS and stalling the PCIe bus. - Remediation: Transition to Low-Watermark Gradual Eviction: lower the eviction threshold to 80% and run a background daemon utilizing token-bucket rate limiting to stage cold blocks to NVMe disks smoothly without I/O bursts. --- Layer 9: End-to-End Production Implementation Blueprint: From Bare Metal to Cluster Acceptance Architecture and algorithmic theory must translate into actionable commands and precise physical datacenter cabling. This section provides an end-to-end implementation manual guiding infrastructure teams from hardware procurement and lossless fabric configuration to cluster orchestration, prefix-affinity gateway deployment, and SLA acceptance verification. flowchart LR P1["Phase 1: Hardware BOM<br/>& Lossless Cabling"] --> P2["Phase 2: Host OS, OFED Drivers<br/>& NUMA Initialization"] P2 --> P3["Phase 3: DeepInfer Kernel Build<br/>& Container Image Packaging"] P3 --> P4["Phase 4: Prefill Pool<br/>& Decode Supernode Launch"] P4 --> P5["Phase 5: deepseek-recipe Gateway<br/>& Prefix Affinity Routing"] P5 --> P6["Phase 6: End-to-End Benchmarking<br/>& SLA Monitoring Dashboards"] --- Phase 1: Hardware Procurement, Rack Planning & Lossless Cabling To host DeepSeek-V4.1-Flash across a standard production validation footprint, the recommended baseline cluster comprises 96 GPUs (12 8-GPU physical servers), partitioned into 4 Prefill nodes and 1 64-GPU Decode supernode: 1. Hardware Bill of Materials (BOM) - Compute Nodes (12x HGX H100/B200 or Huawei Ascend 950 8-way Servers): - CPU: Dual-socket AMD EPYC 9654 (192 physical cores) or Intel Xeon Platinum 8480+; - Memory: 2TB DDR5-4800 ECC Registered RAM (ensuring sufficient host shared memory); - Local Storage: 4x 7.68TB NVMe PCIe 5.0 SSDs (configured in RAID 0 for Tier 3 KV swapping); - GPUs: 8x NVIDIA H100 SXM5 80GB (fully meshed intra-node via NVLink at 900 GB/s); - NICs: 8x NVIDIA ConnectX-7 400Gbps OSFP single-port HCAs (dedicated to inter-GPU RDMA traffic) + 2x 100Gbps NICs (management/public traffic). - Network Fabric (Leaf-Spine Lossless Topology): - Leaf Switches: 2x NVIDIA Quantum-2 QM9700 64-port 400G NDR InfiniBand switches (or 400G/800G RoCEv2 equivalents); - Spine Switches: 2x QM9700 spine switches, establishing a non-blocking 1:1 dual-uplink fabric. 2. Network Cabling & Switch Configuration Ensure strict PCIe local affinity between GPUs and NICs, eliminating cross-socket traffic: - GPU 0 connects to the PCIe switch hosting NIC 0; GPU 1 connects to NIC 1, and so forth. - Lossless Fabric Switch Configuration (RoCEv2 Parameters): # Enable Priority Flow Control (PFC) on priority 3 and ECN marking on access ports interface Ethernet1/1-1/64 flowcontrol priority 3 pfc congestion-control ecn minimum-threshold 150000 maximum-threshold 1500000 mark-probability 10 --- Phase 2: Host Operating System, Driver & NUMA Optimization Apply uniform kernel and fabric tuning across all 12 physical nodes. 1. Kernel Network and Memory Configurations (/etc/sysctl.d/99-ai-infra.conf) # Disable transparent hugepage compaction to protect pinned GPU memory allocations vm.transparent_hugepage = never # Expand socket read/write buffers and network backlog net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.core.rmem_default = 67108864 net.core.wmem_default = 67108864 net.core.optmem_max = 67108864 net.core.netdev_max_backlog = 250000 # Enable ECN congestion notification net.ipv4.tcp_ecn = 1 # Expand memory map limits vm.max_map_count = 2621440 Apply via: sudo sysctl --system 2. Hardware Topology & GPUDirect RDMA Verification Execute self-checks to ensure all 8 adapters operate at full bandwidth: # 1. Verify InfiniBand/RoCEv2 adapter state and link speeds (400 Gbps Active) ibstat | grep -E "(CA '|State:|Physical state:|Rate:)" # 2. Verify GPU topology (All links must indicate NVLink NV18/NV9 interconnects) nvidia-smi topo -m # 3. Lock GPUDirect RDMA levels and export topology cache export NCCL_NET_GDR_LEVEL=5 export NCCL_TOPO_DUMP_FILE=/etc/ai-infra/nccl_topo.xml export NCCL_SET_STACK_SIZE=1 --- Phase 3: DeepInfer Kernel Compilation & Container Image Packaging To avoid dependency conflicts in production, encapsulate custom kernels (FlashMLA, DeepGEMM, DeepEP, TileLang) in a multi-stage Docker build: # syntax=docker/dockerfile:1.4 FROM nvidia/cuda:13.0.0-devel-ubuntu24.04 AS builder WORKDIR /build # Install compilation toolchains and Rust RUN apt-get update && apt-get install -y --no-install-recommends \ git cmake ninja-build python3-dev python3-pip curl pkg-config libssl-dev && \ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y ENV PATH="/root/.cargo/bin:${PATH}" # 1. Build DeepInfer custom kernels RUN git clone --recursive https://github.com/deepseek-ai/DeepInfer.git /build/DeepInfer && \ cd /build/DeepInfer && \ pip install --no-cache-dir -r requirements.txt && \ python3 setup.py bdist_wheel # 2. Build deepseek-recipe protocol parser RUN git clone https://github.com/deepseek-ai/deepseek-recipe.git /build/deepseek-recipe && \ cd /build/deepseek-recipe && \ cargo build --release && \ python3 -m pip install maturin && maturin build --release # Production runtime image FROM nvidia/cuda:13.0.0-runtime-ubuntu24.04 WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends \ python3 python3-pip numactl infiniband-diags ibverbs-utils libibverbs1 && \ rm -rf /var/lib/apt/lists/* COPY --from=builder /build/DeepInfer/dist/*.whl /tmp/ COPY --from=builder /build/deepseek-recipe/target/wheels/*.whl /tmp/ RUN pip install --no-cache-dir /tmp/*.whl && rm -rf /tmp/*.whl ENV PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:128" ENV NCCL_NET_GDR_LEVEL=5 ENTRYPOINT ["python3", "-m", "deepinfer.entrypoints.server"] --- Phase 4: Cluster Orchestration and Distributed Launch Scripts Prefill nodes and Decode supernodes initialize with distinct topology roles. 1. Prefill Node Pool Launch Script (launch_prefill_pool.sh) Execute across 4 Prefill nodes (IPs: 192.168.10.101 ~ 104): #!/usr/bin/env bash # Prefill node: Intra-node TP=8, RDMA one-sided write enabled export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export HOST_IP=192.168.10.101 numactl --interleave=all python3 -m deepinfer.entrypoints.server \ --model-path /models/DeepSeek-V4.1-Flash \ --role prefill \ --host 0.0.0.0 \ --port 8001 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 1 \ --max-model-len 131072 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.90 \ --kv-transfer-protocol rdma \ --rdma-device-names mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 2. Decode 64-GPU Supernode Launch Script (launch_decode_supernode.sh) Execute across 8 Decode physical servers (IPs: 192.168.10.111 ~ 118) with TP=8, EP=8: #!/usr/bin/env bash # On Decode Head Node (192.168.10.111) and 7 companion workers export NCCL_SOCKET_IFNAME=eth0 export NCCL_IB_GID_INDEX=3 export NCCL_ALGO=Tree python3 -m deepinfer.entrypoints.server \ --model-path /models/DeepSeek-V4.1-Flash \ --role decode \ --host 0.0.0.0 \ --port 8002 \ --cluster-nodes 8 \ --tensor-parallel-size 8 \ --expert-parallel-size 8 \ --max-model-len 131072 \ --max-num-seqs 128 \ --gpu-memory-utilization 0.92 \ --dspark-enable \ --dspark-confidence-threshold 0.85 \ --dspark-max-draft-tokens 6 \ --swa-bounded-replay \ --kv-cache-ttl-hours 72 \ --tiered-kv-storage-path /mnt/nvme_tier3_kv/ 3. Kubernetes / KubeRay Manifest Configuration In Kubernetes environments, mount /dev/shm backed by memory alongside RDMA device limits: apiVersion: ray.io/v1

DeepSeek Opens DeepInfer: What Developers Still Need to Build a Serving Stack DeepSeek has open-sourced DeepInfer, the inference framework behind its large-scale production services, alongside deepseek-recipe, a component for converting API requests into model inputs and parsing outputs. For developers considering self-hosted DeepSeek applications, the release exposes two useful layers: model execution and protocol handling. The engineering question is how much infrastructure remains between those layers and a reliable application endpoint. DeepSeek’s deployment Q&A makes that boundary explicit. Parts of its production system, including KV cache management and load balancing, remain tied to internal infrastructure and are not yet fully open-sourced. That distinction matters when estimating both implementation effort and potential savings. The release gives infrastructure teams a concrete starting point. The DeepInfer repository currently documents support for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It provides offline inference and is licensed under Apache 2.0. Those requirements narrow the immediate audience. A team with access to supported accelerators can investigate the execution engine directly. A developer working on a laptop should begin by studying the architecture or experimenting with the separate protocol component, rather than budgeting for a local DeepInfer deployment. DeepSeek describes an architecture that separates vision encoding, prefill, and decode, allowing resources to be allocated around the demands of each stage. Its Q&A also describes distributed execution and speculative decoding. These are useful implementation references for teams already operating inference infrastructure, but they do not establish a universal cost reduction for every workload. Consider a hypothetical application that generates campaign variants overnight. Its initial requirement is a batch of completed outputs, so offline inference offers a relevant starting point. An interactive campaign assistant has additional needs: admission control, streaming delivery, cancellation, and predictable latency under concurrent traffic. Those requirements should appear in the project estimate before a team promises a production migration. Protocol conversion is a separate integration opportunity. The deepseek-recipe repository contains Rust libraries with Python bindings. It converts Messages, Chat Completions, and Responses requests into a shared conversation representation, renders DeepSeek prompts, encodes tokens with an attached tokenizer, and converts generated output into the corresponding response format. The project uses the MIT license. For example, imagine a product with several clients that already submit different request formats. A shared conversion layer could reduce duplicated prompt-rendering and response-parsing code. That is an implementation opportunity, not evidence that switching an existing service will require no changes. The repository explicitly assigns model inference, tool execution, and HTTP transport to the caller. An application receiving a tool call must therefore retain its own execution policy and dispatch logic. Supporting the representation of a tool call does not supply the application that executes it. Structured output has a similar boundary. Recipe can convert format requirements and produce an output constraint, but the caller decides whether the inference backend enforces that constraint through constrained decoding. For a lead-enrichment application expecting JSON, developers should test the complete request-to-response path and validate required business fields before passing results downstream. A first experiment can stay deliberately small. For DeepInfer, the documented build prerequisites are Linux, GCC 13 or newer, and Python 3.12, with CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The getting-started instructions describe selecting the accelerator and building from source in an empty Python environment. A practical evaluation sequence is: 1. Check out the repository and record the commit used. Confirm that the accelerator and software environment match the documented prerequisites. 2. Follow the build instructions, then run python examples/offline_inference.py --help to inspect the available arguments before choosing a configuration. 3. Run the supplied offline example on the supported environment. The default checkpoint is downloaded into the Hugging Face cache; the example also accepts a local model directory. 4. Enable --check_structured_output when evaluating constrained generation. The documented checks cover regex, JSON, and EBNF outputs and normal end-of-sequence termination. 5. Add a small application-specific input set and compare output validity, completion time, and resource use against the service you currently operate. The checkpoint handling and structured-output checks are documented in the official examples. These commands and steps have not been executed for this article; they describe a starting procedure, not a tested deployment recipe or a statement of hardware capacity. Keep the first workload representative. For the campaign example, include short product descriptions, longer brand guidance, and requests for several output variants. Record input and output lengths so that a faster run is not mistaken for an improvement when it simply generated less text. The business case depends on the service around the engine. DeepInfer’s README lists a KV cache system, vision/prefill/decode disaggregation, and load balancing across data-parallel replicas among the requirements for a complete service that are not included in the release. Although much of the engine functionality needed for disaggregation is implemented, assembling the surrounding system remains substantial work. For a growth team, the useful economic measure is cost per accepted result. Include accelerator rental, idle capacity, integration work, and operational support, then account for outputs rejected by application validation. A lower engine execution time alone does not demonstrate a lower cost for a usable campaign or completed workflow. Teams with supported hardware and inference expertise now have access to DeepSeek’s execution framework and a reusable protocol layer. The immediate opportunity is to build a bounded offline prototype, identify which surrounding components it needs, and use measured application results to decide whether operating the full service is worthwhile.