Mesh-LLM

Mesh LLM: Distributed AI/LLM for the people (3.7k stars) What it does All text below is quoted directly from the project README. >Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat. How the mesh works: - Single-machine fit first. If one node can host the full model, it serves the model locally without stage traffic. - Mesh routing. Every node exposes the same /v1 API. Requests are routed by the model field to the peer that can serve that model. - Encrypted peer transport. QUIC end-to-end encrypts traffic between Mesh nodes, including inference requests, responses, and split-model activations. - Skippy stage splits. Large dense models can load as package-backed layer stages. - Layer packages. Package repositories contain model-package.json plus GGUF fragments so peers fetch only the pieces needed for their assigned stage. - Public discovery. Published meshes advertise through Nostr discovery; private meshes stay invite-token based. >curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash Repository Mesh-LLM/mesh-llm Repository stats |Metric|Value| |:-|:-| |Stars|3,660| |Forks|476| |Last release|v0.78.1 (2026-10-05)| |Languages|Rust 86.1%, TypeScript 9.9%, Shell 1.4%| |Contributors|51| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|@Copilot (74 commits), @Copilot (48 commits), @claude (22 commits), @codex (1 commits)| |AI co-authored commits|Claude: 22, Codex: 1, Copilot: 49| Main contributor |Field|Value| |:-|:-| |Name|James Dumay| |GitHub|@i386| |Commits to this repo|1,068 (top contributor)| |Bio|A coder| |Company / Location|Sydney, Australia| |Followers|146| ────────────────────────────── 📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com 🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.

单机显存塞不下大模型，多台机器又各跑各的？ mesh-llm 把多机 GPU 和内存拼成一张网，对外就是一个 OpenAI 兼容接口，模型太大还能分段跑，节点随时加。 适合手头有几台机器想合起来跑模型的人，先从两台试起。 你会先拼几台，两台还是更多？ #分布式 #LLM @grok Mesh-LLM/mesh-llm

mesh-llm GitHub Overview: How Private Layer Splits Pool Local AI Memory mesh-llm by mesh-llm: mesh-llm/mesh-llm Mesh-LLM is an open-source distributed inference system for running open-weight language models across owner-controlled computers when a model exceeds one machine's memory. This overview explains how contiguous transformer layers can be split across two hosts, with each host fetching its assigned layer fragments and exposing an OpenAI-compatible local endpoint. Private invite tokens and encrypted QUIC connect the mesh, while model weights stay resident on their assigned machines and compact activation tensors cross the stage boundary. The architecture exchanges aggregate memory for generation speed: sequential network latency compounds per token, with documented examples moving from 68 tokens per second on one machine to 21 across two and 13 across three. The 80 ms round-trip cap and single-machine-fit-first policy frame when distributed inference is appropriate. TimeStamps: 0:00 Why large language models exceed one machine 0:58 Mesh-LLM as a distributed memory bridge 1:30 Stage splitting across two computers 1:40 Private invite-token mesh formation 1:59 Selective layer packages and local weights 2:39 Activation handoff through the inference pipeline 3:26 Sequential latency and throughput trade-offs 4:19 The 80 ms cap and single-machine fit first 🧠 Aggregate memory for models that exceed one host 🔐 Invite-token access and encrypted peer transport 🧩 Layer packages keep weights at their assigned stages ⚡ Activation handoff trades speed for capacity The key takeaway is architectural: mesh-llm makes memory capacity the reason to distribute inference, not speed. When a model fits on one machine, local execution remains the baseline; when it does not, private layer packages, stage-local weights, encrypted activation transport, and an OpenAI-compatible endpoint provide a structured path to local multi-host serving. #MeshLLM #DistributedInference #LocalAI [de-DE] YNAS ynp_01841edc90d64da4 Cinematic Privater automatisierter Staging-Upload. Paketidentität: ynp_01841edc90d64da4; Ausgabe: filmisch. [es-US] YNAS ynp_01841edc90d64da4 cinemático Carga de entorno de prueba de automatización privada. Identidad del paquete: ynp_01841edc90d64da4; salida: cinematográfica. [fr-FR] YNAS ynp_01841edc90d64da4 cinématique Téléchargement de l'environnement de test d'automatisation privé. Identifiant du package : ynp_01841edc90d64da4 ; sortie : cinématique. [hi] YNAS ynp_01841edc90d64da4 सिनेमैटिक निजी स्वचालन स्टेजिंग अपलोड। पैकेज पहचान: ynp_01841edc90d64da4; आउटपुट: सिनेमाई। [id] YNAS ynp_01841edc90d64da4 sinematik Unggahan staging otomatisasi pribadi. Identitas paket: ynp_01841edc90d64da4; output: sinematik. [it] YNAS ynp_01841edc90d64da4 cinematic Caricamento di automazione privata. Identità del pacchetto: ynp_01841edc90d64da4; output: cinematografico. [pt-BR] YNAS ynp_01841edc90d64da4 cinematográfico Upload de teste de automação privada. Identificação do pacote: ynp_01841edc90d64da4; saída: cinemática.

@AbdelStark Working on something similar mesh-llm#2665

@JSimonDev @kimmonismus It already exists just check out mesh-llm: Mesh-LLM/mesh-llm meshllm.cloud

@sudoingX @niklaslenz_ai What are you working on? Have y’all seen: Mesh-LLM/mesh-llm Could be helpful

@Cyndesama @alxfazio With Mesh LLM! Baked into Buzz. Share your compute with your friends to power your agents engineering.block.xyz Mesh-LLM/mesh-llm