Building a Distributed LLM Cluster from Idle Apple Silicon Macs (Corrected Guide)
> Correction (October 2026): an earlier version of this post contained invented commands (exo cluster init --config), an impossible llama.cpp setting ("35 of 32 layers"), a Ray GPU-scheduling setup that doesn't apply to Apple GPUs, unsupported performance and electricity figures, and a link to an unrelated product. It has been rewritten; everything below uses real tools and flags.
Most developers with more than one Apple Silicon Mac leave the extra machines idle. Because the CPU and GPU share one pool of unified memory, even a mid-range Mac can hold a quantized model that would need a large discrete GPU elsewhere. Here is how to turn several of them into one LLM service, using tools that actually exist today.
First, two ways to "cluster"
- Replicas (throughput): each Mac runs its own copy of a model that fits in its memory, and a load balancer spreads requests. More users served, same latency per request.
- Split (capacity): one model is too big for any single Mac, so its layers are spread across machines. This lets you run a bigger model; it does not make it faster than one Mac that could hold it, because every token crosses the network.
Pick replicas unless the model genuinely doesn't fit.
Inventory each node
On every Mac:
system_profiler SPHardwareDataType | grep -E "Chip|Memory"
Write down chip and memory per node. Memory decides what each node can serve. A rough rule for Q4_K_M quantization is ~0.6 GB per billion parameters plus a few GB for context: an 8B model is a ~5 GB file, a 70B model ~43 GB.
Inference: llama.cpp, natively
Run the server natively, not in Docker: containers on macOS run in a Linux VM with no access to the Apple GPU.
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
./build/bin/llama-server -m ~/models/model.gguf -ngl 99 -c 8192 --host 0.0.0.0 --port 8080
Metal is enabled by default on macOS. -ngl 99 simply means "offload all layers to the GPU"; on Apple Silicon there's no separate VRAM to budget, the limit is how much unified memory macOS lets the GPU wire (adjustable on macOS 14+ with sudo sysctl iogpu.wired_limit_mb=<MB>, leaving several GB for the OS).
Replicas: one endpoint with nginx
Run llama-server on each Mac, then put nginx in front on one of them:
upstream llama {
least_conn;
server 10.0.1.10:8080;
server 10.0.1.11:8080;
}
server {
listen 8000;
location / {
proxy_pass http://llama;
proxy_buffering off; # keep streamed tokens flowing
proxy_read_timeout 600s;
}
}
llama-server exposes an OpenAI-compatible /v1/chat/completions, so clients just point at port 8000.
Split: llama.cpp RPC
On each worker, expose its backend over RPC — bind it to a private interface, because rpc-server has no authentication:
./build/bin/rpc-server -H 10.0.9.2 -p 50052
On the main node, load the model across itself and the workers:
./build/bin/llama-server -m ~/models/big-model.gguf -ngl 99 --rpc 10.0.9.2:50052
Two Macs linked by a Thunderbolt cable get a Thunderbolt Bridge network interface; giving it static IPs and using it for RPC traffic gives far more bandwidth than gigabit Ethernet.
An alternative for split inference is exo, which discovers other exo nodes on the local network automatically and shards a model across them; follow its README for the current install steps, as the project moves fast.
Keep it running and observable
- Supervise llama-server with a launchd agent (KeepAlive + RunAtLoad) so it survives crashes and reboots.
- Keep inference Macs awake: sudo pmset -a sleep 0 disksleep 0, and enable Wake for network access if you want to wake them with a Wake-on-LAN packet.
- llama-server has a /health endpoint for checks; sudo powermetrics --samplers gpu_power -i 1000 -n 1 shows GPU power on a node if you want to feed your own metrics exporter.
For cost, do the arithmetic for your own hardware: measured watts × hours × your electricity price. For example, a hypothetical cluster averaging 120 W for 24 hours uses 2.88 kWh.
---
If you want the full setup — Ansible playbooks, launchd templates, nginx config and RPC split mode — I packaged it as Turn 2+ Macs Into One LLM Server.
This article was drafted with AI assistance.
Mentions
6) exo joins your devices into one AI machine You open it on two Macs on the same network and they find each other with no setup. Together they can run models too big for one machine. The Mac app needs Apple silicon. exo-explore/exo
For those pointing out interconnect etc. etc. Yes good points, but see: exo-explore/exo and from Apple themselves: machinelearning.apple.com (thank you to @someRandomDev5 for this link)
@CloudyBRM From Apple themselves: machinelearning.apple.com Also see: exo-explore/exo
@cozy_compute Yes and no... check @exolabs exo-explore/exo
家里有两三台 Mac，想在本地跑个大点的模型，单台的内存总是差一截。 exo 能把这些设备连成一个 AI 集群，一台装不下的模型，拆开分给几台一起跑，已狂揽 4.8 万多 Star。 设备之间会自动发现，不用手动配置，它会看每台机器的内存和网速，自己决定模型怎么切。 GitHub：exo-explore/exo 设备越多跑得越快，官方给的数据是 2 台提速 1.8 倍，4 台提速 3.2 倍。 README 里放了 4 台 Mac Studio 一起跑 6710 亿参数 DeepSeek v3.1 的实测，几台机器摆在一起跑模型，画面挺有冲击力。 Mac 上有现成的 App，在后台运行，自带管理面板，能看集群状态，也能直接跟模型聊天。 接口兼容 OpenAI、Claude 和 Ollama 的格式，手上已经在用的工具直接连过去就行。
Building a Distributed LLM Cluster from Idle Apple Silicon Macs (Corrected Guide) > Correction (October 2026): an earlier version of this post contained invented commands (exo cluster init --config), an impossible llama.cpp setting ("35 of 32 layers"), a Ray GPU-scheduling setup that doesn't apply to Apple GPUs, unsupported performance and electricity figures, and a link to an unrelated product. It has been rewritten; everything below uses real tools and flags. Most developers with more than one Apple Silicon Mac leave the extra machines idle. Because the CPU and GPU share one pool of unified memory, even a mid-range Mac can hold a quantized model that would need a large discrete GPU elsewhere. Here is how to turn several of them into one LLM service, using tools that actually exist today. First, two ways to "cluster" - Replicas (throughput): each Mac runs its own copy of a model that fits in its memory, and a load balancer spreads requests. More users served, same latency per request. - Split (capacity): one model is too big for any single Mac, so its layers are spread across machines. This lets you run a bigger model; it does not make it faster than one Mac that could hold it, because every token crosses the network. Pick replicas unless the model genuinely doesn't fit. Inventory each node On every Mac: system_profiler SPHardwareDataType | grep -E "Chip|Memory" Write down chip and memory per node. Memory decides what each node can serve. A rough rule for Q4_K_M quantization is ~0.6 GB per billion parameters plus a few GB for context: an 8B model is a ~5 GB file, a 70B model ~43 GB. Inference: llama.cpp, natively Run the server natively, not in Docker: containers on macOS run in a Linux VM with no access to the Apple GPU. git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp cmake -B build -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release cmake --build build -j ./build/bin/llama-server -m ~/models/model.gguf -ngl 99 -c 8192 --host 0.0.0.0 --port 8080 Metal is enabled by default on macOS. -ngl 99 simply means "offload all layers to the GPU"; on Apple Silicon there's no separate VRAM to budget, the limit is how much unified memory macOS lets the GPU wire (adjustable on macOS 14+ with sudo sysctl iogpu.wired_limit_mb=<MB>, leaving several GB for the OS). Replicas: one endpoint with nginx Run llama-server on each Mac, then put nginx in front on one of them: upstream llama { least_conn; server 10.0.1.10:8080; server 10.0.1.11:8080; } server { listen 8000; location / { proxy_pass http://llama; proxy_buffering off; # keep streamed tokens flowing proxy_read_timeout 600s; } } llama-server exposes an OpenAI-compatible /v1/chat/completions, so clients just point at port 8000. Split: llama.cpp RPC On each worker, expose its backend over RPC — bind it to a private interface, because rpc-server has no authentication: ./build/bin/rpc-server -H 10.0.9.2 -p 50052 On the main node, load the model across itself and the workers: ./build/bin/llama-server -m ~/models/big-model.gguf -ngl 99 --rpc 10.0.9.2:50052 Two Macs linked by a Thunderbolt cable get a Thunderbolt Bridge network interface; giving it static IPs and using it for RPC traffic gives far more bandwidth than gigabit Ethernet. An alternative for split inference is exo, which discovers other exo nodes on the local network automatically and shards a model across them; follow its README for the current install steps, as the project moves fast. Keep it running and observable - Supervise llama-server with a launchd agent (KeepAlive + RunAtLoad) so it survives crashes and reboots. - Keep inference Macs awake: sudo pmset -a sleep 0 disksleep 0, and enable Wake for network access if you want to wake them with a Wake-on-LAN packet. - llama-server has a /health endpoint for checks; sudo powermetrics --samplers gpu_power -i 1000 -n 1 shows GPU power on a node if you want to feed your own metrics exporter. For cost, do the arithmetic for your own hardware: measured watts × hours × your electricity price. For example, a hypothetical cluster averaging 120 W for 24 hours uses 2.88 kWh. --- If you want the full setup — Ansible playbooks, launchd templates, nginx config and RPC split mode — I packaged it as Turn 2+ Macs Into One LLM Server. This article was drafted with AI assistance.
Interesting, so this is basically Apple's version of exo ? exo-explore/exo