Workflow · cluster

Distributed inference across Macs

Run a model that does not fit any single machine by sharding its layers across two or more Macs over Thunderbolt. One OpenAI-compatible endpoint, the same serving code path a single Mac uses.

What this is for

Pipeline parallelism is a capacity feature, not a speed feature. Rank 0 runs its layers, hands the activation to rank 1, and waits. There is no overlap within a single request, so every token pays a network hop and you gain nothing by splitting a model that already fits.

Read this first A model that fits on one Mac is slower when sharded. We measure Qwen3.5-0.8B at 5.0 ms/token on one Mac and 13.4 ms/token across two, 2.7× worse. Reach for a cluster when the alternative is a degraded path: streaming experts off the SSD, a smaller quant, or not running the model at all.

Where it does win: Qwen3.5-122B-A10B-OptiQ-2bit is 42.8 GiB of weights. On a single 36 GB Mac it only runs by streaming MoE experts from the SSD, at 4.9 tok/s. Split across a 36 GB M3 Max and a 24 GB M4 it is fully resident and decodes at 20.5 tok/s. That is 4.2× faster, because the comparison is against a crippled path rather than a healthy one.

Setup

You need SSH between the Macs, a Thunderbolt cable, and the model cached on each node (or the peers download it on first serve). Connect the Macs and macOS creates a bridge0 interface with link-local addresses; OptiQ finds them itself.

on every machinebash

pip install mlx-optiq
optiq cluster up          # advertise this Mac over Bonjour

on the machine you will drivebash

optiq cluster peers       # who is reachable, on what link

Link-local IPs change on reboot. Discovery re-resolves them every time; nothing is pinned to a file.

Install at the same relative path on every Mac The ring is launched by ssh-ing into each peer and running cd <dir> && <dir>/<venv>/bin/python3 from that peer's home. Usernames differ between machines, so the path is resolved relative to each home. Put the project (and its virtualenv) in the same place on both, for example ~/optiq/.venv on each. Or point at it explicitly with --cwd / OPTIQ_CLUSTER_CWD. By default OptiQ uses the virtualenv it is running inside.

optiq cluster serve measures the real round-trip to each peer and refuses a slow link by default. Pipeline inference over Wi-Fi is slower than one Mac, because every token pays the latency. It fails loudly instead of quietly degrading. Pass --link any if you want it anyway. Thunderbolt is ~0.6 ms; Wi-Fi is ~85 ms.

From the Lab

Settings → Cluster shows the discovered Macs on the ring with each node's free memory, swap pressure, and how many layers it would take. The number it reports is what the ring can spare right now, not the RAM you bought.

OptiQ Lab cluster page showing two Macs on a Thunderbolt ring with per-node memory — Discovered peers, free memory per node, and the combined capacity.

Enter a model and press Start serving. A 40 GB model takes time to page in, so chat does not switch to the cluster until the endpoint answers. You get a loading state rather than a connection error.

The cluster page after the model has loaded, showing it as the active inference backend — Once every rank has materialized its shard, the cluster becomes the Lab's inference backend.

Chat then talks to the ring exactly as it talks to a single Mac: streaming, tool calls, structured output, LoRA adapters. Below, the 122B writing a flight simulator, its HTML running in the Lab's sandboxed artifact panel.

Lab chat streaming code from the sharded 122B, with the resulting page running in the artifact panel — Qwen3.5-122B-A10B-OptiQ-2bit, resident across two Macs, at ~20 tok/s.

From the CLI

serve a sharded modelbash

optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --port 8100

outputtext

[rank 0] materialized  8/30 layers | resident  9.6 GiB
[rank 0] materialized 16/30 layers | resident 15.7 GiB
[rank 0] materialized 24/30 layers | resident 21.7 GiB
[rank 1/2] layers 30-47 (18/48) | loaded 7.4s | resident 16.6 GiB | cap 17.3 GiB (free 21.4, GPU limit 22.0)
[rank 0/2] layers  0-29 (30/48) | loaded 7.8s | resident 26.2 GiB | cap 26.6 GiB (free 27.0, GPU limit 30.0)
[cluster] OpenAI endpoint: http://127.0.0.1:8100/v1/chat/completions
[cluster] 2 nodes, model=mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit. Ctrl+C to stop.

42.8 GiB resident across two Macs, loaded in 26 s. The endpoint is ordinary OpenAI; point any client at it.

Rank 0 owns the embedding, the final norm, and the LM head, and drives the ring. The last rank returns a hidden state rather than logits, so what crosses the wire each token is ~12 KB instead of ~1 MB of vocabulary.

The memory rules

Layers are held resident, so the model must fit the memory the cluster can spare at that moment. Boundaries follow each node's free memory and the real per-layer byte counts. The Mac you are working on takes fewer layers than an idle one, and a MoE's heavier late blocks are counted as they actually are.

Each rank keeps a run-reserve free for activations, the KV cache, and MLX's buffer pool. Before a single weight is evaluated, OptiQ prices the model against what the ring can spare and refuses to load rather than swap a machine to death:

preflight refusaltext

mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit does not fit the cluster's *currently free* memory.
  rank 0: needs  24.4 GiB | can spare  25.0 GiB (free 25.9 GiB - 0.9 GiB to run)
  rank 1: needs  18.4 GiB | can spare  18.3 GiB (free 21.5 GiB - 3.2 GiB to run)   <-- does not fit

Weights 42.8 GiB vs 43.3 GiB spare across 2 node(s).
Fix: free memory on the node(s) above (quit apps; a Mac whose swap is full needs a
reboot), use a smaller quant, add a Mac, or lower the run-reserve with
OPTIQ_CLUSTER_HEADROOM_GB=<gb> (risks swapping).

Do not tune this to force a fit OPTIQ_CLUSTER_HEADROOM_GB exists so you can raise the reserve, not shrink it. We forced a 42.8 GiB model onto a pair that could spare 42.3 GiB by dropping the reserve to 0.8 GiB per node. It loaded cleanly, then the smaller Mac swapped 2.1 GB and one generation ran for 22 minutes without finishing. A refusal costs you nothing; a swap storm costs you the machine.

The smaller Mac has a residency ceiling

There is a second limit, and it is the one that surprises people. Past roughly 73% of its RAM in resident weights, a Mac's GPU throughput collapses. System memory is free, nothing swaps, and no error is raised. On a 24 GB M4 the cliff sits near 17.5 GiB:

M4 (24 GB) · resident shard vs throughput

Resident weights	iogpu.wired_limit_mb	Throughput
16.6 GiB	20480	23.3 tok/s
17.3 GiB	20480	15.1 tok/s
18.1 GiB	20480	0.1 tok/s
18.1 GiB	22528	0.1 tok/s
19.6 GiB	22528	0.05 tok/s (swapping)

Raising iogpu.wired_limit_mb does not move it. 18.1 GiB reads 0.1 tok/s at both a 20 and a 22 GiB limit. If a ring loads but crawls, the smallest node is holding too many layers. Give it a larger reserve so it takes fewer. That is the opposite of what instinct suggests.

push layers off the smaller nodebash

# per-rank reserve, in the ring's rank order
OPTIQ_CLUSTER_HEADROOM_GB="0.4,4.1" optiq cluster serve --model <repo>

rank 0 (36 GB) takes 30 layers / 26.2 GiB; rank 1 (24 GB) takes 18 / 16.6 GiB. 20.5 tok/s.

What to expect

Qwen3.5-122B-A10B-OptiQ-2bit · 42.8 GiB · M3 Max 36 GB + M4 24 GB

Path	Resident	Steady throughput
Two Macs, sharded	42.8 GiB across 2	20.5 tok/s
One Mac, SSD expert streaming	9.97 GiB	4.9 tok/s
One Mac, fully resident	does not fit	—

Load takes 26 s end to end; weights are memory-mapped and materialized layer by layer. Throughput is flat in context. The gap between the two paths does not close as the conversation grows.

Benchmarking note The first forward compiles Metal kernels, which takes 13–79 s depending on memory pressure. A single-shot throughput measurement on a freshly loaded ring reads ~0.1 tok/s on a perfectly healthy cluster. Always warm up with a short request and time the second one.

Troubleshooting

It refuses to load

Believe it. Quit apps on the node flagged <-- does not fit. A Mac whose swap is full stays degraded until it reboots. macOS never shrinks the swap file. The Lab's cluster page shows swap per node and will tell you which machine to restart.

It loads, then crawls

The smallest node is over its residency ceiling. Raise its reserve so it takes fewer layers. Do not raise iogpu.wired_limit_mb; it will not help.

It refuses the link

You are on Wi-Fi. Connect the Thunderbolt cable and check ifconfig bridge0 reports status: active on both machines.

A peer cannot find python

zsh: no such file or directory: .venv/bin/python3 means the peer does not have mlx-optiq where this Mac does. The path is relative to each machine's home directory. Install it at the same relative location on every node, or pass --cwd.

The ranks disagree about the model

Every node needs the same complete checkout in its Hugging Face cache. A partially downloaded copy on one peer makes the ranks compute different shard boundaries and the ring hangs at startup.