A 122B model, two Macs, and a flight simulator.

Qwen3.5-122B-A10B ships as 244 GB of bf16 weights. Below, a 2-bit quant of it is flying a flight simulator that it wrote, in a browser, on two laptops connected by a cable.

42.8 GiB of weights, held resident across a 36 GB M3 Max and a 24 GB M4 over Thunderbolt. Neither machine can hold the model on its own. Together they decode at 20.5 tok/s. Loading takes 26 seconds. OptiQ 0.3 makes that one command.

optiq cluster up                       # on every Mac
optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit

One model behind the ring

The distributed path is not a second implementation of generation. DistributedModel exposes the whole ring to mlx-lm as a single model: rank 0 owns the embedding, some layers, the final norm and the LM head, and implements the ordinary model(inputs, cache) forward by driving the ring. It runs its layers, passes the activation down the line, and the last rank hands back a hidden state that rank 0 projects through the head.

mlx_lm.stream_generate never learns any of this happened. Token streaming, tool calls, structured output, samplers and stop tokens all work through the exact code path a single Mac uses, with no cluster-specific reimplementation. What crosses the wire each token is a 12 KB hidden state rather than a megabyte of vocabulary logits.

[rank 1/2] layers 30-47 (18/48) | loaded 7.4s | resident 16.6 GiB
[rank 0/2] layers  0-29 (30/48) | loaded 7.8s | resident 26.2 GiB
[cluster] OpenAI endpoint: http://127.0.0.1:8100/v1/chat/completions

What you actually get

Throughput is flat in context: the gap between the two paths does not close as the conversation grows. Nine end-to-end checks pass against the live ring, including greedy determinism across ranks, a multi-turn KV cache that survives the pipeline split, and a 1.2k-token prompt prefilling across both shards.

The demo

We asked it, through OptiQ Lab's chat, for "a beautiful, relaxing flight simulator in a single html page." It streamed 12 KB of self-contained canvas code at 17 tok/s, and the Lab ran it in a sandboxed panel next to the transcript. That is the animation at the top of this page: a plane the model drew, banking over terrain the model generated, driven by a physics loop the model wrote, decoded by 122 billion parameters spread across two laptops.

Quality varies. Six samples came off the ring and two ran. Two reached for Three.js and got it wrong. One imported an addon by bare specifier. One called relativeCameraOffset.applyMatrixMatrix(), a method that does not exist. One drew a sky, terrain and a HUD, and no plane. One drew the plane, then wrote // Boundaries (Soft clamp) — we don't want to lose the player and never implemented the clamp. The plane flew off-screen in seconds.

That last failure is the interesting one: the model knew what it needed, wrote the comment, and did not do it. Adding "never let the plane leave the visible canvas" and "no external scripts" to the prompt fixed both classes at once, and the next sample ran clean. Two of six is a more honest number for what a 2-bit quant does when asked to write a working program than any benchmark score, and it is better than it sounds, because the failures are cheap to detect and a retry costs three minutes.

The memory rules

Layers are held resident, so the model must fit the memory the cluster can spare at that moment. Boundaries follow each node's free memory and the real per-layer byte counts, so the Mac you are working on takes fewer layers than an idle one. Before a single weight is evaluated, a preflight prices the model against what the ring can spare and refuses to load rather than swap a machine to death:

mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit does not fit the cluster's *currently free* memory.
  rank 0: needs  24.4 GiB | can spare  25.0 GiB (free 25.9 GiB - 0.9 GiB to run)
  rank 1: needs  18.4 GiB | can spare  18.3 GiB (free 21.5 GiB - 3.2 GiB to run)   <-- does not fit

That refusal exists because we once overrode it. Dropping the per-node run-reserve to 0.8 GiB made a 42.8 GiB model fit a pair with 42.3 GiB to spare. It loaded cleanly. Then the M4 swapped 2.1 GB and a single generation ran for twenty-two minutes without finishing.

The smaller Mac has a residency ceiling

There is a second limit, and it is the one that surprises people. Past roughly 73% of its RAM in resident weights, a Mac's GPU throughput collapses. System memory is free, nothing swaps, and no error is raised. On a 24 GB M4 the cliff sits near 17.5 GiB:

Raising iogpu.wired_limit_mb does not move it: 18.1 GiB collapses identically at a 20 GiB limit and a 22 GiB one. If a ring loads but crawls, the smallest node is holding too many layers. Give it a larger reserve so it takes fewer. That is the opposite of what instinct suggests.

Engineering notes

Almost none of the two evenings this took went into the sharding code. It went into four wrong explanations, each of which came from a measurement that looked healthy.

The loader that quantized noise

mlx-lm builds the model with randomly initialized float weights, calls nn.quantize() on them, and only then loads the real tensors. The middle step is pure waste: a full copy of the model is built and evaluated before being discarded. On a 42.8 GiB checkpoint that transient peaks near 53 GiB, so a 36 GB Mac swaps ~35 GB to open a model it can comfortably hold, and a 24 GB Mac is killed inside load().

optiq.runtime.fast_load installs the checkpoint's own memory-mapped arrays into the quantized modules instead of quantizing noise. On the 122B on a 24 GB M4 the peak falls from 52.73 to 0.00 GiB, the load from 72 s to 1.3 s, and swap from 20.9 GB to nothing. It was never a cluster bug. It bites every caller of mlx_lm.load, which is why single-Mac SSD streaming also went from a 103 s load to 4.4 s.

The cliff, and the wrong explanation

We blamed the collapse above on Metal's wired-memory limit. It is a real thing, and the symptom fits exactly. We raised it on the smaller Mac. Twice. It did nothing, and that null result is the useful one: 18.1 GiB collapses at both a 20 and a 22 GiB limit, so the sysctl is not the lever. The machine simply cannot hold much more than 17.5 GiB of resident weights and keep its GPU fed.

The probe that lied

Twice we concluded a healthy cluster was broken, because we measured the first request. The first forward through a freshly loaded model compiles Metal kernels: 2.3 s on a small model, 79 s on the 122B under memory pressure. A 40-token probe issued right after load reports ~0.1 tok/s on a ring that is about to sustain 20.

Worse, it poisoned an earlier diagnosis. We had profiled the ring, seen a 229 ms per-shard cost, and shipped a commit blaming lazy KV cache graphs. Then we instrumented properly: a flat 18.5 ms per step across 64 steps, with step 1 costing 2288 ms. The profiler had been averaging the warm-up in. 2288/14 ≈ 163 ms, the exact size of the phantom. The commit was reverted and the function left in place as an honest no-op with a docstring saying so.

The bug curl could not see

The Lab could chat with the cluster and stream tokens. But when the model wrote an HTML page, the page never rendered. The Lab sat there, apparently mid-stream, forever. Meanwhile curl against the same endpoint returned a complete SSE stream ending in data: [DONE].

The cluster's SSE handler is built on Python's BaseHTTPRequestHandler, which speaks HTTP/1.0. It answered the browser's Connection: keep-alive in kind. An SSE body has no Content-Length, and HTTP/1.0 has no chunked encoding, so nothing framed the response. The end of the body is the close of the socket, and the socket never closed. fetch()'s reader never reported done. The assistant message was never pushed to the transcript. The renderer, which only runs on a completed message, never ran.

self.send_header("Connection", "close")
self.end_headers()
self.close_connection = True

Three characters of protocol, most of an evening. The lesson is not about SSE. It is that curl and a browser are different clients, and a tool that tolerates a malformed response will happily tell you the response is fine.

The through-line of all four is the same. Free memory said fine. curl said fine. The profiler said 229 ms. Every one of those signals was measuring something adjacent to the thing we cared about, and each was confident. The cluster works now, and a surprising amount of what makes it work is the code that refuses to run.