Troubleshooting

Error messages you can hit running quantized models on Apple Silicon, what causes each one, and what to do about it. Several are fixed in a specific release, so the answer is often to upgrade.

pip install -U mlx-optiq resolves most of what follows. The current release is listed on the changelog.

There is no Stream(gpu, N) in current thread

Raised while serving, usually on the first request, and it can take the whole server process down rather than failing the one request.

MLX streams belong to the thread that created them. An array that has been built but not yet evaluated carries the stream of its creating thread, and evaluating it on another thread raises this. optiq serve loads the model on the main thread and generates on mlx-lm's own thread, so any array left unevaluated after loading will trigger it. On Gemma-4 the array in question was the RoPE frequency table on the global-attention layers.

Fixed in 0.4.23. If you see it on a later version, it is a different array and worth reporting.

scaled_dot_product_attention(): incompatible function arguments … array, tuple, tuple

Seen on Gemma-4 with quantized KV, typically once context grows past roughly ten thousand tokens, and sometimes as [broadcast_shapes] Shapes (1,1,1,0) and (1,16,1,10764) cannot be broadcast.

Gemma-4 shares KV across layers. A consumer layer receives the producing layer's (packed, scales, biases) tuple while holding no quantized cache of its own, and upstream mlx-lm decides which kernel to use by checking the cache rather than the tensors. The tuple then reaches the fp16 kernel, which cannot accept it.

Fixed in 0.4.25. Verified past fourteen thousand tokens on gemma-4-26B-A4B-it-OptiQ-4bit with a mixed per-layer KV config.

[METAL] Command buffer execution failed: Insufficient Memory

The GPU refused an allocation. The exception is thrown on a Metal callback thread where there is no Python frame to catch it, so the server exits rather than returning an error for that request.

The usual cause is a KV window sized against free system RAM when the limit that actually applies is Metal's recommended working set, which is shared by every process using the GPU. Two servers on one machine can each size a window that fits, and together exceed it.

From 0.4.25 the automatic context cap is bounded by both limits. If you are running more than one model at a time, set --max-context explicitly rather than relying on auto.

NotImplementedError from RotatingKVCache.to_quantized

Hit when quantizing the KV cache of a model with sliding-window attention, which includes every Gemma-4. Upstream mlx-lm raises this in v0.1.2 and earlier.

OptiQ installs a rotating cache that supports quantization and uses it automatically when optiq serve or optiq kv-cache needs one. Reaching the upstream error means the model is being loaded through stock mlx-lm rather than through OptiQ.

Model type <arch> not supported, or No module named 'mlx_lm.models.<arch>'

Reported for gemma4_unified, diffusion_gemma, mage_vl, mistral4, llada2 and dhara_ar, among others.

These architectures are not in stock mlx-lm. OptiQ registers them, so loading through optiq serve, optiq lab or optiq code works. A bare mlx_lm.load() in your own script will not, unless you import OptiQ first so the registration runs.

Tool calls are ignored when serving Devstral or another Mistral-family model

The model appears to answer normally but never actually calls a tool, and the server log shows Failed to parse tool call … Could not parse tool call from: read_file[ARGS]{….

Mistral models emit tool calls as [TOOL_CALLS]name[ARGS]{json}. mlx-lm's server splits the output on that marker and passes on only what follows, and OptiQ's parser required the marker to be present, so every served call was discarded. A second problem meant only the first of several chained calls ran.

The dropped calls are fixed in 0.4.24 and chained calls in 0.4.25. Both affect agentic use specifically; ordinary chat was unaffected.

A LoRA adapter that trains successfully but contains NaN

Training reports completion and the adapter mounts, but the model produces nothing usable afterwards.

A batch that contributes no unmasked target tokens makes the loss compute zero divided by zero. That happens more often than it sounds: with prompt masking on, any example whose response falls past --max-seq-length masks out completely. The resulting NaN spreads to every LoRA factor through the gradient-norm clip.

Fixed in 0.4.25, which also refuses to report an adapter as trained if any tensor in the saved file is non-finite. On an earlier version, raising --max-seq-length or shortening the training examples avoids it.

--stream-experts appears to do nothing

The flag is accepted, the model loads, and memory use is unchanged.

On Gemma-4 MoE models the expert tensors are stored under a path OptiQ's matcher did not recognise, so it found nothing to stream and loaded the model resident without reporting a problem.

Fixed in 0.4.23. Expert streaming is described on the serve page, and the background is in Quantizing a mixture-of-experts model on MLX.

The mlx-vlm batched vision path does not support KV cache quantization yet

A genuine limitation rather than a bug. Quantized KV and batched image input cannot currently be combined.

Serve vision models without --kv-bits or --kv-config, or send image requests one at a time. OptiQ's vision path already routes image requests off the batch path for this reason; the message means something is reaching mlx-vlm's batched path directly.

Choosing --max-kv-size, --kv-bits and the wired limit

These interact and it is easy to set one against another.

--max-kv-size bounds how many tokens of KV are kept, --kv-bits sets the precision the cache is stored at, and the wired limit governs how much memory Metal will hold resident. Quantizing the cache to 4 bits roughly quarters its size, which usually matters more than capping the window. --max-context auto sizes the window from available memory and the device's working set, and is the right default unless you are running several models at once.

Full descriptions are on the serve page.

Something else

If you hit an error that is not here, the changelog lists what each release fixed, and reporting it on the model card discussion for the quant you are running is the fastest way to get it looked at.