# mlx-optiq.com — full content dump > Generated llms-full.txt. Sister file to llms.txt. This is the complete docs corpus as plain text, suitable for one-fetch AI crawler ingestion (SearchGPT, AI Overview, Perplexity, Claude search). For the structured index see https://mlx-optiq.com/llms.txt Regenerate with `python site/_build_llms_full.py` after editing any page. --- ## Run any LLM, right on your Mac. Source: https://mlx-optiq.com Local LLMs on Apple Silicon Run any LLM, right on your Mac. OptiQ is the local-LLM stack for Apple Silicon. Quantize open models down to a size your Mac can run, then use them from the command line, a local web app, or a coding agent in your terminal. $ pip install mlx-optiq copy CLIoptiq servequantize & serve LABoptiq labthe workbench CODEoptiq codecoding agent BOOSToptiq cloud loginfrontier when needed 200+ models on Hugging Face 1M+ downloads / month The OptiQ stack Everything talks to the same server. Compile a model once with the CLI, then point whatever you like at it: the Lab in a browser, the coding agent in your terminal, or the editor you already use. Boost rides on that same server, which is why none of them needs a plugin. CLI OptiQ CLI Quantize · fine-tune · serve The optimizing compiler and toolkit. Sensitivity-driven mixed-precision quantization, sensitivity-aware LoRA, and a server that speaks both the OpenAI and Anthropic APIs, MLX-native, no PyTorch. Explore the CLI → LAB OptiQ Lab A local model workbench One pip install for a browser UI over the whole workflow: chat with sandboxed tools and images, a quantize wizard, SFT and DPO fine-tuning with a dataset designer, and a two-model arena. Explore the Lab → CODE OptiQ Code A coding agent for your model A terminal coding agent that drives whatever you are serving, offline, on your Mac. Built for weak local models: it never returns an empty patch and recovers from failed edits. Explore OptiQ Code → CLOUD OptiQ Cloud Boost a hard turn A frontier model on tap for the turns your local one cannot finish. Boost lives in optiq serve, so Claude Code, Codex, OpenCode, Hermes and the Lab all get it with no plugin, and a whole tool loop costs one Boost. See how Boost works → OptiQ Cloud One hard turn shouldn’t cost you the session. Your local model handles the everyday work. When a turn starts going in circles, Boost hands that single episode to a frontier model and gives control straight back, in the same editor. You pay for the few turns that need it instead of a second subscription. Eighteen turns on your own model. One Boost, and your code never left the machine except for that episode. $20for 200 Boosts A whole tool loop1 Boost Free to start20 Boosts Works withany coding agent Start with 20 free Boosts No card to try it. Boost lives in the server, so nothing you already use needs a plugin. Pre-built models Drop-in 4-bit quants at the same size. Production OptiQ-quantized LLMs on Hugging Face, across 12+ families: Qwen, Gemma-4, Muse-Glimmer, Nemotron, Mistral / Devstral, MiniCPM5, Laguna, Nanbeige, and a discrete-diffusion Gemma. They run from 1 B dense models to 35 B-A3B mixture-of-experts. Most load directly into stock mlx-lm; a few vendored architectures register with a one-line import optiq. Qwen · new Qwen3.8-27B-OptiQ-4bit The newest Qwen and the first we ship with image input, with a bundled MTP head for speculative decoding. 18.9 GB on disk 87.98 Capability vision + MTP Muse-Glimmer Muse-Glimmer-30B-OptiQ-4bit A 30B image-text reasoning model that thinks in a separate channel before it answers. Capability Score 87.36. 22.1 GB on disk 87.36 Capability 100% HashHop Gemma-4 gemma-4-31B-it-OptiQ-4bit The largest single dense quant we ship. 31 B parameters in 20.8 GB with Capability Score 79.7 (+3.5 vs uniform-4-bit). Pair with the matching -assistant-bf16 drafter for speculative decoding. 20.8 GB on disk 79.7 Capability +3.5 vs U4 Nemotron NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit A Mamba-attention hybrid with a 128-expert MoE, 6 active per token. 22.8 GB on disk, down from 65.8 GB, and the MTP speculative head is preserved in the sidecar for optiq serve --mtp. 22.8 GB on disk 73.94 Capability MoE + MTP browse all models → How it compares Where mlx-optiq sits among the Mac LLM options. A snapshot of how the popular paths stack up on what actually moves memory footprint and decode speed on Apple Silicon. mlx-optiq mlx-lm llama.cpp Per-layer mixed-precision weights Yes, calibration-driven Uniform 4-bit Block-wise K-quant Per-layer mixed-precision KV cache Yes Uniform 4 / 8 / fp16 Group-wise int8 only Sensitivity-aware LoRA fine-tuning Rank scaled by per-layer bits Constant rank LoRA Inference only OpenAI and Anthropic compatible server One process, both OpenAI only llama-server (OpenAI shim) Text and image input Yes Text only Image via separate build Sandboxed tool support for chat Three tools: web search, Python, terminal None None Reading the table mlx-optiq is the only path on this list that uses calibration-driven, per-layer bit allocation for both weights and KV cache, in the native MLX runtime, with serving, fine-tuning, and image input on the vision models in the same package. The others target different goals. Scores and sizes, measured on Gemma-4-12B → FAQ Common questions. Can I run LLMs locally on a Mac? Yes. mlx-optiq runs large language models natively on Apple Silicon, from M1 to M5, using Apple's MLX framework. Install it from PyPI with pip install mlx-optiq, then quantize, prune, fine-tune, and serve models entirely on your Mac, fully offline. Do I need a GPU or PyTorch to run LLMs on a Mac? No. There is no PyTorch and no discrete GPU in the path. mlx-optiq is MLX-native and uses the unified memory of Apple Silicon directly, so a MacBook, Mac mini, or Mac Studio is enough. No CUDA, no cloud, no API key. How much RAM do I need to run an LLM on a Mac? It depends on the model. A 4-bit OptiQ quant of a 4B model needs roughly 3 GB; a 9B needs about 6 GB; larger mixture-of-experts models need more. Mixed-precision 4-bit quantization is what lets bigger models fit in a Mac's memory while staying close to full-precision quality. What is mlx-optiq? An MLX-native toolkit to quantize, prune, fine-tune, and serve LLMs locally on Apple Silicon. Its core is data-driven mixed-precision quantization: it measures each layer's sensitivity and assigns per-layer bit-widths, so quants keep more quality than uniform 4-bit at the same size. It also ships a local web UI (OptiQ Lab) and an OpenAI and Anthropic compatible server. Get started Make your Mac an LLM workstation. Pick a model and grab a snippet. The docs cover every supported family, fine-tuning recipes, and the OpenAI-compatible serving stack. read the docs → browse models --- ## OptiQ quants, ready to load. Source: https://mlx-optiq.com/models Pre-built quants · Hugging Face OptiQ quants, ready to load. Every model is a standard MLX checkpoint. Load it with mlx_lm.load(...), no special runtime and no PyTorch. OptiQ measures each layer's sensitivity and spends bits where they matter, so a mixed-precision quant keeps the accuracy a flat uniform quant gives up, at the same size. Load any quant · with optiq import optiq # unlocks every OptiQ arch from mlx_lm import load, generate model, tok = load( "mlx-community/Laguna-XS-2.1-OptiQ-4bit") print(generate(model, tok, prompt="Hi")) $ optiq convert Qwen/Qwen3.6-27B --target-bpw 4.5 copy 12+ model families 2–8 bit, mixed precision Model families Pick a family. Details in the docs. Each family page lists every quant, its size, and its six-metric Capability Score against stock uniform 4-bit. Sparse-MoE families also ship REAP variants: half the routed experts removed, the same active parameters per token, roughly 40 % off disk and memory. What pruning costs What mixed precision buys, on Gemma-4-12B Muse-Glimmer A 30B image-text reasoning model, Capability Score 87.36. Sliding-window attention on three layers in four keeps memory flat as context grows. 30B · vision · read the docs → Nemotron NVIDIA's Mamba-attention hybrid. A dense 4B, a 30B-A3B MoE, and the 3.5 Lightning 30B-A3B, each with a KV config for its few attention layers. 4B + two 30B MoE · read the docs → LFM2.5 Liquid AI's on-device series, 230M through 8B, plus a 3B that takes images. Convolutional blocks alternate with full attention, so only a few blocks hold a KV cache and it stays small as context grows. 230M to 8B · vision · on-device · read the docs → Qwen Three generations in one family: 3.8 with image input, 3.6, and 3.5, the daily driver behind Conjure and most of our agent work. 0.8B through 122B. 3.5 · 3.6 · 3.8 · dense + MoE · vision · read the docs → Gemma-4 Google's instruct and QAT series, e2b through 31B, with image input on the multimodal quants. e2b to 31B · vision · read the docs → DeepSeek-V4 A 43-layer sparse-attention MoE that runs three different attention modes depending on the layer. The routed experts carry almost all the weight, so they go to 2-bit while attention stays higher. Flash · 2-bit experts · read the docs → Laguna poolside's Laguna-XS-2.1, a sparse-MoE reasoning model for coding and agents. Its OptiQ quant nearly ties our best Capability Score. MoE · coding · read the docs → Diffusion LLM Our first non-autoregressive family. A discrete-diffusion Gemma that denoises a canvas of tokens instead of decoding left to right. 26B-A4B · non-autoregressive · read the docs → Mistral · Devstral Devstral-Small-2, the tool-calling coding model. It was the first OptiQ quant of a Mistral-family base. 24B · agentic coding · read the docs → MiniCPM5 Two on-device bases with a hybrid reasoning mode: the sub-gigabyte 1B, and the 1.8 GB 2B that scores level with the Qwen3.5-4B quant. 1B · 2B · on-device · read the docs → Spark-X2.5 XHToken's dense 4B with a 3:1 sliding-window / full-attention layout and a native 1M-token context, a new architecture ported for MLX. 2.8 GB, thinking by default. 4B · 1M context · read the docs → Nanbeige A looped transformer. It runs a 22-layer stack twice with shared weights, so it computes like a deeper network without the parameters. 3B · looped · read the docs → Mage-VL Microsoft's 5B vision-language model, the first video model in the lineup. It reads images and video, with the vision tower kept at bf16 in a sidecar. 5B · video + image · read the docs → And dozens more across the ecosystem, from us and from developers publishing their own quants. Browse every OptiQ quant on Hugging Face → Load Standard MLX. Run mlx_lm.load("mlx-community/") and generate. Vendored-arch families (Laguna, Nanbeige, dhara) need import optiq first to register the arch. Using quants → Serve Point optiq serve at any repo for an OpenAI and Anthropic-compatible endpoint with a mixed-precision KV cache. KV-quant serving → Tune Sensitivity-aware LoRA, hot-swappable adapters, and a full local Lab, all on your Mac. LoRA fine-tuning → --- ## Four 4-bit quants of Gemma-4-12B Source: https://mlx-optiq.com/compare Measured · Gemma-4-12B Four 4-bit quants of Gemma-4-12B Three repositories on Hugging Face put 4bit in the name of the same Gemma-4-12B. They run from 6.3 GB to 10.2 GB and no two quantize it the same way. Here is what each one scores. vs uniform 4-bit What the extra 2 GB buys Both start from google/gemma-4-12B-it. One gives every layer 4 bits. The other measures each layer first and raises the ones that cannot take it. Benchmark OptiQ mixed 4/8 Uniform 4-bit Delta MMLU 42.6% 34.4% +8.3 GSM8K 93.4% 90.1% +3.3 IFEval 73.9% 71.2% +2.8 BFCL-V3 simple 71.0% 71.5% −0.5 HumanEval 88.4% 76.8% +11.6 HashHop 40.0% 27.0% +13.0 Capability Score 68.23 61.83 +6.40 On-disk size 8.3 GB 6.3 GB +2.0 Six points on the mean, for 2.0 GB more on disk. BFCL goes the other way by half a point. At 200 calls that is inside the confidence interval. vs QAT How QAT compares Read this first QAT is not a switch you flip. Somebody has to run quantization-aware training while the model trains, then publish those weights. Google does it for Gemma-4. Very few others do. Build Base Capability Score Uniform 4-bit instruct 61.83 OptiQ mixed 4/8 instruct 68.23 Uniform 4-bit QAT 68.27 OptiQ mixed 4/8 QAT 69.64 Those two middle rows came from separate runs against different baselines, so read them as level. They stack: the sweep on the QAT base reaches 69.64. The margin is smaller there, +1.37 against +6.40. QAT has already taken out most of what the sweep goes looking for. gemma-4-12B-it-qat-4bit is not a uniform 4-bit quant either. It pins 144 components at 8 bits and weighs 10.2 GB. Comparing scores across the three published repos means comparing three different sizes. Method Six benchmarks at fixed sample counts, greedy decoding throughout. Each quant is scored against a uniform 4-bit quant of its own base. That isolates the bit allocation from everything else the base brings. MMLU reads low for this family because its default scoring takes the answer letter by logit argmax. It still separates two quants of the same base. That is all it does here. reproducebash $ optiq eval mlx-community/gemma-4-12B-it-OptiQ-4bit --task all --score Every quant carries the same table on its Hugging Face page, all of them linked from the model list. The eval-framework write-up covers the limits of the Capability Score. --- ## Documentation Source: https://mlx-optiq.com/docs Documentation Documentation mlx-optiq is built around one measurement: per-layer KL-divergence sensitivity, computed once on calibration data. That signal drives three optimization passes: mixed-precision weight quantization, mixed-precision KV-cache allocation, and sensitivity-aware LoRA rank scaling. Around them sit the rest of the toolkit: hot-swap LoRA adapters, a dual-protocol (OpenAI + Anthropic) inference server, and a sandboxed code-execution helper for agent workflows. This site is the canonical reference. Every page is self-contained: code examples are copy-paste runnable on a stock Mac with Python 3.11+ and 16 GB+ RAM. The products One pip install mlx-optiq gives you four ways to work with a local model, all sharing the same MLX-native core: OptiQ CLI, the optimizing compiler and toolkit: quantize, prune, fine-tune, and serve from the command line. OptiQ Lab, a local web workbench for chat, quantize, fine-tune, and dataset work in the browser. OptiQ Code, a terminal coding agent that drives whatever model you are serving. OptiQ Cloud, a Boost that hands one hard turn to a frontier model and gives control straight back. Pick a path I want to use a pre-built quant Start with Installation, then jump to your model family: Nemotron, Qwen3.5, Qwen3.6, MiniCPM5, Spark-X2.5, or Gemma-4. Each has a 5-minute hello-world plus model-specific tips (chat template, sampling defaults, recommended context length). I want to code with a local model Serve a model, then point OptiQ Code at your repo. It drives the served model through a read / edit / run-tests loop in your terminal, engineered to stay reliable on small local models. See the product overview. I want to quantize my own model Read How sensitivity works to understand the algorithm, then the convert CLI reference. The --reference auto flag picks bf16 when it fits and a uniform-4-bit baseline when it doesn't. I want to fine-tune with LoRA The LoRA fine-tuning guide covers PEFT-compatible adapter output, sensitivity-aware rank scaling, and the empirical training-ceiling map for a 36 GB Mac across all 12 supported models. I want to serve an LLM The KV-quant serving guide covers running optiq serve with both the OpenAI /v1/chat/completions and Anthropic /v1/messages endpoints from the same process, plus mixed-precision KV cache and a mounted LoRA adapter. I want to send images The vision guide covers image+text on Gemma-4: one bf16 sidecar makes the same repo load text-only under stock mlx-lm and full image+text under OptiQ, with no mlx-vlm dependency. Upload a picture in the Lab, or send an image_url to optiq serve. I want to evaluate a quant The eval CLI ships a smoketest (KL + GSM8K-50, ~5 min on 27B) for triage and a full benchmark suite (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop, ~2 h on 27B) that produces the Capability Score on every model card. HumanEval runs in a layered sandbox (container via Docker, Podman or Apple container → macOS sandbox-exec → subprocess + rlimit). HashHop checks long-context multi-hop retrieval via chained key/value lookups. Methodology is in the eval-framework write-up. For agents and IDEs The full library reference is also published as a single Markdown file: /llms.txt. Drop it into Claude Code, Cursor, or any agent context window. The full reference in ~12 KB. --- ## Installation Source: https://mlx-optiq.com/docs/install Documentation Installation mlx-optiq is a pure-Python package on PyPI and needs Python 3.11+. Quantizing and local inference need Apple Silicon, because MLX is an Apple framework. The coding agent and the Lab run on Linux and Windows too, pointed at any OpenAI-compatible endpoint. See Windows and Linux. One-line install terminalbash $ pip install mlx-optiq That's it. The base install pulls in mlx, mlx-lm, huggingface-hub, click and a handful of small utilities. ~80 MB on disk including all dependencies. What the install gives you pip install mlx-optiq is the whole product. Every command works out of the box: optiq convert, optiq serve, optiq eval, optiq lora, optiq kv-cache, optiq latency, the optiq code coding agent, and optiq lab, the local web UI. Loading and running any published quant works immediately. The dev extra One extra exists, and only for working on OptiQ rather than with it: terminalbash # test suite, browser tests, and the benchmark datasets $ pip install 'mlx-optiq[dev]' It carries pytest and Playwright, the datasets library that optiq eval uses to pull MMLU/GSM8K/IFEval/HumanEval, and the two test oracles (mlx-vlm, torch) that the vendored vision tower is validated against. Nothing on the convert, serve, or agent path imports any of it. Changed in 0.4.16 The [lab], [game], [convert], [cli] and [all] extras are gone: everything they installed now ships by default. The names still resolve so old install lines keep working, they just no longer add anything. Running optiq eval without [dev] tells you exactly what to install. Verify the install terminalbash $ optiq --version # mlx-optiq, version 0.4.33 $ python -c "import optiq; print(optiq.__version__)" # 0.4.33 System requirements What you need depends on what you are doing. Quantizing needs Metal; driving a model over an API does not. You want to Runs on optiq code and optiq lab against an API (OpenAI-compatible base_url) macOS, Linux, Windows Run a quant locally (optiq serve, Lab chat on a local model) Apple Silicon Quantize (optiq convert, sensitivity analysis, LoRA) Apple Silicon Python: 3.11 or newer, everywhere. Distro Python is often 3.10 or older — uv, pyenv or conda will get you a newer one without touching the system install. Apple Silicon: macOS 14 (Sonoma) or newer, M1 through M5, any tier. RAM (local inference): 16 GB for small quants (0.8–4 B), 24 GB for 9 B, 36 GB+ for 27–35 B and fine-tuning. Disk: pre-built quants are 0.5–20 GB each. Windows and Linux MLX is an Apple framework, so local inference and quantization are Mac-only. The coding agent and the Lab are not: point them at any OpenAI-compatible endpoint and they are ordinary Python. Install the same package and set two variables. terminalbash $ pip install mlx-optiq # any OpenAI-compatible endpoint: a Mac running `optiq serve` on your # network, vLLM, Ollama, OpenRouter, or a hosted API $ export OPTIQ_CODE_BASE_URL=http://your-host:8080/v1 $ export OPTIQ_CODE_API_KEY=your-key $ export OPTIQ_CODE_MODEL=your-model $ optiq code # the agent, driving that endpoint $ optiq lab # the web UI, same idea Every setting has an OPTIQ_CODE_ variable, or you can put them in ~/.optiq/code/config.json. optiq code config prints each value and where it came from. Verified This path was tested on Linux x86_64 with MLX installed but non-functional (the wheel installs; libmlx.so does not load). import optiq, the agent loop, its tools, and the Lab web server all work regardless, because nothing on that path imports mlx.core. Commands that do need Metal will tell you so rather than failing obscurely. Install a container runtime for the sandbox The Lab's python and terminal tools run code the model wrote. On macOS the OS provides sandbox-exec, which denies network access and confines writes to a scratch directory. Linux and Windows have no equivalent, so there OptiQ uses a container instead, and without one it falls back to running the code as you, in a temp directory. Install Docker or Podman and the strongest tier turns on by itself. Nothing to configure. Platform Tier used Isolation Any, with Docker, Podman or Apple container container Separate filesystem, no network, memory cap macOS sandbox-exec No network, writes confined to the scratch directory Linux or Windows, no runtime subprocess No filesystem or network isolation On that last row the AST checks and the blocked-command list are the only thing left, and they are not a sandbox. Treat it as running the model's code with your own permissions, because that is what it is. The Lab shows the active tier so you can tell which row you are on. Two details specific to Windows. The terminal tool emits sh one-liners, so a container runtime (or Git Bash, or WSL) is what makes it work at all; without one it says so instead of failing strangely. And on macOS, Docker is not preferred over sandbox-exec, because sandbox-exec is already strong there and starts instantly while Docker boots a Linux VM per call. OPTIQ_SANDBOX_CONTAINER=0 turns the container tier off; naming a runtime, as in OPTIQ_SANDBOX_CONTAINER=podman, forces that one. Working in a virtualenv Strongly recommended. uv is the fastest path: terminalbash $ uv venv .venv $ source .venv/bin/activate $ uv pip install mlx-optiq Or stock venv: terminalbash $ python3.11 -m venv .venv $ source .venv/bin/activate $ pip install mlx-optiq Upgrade terminalbash $ pip install --upgrade mlx-optiq Already have a quant downloaded? Pre-built quants live in your local Hugging Face cache (~/.cache/huggingface/hub). They're independent of the mlx-optiq version. Upgrading the package doesn't re-download anything. Troubleshooting "No matching distribution found" You're probably on Linux, Windows, or Intel macOS. mlx-optiq requires Apple Silicon. There's no fundamental reason it couldn't work on Linux too, but it depends on MLX, which is macOS-only. Slow first model download Hugging Face downloads can be slow from some regions. Set HF_HUB_ENABLE_HF_TRANSFER=1 and install hf_transfer for ~5× speedups on large models: terminalbash $ pip install hf_transfer $ export HF_HUB_ENABLE_HF_TRANSFER=1 "Metal command-buffer timeout" while quantizing 27 B+ Long Metal kernels can time out on the macOS GPU watchdog. mlx-optiq patches around this internally for the convert path; if you hit it during fine-tuning, lower --max-seq-length. See the fine-tuning guide's training-ceiling map. Next: pick a model family. Qwen3.5, Qwen3.6, Gemma-4, or jump to Using mlx-optiq quants. --- ## Using mlx-optiq quants Source: https://mlx-optiq.com/docs/quants Documentation Using mlx-optiq quants mlx-optiq-quantized models are standard MLX checkpoints. They load with the unmodified mlx_lm.load function and generate with mlx_lm.generate. The only difference from a uniform-4-bit checkpoint is the per-layer bit-width recorded in metadata. This is true for the standard-architecture families (Qwen3.5, Qwen3.6, Gemma-4, Mistral / Devstral, MiniCPM5), which mlx-lm already knows how to load. Nothing extra is required. Vendored-arch quants: import optiq first A few families use architectures that stock mlx-lm does not ship: Laguna, Nanbeige, and dhara. For those, import optiq once before you call mlx_lm.load. Importing OptiQ registers the vendored architecture into mlx-lm, so the plain mlx_lm.load call then works. Without the import, the load raises an unknown-model_type error. (Nemotron 3 loads on stock mlx-lm and needs nothing extra; Diffusion LLMs use OptiQ's own decode loop, not mlx_lm.load.) vendored_arch.pypython import optiq # registers Laguna / Nanbeige / dhara into mlx-lm from mlx_lm import load, generate model, tok = load("mlx-community/Laguna-XS-2.1-OptiQ-4bit") print(generate(model, tok, prompt="Hello", max_tokens=100)) Standard-arch quants (Qwen, Gemma, Mistral, MiniCPM5) do not need this; the examples below load without it. One-shot generation oneshot.pypython from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") out = generate(model, tok, prompt="Why is mixed-precision quantization a good idea?", max_tokens=300) print(out) Streaming generation For interactive UIs and CLIs, stream tokens as they come: streaming.pypython from mlx_lm import load, stream_generate from mlx_lm.sample_utils import make_sampler model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") sampler = make_sampler(temp=0.6, top_p=0.95) for response in stream_generate( model, tok, prompt="Write a haiku about Apple Silicon.", max_tokens=200, sampler=sampler, ): print(response.text, end="", flush=True) Chat templates Instruction-tuned models (Qwen3.5-*-Instruct, Qwen3.6-*, Gemma-4-*-it) need their chat template applied. Always do this for chat-style use: chat.pypython from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.6-27B-OptiQ-4bit") messages = [ {"role": "system", "content": "You are a concise expert."}, {"role": "user", "content": "Explain RoPE in 3 bullet points."}, ] prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, ) out = generate(model, tok, prompt=prompt, max_tokens=400) print(out) Reasoning models Qwen3.5 and Qwen3.6 instruct variants have a built-in ... reasoning channel. Pass enable_thinking=False to apply_chat_template to skip it (much faster, slightly less accurate on math/logic), or leave it on for best quality. Multi-turn chat loop chat_loop.pypython from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") messages = [{"role": "system", "content": "You are helpful."}] while True: user = input("> ") if not user: break messages.append({"role": "user", "content": user}) prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) reply = generate(model, tok, prompt=prompt, max_tokens=800) print(reply) messages.append({"role": "assistant", "content": reply}) Inspecting mlx-optiq metadata Each mlx-optiq checkpoint records its per-layer bit assignment in config.json under the quantization field, plus optiq/metadata.json with the per-layer allocation and optiq/sensitivity.json with the measured KL sweep behind it. (Quants published before 0.4.18 carry the metadata at the root as optiq_metadata.json and no sweep.) The methodology behind the per-layer assignment is in our research write-up; the calibration mix that drives it is described in the optiq.jsonl post. inspect.pypython import json from huggingface_hub import snapshot_download local = snapshot_download( "mlx-community/Qwen3.5-9B-OptiQ-4bit", allow_patterns=["optiq/metadata.json", "config.json"], ) meta = json.load(open(f"{local}/optiq/metadata.json")) print("Method: ", meta["method"]) print("Achieved BPW: ", round(meta["achieved_bpw"], 3)) print("8-bit layers: ", meta["n_high_bits"]) print("4-bit layers: ", meta["n_low_bits"]) Memory ceilings on a 36 GB Mac Approximate working-set sizes during single-prompt inference at moderate context (≤8 k tokens). Add ~2 GB for the framework and OS reserve. Approximate footprint at 8 k context Model Disk Inference RAM (≤8 k) Recommended Mac Qwen3.5-0.8B 0.5 GB ~1.5 GB 8 GB+ Qwen3.5-2B 1.4 GB ~3 GB 16 GB+ Qwen3.5-4B 2.8 GB ~5 GB 16 GB+ Qwen3.5-9B 5.6 GB ~9 GB 24 GB+ Qwen3.5-27B 15.7 GB ~22 GB 36 GB+ Qwen3.6-27B 15.7 GB ~22 GB 36 GB+ Qwen3.5-35B-A3B 20.1 GB ~26 GB 36 GB+ Qwen3.6-35B-A3B 20.1 GB ~26 GB 36 GB+ gemma-4-26B-A4B-it 14.9 GB ~21 GB 32 GB+ gemma-4-31B-it 18.1 GB ~24 GB 36 GB+ For 64 k context inference on the 9 B and larger models, use mixed-precision KV serving. It both shrinks the cache and speeds decode. Next: dive into how sensitivity works, or jump to your model family's getting-started guide. --- ## How sensitivity works Source: https://mlx-optiq.com/docs/sensitivity Documentation · methodology How sensitivity works Per-layer KL-divergence sensitivity is mlx-optiq's measurement for deciding which transformer layers can safely be quantized to 4 bits and which need 8. Not every layer in a transformer is equally fragile under quantization. Some tolerate 4-bit; others lose accuracy. mlx-optiq measures which is which by perturbing one layer at a time and watching the model's output distribution. One measurement drives weight, KV, and LoRA allocation. The same per-layer sensitivity number informs weight bit-width, KV bit-width, and LoRA rank. Two methods: optiq and static The per-layer allocation comes from one of two signals, selected with optiq convert --method. They trade measurement fidelity against convert time and memory. Method Signal Convert cost Best for optiq · default Exact logit KL from calibration forward passes High, n_layers × n_bits × n_samples forward passes The gold-standard signal; small / medium models static Structural rules from architecture, no measurement Negligible. No forward passes, loads lazily, respects target BPW The fast path; large bases where exact KL is impractical Measured on the smallest base, Qwen/Qwen3.5-0.8B (GSM8K, 3-shot, 200 questions, target 5.0 BPW with candidate bits 4 and 8): Method Convert time Achieved BPW GSM8K optiq 899 s 5.59 34.5% static 7.2 s 5.15 34.5% static does no measurement at all, yet its structural rules, high bits to the embedding and output head, the first and last block, attention and the MoE router; low bits to the dense MLP and routed experts, match optiq here at a lower BPW and 125× faster. For a typical transformer, architecture predicts most of the sensitivity, so the layers optiq would pick out are largely known in advance. Use optiq when you want a measured signal or are characterising a new family. And because static loads lazily and runs no forward passes, it is the practical choice for the very large bases, a 2/4-bit mixed quant of a 122 B-A10B MoE that runs on a laptop through SSD expert streaming, for instance. Choosing a method Default to optiq for the gold-standard signal on small and medium models. Use static for a fast convert, and for any base too large for exact sensitivity to be practical. It matches optiq on typical architectures and is the path for 100 B-class MoEs. method.shbash # default: exact KL sensitivity $ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8 # fast structural allocation for a large MoE, mixed 2/4-bit $ optiq convert mlx-community/Qwen3.5-122B-A10B-bf16 \ --method static --candidate-bits 2,4 --target-bpw 2.5 The measurement (optiq method) For each pair (layer L, candidate bits b): Forward-pass calibration data through the model with all weights in their reference precision. Record the output logits. Replace just L's weight with a simulate-quantized version at b bits (round-trip: quantize → dequantize). All other layers stay at reference precision. Forward-pass the same calibration data again. Record the perturbed logits. Compute KL divergence between the reference and perturbed logit distributions, averaged across calibration samples and tokens. Restore L to its reference precision and move on to the next layer. The result is a table: for every layer, the KL cost of dropping it from reference to each candidate bit-width. This is mlx-optiq's per-layer sensitivity signal. Two reference modes What does "reference precision" mean concretely? Two options, picked automatically by --reference auto based on whether the bf16 weights fit in your Mac's RAM: 1. bf16 reference: gold standard Load the original bf16 model into RAM. Each sensitivity probe swaps a single layer between bf16 and a quantized copy. Highest-fidelity measurement. Required RAM ≈ model size in bf16 (≈ 2 × parameters in GB). Used automatically when bf16 fits in ~70% of available RAM. On a 36 GB Mac, this means models up to ~10 B parameters. 2. uniform_4bit reference: for big models Build a uniform-4-bit MLX baseline first. Load that as the running model (~25% of bf16 size). Stream bf16 weights off disk, one layer at a time, swapping each in for its sensitivity probe. The signal is slightly weaker. You're measuring KL relative to uniform-4 instead of bf16. But it lets 27 B+ models still get a calibration-driven mixed-precision allocation on a 36 GB Mac. Used automatically when bf16 doesn't fit. The bf16 weights still need to be on disk for the streaming probes; only RAM is the constraint. Auto-routing in practice convert.shbash # auto picks bf16 if it fits, else uniform_4bit $ optiq convert Qwen/Qwen3.5-9B \ --target-bpw 5.0 --candidate-bits 4,8 \ --reference auto # force bf16 (will OOM if model doesn't fit) $ optiq convert Qwen/Qwen3.5-9B --reference bf16 # force uniform-4-bit reference (works on 27 B+ on 36 GB) $ optiq convert Qwen/Qwen3.5-27B --reference uniform_4bit How static assigns bits The static method does no measurement. It scores each layer by architecture alone, from a small set of priors that hold across transformer families: The embedding and output head rank highest. The first and last transformer block are protected. Attention and the MoE router rank above the dense MLP. The routed experts and mid-network layers rank lowest. Those priorities feed the same allocator the measured path uses, at the requested candidate bits and target BPW. So static produces a real mixed-precision quant, not a fixed recipe. It reads only layer names and shapes (the bf16 model is memory-mapped, never resident) and runs no forward passes, which is what lets it allocate a 100 B-class base on a 36 GB Mac in seconds. On a sparse-MoE base it puts the high bits on the router and the protected blocks and keeps the large routed experts low, the right call when the expert budget is what makes the quant fit through SSD streaming. Calibration data mlx-optiq ships a hand-curated 6-domain mix at optiq/calibration/data/optiq.jsonl: 40 samples spanning prose, reasoning (with blocks), code, multi-turn agent loops, function-calling traces, and constraint-bearing instructions. Chat samples are auto-rendered through the target model's tokenizer.apply_chat_template() before tokenization, so the activated subspace matches production. The file lives in the wheel, so no network at convert time. Pass --calibration-mix /path/to/your.jsonl to override with a domain-specific mix; rebuild the default with python scripts/build_calibration.py. See the calibration-mix blog post for the why. The allocator Once you have the sensitivity table, allocating per-layer bits is a knapsack. mlx-optiq uses a greedy heuristic that's optimal in practice for the small number of layers and bit-widths involved: Start every layer at the lowest candidate bit-width (e.g. 4-bit). Compute the average bit-budget so far. If under target BPW: find the layer where upgrading by one bit-width tier buys the largest KL reduction per extra bit. Upgrade it. Repeat until the average BPW reaches the target. Some layers are protected. They always get the highest bit-width regardless of the knapsack. By default these are lm_head, embed_tokens, the first attention block and the last attention block. They're cheap (small parameter share) and pathological to quantize. The output mlx-optiq hands the per-layer bit map to mlx_lm.convert as a quant_predicate. The output is a standard MLX checkpoint indistinguishable from a uniform-quantized one to mlx_lm.load, except that some layers are at 8-bit and others at 4-bit. Why KL and not perplexity? Perplexity is a scalar. It is too coarse to distinguish layers that fail in different ways. KL divergence on the full output distribution captures shifts in which tokens get mass, not just how confident the model is on the chosen one. The KL signal correlates strongly with downstream task accuracy (GSM8K, MMLU) and is much cheaper to compute than running a full eval. What about MoE models? Sparse mixture-of-experts (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, gemma-4-26B-A4B) walk through the same sensitivity knapsack, with three structural rules layered on top: The fused expert tensor (shape (num_experts, hidden, intermediate)) is treated as one knapsack entry per (block, projection). So the routed experts in a block share a bit-width, but each block × projection picks its own. mlx-optiq walks the MoE structure with arch-aware naming: Gemma's switch_glu keeps the experts segment, Qwen's switch_mlp drops it. Router projections (Qwen3's mlp.gate, Gemma's mlp.router) are explicitly protected at the high bit-width. The router's output controls which experts a token visits, so quantization noise here propagates to which paths get taken, not just the magnitude of activations on a fixed path. Shared experts (any layer matching *.shared_experts.*) are also protected at the high bit-width. These fire for every token like a dense MLP, so they're sensitive in the per-token sense even though they sit inside an MoE block. Empirically, even before these explicit rules landed, the KL knapsack put routers and shared experts at 8-bit on every shipped MoE quant: the per-layer KL signal is large at exactly those layers. The protection rule encodes the same outcome by design so it does not depend on the calibration mix happening to land that way. Next up: see the algorithm in action in our research write-up, or get started with a model family. --- ## Nemotron on Apple Silicon Source: https://mlx-optiq.com/docs/nemotron Family guide · Nemotron Nemotron on Apple Silicon NVIDIA's Nemotron 3 is a hybrid: it interleaves Mamba2 state-space blocks with a handful of full-attention layers, and the MoE variants add a sparse expert mixture. In the 4B (dense), only four of the 42 backbone blocks are true attention; the rest are linear-attention SSM or MLP. The 30B-A3B routes through 128 experts at ≈3 B active parameters per token. The Super 120B-A12B goes further: a 512-expert MoE (22 active) that, at 2-bit, runs on a 36 GB Mac by streaming its experts off SSD. All load through mlx-lm's built-in nemotron_h class. Nemotron 3.5 Lightning Lightning-30B-A3B is the newer generation: the same Mamba-attention hybrid, now with a 128-expert MoE and 6 experts active per token. 22.8 GB on disk, down from 65.8 GB, and the MTP speculative head is preserved in the sidecar. Metric Score NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit 22.8 GB 73.94 n/a Newest · 128-expert MoE, MTP head preserved MMLU (5-shot, 969 samples) 78.9% GSM8K (1000 samples) 85.5% IFEval (full set, strict) 68.4% BFCL-V3 simple (200 calls) 83.0% HumanEval (164 problems, pass@1) 87.8% HashHop (long-context retrieval) 40.0% Capability Score (mean of 6) 73.94 The routed experts carry most of the weight, so that is where the bits come from: every 4-bit assignment in the model is an expert tensor, with the rest of the stack at 8-bit. Code generation is its strength; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers predicts. At 22.8 GB it will not sit resident on a 24 GB machine. Stream the experts from SSD instead: terminalbash $ optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory. The quants Model Size on disk Capability Score vs uniform-4 Best for NVIDIA-Nemotron-3-Super-120B-A12B-OptiQ-2bit 47.5 GB n/a n/a Super · 2-bit, SSD-streamed on a 36 GB Mac NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit 20.6 GB 72.32 +2.02 Strongest of the two, math, code, long context NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit 2.9 GB 66.68 +0.24 Small dense assistant, hybrid-SSM experimentation The Super 120B, streamed off SSD The Nemotron-3-Super-120B-A12B is the largest in the family. Its OptiQ quant is a 2-bit static build (47.5 GB) that runs on a 36 GB Mac: ~14 GB resident (the Mamba blocks, attention and shared experts stay in RAM), and the 34 GB of routed experts stream off the SSD, ~3 tok/s. There is no Capability Score for it: at 2-bit that number is not meaningful, so it ships with a coherence check instead: it still writes working code at 2-bit. Read the write-up. Per-benchmark breakdown, 30B-A3B Benchmark uniform-4 OptiQ-4 (mixed) Δ MMLU (5-shot, 1000) 74.8% 76.2% +1.3 GSM8K (3-shot CoT) 78.5% 81.6% +3.1 IFEval (strict) 67.5% 69.1% +1.7 BFCL V3 (simple AST) 74.0% 74.0% 0.0 HumanEval (pass@1) 86.0% 89.0% +3.0 HashHop (overall) 22.0% 25.0% +3.0 Per-benchmark breakdown, 4B Benchmark uniform-4 OptiQ-4 (mixed) Δ MMLU (5-shot, 1000) 63.3% 64.0% +0.7 GSM8K (3-shot CoT) 79.9% 81.5% +1.6 IFEval (strict) 56.0% 56.2% +0.2 BFCL V3 (simple AST) 75.5% 75.5% 0.0 HumanEval (pass@1) 80.5% 77.4% -3.1 HashHop (overall) 25.0% 27.0% +2.0 The MoE 30B is the cleaner win The 30B-A3B clears uniform 4-bit by a full +2.0 Capability Score, winning or tying all six benchmarks, including the fused routed-expert tensors that OptiQ assigns per-layer 4/8-bit (most stay at 4-bit, which keeps the model at 5.05 BPW / 20.6 GB). The dense 4B is a tighter +0.24: it wins four of six and trades a little HumanEval, and its disk delta runs richer because a Mamba2 block carries only two linears so more of them land at 8-bit. Every metric gets one equal vote; disk size sits next to the score as a second axis. See the eval-framework writeup. Hello world hello.pypython from mlx_lm import load, generate model, tok = load("mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit") prompt = tok.apply_chat_template( [{"role": "user", "content": "Explain why hybrid Mamba+attention models scale to long contexts."}], tokenize=False, add_generation_prompt=True, ) print(generate(model, tok, prompt=prompt, max_tokens=300)) Hybrid KV cache Only the four full-attention layers carry a KV cache, the Mamba2 blocks keep recurrent state instead, which is what gives the architecture its flat long-context memory profile. The repo ships a kv_config.json from a real sensitivity pass that covers just those attention layers: three at 4-bit, one at 8-bit, 5.0 average KV bits. Point optiq serve at it for mixed-precision KV. NemotronH KV support NemotronH names its attention module mixer (not self_attn) and skips MLP layers in the prompt cache. optiq kv-cache classifies each layer as attention / SSM / MLP and maps cache slots to the right layer indices. Serving terminalbash $ optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit \ --kv-config kv_config.json --port 8000 # From any OpenAI-compatible client: $ curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit", "messages":[{"role":"user","content":"What is 17 * 23?"}]}' --- ## Muse-Glimmer on Apple Silicon Source: https://mlx-optiq.com/docs/muse-glimmer Family guide · Muse-Glimmer Muse-Glimmer on Apple Silicon Muse-Glimmer-30B is a 30B image-text reasoning model. It answers in two channels, thinking to itself before it replies, and it scores 87.36 on the six-metric Capability Score. The architecture 52 decoder layers with gated attention. Three layers in four use a 2048-token sliding window and the fourth attends globally, which is what keeps memory flat as context grows while still letting the model reach back across a long document. It reasons before it answers, in a separate channel. Nothing else implements this family: stock mlx-lm has no class for it, neither does mlx-vlm, and the checkpoint asks for a newer transformers than OptiQ pins. OptiQ ships an MLX-native port of both towers, checked against the reference implementation and matching to float32 round-off, so answers are the base model's rather than an approximation. The quant OptiQ quantizes only the language tower and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images. Property Value Model Muse-Glimmer-30B-OptiQ-4bit · base meta-models/Muse-Glimmer-30B Architecture 52-layer decoder, gated attention, sliding window 2048 on 3 layers in 4, ~30B params Language tower mixed 4/8-bit · 169 layers at 4-bit, 248 at 8-bit · 5.10 bpw Vision tower bf16, kept in optiq/optiq_vision.safetensors (806 tensors) On disk 18.6 GB language + 3.5 GB vision The sweep measured all 417 projections. Sensitivity falls steadily with depth, so the early layers keep their precision and the back half gets compressed: Layers Mean bits 0–12 6.88 13–25 6.50 26–38 6.27 39–51 5.85 Capability Score The standard OptiQ six-metric text eval. Perfect long-context retrieval, and the strongest GSM8K we have measured. Metric Score MMLU (5-shot, 969 samples) 83.1% GSM8K (1000 samples) 92.1% IFEval (full set, strict) 80.6% BFCL-V3 simple (200 calls) 88.5% HumanEval (164 problems, pass@1) 79.9% HashHop (long-context retrieval) 100.0% Capability Score (mean of 6) 87.36 Two channels Muse-Glimmer reasons before it answers, and it separates the two. The reasoning goes to self and the user-facing reply to user: raw outputtext to=self<|message|>April: 48. May: half as many = 24. Total 72.<|eom|> <|start|>assistant to=user<|message|>In April she sold 48 clips. 48 + 24 = 72 clips altogether. Read the final channel, not the whole string. The reasoning restates the question and floats candidates it then rejects, so anything parsing the raw output will pick up numbers the model did not commit to. The same applies to tool calls, which arrive in an block rather than the more common JSON. Hello world hello.pypython import optiq # registers the muse_glimmer arch + vision sidecar from mlx_lm import load, generate model, tok = load("mlx-community/Muse-Glimmer-30B-OptiQ-4bit") msgs = [{"role": "user", "content": "Explain why the sky is blue."}] prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True) print(generate(model, tok, prompt=prompt, max_tokens=800)) Give it room. It is a reasoning model, and a short token budget cuts it off mid-thought before the answer channel opens. --- ## Qwen on Apple Silicon Source: https://mlx-optiq.com/docs/qwen Family guide · Qwen Qwen on Apple Silicon Three Qwen generations run on OptiQ: Qwen3.8, Qwen3.6 and Qwen3.5. They share a chat template, a hybrid linear+full attention architecture, and the same ... reasoning channel, so most of what follows applies to all three. Where they differ, the section says so. Newest first. Qwen3.8 The newest of the three, and the first Qwen we ship with image input. Qwen3.8-27B is a hybrid: three quarters of its 64 blocks use linear attention and the rest full attention, which is why the sensitivity sweep scores 498 tensors rather than the ~450 a dense model this size would have. Model Size Capability Best for Qwen3.8-27B-OptiQ-4bit 18.9 GB 87.98 Image + text, long-form reasoning 261 of its 498 quantized tensors sit at 8-bit and 237 at 4-bit. The vision tower stays at bf16 in a sidecar under optiq/, so the same repo loads text-only under stock mlx-lm and image+text under OptiQ. Qwen3.6 Qwen3.6 is the successor to Qwen3.5: strong reasoning at sizes that fit on consumer Apple Silicon. We ship two quants: a dense 27 B and a 256-expert MoE with 3 B active per token. Both load with stock mlx_lm.load and reach 89–95 % on GSM8K. Available quants Model Size Capability Best for Qwen3.6-35B-A3B-OptiQ-4bit 20.1 GB 80.03 Sparse MoE, 256 experts, 3B active Qwen3.6-35B-A3B-OptiQ-4bit-REAP-19B 13.9 GB 76.57 Half the experts · 24.5 → 11.6 GB to run Qwen3.6-27B-OptiQ-4bit 15.7 GB 86.04 Strongest dense quant we ship Hello world hello.pypython from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.6-27B-OptiQ-4bit") prompt = tok.apply_chat_template( [{"role": "user", "content": "Compare REINFORCE and PPO in two paragraphs."}], tokenize=False, add_generation_prompt=True, ) print(generate(model, tok, prompt=prompt, max_tokens=600)) Recommended sampling sampling.pypython from mlx_lm.sample_utils import make_sampler # Strong reasoning baseline (Qwen3.6 supports thinking mode) sampler = make_sampler(temp=0.6, top_p=0.95, top_k=20) # Conversational sampler = make_sampler(temp=0.7, top_p=0.9) The 35B-A3B MoE model Qwen3.6-35B-A3B is a 256-expert sparse mixture-of-experts: 35 B total parameters, only 3 B active per token. The fused expert tensor (switch_mlp in MLX terminology) gets quantized as a single layer in mlx-optiq's sensitivity pass, but each expert independently uses the assigned bit-width. Expect MoE inference to be faster than the dense 27 B at the same memory footprint, because only 3 B of weights actually multiply per token. The sensitivity pass is also faster because there are fewer "layers" (the experts collapse into single switch_mlp tensors). Long-context serving serve.shbash # Sensitivity pass (1-2 min, once per model) $ optiq kv-cache mlx-community/Qwen3.6-27B-OptiQ-4bit \ --target-bits 5.0 --candidate-bits 4,8 \ -o ./kv/qwen36_27b # Mixed-precision KV serving $ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit \ --kv-config ./kv/qwen36_27b/kv_config.json \ --max-tokens 32768 --temp 0.6 --top-p 0.95 Fine-tuning 27B fits at max-seq-length=512 on a 36 GB Mac with default rank=8 LoRA on q_proj/v_proj. The 35B-A3B MoE caps at max-seq-length=128 due to per-expert memory overhead but trains 3× faster per iteration than the dense 27B because of sparse activation. The rank-scaling story is in the sensitivity-aware LoRA blog. finetune.shbash # Qwen3.6-27B at T=512, peak ~27.7 GB $ optiq lora train mlx-community/Qwen3.6-27B-OptiQ-4bit \ --data ./my_data \ --max-seq-length 512 \ --rank 8 --rank-scaling by_bits \ --iters 1000 -o ./my_adapter # Qwen3.6-35B-A3B at T=128, peak ~25.3 GB, 32 tok/s $ optiq lora train mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit \ --data ./my_data \ --max-seq-length 128 \ --rank 8 --rank-scaling by_bits \ --iters 2000 -o ./my_moe_adapter Re-quantizing locally Because the bf16 base is a vision-language model with a ~50 GB on-disk footprint, re-quantizing locally requires the bf16 weights cached and uses --reference uniform_4bit automatically (won't fit bf16 in RAM). Plan ~80 GB of disk headroom for a full pass. The pre-built quants on Hugging Face are usually what you want. Next: see how sensitivity works, or read the fine-tuning guide. Qwen3.5 Qwen3.5 is Alibaba's late-2025 release: a six-model dense lineup plus one sparse MoE. We ship mlx-optiq-4-bit quants for all six. They share the same chat template, hybrid linear+full attention architecture, and the same ... reasoning channel. Available quants Model Size Capability Best for Qwen3.5-122B-A10B-OptiQ-2bit 44 GB n/a Largest · 2-bit, SSD-streamed on a 36 GB Mac Qwen3.5-122B-A10B-OptiQ-2bit-REAP-63B 23.8 GB n/a Half the experts · 44 → 24 GB on disk Qwen3.5-35B-A3B-OptiQ-4bit 20.1 GB 77.42 Sparse MoE, 3B active Qwen3.5-35B-A3B-OptiQ-4bit-REAP-19B 12.8 GB n/a Half the experts · same recipe as Qwen3.6 Qwen3.5-27B-OptiQ-4bit 15.7 GB 82.22 Long-form reasoning Qwen3.5-9B-OptiQ-4bit 5.6 GB 69.85 Default daily-driver Qwen3.5-4B-OptiQ-4bit 2.8 GB 68.76 Sweet spot for laptops Qwen3.5-2B-OptiQ-4bit 1.4 GB 50.41 Local-only chat, classifiers Qwen3.5-0.8B-OptiQ-4bit 0.5 GB 38.42 Toy agents, prompt rewriters The 122B, on a laptop Qwen3.5-122B-A10B is the family's largest. Its OptiQ quant is a 2-bit static allocation (44 GB) that runs on a 36 GB Mac by streaming the mixture-of-experts off SSD: about 12 GB resident, ~5 tok/s. At 2-bit its Capability Score would not be meaningful, so it ships with a coherence check instead: it still writes working code at 2-bit. Read the write-up. Hello world hello.pypython from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") prompt = tok.apply_chat_template( [{"role": "user", "content": "What is the capital of Australia?"}], tokenize=False, add_generation_prompt=True, ) print(generate(model, tok, prompt=prompt, max_tokens=200)) Recommended sampling Qwen3.5-Instruct variants behave well at: sampling.pypython from mlx_lm.sample_utils import make_sampler # Reasoning / math / code sampler = make_sampler(temp=0.6, top_p=0.95, top_k=20) # Conversational chat sampler = make_sampler(temp=0.7, top_p=0.9) # Deterministic / classification sampler = make_sampler(temp=0.0) Reasoning channel Qwen3.5 has built-in chain-of-thought via ... tags. Default-on for instruct variants: thinking.pypython # Default: thinking enabled (slower, more accurate on math/logic) prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) # Disable thinking for snappier replies (chat, classification) prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) To programmatically strip the block before showing output to the user: strip_think.pypython import re out = generate(model, tok, prompt=prompt, max_tokens=800) final = re.sub(r".*?", "", out, flags=re.DOTALL).strip() Hybrid attention Qwen3.5 uses a hybrid architecture: most layers are linear-attention (Gated DeltaNet), and a sparse subset are full-attention. Layer indices [3, 7, 11, 15, 19, 23, ...] (every 4th) are full-attention; the rest are linear. This matters for KV-cache serving. Only the full-attention layers carry a KV cache that needs sensitivity-aware quantization. Why this matters for mlx-optiq The sensitivity-aware KV quantization pass focuses on the full-attention layers (the ones with a real KV cache). Linear-attention layers don't have a per-token KV the same way. mlx-optiq's kv-cache command correctly skips them. Long-context serving For 64 k-token contexts, run a one-time KV sensitivity pass and serve with the resulting config: serve.shbash # 1-2 min, once per model $ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \ --target-bits 5.0 --candidate-bits 4,8 \ -o ./kv/qwen35_9b # Serve at :8080 with mixed-precision KV $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --kv-config ./kv/qwen35_9b/kv_config.json \ --max-tokens 32768 --temp 0.6 --top-p 0.95 Fine-tuning recipes Empirical training-ceiling map at iogpu.wired_limit_mb=0 on a 36 GB Mac (default config: q_proj, v_proj, num_layers=16, rank=8, rank_scaling=by_bits). The rank-scaling rationale lives in the sensitivity-aware LoRA blog. Model Max seq len Peak mem Tokens/sec Qwen3.5-0.8B 2,800 23.4 GB 29.2 Qwen3.5-2B 2,400 19.3 GB 38.3 Qwen3.5-4B 1,600 24.8 GB 19.1 Qwen3.5-9B 1,400 25.4 GB 21.6 Qwen3.5-27B 512 27.7 GB 11.4 Qwen3.5-35B-A3B 128 25.3 GB 32.2 finetune.shbash # 9B at T=1400, proven sweet spot, peak 25.4 GB $ optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit \ --data ./my_training_data \ --max-seq-length 1400 \ --rank 8 --rank-scaling by_bits \ --num-layers 16 --iters 1000 \ -o ./my_adapter See the full LoRA fine-tuning guide for the algorithm, rank-scaling explanation and PEFT-compat output format. --- ## Gemma-4 on Apple Silicon Source: https://mlx-optiq.com/docs/gemma-4 Family guide · Gemma-4 Gemma-4 on Apple Silicon Google's Gemma-4 instruct series spans five sizes: two compact dense (e2b, e4b), a dense 12 B (the unified text+vision model, now with image input), one sparse mixture-of-experts (26 B-A4B), and one large dense (31 B). All five ship with mlx-optiq-4-bit quants on Hugging Face, and all five take image input. They share the Gemma chat template and a distinctive shared-KV attention design. Image input Gemma-4 takes image input. When a quant carries an optiq_vision sidecar, optiq serve and the Lab answer image+text prompts with no mlx-vlm dependency. See vision (image input). Available quants Model Size Capability vs uniform-4 Best for gemma-4-31B-it-OptiQ-4bit 18.1 GB 82.77 +3.47 Strongest dense quant gemma-4-26B-A4B-it-OptiQ-4bit 14.9 GB 75.76 +3.06 Sparse MoE, 4 B active gemma-4-26B-A4B-it-OptiQ-4bit-REAP-14B 10.5 GB 68.13 Half the experts · larger loss than Qwen at 50 % gemma-4-12B-it-OptiQ-4bit 8.3 GB 68.23 +6.40 Unified text+vision, image input gemma-4-e4b-it-OptiQ-4bit 6.0 GB 68.92 +13.57 Best small-model recovery gemma-4-e2b-it-OptiQ-4bit 4.0 GB 56.38 +2.12 Compact daily-driver Headline result gemma-4-e4b-it at uniform 4-bit collapses to 23.5% on GSM8K. mlx-optiq recovers it to 55.5%, a +32-point jump at the same 6 GB on disk. That gap is what uniform quantization discards. Full lineup story in the Gemma-4 launch blog. QAT variants Google also ships QAT-unquantized Gemma-4 checkpoints, bf16 weights trained so that 4-bit quantization is near-lossless. OptiQ runs its sensitivity pass on those bases and still beats a uniform 4-bit quant of the same QAT base on the six-metric Capability Score. The scores below compare each OptiQ-QAT quant against a genuine uniform-4 quant of its own QAT base, holding the base fixed, so the delta is purely what the per-layer allocation adds. Model (QAT base) Size Capability Δ vs uniform-4 gemma-4-e2b-it-qat-OptiQ-4bit 4.3 GB 57.23 +2.09 gemma-4-e4b-it-qat-OptiQ-4bit 6.5 GB 68.67 +1.19 gemma-4-12B-it-qat-OptiQ-4bit 8.9 GB 69.64 +1.37 gemma-4-26B-A4B-it-qat-OptiQ-4bit 20.7 GB 74.40 +0.19 gemma-4-31B-it-qat-OptiQ-4bit 22.4 GB 82.01 +1.65 The 31B model posts the largest absolute gain (+1.65, with a +9-point jump on long-context HashHop retrieval). The 26B-A4B sparse-MoE is the one case where 4-bit QAT already saturates the routed experts; reaching them takes a higher average bit-width, so its OptiQ quant ships larger and wins by a smaller margin. Load any of these exactly like the non-QAT quants below, the same mlx_lm.load path. Hello world hello.pypython from mlx_lm import load, generate model, tok = load("mlx-community/gemma-4-31B-it-OptiQ-4bit") prompt = tok.apply_chat_template( [{"role": "user", "content": "What's the difference between TF-IDF and BM25?"}], tokenize=False, add_generation_prompt=True, ) print(generate(model, tok, prompt=prompt, max_tokens=400)) Recommended sampling Gemma-4-it variants prefer slightly higher temperature than Qwen3.x: sampling.pypython from mlx_lm.sample_utils import make_sampler # Default chat (Google's recommended) sampler = make_sampler(temp=1.0, top_p=0.95, top_k=64) # Reasoning / math: slightly tighter sampler = make_sampler(temp=0.7, top_p=0.9) Mixed-precision KV cache Mixed-precision KV cache Gemma-4 uses shared-KV attention plus a sliding-window RotatingKVCache for SWA layers. Upstream mlx-lm raises NotImplementedError: RotatingKVCache Quantization NYI on the rotating path, which blocks mixed-precision KV on the SWA layers. mlx-optiq ships optiq.runtime.kv.RotatingQuantizedKVCache (a drop-in rotating cache with quantized (packed, scales, biases) storage) plus a small SDPA dispatch patch that handles Gemma-4's KV-sharing edge case. The patch is auto-installed by optiq serve and optiq kv-cache whenever quantized KV is requested. Each Gemma-4 OptiQ-4bit repo on Hugging Face bundles a recommended kv_config.json from a real sensitivity-analysis pass. Drop-in via --kv-config: gemma-4 long-context, mixed-precision KVbash $ optiq serve \ --model mlx-community/gemma-4-31B-it-OptiQ-4bit \ --kv-config kv_config.json The 26B-A4B MoE Gemma-4-26B-A4B is a sparse mixture-of-experts: 26 B total, 4 B active per token. Different from Qwen MoE, Gemma uses switch_glu with a fused gate-and-up projection. mlx-optiq's MoE walker handles both layouts. The per-expert sensitivity rolls up into a single switch_glu tensor for allocation purposes. Long-context fp16 KV is still the simplest path; mixed-precision KV (above) gives faster decode at long contexts. Either way: serve.shbash # Stock fp16 KV serving via mlx-optiq (no --kv-config) $ optiq serve --model mlx-community/gemma-4-31B-it-OptiQ-4bit \ --max-tokens 8192 --temp 1.0 --top-p 0.95 Fine-tuning recipes Empirical training-ceiling map at iogpu.wired_limit_mb=0 on 36 GB Mac: Model Max seq len Peak mem Tokens/sec gemma-4-e2b-it 2,400 22 GB ~28 gemma-4-e4b-it 1,800 24 GB ~22 gemma-4-26B-A4B-it 512 27.6 GB 22.2 gemma-4-31B-it 32 21.4 GB 30.9 The 31B-dense is unusually memory-tight at long context due to its larger embedding+vocab footprint. The 26B-A4B MoE actually allows a longer sequence at the same RAM budget. The sparse activation helps at long context. finetune.shbash $ optiq lora train mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit \ --data ./my_data \ --max-seq-length 512 \ --rank 8 --rank-scaling by_bits \ --num-layers 16 --iters 2000 \ -o ./gemma_adapter Next: read about how sensitivity works or the LoRA fine-tuning guide. --- ## DeepSeek-V4 on Apple Silicon Source: https://mlx-optiq.com/docs/deepseek Family guide · DeepSeek-V4 DeepSeek-V4 on Apple Silicon DeepSeek-V4-Flash is a sparse-attention mixture of experts. Across its 43 layers it runs three different attention mechanisms, chosen per layer by a compress_ratios entry in the config: five layers use sliding-window local attention, twenty compress the KV, and twenty-one compress it and then select a top-k subset to attend over. Almost all of its weight sits in the routed experts, which is what makes it quantize well. Vendored architecture Stock mlx-lm has no deepseek_v4 class. Three separate pull requests tried to add one and none were merged, so OptiQ vendors the working implementation and registers it on import optiq. Call import optiq before mlx_lm.load. Running it on a small machine The 2-bit quant is 92.5 GB on disk, which is more than most Macs have in RAM. OptiQ streams the routed experts off SSD instead of holding them resident: the scales and biases stay in memory, the packed expert weights are read as a token routes to them. The model loads in a few seconds at 6.5 GB resident. Peak memory climbs with context rather than with model size, so a short prompt stays near 8.5 GB and a 900-token one reaches about 12.7 GB. Nothing needs configuring. optiq serve and the Python API detect a MoE quant too large to sit resident and switch to streaming on their own. The quant Model bf16 size OptiQ size Resident Context DeepSeek-V4-Flash-0731-OptiQ-2bit 608 GB 92.5 GB 6.5 GB 1M Hello world hello.pypython import optiq # registers the deepseek_v4 arch with mlx-lm from optiq.runtime import moe_stream from mlx_lm import generate model, tok = moe_stream.load_streaming("mlx-community/DeepSeek-V4-Flash-0731-2.4bit-mixed") print(generate(model, tok, prompt="What is 17 * 23?", max_tokens=64)) Expect it to be slow Streaming experts from SSD trades throughput for fitting at all. Decode runs at a few tokens per second on a 24 GB machine, which is fine for a batch job and painful for a chat. With enough RAM to hold the model resident it runs at normal speed. How the bits are allocated The routed experts hold the overwhelming majority of the parameters and tolerate 2-bit; attention, the shared experts, the embeddings and the output head do not, and stay at 4-bit or higher. Quantizing everything uniformly collapses the model into repeated fragments, which is the same result other projects report on this architecture. This is what OptiQ's per-layer allocation does everywhere, but the effect is starker here than on a dense model, because the split between what matters and what does not lines up so cleanly with the expert boundary. Prompt format DeepSeek-V4 ships no chat template, so the prompt is assembled by hand: prompttext <|begin_of_sentence|><|User|>{your message}<|Assistant|> The trailing closes the reasoning block so the model answers directly. Leave it off and it thinks first. --- ## LFM2.5 on Apple Silicon Source: https://mlx-optiq.com/docs/lfm2.5 Family guide · LFM2.5 LFM2.5 on Apple Silicon LFM2.5 is Liquid AI's on-device series, from 230M up to an 8B sparse MoE. It is a hybrid: short convolutional blocks alternate with full-attention blocks, and only the attention blocks carry a KV cache. On the 1.2B that means six layers hold cache out of the whole stack, so memory grows slowly as context does. That is the property that makes the family interesting on a laptop or a phone, and it is why the bundled KV config is short. As of LFM2.5-VL the family also reads images. The VL model pairs a SigLIP2 encoder with the same hybrid language tower, so it inherits the short KV cache. The quants Every row is a six-metric Capability Score, not a perplexity number. Sizes are the artifact on disk. Model bf16 size OptiQ size Compression Capability Score LFM2.5-230M-OptiQ-4bit 443 MB 180 MB 2.5× 24.83 LFM2.5-350M-OptiQ-4bit 681 MB 269 MB 2.5× 26.60 LFM2.5-1.2B-Instruct-OptiQ-4bit 2.34 GB 825 MB 2.8× 47.82 LFM2.5-1.2B-Thinking-OptiQ-4bit 2.34 GB 820 MB 2.9× 54.14 LFM2.5-1.2B-JP-202606-OptiQ-4bit 2.34 GB 838 MB 2.8× 47.83 LFM2.5-2.6B-OptiQ-4bit 5.2 GB 1.93 GB 2.7× 35.19 LFM2.5-VL-3B-OptiQ-4bit 6.3 GB 2.83 GB 2.2× 37.93 LFM2.5-8B-A1B-OptiQ-4bit 15.8 GB 5.46 GB 2.9× 28.69 Which one to reach for The Thinking variant is the one to use if you can spare the tokens. It scores 64.7% on MMLU against the Instruct model's 43.3%, and 82.8% on GSM8K against 69.7%. Reasoning is doing real work at this size, not decorating the answer. The Japanese variant is the interesting one. Its overall score lands at 47.83 against the Instruct model's 47.82, near enough to call identical, but the capability sits in different places: seven points more on MMLU, nearly ten more on HumanEval, and roughly eight less on both GSM8K and IFEval. Same total, redistributed. Below a gigabyte the 230M and 350M are useful for classification, routing and extraction rather than open-ended chat. Both score at chance on long-context retrieval. Vision LFM2.5-VL-3B adds a 27-layer SigLIP2 encoder and a two-layer projector on top of the hybrid language tower. OptiQ quantizes the language path to mixed 4/8-bit and keeps the encoder and projector at bf16 in the optiq/ sidecar, so one artifact does text and images. Property Value Vision tower SigLIP2, 27 layers, hidden 1152, patch 16 Resolution native aspect ratio, position embeddings resampled per image Token budget 2×2 pixel unshuffle before the projector, capped at 256 image tokens Language tower mixed 4/8-bit: 88 layers at 4-bit, 78 at 8-bit On disk 1.98 GB language + 0.85 GB vision Capability Score 37.93. It reads images well and its text scores sit mid-pack for a 3B, but it does not initiate tool calls: BFCL is 0.0% because the model works the answer out itself rather than calling the function. Asked for the area of a triangle it derives the formula, where the text-only 1.2B calls calculate_triangle_area and scores 45.0% through the same harness. It emits correct calls when told to use a tool, so the capability is there and the model simply does not reach for it. For agentic tool use at this size, take the text 1.2B or 2.6B. The vision tower runs natively in MLX, with no PyTorch and no mlx-vlm at runtime. It was checked against the reference implementation across four patch grids and matches to bf16 round-off, so image answers are the base model's, not an approximation of them. Hello world hello.pypython from mlx_lm import load, generate model, tok = load("mlx-community/LFM2.5-1.2B-Thinking-OptiQ-4bit") prompt = tok.apply_chat_template( [{"role": "user", "content": "What is 17 * 23? Think briefly then answer."}], tokenize=False, add_generation_prompt=True, ) print(generate(model, tok, prompt=prompt, max_tokens=2048)) Give the Thinking model room It spends its first few hundred tokens inside the reasoning block, and that comes out of max_tokens. Set the cap too low and the entire allowance is consumed before the answer starts, so you get an empty string back rather than a short reply. 2048 is a sensible floor. Tool calling LFM2.5 does not emit JSON tool calls. It writes Python, between two special tokens: tool calltext <|tool_call_start|>[get_weather(city="Paris")]<|tool_call_end|> Pass tools= to apply_chat_template and the catalog is rendered into the system prompt. OptiQ parses this form in both the evaluator and the agent loop, so a model that calls tools correctly scores as though it does. Serving Each quant ships a kv_config.json sized for its handful of attention layers. On the 1.2B that is six layers at roughly four bits. servebash optiq serve --model mlx-community/LFM2.5-1.2B-Thinking-OptiQ-4bit \ --kv-config kv_config.json --- ## LoRA fine-tuning Source: https://mlx-optiq.com/docs/finetune Workflow · LoRA LoRA fine-tuning mlx-optiq ships a LoRA trainer that reads its own per-layer sensitivity assignments and gives sensitive layers proportionally more adapter capacity. Output is PEFT-compatible (adapter_config.json + adapters.safetensors) plus an mlx-optiq sidecar describing the per-layer rank distribution. The same layers mlx-optiq kept at 8-bit during quantization also get more adapter rank during fine-tuning. The same sensitivity signal drives quantization and LoRA rank. The basic recipe train.shbash # Defaults: all transformer blocks adapted, all 7 trainable linears per # block (Unsloth-aligned), rank 8 with by_bits sensitivity overlay, # alpha = rank, mask_prompt enabled, max_seq=512. LR + iterations are # method-aware: SFT runs 3 epochs at 2e-4 (see "Iterations & learning # rate" below). Pass --iters for an absolute step count instead. $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --data ./my_training_data \ -o ./my_adapter # Show the per-layer rank distribution $ optiq lora info ./my_adapter Preset bundles for quick rank selection: --preset small (r=8, α=16), default (r=8, α=8), medium (r=16, α=16), large (r=32, α=32), xl (r=64, α=64), xxl (r=128, α=128). Presets set the BASE rank; with --rank-scaling by_bits (default), per-layer rank still scales up on layers OptiQ kept at higher bits. Data format JSONL with either messages (chat format) or prompt/completion pairs. Use one of these formats, not bare text. The text format can't expose a prompt/response boundary, so prompt masking falls through to full-sequence loss and degrades quality on tasks where the base model is already competent. data.jsonljson {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]} {"prompt": "...", "completion": "..."} Chat template is applied automatically; do not pre-template the data. mask_prompt is on by default, so loss is computed only on the assistant's response tokens. Layout on disk: directory layoutbash my_training_data/ ├── train.jsonl └── valid.jsonl # optional, used for validation loss SFT and DPO The same optiq lora train command trains both objectives; pick with --method. SFT (--method sft, the default) is standard supervised fine-tuning on your responses. DPO (--method dpo, Direct Preference Optimization) aligns the model on preferences and is the usual second stage: SFT first to teach the format and task, then DPO to prefer better responses over worse ones. dpo.shbash # DPO on top of an SFT adapter (the standard two-stage flow): # --mount-adapter starts DPO from your SFT weights; --dpo-beta is the # KL constraint (default 0.1); LR defaults to 5e-5 for DPO. $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --method dpo \ --data ./my_preference_data \ --mount-adapter ./my_sft_adapter \ --dpo-beta 0.1 \ --fused-dpo \ # chunked logp for >4k context (see below) --iters 200 \ -o ./my_dpo_adapter DPO data is JSONL with a chosen and a rejected completion for the same context. Single-turn uses {prompt, chosen, rejected} strings; multi-turn / agentic uses full message lists: preferences.jsonljson // single-turn {"prompt": "...", "chosen": "...", "rejected": "..."} // multi-turn (chosen/rejected are full {messages} lists) {"chosen": [{"role": "user", ...}, ...], "rejected": [{"role": "user", ...}, ...]} Both live in OptiQ Lab too Prefer a UI? optiq lab → Fine-tune exposes the same knobs: pick sft or dpo, point at a dataset, mount an adapter, and launch. Same engine, same defaults as the CLI. Preference-pair requirement Both chosen and rejected must be plausible completions of the same prompt under the base model. If chosen answers a different context than the prompt introduces, both reward terms drift in lockstep and the margin signal saturates near zero (the "loss=0, rewards drifting to -hundreds" pathology). The trainer prints a one-shot warning if the first validation pass shows this signature. Iterations & learning rate --iters is optional. Omit it and OptiQ trains by epoch (num_epochs × ceil(examples / batch)), defaulting to 3 epochs for SFT and 1 for DPO (one epoch is a preference nudge; more invites the collapse pathology). Pass --num-epochs N to change the count, or --iters N for an absolute step count. An explicit value always wins. Learning rate is method-aware: omit --learning-rate and SFT uses 2e-4, DPO uses 5e-5. These defaults resolve inside OptiqLoraConfig, so the CLI, OptiQ Lab, and direct construction all agree: a method="dpo" config no longer silently trains at the SFT rate. epochs.shbash # No --iters: 3 epochs (SFT default) at 2e-4 over the dataset $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --data ./my_training_data -o ./my_adapter # Two epochs instead of the default three $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --data ./my_training_data --num-epochs 2 -o ./my_adapter NEFTune noisy-embedding SFT NEFTune (Jain et al. 2023) adds uniform noise, scaled by alpha / sqrt(seq_len × embed_dim), to the token embeddings during the SFT forward pass, and nothing at inference. It's a near-free regularizer that stops the adapter from over-memorizing the exact surface form of a small dataset, which measurably improves instruction-following. Pass --neftune-noise-alpha N (the paper suggests 5–15); it's off by default (matching TRL/Unsloth), and SFT only (mirroring TRL, which applies it in the SFTTrainer, not DPO). The noise is gated on training mode, so your validation loss is still measured on clean embeddings. neftune.shbash $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --data ./my_training_data -o ./my_adapter \ --neftune-noise-alpha 5 Early stopping & experiment logging Both are optional and off by default, and both work the same in the CLI and in OptiQ Lab (the Lab surfaces them through the same config, and its live chart keeps updating alongside them). Early stopping. Pass --early-stopping-patience N to halt once the validation loss hasn't improved for N consecutive evaluations (evaluations fire every --steps-per-eval steps). It needs a validation set (valid.jsonl); without one it's a no-op with a warning. Use --early-stopping-min-delta D to require an improvement of at least D to count as progress (so tiny wiggles don't keep training alive). When it fires, the best-validation adapter is promoted to the returned adapter. The trailing steps that tripped the counter are worse than the best, so you get the good checkpoint, not the last one. (OptiQ always snapshots the best under /best/ regardless; early stopping just also makes it the top-level adapter.) Experiment logging. Pass --report-to wandb (needs pip install wandb + a WANDB_API_KEY or wandb login) to stream train/val metrics to Weights & Biases; swanlab is also supported, and you can pass several comma-separated. Set the project with --wandb-project. The tracker runs alongside the normal CLI log and the Lab chart, not instead of them. early-stop.shbash # Stop after 3 evals with no meaningful val improvement; log to W&B $ optiq lora train mlx-community/Qwen3.5-4B-OptiQ-4bit \ --data ./my_training_data -o ./my_adapter \ --steps-per-eval 50 \ --early-stopping-patience 3 --early-stopping-min-delta 0.01 \ --report-to wandb --wandb-project my-optiq-runs Sensitivity-aware rank scaling --rank-scaling by_bits (default) gives each layer an adapter rank proportional to its quantization bit-width. With --rank 8: Layers mlx-optiq quantized at 4-bit get rank 8. Layers mlx-optiq quantized at 8-bit get rank 16. Head-to-head on a 6-category logical-puzzles reasoning dataset (Qwen3.5-4B-OptiQ-4bit, 1 epoch over 200 training samples, 100-sample test split): Config Trainable params Test accuracy Constant rank-8 11.58 M 27 % by_bits (rank 8 / 16) 13.49 M (+16%) 35 % Constant rank-16 22.20 M (+92%) 36 % by_bits matches constant rank-16 on accuracy (35% vs 36%, within noise at n=100) using 39% fewer trainable parameters. Versus constant rank-8 at almost matched param budget, by_bits is +8 absolute accuracy points. Full per-category breakdown in the sensitivity-aware LoRA blog post. Other scaling modes: scaling.shbash # Constant rank (matches Unsloth / PEFT default behaviour) $ optiq lora train ... --rank-scaling constant # Scale by raw KL sensitivity (more aggressive than by_bits) $ optiq lora train ... --rank-scaling by_kl Training-ceiling map (36 GB Mac) Empirical sequence-length and peak-memory ceilings at the system-default iogpu.wired_limit_mb=0. Measured under the conservative recipe: num_layers=16 (only the last 16 transformer blocks adapted), target_modules=q_proj,v_proj, rank=8. The current default (num_layers=-1, all 7 target modules) adapts ~3× more LoRA modules so pushes seq-length proportionally lower for the same model on the same machine; drop num_layers or max_seq_length to land within these ceilings if you need the full capacity headroom for very long contexts. Long context on 24 GB Above max_seq_length=4096 the full [seq, vocab] logit tensor is what OOMs first (a ~250k-token vocab makes it multi-GB). OptiQ auto-enables a fused cut-cross-entropy at that point (set OPTIQ_FUSED_CE=0/1 to override): the vocab head + loss are computed in chunks so the full logit tensor is never materialized, gradient-equivalent to the plain path. This trains 8k context at ~14.7 GB peak on a 24 GB M4 where the plain path OOMs, and scales to ~12k. It composes with all layers adapted; it does not change the attention-side ceilings in the table below. Long context for DPO DPO has the same problem worse: its per-sequence logp materializes the full [seq, vocab] logits four times per step (policy + reference × chosen + rejected), so on a 24 GB Mac the plain path OOMs by ~4k. The --fused-dpo flag (or OPTIQ_FUSED_DPO=1) applies the same chunked-head trick to the logp, gradient-verified exact against autograd. Unlike SFT this is opt-in, not auto: the break point is VRAM-dependent. Measured · Qwen3.5-4B-OptiQ-4bit · M3 Max 36 GB · batch 1, all layers, --grad-checkpoint context SFT step SFT peak DPO step (--fused-dpo) DPO peak 2048 2.8 s 7.7 GB 29 s 8.7 GB 4096 6.7 s 11.4 GB 144 s 10.9 GB 8192 45 s 19.9 GB 216 s 17.3 GB Both fit 8k on this machine, and DPO's peak (17.3 GB) leaves room on a 24 GB Mac. Step time still grows faster than context (attention is O(seq²), and DPO pays it across four forwards) so 4k remains the comfortable working point and 8k is for when you need it, not by default. Tuning knobs you rarely need flag / env default what it does --adapt-experts off Adapt a MoE model's expert pools. Every expert gets its own LoRA, so the adapter grows by the expert count, on a 256-expert model that is 128×, a 1.2 B-parameter adapter at r=8. Attention projections are adapted either way. OPTIQ_FLASH_ATTN auto auto uses MLX's fused SDPA whenever its [B, Hq, T, T] score tensor fits a memory budget, and a tiled FlashAttention-2 backward otherwise. always / never force it. OPTIQ_FLASH_ATTN_BUDGET_GB 25% of wired limit How much of Metal's wired limit stock SDPA's scores may occupy before the tiled backward takes over. OPTIQ_FLASH_BLOCK 128 Query-block size for the tiled backward. Peak memory scales with it; step time does not. OPTIQ_FUSED_CE auto (on) Fused cut-cross-entropy for the SFT LM head. --fused-dpo off The same chunked head for DPO's four logp passes. Needed beyond ~4k on a 24 GB Mac. These numbers changed in 0.3.0 Earlier docs quoted ~30 min per DPO step at 4k and called anything beyond 4k impractical. Those figures reflected a slow kernel that ran on every step (fixed in 0.3.0): OptiQ's flash-attention Metal kernel is 14–137× slower than MLX's fused SDPA. It is now taken only when stock SDPA's [B, Hq, T, T] score tensor would not fit in memory. Same math, ~13× faster DPO at 4k. See the changelog. directory layoutbash my_adapter/ ├── adapter_config.json # PEFT-compatible config ├── adapters.safetensors # PEFT-compatible weights └── optiq_lora_config.json # mlx-optiq sidecar with per-layer ranks Inspect the per-layer rank distribution: terminalbash $ optiq lora info ./my_adapter # OptiQ LoRA adapter # base: mlx-community/Qwen3.5-4B-OptiQ-4bit # rank: 8 (scaling: by_bits) # scale: 1.0 dropout: 0.0 # rank distribution: {8: 101, 16: 27} across 128 adapted modules Loading an adapter load_adapter.pypython from mlx_lm import load, generate model, tok = load( "mlx-community/Qwen3.5-9B-OptiQ-4bit", adapter_path="./my_adapter", ) print(generate(model, tok, prompt="...", max_tokens=200)) Hot-swap adapters at serve time mlx-optiq's mounted-LoRA primitive lets you keep N adapters resident on one base, switching per-request. See the serving guide. --- ## KV-quant serving Source: https://mlx-optiq.com/docs/serve Workflow · serving KV-quant serving optiq serve is a drop-in replacement for mlx_lm.server. It exposes both the OpenAI /v1/chat/completions endpoint and the Anthropic /v1/messages endpoint from the same process. Point Claude Code, the OpenAI SDK, the Anthropic SDK, or plain curl at the same local URL. On top of that: sensitivity-aware quantized KV cache for long-context throughput, automatic prompt caching (multi-turn prefix reuse), structured JSON/regex output, model switching per request, and mounted LoRA adapters that swap per request. Quickstart terminalbash # Stock fp16 KV serving. Works for any mlx-optiq quant. $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --port 8080 Then call it like any OpenAI endpoint: curl_chat.shbash $ curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mlx-community/Qwen3.5-9B-OptiQ-4bit", "messages": [{"role": "user", "content": "What is RoPE?"}], "max_tokens": 300, "stream": false }' Mixed-precision KV cache Default optiq serve uses fp16 KV. Pass --kv-bits 4 for uniform 4-bit KV or --kv-config kv_config.json for OptiQ mixed-precision KV. Both automatically install the streaming-converter and FlashAttention-2 patches that prevent stock mlx-lm's OOM at long context: on a 24 GB Mac at 32k context, peak drops from 16.35 GB (stock u4) to 7.60 GB (ours), 34% below the fp16 KV path. Speed is within ±2% of fp16 on Qwen3.5-9B-class models. At hash-hop hops=3 on Qwen3.5-9B, OptiQ mixed-precision scores 33% better than uniform 4-bit (32% vs 24% retention of fp16's 36%). For the engineering write-up see Why u4 KV cache OOMs harder than fp16; the underlying research on per-layer KV sensitivity is in Not All Layers Are Equal. Step 1: measure terminalbash # 1-2 min. Once per model. $ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \ --target-bits 5.0 \ --candidate-bits 4,8 \ -o ./kv/qwen35_9b # writes ./kv/qwen35_9b/kv_config.json # [{"layer_idx": 3, "bits": 8, "group_size": 64}, ...] Step 2: serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --kv-config ./kv/qwen35_9b/kv_config.json \ --max-tokens 32768 --temp 0.6 --top-p 0.95 Why this works Layer 0's KV is often 56× more sensitive than the layer-average. Uniform 4-bit KV is catastrophic. A single 8-bit layer (the most KV-sensitive one, often layer 3 in Qwen3.5's hybrid attention) protects quality while every other layer runs 4-bit. Apple Silicon's mx.quantized_matmul also handles the 8-bit fast path more efficiently than 4-bit, so protecting that one layer also flips it onto a faster kernel. So the 8-bit layer improves both quality and speed. Prompt caching: automatic prefix reuse Multi-turn conversations reuse the KV cache of their shared prefix automatically. No flag. When a request extends a conversation the server has already seen (the agentic pattern, where each turn appends to the history), the server matches the longest cached prefix and prefills only the new tokens. So the time-to-first-token of turn N stops growing with conversation length. Measured on Qwen3.5-0.8B with a ~4.3k-token context: turn 1 (cold) prefills in ~0.97s; turn 2, extending it, reuses 4306 of 4331 tokens and drops to ~0.24s, a ~4× TTFT cut that grows with model size (bigger models have slower prefill). This is what makes Claude Code and other agents feel responsive against a local model: every turn after the first is near-instant to first token. The cache is an LRU across conversations, held in RAM under a byte budget set by --prompt-cache-bytes. It's inherited from mlx-lm and on by default. Nothing to configure for the common case; raise the budget to keep more concurrent conversations warm: terminalbash $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --prompt-cache-bytes 8000000000 --port 8080 Each response reports how many prompt tokens were served from cache in usage.prompt_tokens_details.cached_tokens. Speculative decoding Decode faster without changing what the model writes. --ngram-draft 8 works on any model and pays most on coding agents, which keep copying text already in the conversation (1.3-1.5× on real coding turns). Qwen3.5/3.6 quants can use their bundled MTP head with --mtp, and Gemma-4 E4B its assistant drafter with --drafter. terminalbash $ optiq serve --model mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit-REAP-19B \ --ngram-draft 8 --port 8080 The speculative decoding guide compares the three, with measurements and when each one pays off. LoRA adapter at serve time The CLI accepts one adapter per server process via --adapter. The argument is either a HuggingFace repo id (auto-downloaded into the OptiQ adapter cache) or a local directory: terminalbash # Local path $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --adapter ./my_adapter # HF repo id (downloaded on first use) $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --adapter your-org/my-adapter OptiQ-trained adapters surface their optiq_lora_config.json sidecar (rank, rank distribution, scaling mode) in the startup log. Multi-adapter, hot-swap at the Python layer The mounted-LoRA primitive supports multiple adapters resident at once, switched per call via a ContextVar. This is a programmatic API, not a CLI flag. Embed it in your own server or notebook to serve N adapters from one base: mount.pypython from mlx_lm import load, generate from optiq.adapters.mount import ( prepare_model_for_mounted_lora, mount_adapter_on_model, AdapterActivation, ) model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") prepare_model_for_mounted_lora(model) mount_adapter_on_model(model, "agent-A", "./adapter_a") mount_adapter_on_model(model, "agent-B", "./adapter_b") with AdapterActivation("agent-A"): out_a = generate(model, tok, prompt=p, max_tokens=100) with AdapterActivation("agent-B"): out_b = generate(model, tok, prompt=p, max_tokens=100) Mounted adapters stay separate from the base (unlike mlx-lm's load_adapters, which merges weights). The ContextVar means concurrent asyncio tasks or threads with different active adapters don't step on each other. Memory math One Qwen3.5-9B-OptiQ-4bit base is ~5.6 GB. Each LoRA adapter is ~50 MB. 10 adapters co-resident ≈ 6.1 GB, vs ~56 GB if you spun up one full model copy per adapter. Switching is free. No weight reload, no GPU re-upload. Embedding the server in your own process If you want to install the OptiQ KV-cache hooks and Anthropic endpoint into a self-managed mlx_lm.server launch: embed_serve.pypython from optiq.serve import _load_kv_config, install_mixed_kv from optiq.anthropic_server import install_anthropic_endpoint import sys # Per-layer mixed-precision KV cache configs = _load_kv_config("./kv/qwen35_9b/kv_config.json") install_mixed_kv(kv_configs=configs, quantized_kv_start=0) # Anthropic /v1/messages alongside OpenAI /v1/chat/completions install_anthropic_endpoint() # Hand off to mlx_lm.server sys.argv = ["mlx_lm.server", "--model", "mlx-community/Qwen3.5-9B-OptiQ-4bit", "--port", "8080"] from mlx_lm.server import main main() OpenAI client compatibility Use the official openai Python client by pointing it at your local server: openai_client.pypython from openai import OpenAI client = OpenAI( base_url="http://localhost:8080/v1", api_key="not-used", # local server, but key is required ) resp = client.chat.completions.create( model="mlx-community/Qwen3.5-9B-OptiQ-4bit", messages=[{"role": "user", "content": "hi"}], stream=True, ) for chunk in resp: print(chunk.choices[0].delta.content or "", end="") Structured / JSON output Pass an OpenAI response_format and the server constrains generation so the model can only emit tokens that keep the output valid. Both json_object (any valid JSON) and json_schema (a specific shape) are supported, plus the vLLM-style guided_regex and guided_choice extensions. Output is constrained to stay valid, so it parses without retries. curl_json_schema.shbash $ curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [{"role": "user", "content": "A person: Alice, 30."}], "response_format": { "type": "json_schema", "json_schema": {"schema": { "type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"] }} } }' # -> {"name": "Alice", "age": 30} The vLLM extensions take a raw regex or a fixed choice set: guided.shbash # constrain to a regex $ curl ... -d '{"messages": [...], "guided_regex": "[0-9]{3}-[0-9]{3}-[0-9]{4}"}' # -> 123-456-7890 # constrain to a fixed set of answers $ curl ... -d '{"messages": [...], "guided_choice": ["yes", "no"]}' # -> yes In the OptiQ Lab The Lab Chat surface exposes the same constraint without curl. In Model & params, set JSON mode to any valid JSON or match schema (paste a JSON Schema in the box that appears), then send a message. The reply is guaranteed-valid JSON, parsed the same way the curl examples above are constrained. Tools are turned off while JSON mode is on, since a single constrained response and a tool-call loop are mutually exclusive. The Lab's server installs the constraint automatically, so any model you load supports it. No PyTorch Structured output is built on lm-format-enforcer (pure-Python: pydantic + interegular), not xgrammar, which hard-requires PyTorch and would break OptiQ's MLX-native, no-torch runtime. The constraining logit mask adds about 1 ms/token; free-form requests (no response_format) are untouched. For reasoning models, thinking is auto-disabled when a spec is present so the constrained output lands in content, not reasoning. Tool-call healing Quantized open-weight models often emit a malformed tool call instead of the clean structured form: a Hermes tag, a fenced JSON block, a bare object, trailing commas, fancy quotes, or function-call syntax. mlx-lm's parser only recognizes the canonical format, so a malformed call leaks into the message content as raw text and the client never gets a tool_calls array. OptiQ heals these server-side. On a non-streaming completion that carried tools, it scans the output for the six common malformed shapes, recovers them into proper OpenAI tool_calls, strips the leftover "I'll call X" preamble from content, and sets finish_reason to tool_calls. Unknown tool names are rejected so the model cannot invent a tool. Always on; requests without tools are untouched. It is the same healer the OptiQ Lab uses, lifted to the server layer so any client (an agent, or Claude Code through the OpenAI endpoint) gets clean calls, not just the Lab UI. Prevent vs repair Two routes to reliable tool calls: structured output above constrains the model so it can only emit valid calls, and healing repairs malformed ones after the fact. Healing runs on non-streaming completions, where the full output is in hand. The model field: single-model by default optiq serve --model X hosts one model, but the OpenAI/Anthropic protocol requires a model field on every request. By default the server treats it as a label: every request is served by --model X, whatever the client sends. So a client sending the model's basename, a friendly alias, or a wrong default (Claude Code's claude-…) is served the one model instead of the server trying to download that id and returning a 404. This matches how the OptiQ Lab serves. One model per process, switched by restarting the server. To let one running server hot-swap between cached models per request, pass --allow-model-switch (auto-enabled by --models-dir). Then the model field selects the model to load; an unknown id errors rather than being pinned. /v1/models lists every MLX model in your HuggingFace cache plus the served one. switch.shbash # opt into per-request switching $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --allow-model-switch --port 8080 # now naming a different (cached) model switches to it $ curl http://localhost:8080/v1/chat/completions -d '{ "model": "mlx-community/Qwen3.6-27B-OptiQ-4bit", "messages": [{"role": "user", "content": "hi"}] }' Locally-built quants that were never pushed to the hub won't show up in the cache scan. Point --models-dir at the directory holding them and they're advertised in /v1/models too, switchable by passing their path as the request model: terminalbash $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --models-dir ./optiq_output --port 8080 Model variants: thinking & sampling presets by name Append a variant suffix to the model id and the server strips it before load (it maps to the real model at zero extra memory) and applies the preset to that request. So any OpenAI client selects it by name, without the non-standard chat_template_kwargs/sampler fields many clients don't expose. Two kinds of variant: Thinking: :no-think (direct answers) / :think (full reasoning), the reasoning-model toggle. Sampling presets: :precise (temp 0, deterministic), :creative (temp 0.8, top_p 0.95), :balanced (temp 0.4, top_p 0.9). The served model's variants are listed in /v1/models. variants.shbash # direct answer, no reasoning trace (faster, no rambling) $ curl http://localhost:8080/v1/chat/completions -d '{ "model": "mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit:no-think", "messages": [{"role": "user", "content": "Write a bubble sort in Python."}] }' On families whose template has no thinking toggle (e.g. Gemma-4) the suffix is a harmless no-op; an unknown suffix (or a repo-id / path that legitimately contains a colon) passes through untouched. Works identically in optiq serve and the OptiQ Lab's API server. Anthropic API: point Claude Code at your local quant The same server simultaneously answers Anthropic's /v1/messages endpoint with the exact response shape Claude clients expect. This means you can drive a local mlx-optiq quant from any tool that speaks the Anthropic API: Claude Code, the official anthropic Python SDK, or your own integrations. terminalbash # Same optiq serve invocation. No extra flag needed. $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --port 8080 Anthropic SDK against your local quant: anthropic_client.pypython from anthropic import Anthropic client = Anthropic( base_url="http://localhost:8080", api_key="not-used", ) resp = client.messages.create( model="mlx-community/Qwen3.5-9B-OptiQ-4bit", max_tokens=300, messages=[{"role": "user", "content": "hi"}], ) print(resp.content[0].text) Claude Code via env var (one line): claude_code.shbash export ANTHROPIC_BASE_URL="http://localhost:8080" export ANTHROPIC_API_KEY="not-used" $ claude # now driven by your local quant Context scaling for smaller-context models. Claude Code auto-compacts based on the token usage the server reports vs the context window it assumes (~200k). A smaller-context local model would overflow before it compacts. --context-scale FACTOR multiplies the reported usage so compaction fires at the right real-token point. Set it to (window the client assumes) / (your model's context), e.g. 6.25 for a 32k model behind a 200k assumption. Only the reported usage is scaled; generation is untouched. See the Claude Code guide. What's translated The shim accepts Anthropic-shaped requests, translates them into the OpenAI request the underlying mlx-lm engine wants, runs generation, and translates the response back into Anthropic shape, including streaming events. system, messages, max_tokens, stream, temperature, and top_p all work. Tool-use parameters are accepted but route through the same generation path (the underlying model does what it does; there's no server-side function-calling router). Memory & resilience on Apple Silicon Three flags keep a server healthy on unified memory, where the model, KV cache, and everything else share one RAM pool. Idle auto-unload A served model sits in unified memory even while idle, so the machine can't use that RAM for anything else. --idle-timeout SECONDS drops the model after that many seconds with no requests and reloads it lazily on the next call, the same lazy first-request load, re-triggered. A generation already in flight finishes safely (it holds its own reference to the model); only the next request pays the reload. Set the timeout longer than your longest single generation so a normal long decode is never interrupted. Off by default. free the model after 10 min idlebash optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --idle-timeout 600 Memory-aware context cap A request that runs to a big model's full native context (128k–256k) can allocate more KV cache than unified memory holds and kill the whole server mid-generation. --max-context auto (the default) reads the model's KV geometry and free RAM and engages a cap only when the full native context wouldn't fit, otherwise it's a no-op, so a machine with enough RAM sees no change. Once a prompt exceeds the cap, the KV window rotates (mlx-lm's RotatingKVCache) instead of crashing; batching, prompt-cache reuse, and KV quantization all keep working. Pass an integer for a hard token cap, or off to disable. Sliding-window models (Gemma-4, Qwen3-Next) manage their own KV and are untouched. Resilient downloads, Xet → HTTPS failover HuggingFace's Xet high-performance transfer is fast when it works but can stall partway through a large shard on corporate proxies, TLS-inspection middleboxes, or flaky links. OptiQ retries the normal path, then, as a last resort, forces the plain HTTPS path (HF_HUB_DISABLE_XET=1) for one final attempt, resuming from cache, so completed shards are never re-fetched. On by default for every optiq serve / optiq convert / adapter download; nothing to configure. Production tips Bind to 127.0.0.1 for local-only use, or behind a reverse proxy. Don't expose the raw 0.0.0.0 binding to the public internet; there's no auth. Cap concurrency with --max-concurrent (default 8). Concurrent requests are batched, but each in-flight request holds its own KV cache, so mlx-lm's datacenter default of 32 can OOM-crash unified memory under a burst. OptiQ injects a Mac-safe cap; lower it further for big models / long contexts (excess requests queue), raise it if you have RAM to spare. Cap context with --max-context (default auto). Keeps a runaway-length prompt from OOM-crashing the server; a no-op unless the model's full native context wouldn't fit RAM. Reclaim idle RAM with --idle-timeout. On a shared machine, free the model when the server goes quiet; it reloads on the next request. Tune --max-tokens conservatively. Each in-flight request keeps a KV cache resident; long contexts dominate memory. Pre-load adapters. Loading a new adapter mid-flight stalls all in-flight requests. Mount everything you need at startup. --- ## Vision (image input) Source: https://mlx-optiq.com/docs/vision Workflow · vision Vision (image input) mlx-optiq answers image and text prompts on the Gemma-4 and Qwen3.5 / Qwen3.6 families. The language tower is still OptiQ mixed-precision quantized and decoded by mlx-lm; the vision tower is vendored into mlx-optiq (no mlx-vlm runtime dependency) and kept at bf16 in a sidecar that rides alongside the quantized weights. At a glance Vision support is on the Gemma-4 family (e2b, e4b, 12B, 26B-A4B, 31B) and the Qwen3.5 / Qwen3.6 family. The vision/audio towers stay at bf16; only the language tower is quantized. Audio (speech) input is not supported. Also video Mage-VL is a separate vision family with its own Mage-ViT encoder. It reads still images and video, and it is the first video model in the lineup. It uses the same bf16-sidecar approach described here. See the Mage-VL guide. One artifact, two ways to load it OptiQ stores the vision and audio towers, at bf16, in a sidecar file named optiq_vision.safetensors next to the quantized language shards. mlx-lm selects its weights with glob("model*.safetensors"), so it never matches the sidecar. The result is a single published repo that loads two ways: Loader Reads You get stock mlx-lm model*.safetensors Text-only model (sidecar ignored) OptiQ model*.safetensors + optiq_vision.safetensors Full image + text There is no separate vision build. Vision stays at bf16 because 4-bit vision degrades OCR and fine detail; the language tower, where almost all of the size lives, is still fully quantized. Supported models Pre-built quants take image input today, across three vision architectures. Every preprocessing step and vision tower is vendored from mlx-vlm and reproduces its outputs to a maximum absolute difference of zero. Family Models with image input Vision tower Gemma-4 e2b, e4b, 12B, 26B-A4B, 31B SigLIP tower (e2b/e4b/26B/31B); encoder-free unified backbone (12B) Qwen3.5 0.8B, 2B, 4B, 9B, 27B, 35B-A3B Qwen3-VL encoder Qwen3.6 27B, 35B-A3B Qwen3-VL encoder Nemotron 3 and MiniCPM5 are text-only and carry no sidecar. Audio input is not supported on any model. Serving images When a model carries the sidecar, optiq serve turns on image support automatically. Send an OpenAI-style image_url content part (a data URL or an http(s) URL): terminalbash optiq serve --model mlx-community/gemma-4-e2b-it-OptiQ-4bit curl http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":[ {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}, {"type":"text","text":"What is in this image?"}]}]}' Text-only requests are unchanged: the vision path only runs when a request actually carries an image, so MTP speculation, mounted LoRA adapters, KV-cache quantization, and plain text generation all behave exactly as without it. In the Lab The Lab's Chat tab takes image uploads directly. Run optiq lab --model , open Chat, click attach, drop in a picture, and ask a question. gemma-4-e2b at 4-bit, reading an uploaded image in the Lab. Python API The engine takes images= (paths, URLs, data URLs, or PIL images) or a full messages= list with image_url parts: pythonpy from mlx_lm import load from optiq.runtime.engine import OptiqEngine model, tok = load("mlx-community/gemma-4-e2b-it-OptiQ-4bit") eng = OptiqEngine.from_loaded(model, tok, "mlx-community/gemma-4-e2b-it-OptiQ-4bit") st = eng.generate("What is in this image?", images=["cat.jpg"], max_tokens=128) print(st.text) Adding the sidecar to a quant If you have an existing OptiQ language quant and the bf16 base it came from, attach a vision sidecar with one call. It extracts the bf16 vision and audio towers, writes optiq_vision.safetensors into the quant directory, and restores the multimodal config keys: pythonpy from optiq.vlm import build_vision_sidecar build_vision_sidecar( base="google/gemma-4-e2b-it", # bf16 base with the towers quant_dir="./gemma-4-e2b-it-OptiQ-4bit", # existing OptiQ language quant ) How it works The vision front-end preprocesses the pixels, runs the vendored vision tower, projects the result into the language model's hidden space, and scatters those soft tokens into the text-embedding sequence at the image-placeholder positions. The merged embeddings go to mlx-lm's language model through its input_embeddings hook, and decode proceeds with the same quantized weights, KV cache, and sampler as text. mlx-optiq resolves the right front-end per model_type: gemma4 (SigLIP tower), gemma4_unified (the encoder-free 12B), and qwen3_5 (Qwen3-VL tower). The vendoring is validated against mlx-vlm tensor for tensor: feeding mlx-vlm's own pixel values through mlx-optiq's preprocessing and vision tower reproduces its outputs to a maximum absolute difference of zero, on every architecture. Each tower needs the backbone to treat its visual tokens correctly. For the SigLIP towers, the tokens are self-contained, so the standard causal decode is enough (one detail: gemma4_text rescales the incoming embeddings by embed_scale, so the vision features are pre-divided to compensate). The unified Gemma-4 12B has no separate tower and was trained to attend bidirectionally over the image span, so OptiQ makes that span bidirectional with a one-shot mask wrapper (text and decode stay causal). Qwen's tower already carries 2D rotary positions internally, so its visual tokens arrive position-aware and need nothing special from the backbone. See also The release write-up: mlx-optiq can see. Model lists and sampling defaults: the Gemma-4 and Qwen3.5 family guides. --- ## Speculative decoding on Apple Silicon Source: https://mlx-optiq.com/docs/speculative Workflow · Speculative decoding Speculative decoding on Apple Silicon A language model writes one token per forward pass. Speculative decoding guesses several tokens ahead, then checks all of them in a single pass of the full model. Every guess the model agrees with is a token it did not have to generate on its own; the first guess it disagrees with is thrown away and the model’s own token is used. The output stays the model’s output. What changes is how many passes it takes. OptiQ has three places to get the guesses from. They share the same verify loop and differ in what does the guessing, which decides where each one pays off. Method Guesses come from Models Flag Measured N-gram lookup Text already in the conversation Any model --ngram-draft 8 1.3-1.5x on coding-agent turns MTP head A small prediction head bundled in the quant Qwen3.5 / Qwen3.6, 4B and up --mtp 1.20-1.40x Assistant drafter A separate small model trained to match the target Gemma-4 E4B --drafter 1.18x geomean One method runs per server. Use n-gram lookup when the work repeats its context: coding agents reading and editing files, tool calls that name paths from earlier turns, answers that quote code. It needs nothing extra and works on models the other two don’t cover. On the families that have one, the MTP head or the drafter also speeds up new prose, where there is little to copy. N-gram lookup: any model Agents repeat themselves. A file edit copies most of the file it just read, a tool call names paths already in the context, a final answer quotes the code it wrote. --ngram-draft turns that into speed: when the last few tokens match something earlier in the conversation, the server drafts the tokens that followed it and checks them all in one forward pass. Accepted drafts cost a fraction of a normal step each. A rejected draft is discarded, and the model’s own token is used instead. terminalbash $ optiq serve --model mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit-REAP-19B \ --ngram-draft 8 --port 8080 There is no draft model to download or keep in memory, so it works with every model the server loads: dense, MoE and hybrid linear-attention alike. The draft length is not fixed. The server times one verification pass per model when it first loads (a few seconds), then picks how many tokens to draft on each step from how often recent drafts were accepted. It drafts long while text is being copied and stops drafting during new prose, where a wrong guess would only add work. On Qwen3.5 and Qwen3.6 hybrids, a rejected draft rolls the recurrent state back to the last accepted token directly, with no second pass. Measured by replaying real OptiQ Code and OptiQ Lab conversations, where the model regenerates a turn it once wrote (methodology below): Workload Standard attention Qwen3.5/3.6 hybrids Coding agent turns (120 conversations) 1.47× 1.45× Chat and research turns (120 conversations) 1.36× 1.34× Live on an M3 Max with Qwen3.6-35B-A3B-REAP-19B, coding turns decoded at 84.8 tok/s against 64.7 plain (1.31×), and rewriting a file went from 59 to 105 tok/s. Short replies and prose that copies nothing gain little. The gate keeps those close to plain speed. Every emitted token is the model’s own choice at that position. In float32 the output is identical to plain decoding. On quantized weights, checking several tokens in one pass uses different kernels than one token at a time, so a near-tie between two tokens can occasionally resolve the other way. Plain decoding does the same when a prompt is prefilled in different chunk sizes. --ngram-draft N: most tokens to draft per step. 8 is a good value; 0 (default) turns it off. --no-ngram-gate: always draft N tokens when a match is found. For measurement; the gate is faster. --ngram-min N: shortest match that triggers a draft. Default 3. MTP head: Qwen3.5 and Qwen3.6 Qwen3.5 and Qwen3.6 ship an extra small prediction head in their weights, which the literature calls MTP, for Multi-Token Prediction. OptiQ quants keep it, and OptiQ uses it as the guesser. The base model still produces every output token; on the steps where the head’s guess matches, that token comes for free. terminalbash $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp The server reads the model’s recommended sampling settings from generation_config.json and applies them unless you pass your own via --temp, --top-p and --top-k. We report greedy here because it isolates the speedup measurement from sampling noise and matches how unsloth and the upstream MTP literature publish their headlines. Model Base tok/s MTP tok/s Speedup Acceptance Qwen3.5-4B 29.2 35.0 1.20x 67% Qwen3.5-9B 19.5 25.8 1.32x 66% Qwen3.6-27B 6.0 8.4 1.40x 72% With Qwen's recommended production sampler (temp=1.0, top_p=0.95, top_k=20), the speedup is smaller but still positive everywhere: Model Base tok/s MTP tok/s Speedup Acceptance Qwen3.5-4B 28.7 31.2 1.09x 56% Qwen3.5-9B 19.1 22.3 1.17x 56% Qwen3.6-27B 6.2 8.0 1.30x 56% Acceptance is the literature definition, drafts_accepted / drafts_attempted, read straight from the engine. Where MTP does not pay off For Qwen3.5 0.8B the base model is already at 130 tokens per second, and the speculation overhead per cycle eats more than the head can give back. We measured a regression to about 0.7x. The 2B model lands close to break-even. Skip MTP at these sizes. 4B and up consistently win. About depth The default is depth 1, meaning one drafted token per cycle. Depth 2 and above does not help on Apple Silicon. The reason is that Metal's K-token verify forward scales close to linearly with K, while on CUDA the same forward is nearly free due to spare matmul throughput on Tensor Cores. We measured depth 2 through 4 and depth 1 wins every single configuration. We also tried HuggingFace's adaptive depth heuristic (raise K after a clean cycle, lower it on partial accept). It lost 4 to 17 percent depending on the model and sampler. So we ship a fixed depth 1. Assistant drafter: Gemma-4 Gemma-4 has no MTP head. Google instead publishes a separate small drafter, the -assistant variant: a 4-layer model that shares K and V with two layers of the target. OptiQ loads it next to the target and runs the same verify loop. The plumbing differs (typed K/V sharing, target-hidden conditioning, two kinds of cache layer), but from the serving side it is just another source of guesses. This path is greedy only for now. terminalbash $ optiq serve --model mlx-community/gemma-4-E4B-it-4bit \ --drafter mlx-community/gemma-4-E4B-it-assistant-bf16 Greedy, γ=1, 200-token generation, five prompt categories, median of three runs each on M4 Pro 24 GB: Greedy, γ=1, 200-token generation, five prompt categories, median of three runs each on M4 Pro 24 GB: Prompt type Base tok/s Spec tok/s Speedup Acceptance math 29.97 38.66 1.29x 37.5% code 29.68 37.09 1.25x 34.0% prose 31.19 36.65 1.18x 30.3% dialogue 31.66 35.17 1.11x 29.5% reasoning 30.43 32.14 1.06x 25.5% Geomean 1.18x 31.4% Acceptance is lower than Qwen MTP for two reasons. First, the Gemma drafter is a separately trained Q-only model that has to predict the target's distribution from a few shared cache layers, not a head that was co-trained on the target's loss. Second, mlx-lm's multi-token verify forward is not bit-identical to the equivalent sequence of single-token forwards due to bf16 attention precision; the largest diff we measured was 0.68 in logit magnitude at the second of two positions. This means a draft the target would have accepted in a single-token verify can be rejected in a multi-token verify. Greedy outputs still match a baseline greedy run for a long prefix (200 tokens identical on our reasoning prompt), then drift on longer sequences. γ-sweep on the math prompt above (200 tokens, median of three): γ Spec tok/s Speedup baseline 29.09 1.00x 1 39.02 1.34x 2 36.91 1.27x 3 28.04 0.96x 4 23.83 0.82x 5 20.39 0.70x γ=1 is optimal on Metal for the same reason it is optimal for Qwen MTP: the K-token verify forward scales near-linearly with K, while acceptance stays roughly constant per draft slot. The math is in Getting MTP to actually work on Apple Silicon, “What about depth 2 or higher”. γ>1 is implemented and lossless within the same bf16 precision bound as γ=1, but ships defaulted to γ=1. Serving and the Lab All three work behind every serving endpoint (OpenAI, Anthropic, Responses), so clients need no knowledge that speculation is happening. Chat templates work the same, and enable_thinking=False still applies. Weight quantization is unaffected: the MTP head ships as a 4-bit projection with a bf16 final layer, matching the host model. The flags are mutually exclusive; pick one per loaded model. In OptiQ Lab, the Server page has a checkbox for the Qwen MTP path and a “Spec drafter” picker for the Gemma path; n-gram lookup is a server flag. With n-gram lookup on, requests are decoded one at a time, which matches how one person runs a local model. Compatibility Family Spec backend Status Every family N-gram lookup Yes. Checked on 14 models across 7 architectures, including dense, MoE, and the Qwen3.5/3.6 and LFM2.5 hybrids. Qwen3.5 Bundled MTP head Yes for 4B and up. 0.8B and 2B regress; skip. Qwen3.6 Bundled MTP head Yes for 27B. Gemma-4 E4B External -assistant drafter Yes. Greedy, γ configurable; γ=1 default and optimal on Metal. 1.18x geomean on M4 Pro 24 GB. Gemma-4 E2B / 26B / 31B No drafter published Google has not released matching -assistant weights for the other Gemma-4 sizes. Methodology Each Qwen measurement runs in its own subprocess for clean memory state. The prompt is a 166-token Python question. We generate 512 tokens. Decode tokens per second comes from mlx-lm's GenerationResponse.generation_tps, which is measured after prefill so it captures only the decoding phase. For Gemma-4 we run five chat-templated prompts (one each from math, code, prose, dialogue, reasoning), 200-token generations, median of three runs per prompt, all in a single Python process. Decode tokens per second is wall-clock n_tokens / elapsed for both baseline and spec, with a warm-up generation discarded. Qwen acceptance comes from the engine’s drafts_accepted and drafts_attempted counters. Gemma acceptance comes from the drafter loop’s equivalent counters. Both follow the standard literature definition, so the comparisons are apples to apples with unsloth and llama.cpp numbers. Hardware for MTP and the drafter: Apple M4 Pro, 24 GB unified memory, 19.1 GB Apple-recommended working set. For n-gram lookup, 120 OptiQ Code and 120 OptiQ Lab conversations are cut before an assistant turn and that turn is replayed through the real drafting and gate code. The cost of each verification pass comes from timings measured on Qwen3.6-35B-A3B-REAP-19B, with and without the hybrid rollback, so the replay is free of machine-load noise. The live check ran the same model on an M3 Max, 36 GB, over 12 conversations, plain and speculative interleaved from identical prefilled caches, best of two runs each. For the longer story of how we got MTP working correctly on this stack, see the blog post on Apple Silicon MTP. For the Gemma -assistant path, see Gemma-4 spec decoding on Apple Silicon. --- ## Distributed inference across Macs Source: https://mlx-optiq.com/docs/cluster Workflow · cluster Distributed inference across Macs Run a model that does not fit any single machine by sharding its layers across two or more Macs over Thunderbolt. One OpenAI-compatible endpoint, the same serving code path a single Mac uses. What this is for Pipeline parallelism adds capacity but does not make a single request faster. Rank 0 runs its layers, hands the activation to rank 1, and waits. There is no overlap within a single request, so every token pays a network hop and you gain nothing by splitting a model that already fits. Read this first A model that fits on one Mac is slower when sharded. We measure Qwen3.5-0.8B at 5.0 ms/token on one Mac and 13.4 ms/token across two, 2.7× worse. Reach for a cluster when the alternative is a degraded path: streaming experts off the SSD, a smaller quant, or not running the model at all. Where it does win: Qwen3.5-122B-A10B-OptiQ-2bit is 42.8 GiB of weights. On a single 36 GB Mac it only runs by streaming MoE experts from the SSD, at 4.9 tok/s. Split across a 36 GB M3 Max and a 24 GB M4 it is fully resident and decodes at 20.5 tok/s, 4.2× faster. The gain is against the SSD-streaming path, not against a model that already fits in RAM. Setup You need SSH between the Macs, a Thunderbolt cable, and the model cached on each node (or the peers download it on first serve). Connect the Macs and macOS creates a bridge0 interface with link-local addresses; OptiQ finds them itself. on every machinebash pip install mlx-optiq optiq cluster up # advertise this Mac over Bonjour on the machine you will drivebash optiq cluster peers # who is reachable, on what link Link-local IPs change on reboot. Discovery re-resolves them every time; nothing is pinned to a file. Install at the same relative path on every Mac The ring is launched by ssh-ing into each peer and running cd && //bin/python3 from that peer's home. Usernames differ between machines, so the path is resolved relative to each home. Put the project (and its virtualenv) in the same place on both, for example ~/optiq/.venv on each. Or point at it explicitly with --cwd / OPTIQ_CLUSTER_CWD. By default OptiQ uses the virtualenv it is running inside. optiq cluster serve measures the real round-trip to each peer and refuses a slow link by default. Pipeline inference over Wi-Fi is slower than one Mac, because every token pays the latency, so it refuses by default rather than run slowly without warning. Pass --link any if you want it anyway. Thunderbolt is ~0.6 ms; Wi-Fi is ~85 ms. From the Lab Settings → Cluster shows the discovered Macs on the ring with each node's free memory, swap pressure, and how many layers it would take. The number it reports is the memory the ring can spare right now, not each Mac's total RAM. Discovered peers, free memory per node, and the combined capacity. Enter a model and press Start serving. A 40 GB model takes time to page in, so chat does not switch to the cluster until the endpoint answers. You get a loading state rather than a connection error. Once every rank has materialized its shard, the cluster becomes the Lab's inference backend. Chat then talks to the ring exactly as it talks to a single Mac: streaming, tool calls, structured output, LoRA adapters. Below, the 122B writing a flight simulator, its HTML running in the Lab's sandboxed artifact panel. Qwen3.5-122B-A10B-OptiQ-2bit, resident across two Macs, at ~20 tok/s. From the CLI serve a sharded modelbash optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --port 8100 outputtext [rank 0] materialized 8/30 layers | resident 9.6 GiB [rank 0] materialized 16/30 layers | resident 15.7 GiB [rank 0] materialized 24/30 layers | resident 21.7 GiB [rank 1/2] layers 30-47 (18/48) | loaded 7.4s | resident 16.6 GiB | cap 17.3 GiB (free 21.4, GPU limit 22.0) [rank 0/2] layers 0-29 (30/48) | loaded 7.8s | resident 26.2 GiB | cap 26.6 GiB (free 27.0, GPU limit 30.0) [cluster] OpenAI endpoint: http://127.0.0.1:8100/v1/chat/completions [cluster] 2 nodes, model=mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit. Ctrl+C to stop. 42.8 GiB resident across two Macs, loaded in 26 s. The endpoint is ordinary OpenAI; point any client at it. Rank 0 owns the embedding, the final norm, and the LM head, and drives the ring. The last rank returns a hidden state rather than logits, so what crosses the wire each token is ~12 KB instead of ~1 MB of vocabulary. The memory rules Layers are held resident, so the model must fit the memory the cluster can spare at that moment. Boundaries follow each node's free memory and the real per-layer byte counts. The Mac you are working on takes fewer layers than an idle one, and a MoE's heavier late blocks are counted as they actually are. Each rank keeps a run-reserve free for activations, the KV cache, and MLX's buffer pool. Before a single weight is evaluated, OptiQ prices the model against what the ring can spare and refuses to load rather than swap a machine to death: preflight refusaltext mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit does not fit the cluster's *currently free* memory. rank 0: needs 24.4 GiB | can spare 25.0 GiB (free 25.9 GiB - 0.9 GiB to run) rank 1: needs 18.4 GiB | can spare 18.3 GiB (free 21.5 GiB - 3.2 GiB to run) <-- does not fit Weights 42.8 GiB vs 43.3 GiB spare across 2 node(s). Fix: free memory on the node(s) above (quit apps; a Mac whose swap is full needs a reboot), use a smaller quant, add a Mac, or lower the run-reserve with OPTIQ_CLUSTER_HEADROOM_GB= (risks swapping). Do not tune this to force a fit OPTIQ_CLUSTER_HEADROOM_GB exists so you can raise the reserve, not shrink it. Dropping the reserve to 0.8 GiB per node let a 42.8 GiB model load onto a pair that could spare only 42.3 GiB; the smaller Mac then swapped 2.1 GB and a single generation ran for 22 minutes without finishing. Refusing to load is safe; forcing a fit can hang the machine. The smaller Mac has a residency ceiling There is a second limit: the residency ceiling. Past roughly 73% of its RAM in resident weights, a Mac's GPU throughput collapses. System memory is free, nothing swaps, and no error is raised. On a 24 GB M4 the cliff sits near 17.5 GiB: M4 (24 GB) · resident shard vs throughput Resident weights iogpu.wired_limit_mb Throughput 16.6 GiB 20480 23.3 tok/s 17.3 GiB 20480 15.1 tok/s 18.1 GiB 20480 0.1 tok/s 18.1 GiB 22528 0.1 tok/s 19.6 GiB 22528 0.05 tok/s (swapping) Raising iogpu.wired_limit_mb does not move it. 18.1 GiB reads 0.1 tok/s at both a 20 and a 22 GiB limit. If a ring loads but crawls, the smallest node is holding too many layers. Give it a larger reserve so it takes fewer. push layers off the smaller nodebash # per-rank reserve, in the ring's rank order OPTIQ_CLUSTER_HEADROOM_GB="0.4,4.1" optiq cluster serve --model rank 0 (36 GB) takes 30 layers / 26.2 GiB; rank 1 (24 GB) takes 18 / 16.6 GiB. 20.5 tok/s. What to expect Qwen3.5-122B-A10B-OptiQ-2bit · 42.8 GiB · M3 Max 36 GB + M4 24 GB Path Resident Steady throughput Two Macs, sharded 42.8 GiB across 2 20.5 tok/s One Mac, SSD expert streaming 9.97 GiB 4.9 tok/s One Mac, fully resident does not fit n/a Load takes 26 s end to end; weights are memory-mapped and materialized layer by layer. Throughput is flat in context. The gap between the two paths does not close as the conversation grows. Benchmarking note The first forward compiles Metal kernels, which takes 13–79 s depending on memory pressure. A single-shot throughput measurement on a freshly loaded ring reads ~0.1 tok/s on a perfectly healthy cluster. Always warm up with a short request and time the second one. Troubleshooting It refuses to load If it refuses to load, the estimate is real. Quit apps on the node flagged <-- does not fit. A Mac whose swap is full stays degraded until it reboots. macOS never shrinks the swap file. The Lab's cluster page shows swap per node and will tell you which machine to restart. It loads, then crawls The smallest node is over its residency ceiling. Raise its reserve so it takes fewer layers. Do not raise iogpu.wired_limit_mb; it will not help. It refuses the link You are on Wi-Fi. Connect the Thunderbolt cable and check ifconfig bridge0 reports status: active on both machines. A peer cannot find python zsh: no such file or directory: .venv/bin/python3 means the peer does not have mlx-optiq where this Mac does. The path is relative to each machine's home directory. Install it at the same relative location on every node, or pass --cwd. The ranks disagree about the model Every node needs the same complete checkout in its Hugging Face cache. A partially downloaded copy on one peer makes the ranks compute different shard boundaries and the ring hangs at startup. --- ## Frequently asked questions Source: https://mlx-optiq.com/docs/faq Documentation Frequently asked questions Common questions about running quantized LLMs on Apple Silicon with mlx-optiq. What is mlx-optiq? mlx-optiq is an MLX-native optimizing compiler that quantizes LLMs for Apple Silicon. Instead of giving every layer the same bit-width, it measures each layer's sensitivity on calibration data and allocates higher precision to layers that need it, lower precision to layers that don't. The result is a mixed-precision quant that runs in stock mlx-lm with no custom runtime, typically at higher accuracy than uniform 4-bit at the same average bits-per-weight. How is it different from uniform 4-bit quantization? Uniform 4-bit treats every layer the same. mlx-optiq measures per-layer KL sensitivity on a calibration mix, then runs a greedy knapsack to assign bit-widths (4 / 8 bits by default) such that the total budget hits a target average BPW. Sensitive layers stay at 8-bit, robust layers go to 4-bit. On the small end the recovery is dramatic: gemma-4-e4b drops to 23.5% GSM8K at uniform 4-bit; mlx-optiq lifts it to 55.5% at the same disk size. Full methodology in the research post. How do I install mlx-optiq? pip install mlx-optiq. Requires Python 3.11+. Quantizing and running models locally need Apple Silicon (M1 or later); optiq code and optiq lab also run on Linux and Windows when pointed at an OpenAI-compatible base_url. One install gives you convert, serve, eval, the Lab web UI and the coding agent. terminalbash $ pip install mlx-optiq # everything $ pip install 'mlx-optiq[dev]' # plus tests + benchmark datasets Full install guide. How do I quantize Qwen3.5 on a Mac? terminalbash $ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8 The CLI downloads bf16 weights, runs per-layer sensitivity, allocates bits to hit the 5.0 BPW target, converts via mlx_lm.convert, and writes the artifact to ./optiq_output. Add --reference uniform_4bit if you're memory-constrained; that path streams bf16 layers from disk instead of holding them in RAM. Pre-built Qwen3.5 quants from 0.8B to 35B-A3B are on mlx-community. What is mixed-precision KV cache? The KV cache is the activations stored across the attention layers during long-context generation. Uniform 4-bit KV is catastrophic because layer 0's KV is roughly 56x more sensitive than the average layer; quantizing it the same as every other layer wrecks long-context accuracy. mlx-optiq runs a separate per-layer sensitivity pass on the KV cache and assigns higher bit-widths to the few layers that need it. The result: 4-bit average KV that matches fp16 quality on hash-hop and reaches 34% lower peak memory than fp16 at 32k context. Full methodology and benchmarks. Does mlx-optiq work with Claude Code, Codex, or other coding agents? Yes. optiq serve starts a local server that speaks both the OpenAI Chat Completions API and Anthropic's Messages API on the same port. Claude Code and OpenClaw point at /v1/messages, Codex and OpenCode at /v1/chat/completions, all using a fake sk-optiq-local API key. Copy-paste configs at Integrations. Can I fine-tune a quantized model with LoRA? Yes. The trainer is MLX-native (no PyTorch detour) and supports sensitivity-aware rank scaling: layers mlx-optiq kept at 8 bits get proportionally higher LoRA rank than layers it quantized to 4 bits, so adapter capacity matches the bit budget. terminalbash $ optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit \ --data ./jsonl_dir --rank 8 --rank-scaling by_bits Output is a standard mlx-lm adapter directory you can serve via optiq serve --adapter. Fine-tuning docs. What is MTP speculative decoding? Multi-Token Prediction is a speculative-decoding scheme that uses the model's own auxiliary head as the draft model. Greedy generation on Apple Silicon: 1.20x on Qwen3.5-4B, 1.32x on 9B, 1.40x on 27B. Enabled with optiq serve --mtp on a model whose checkpoint includes mtp.safetensors. Detailed methodology. Does mlx-optiq run on Linux or NVIDIA GPUs? No. mlx-optiq targets Apple Silicon (M1 and later) and uses Apple's MLX runtime. For NVIDIA GPUs you would want bitsandbytes, GPTQ, or AWQ paired with vLLM or Transformers. Where can I find pre-built quants? On Hugging Face under mlx-community, with the OptiQ-4bit suffix. OptiQ-4bit quants ship across these families: NVIDIA Nemotron 3 Nano (4B + 30B-A3B), MiniCPM5-1B, Qwen3.5 (0.8B, 2B, 4B, 9B, 27B, 35B-A3B), Qwen3.6 (27B, 35B-A3B), Gemma-4 (e2b, e4b, 12B, 26B-A4B, 31B). The full catalog with size and accuracy numbers is at /models. Which inference tools can load OptiQ quants? mlx-optiq and mlx-lm are the two supported paths, and together they cover what matters. In mlx-optiq everything runs: text, image input on the VLM families, and MTP speculative decoding. In stock mlx-lm the text path loads and generates directly with mlx_lm.load("mlx-community/-OptiQ-4bit"), since mlx-lm is the library OptiQ quantizes with. Other Mac front-ends such as mlx-vlm, LM Studio, and oMLX load MLX weights through their own stack and wrap mlx-vlm for image models, so whether an OptiQ quant runs there depends on that stack rather than on our files. Text usually works. Image input and the newer architectures can lag, because mlx-vlm's per-layer mixed-precision handling and its Gemma-4 shared-KV support have to catch up upstream. The language weights load and generate correctly in mlx-lm and mlx-optiq everywhere, so a failure in another tool is a gap in that loader, not a broken quant. mlx-optiq is the path that runs text, vision, and MTP today. Can I run code execution and web search from a local chat? Yes. OptiQ Lab ships a chat surface with three tools the model can call: web_search (DuckDuckGo, no API key), python (AST-checked sandbox with PNG-inline matplotlib output), and terminal (bash one-liner with token-aware command blocking). The tool-call orchestrator includes a healer for six malformed shapes that quantized models commonly emit. Full description. --- ## Troubleshooting Source: https://mlx-optiq.com/docs/troubleshooting Documentation Troubleshooting Error messages you can hit running quantized models on Apple Silicon, what causes each one, and what to do about it. Several are fixed in a specific release, so the answer is often to upgrade. pip install -U mlx-optiq resolves most of what follows. The current release is listed on the changelog. There is no Stream(gpu, N) in current thread Raised while serving, usually on the first request, and it can take the whole server process down rather than failing the one request. MLX streams belong to the thread that created them. An array that has been built but not yet evaluated carries the stream of its creating thread, and evaluating it on another thread raises this. optiq serve loads the model on the main thread and generates on mlx-lm's own thread, so any array left unevaluated after loading will trigger it. On Gemma-4 the array in question was the RoPE frequency table on the global-attention layers. Fixed in 0.4.23. If you see it on a later version, it is a different array and worth reporting. scaled_dot_product_attention(): incompatible function arguments … array, tuple, tuple Seen on Gemma-4 with quantized KV, typically once context grows past roughly ten thousand tokens, and sometimes as [broadcast_shapes] Shapes (1,1,1,0) and (1,16,1,10764) cannot be broadcast. Gemma-4 shares KV across layers. A consumer layer receives the producing layer's (packed, scales, biases) tuple while holding no quantized cache of its own, and upstream mlx-lm decides which kernel to use by checking the cache rather than the tensors. The tuple then reaches the fp16 kernel, which cannot accept it. Fixed in 0.4.25. Verified past fourteen thousand tokens on gemma-4-26B-A4B-it-OptiQ-4bit with a mixed per-layer KV config. [METAL] Command buffer execution failed: Insufficient Memory The GPU refused an allocation. The exception is thrown on a Metal callback thread where there is no Python frame to catch it, so the server exits rather than returning an error for that request. The usual cause is a KV window sized against free system RAM when the limit that actually applies is Metal's recommended working set, which is shared by every process using the GPU. Two servers on one machine can each size a window that fits, and together exceed it. From 0.4.25 the automatic context cap is bounded by both limits. If you are running more than one model at a time, set --max-context explicitly rather than relying on auto. NotImplementedError from RotatingKVCache.to_quantized Hit when quantizing the KV cache of a model with sliding-window attention, which includes every Gemma-4. Upstream mlx-lm raises this in v0.1.2 and earlier. OptiQ installs a rotating cache that supports quantization and uses it automatically when optiq serve or optiq kv-cache needs one. Reaching the upstream error means the model is being loaded through stock mlx-lm rather than through OptiQ. Model type not supported, or No module named 'mlx_lm.models.' Reported for gemma4_unified, diffusion_gemma, mage_vl, mistral4, llada2 and dhara_ar, among others. These architectures are not in stock mlx-lm. OptiQ registers them, so loading through optiq serve, optiq lab or optiq code works. A bare mlx_lm.load() in your own script will not, unless you import OptiQ first so the registration runs. Tool calls are ignored when serving Devstral or another Mistral-family model The model appears to answer normally but never actually calls a tool, and the server log shows Failed to parse tool call … Could not parse tool call from: read_file[ARGS]{…. Mistral models emit tool calls as [TOOL_CALLS]name[ARGS]{json}. mlx-lm's server splits the output on that marker and passes on only what follows, and OptiQ's parser required the marker to be present, so every served call was discarded. A second problem meant only the first of several chained calls ran. The dropped calls are fixed in 0.4.24 and chained calls in 0.4.25. Both affect agentic use specifically; ordinary chat was unaffected. A LoRA adapter that trains successfully but contains NaN Training reports completion and the adapter mounts, but the model produces nothing usable afterwards. A batch that contributes no unmasked target tokens makes the loss compute zero divided by zero. That happens more often than it sounds: with prompt masking on, any example whose response falls past --max-seq-length masks out completely. The resulting NaN spreads to every LoRA factor through the gradient-norm clip. Fixed in 0.4.25, which also refuses to report an adapter as trained if any tensor in the saved file is non-finite. On an earlier version, raising --max-seq-length or shortening the training examples avoids it. --stream-experts appears to do nothing The flag is accepted, the model loads, and memory use is unchanged. On Gemma-4 MoE models the expert tensors are stored under a path OptiQ's matcher did not recognise, so it found nothing to stream and loaded the model resident without reporting a problem. Fixed in 0.4.23. Expert streaming is described on the serve page, and the background is in Quantizing a mixture-of-experts model on MLX. The mlx-vlm batched vision path does not support KV cache quantization yet A genuine limitation rather than a bug. Quantized KV and batched image input cannot currently be combined. Serve vision models without --kv-bits or --kv-config, or send image requests one at a time. OptiQ's vision path already routes image requests off the batch path for this reason; the message means something is reaching mlx-vlm's batched path directly. Choosing --max-kv-size, --kv-bits and the wired limit These interact and it is easy to set one against another. --max-kv-size bounds how many tokens of KV are kept, --kv-bits sets the precision the cache is stored at, and the wired limit governs how much memory Metal will hold resident. Quantizing the cache to 4 bits roughly quarters its size, which usually matters more than capping the window. --max-context auto sizes the window from available memory and the device's working set, and is the right default unless you are running several models at once. Full descriptions are on the serve page. Something else If you hit an error that is not here, the changelog lists what each release fixed, and reporting it on the model card discussion for the quant you are running is the fastest way to get it looked at. --- ## CLI reference Source: https://mlx-optiq.com/docs/cli Reference · CLI CLI reference The optiq command-line interface drives every mlx-optiq workflow on Apple Silicon: quantization, KV-cache config, LoRA fine-tuning, OpenAI- and Anthropic-compatible serving, evaluation, and latency benchmarking. Twelve top-level commands cover the full workflow: optiq convert: quantize an HF model with mixed-precision sensitivity optiq kv-cache: measure per-layer KV sensitivity, write a serving config optiq lora train / info: sensitivity-aware LoRA training optiq serve: dual-protocol inference server (OpenAI + Anthropic) optiq eval: smoketest + full benchmark suite (MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop) optiq benchmark: perplexity + throughput on a converted model optiq latency: Apple Silicon roofline throughput predictor optiq prune-experts: remove low-value routed experts from a quantized MoE optiq lab: local web UI for quantize / fine-tune / dataset / chat optiq code: terminal coding agent driven by your local model optiq cluster: serve one model across two Macs over Thunderbolt optiq config: show every setting and where its value came from optiq prune-experts Remove low-value routed experts from a quantized MoE checkpoint. Implements REAP (Cerebras Research, ICLR 2026) in the quantized domain: it runs directly on a quantized model, needs no BF16 parent, and copies retained experts bit-for-bit rather than dequantizing them. prune.shbash $ optiq prune-experts MODEL --retain 128 --output ./pruned Required MODEL: HF repo ID or local path to an MLX-format MoE checkpoint. OptiQ quants, stock mlx-community quants and unquantized MLX conversions all work. A raw Hugging Face checkpoint stores its experts unfused and is rejected rather than silently skipped. --output: directory for the pruned model. The source is never modified. Options --retain N: experts to keep per layer. Must be at least the model’s top-k. --target-retention F: keep a fraction instead, e.g. 0.5. --profile PATH: reuse a previous profile instead of re-measuring. --n-samples N: calibration samples to profile on (default 8). --calibration-mix: which mix to rank experts on. Pruning is far more sensitive to this than quantization — an expert the mix never exercises is an expert that gets deleted. --profile-only: write the profile and stop. --keep-mtp / --drop-mtp: carry the MTP sidecar (default: keep). What it costs Experts are ranked by router weight times expert output norm, measured on calibration data; the lowest-ranked are removed and the router is sliced to match. Active parameters per token do not change, so the model gets smaller without getting slower. How that score is aggregated depends on the router. Where the router softmaxes over every expert, as Qwen’s does, the routing weight is an expert’s share of the whole distribution and already expresses the router’s preference, so the score is a mean over the times the expert fired — a rarely-chosen but strong expert outranks a common weak one. Where the router softmaxes over only the selected top-k, as Gemma-4’s does, every chosen expert gets roughly 1/k whatever the router thought of it, and the preference survives only in how often it was chosen; there the score is weighted by selection frequency instead. Using the wrong one is not a small loss. On Gemma-4 at 50 % retention the frequency-weighted ranking gives a KL of 0.56 against the unpruned model where the conditional mean gives 1.22, against 1.93 for picking at random — and at that level the conditionally-ranked model generates degenerate text while the other does not. The rule is chosen per architecture; there is no flag to get wrong. On Qwen3.6-35B-A3B-OptiQ-4bit at 50 % retention: 22.1 GB to 12 GB on disk, 24.5 GB to 11.5 GB peak memory, 47 to 54.7 tok/s, and a Capability Score of 80.03 to 76.57. Procedural ability holds — tool calling −1.0, HumanEval −1.3, GSM8K +2.6, IFEval +4.3 — while broad factual knowledge is the first thing to go, MMLU −21.4. Check MMLU against your own workload before relying on a pruned variant. On gemma-4-26B-A4B-it-OptiQ-4bit at the same 50 % retention: 17.5 GB to 10.4 GB on disk, 16.4 GB to 9.5 GB peak memory. Published OptiQ variants all target 50 % retention, the same way every quant targets 4-bit: one number across the lineup, chosen because that is where the saving is worth having rather than because it is free. What it costs varies sharply by model, and not in the way you would guess: it is not the expert count. Laguna-XS routes over 256 experts, exactly as many as Qwen3.6, and loses far more at the same fraction. The command measures it directly, reporting the KL divergence against the unpruned model before it writes anything — 0.07 and 0.13 for the two Qwens, 0.56 for Gemma-4, 1.9 for Laguna-XS — and warns when that number crosses 1.0. Every checkpoint measured above it wrote fluent, confident, factually wrong text while still handling code and arithmetic correctly, which is the kind of failure a quick read does not catch. If a model lands above the threshold, retain more experts: --retain takes any count. optiq convert Quantize a Hugging Face model with mixed-precision sensitivity-driven bit allocation. convert.shbash $ optiq convert MODEL [OPTIONS] Required MODEL: HF repo ID (e.g. Qwen/Qwen3.5-9B) or local path to a bf16 checkpoint. Options --method [optiq|static]: Bit-allocation method. Default: optiq (exact calibration-driven KL sensitivity). static allocates by structural rules (embedding/head, first/last block, attention and the MoE router get the high bits) at the requested candidate bits and target BPW, with no calibration or forward passes. It matches optiq on typical architectures, loads lazily, and is the fast path for large bases where exact KL is impractical. See the methods comparison. --target-bpw FLOAT: Average bits per weight. Default: 5.0. --candidate-bits TEXT: Comma-separated bit-widths to choose from. Default: 4,8. --reference [auto|bf16|uniform_4bit]: Reference precision for sensitivity probes (optiq method only). Default: auto (bf16 if it fits in RAM, else uniform_4bit). --calibration-mix TEXT: Calibration set. Default: optiq (the bundled 6-domain mix). Pass a path to override with a custom JSONL file. --group-size INTEGER: Quantization group size. Default: 64. --n-calibration INTEGER: Calibration sequences for sensitivity (drawn from the 40-sample mix). Default: 24 (roughly 4 samples from each of the 6 domains). Higher = more stable per-layer KL ranking, linearly slower convert. --n-floor-per-block INTEGER: Block-aware floor: minimum components per transformer block kept above the lowest bit. Prevents lowest-bit assignments from concentrating in the middle. Default: 2; 0 disables. --skip-baselines: Don't also build the uniform-4-bit comparison artifact. -o, --output TEXT: Output directory. Default: optiq_output/. Examples terminalbash # Standard 5.0 BPW mix on a 9B (auto-routes to bf16 reference) $ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 # 27B+ on a 36 GB Mac (auto-routes to uniform_4bit reference) $ optiq convert Qwen/Qwen3.5-27B --reference uniform_4bit # Custom bit set (3-bit / 6-bit mix at 4 BPW average) $ optiq convert Qwen/Qwen3.5-4B \ --target-bpw 4.0 --candidate-bits 3,4,6,8 # Fast structural allocation for a large MoE, mixed 2/4-bit $ optiq convert mlx-community/Qwen3.5-122B-A10B-bf16 \ --method static --candidate-bits 2,4 --target-bpw 2.5 optiq kv-cache Measure per-layer KV-cache sensitivity and write a per-layer KV bit-width config that optiq serve --kv-config consumes. kv-cache.shbash $ optiq kv-cache MODEL [OPTIONS] Options --target-bits FLOAT: Average KV bits across full-attention layers. Default: 5.0. --candidate-bits TEXT: Default: 4,8. --n-samples INTEGER: Calibration samples for KV sensitivity. Default: 5. --seq-len INTEGER: Calibration sequence length. Default: 512. --group-size INTEGER: Default: 64. -o, --output TEXT: Where to write kv_config.json and kv_sensitivity.json. Default: optiq_output/kv_cache. Gemma-4 support Upstream mlx-lm raises NotImplementedError: RotatingKVCache Quantization NYI on sliding-window models. optiq kv-cache installs optiq.runtime.kv.RotatingQuantizedKVCache at probe time so sensitivity analysis (and serve) work on all sliding-window models, Gemma-4, Cohere R2, OLMo 3, Phi-3/4, EXAONE, Ministral 3, etc. optiq lora Train and inspect sensitivity-aware LoRA adapters. optiq lora train lora-train.shbash $ optiq lora train MODEL [OPTIONS] --data PATH: Directory containing train.jsonl (and optional valid.jsonl). Required. --rank INTEGER: Base LoRA rank. Default: 8. --scale FLOAT: LoRA alpha scaling (alpha = rank * scale). Default: 20.0. --dropout FLOAT: LoRA dropout. Default: 0.0. --rank-scaling [constant|by_bits|by_kl]: How to scale rank across layers. by_bits uses the per-layer bit assignments OptiQ recorded at convert time; by_kl scales by raw KL sensitivity; constant is uniform. Default: by_bits. --num-layers INTEGER: Last N transformer blocks to adapt; -1 for all. Default: 16. --target-modules TEXT: Comma-separated module suffixes to adapt. Default: q_proj,v_proj. --use-dora: Use DoRA in place of LoRA (currently NotImplementedError on mlx-lm 0.31.x). --max-seq-length INTEGER: Tokens per training sample. Default: 1600 (M3 Max 36 GB safe ceiling for 9B). See the fine-tuning training-ceiling map for per-model recipes. --iters INTEGER: Training iterations. Default: 1000. --learning-rate / --lr FLOAT: Default: 1e-4. --batch-size INTEGER: Default: 1 (Mac UMA constraint). --grad-accumulation-steps INTEGER: Default: 1. --grad-checkpoint / --no-grad-checkpoint: Enable activation checkpointing. Default: on. --val-batches INTEGER: Default: 25. --steps-per-report INTEGER: Default: 10. --steps-per-eval INTEGER: Default: 200. --steps-per-save INTEGER: Default: 100. -o, --output PATH: Adapter output directory. Default: ./optiq_lora_adapters. optiq lora info lora-info.shbash $ optiq lora info ADAPTER_DIR Prints the per-layer rank distribution and trainable parameter count. optiq serve Dual-protocol inference server. Wraps mlx_lm.server and exposes both the OpenAI /v1/chat/completions endpoint and the Anthropic /v1/messages endpoint from the same process. Drive it with the OpenAI SDK, the Anthropic SDK, plain curl, or Claude Code (via ANTHROPIC_BASE_URL). Adds OptiQ-aware mixed-precision KV cache and a mounted LoRA adapter on top. serve.shbash $ optiq serve [OPTIONS] -- [mlx_lm.server flags] OptiQ-specific options --kv-config PATH: JSON config from optiq kv-cache. Per-layer mixed-precision KV. Overrides --kv-bits. --kv-bits INTEGER: Uniform KV quantization (4 or 8). Omit both --kv-config and --kv-bits for fp16 KV. --kv-group-size INTEGER: KV quantization group size. Default: 64. --quantized-kv-start INTEGER: Token offset at which KV quantization kicks in. Default: 0. --adapter PATH-OR-REPO: Apply a LoRA adapter at startup. Accepts a HuggingFace repo id (auto-downloaded) or a local directory. OptiQ-trained adapters surface their optiq_lora_config.json sidecar in startup logs. --models-dir DIR: Advertise locally-built quants under DIR in /v1/models (alongside HF-cache models and the served one), switchable per request by path. The server already hot-swaps to any requested model. --ngram-draft INTEGER: Prompt-lookup speculative decoding. Drafts up to N tokens copied from the conversation and verifies them in one pass; no draft model needed. 8 is a good value. Default: 0 (off). Mutually exclusive with --mtp and --drafter. See Serve. --ngram-gate / --no-ngram-gate: Pick the draft length each step from recent acceptance and the model’s measured verification cost. Default: on. --ngram-min INTEGER: Shortest n-gram match that triggers a draft. Default: 3. --anthropic / --no-anthropic: Expose the Anthropic /v1/messages endpoint alongside OpenAI's. Default: on. Pass --no-anthropic to disable. Forwarded to mlx_lm.server All other flags pass through unchanged. Common ones include --model, --host, --port, --max-tokens, --temp, --top-p, --top-k. See python -m mlx_lm.server --help for the full upstream list and current defaults. Examples terminalbash # 1. Generate a per-layer KV config (once per model) $ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit --target-bits 5.0 # 2. Serve with mixed-precision KV + Anthropic on by default $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --kv-config optiq_output/kv_cache/kv_config.json \ --port 8080 # 3. Drive Claude Code through it export ANTHROPIC_BASE_URL="http://localhost:8080" export ANTHROPIC_API_KEY="not-used" $ claude optiq eval Two-stage evaluation harness. --task smoketest runs a fast triage (KL + GSM8K-50, ~5 min on 27B). --task all runs the full benchmark suite (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop, ~2 h on 27B). Add --score for the unweighted-mean Capability Score. Individual tasks are addressable on their own. KL is not part of that suite: it is the one step that holds a reference model in memory alongside the candidate, which is how a large quant gets killed on a memory-tight Mac, and it does not enter the Capability Score. Ask for it with --with-kl, or run it alone as --task kl. The methodology is in the eval-framework write-up. eval.shbash $ optiq eval MODEL_PATH --task [TASK] [OPTIONS] Tasks smoketest: KL on 64 prompts × 256 tokens, plus GSM8K on 50 samples. Fast triage. all: full benchmark suite below, in one run. kl: KL divergence vs reference (default 64 prompts). mmlu: 5-shot MMLU on 1000 samples. gsm8k: GSM8K on 1000 samples (default), CoT 3-shot. Default task if none specified. gsm8k-50: GSM8K on 50 samples (smoketest tier). ifeval: full 540-prompt IFEval (instruction following). bfcl: BFCL-V3 simple, 200 questions (function calling). humaneval: full 164 problems, code execution sandboxed (container via Docker, Podman or Apple container → macOS sandbox-exec → subprocess + rlimit). hashhop: HashHop Long-Context Evaluation. 25 instances per hop count × 4 hop counts (~100 total) at ~12 k context. Multi-hop key→value retrieval. Options --task TEXT: One of the tasks above. Default: gsm8k. --n-samples INTEGER: Override the per-task default sample count. --baseline PATH: Side-by-side comparison against another model (gsm8k task only). --reference-model PATH: Explicit reference for the KL eval. If unset, auto-resolved. --reference-mode [auto|bf16|uniform_4bit]: KL reference selection strategy. Default: auto (bf16 if it fits in RAM via HfApi.model_info per-shard sizes, else mlx-community uniform-4-bit baseline). --score: With --task all, also compute the Capability Score (unweighted mean of MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop). --reasoning: Score an always-on reasoning model fairly. Lets the model emit its block (no suppression), gives the generation tasks a large per-question budget so the trace completes, strips the think block before extracting the answer, and scores MMLU generatively (parse the answer letter) instead of by first-letter logit argmax, which collapses to chance for a model trained to reason before answering. Opt-in; default eval behavior is unchanged. --reasoning-max-tokens INTEGER: Per-question budget in --reasoning mode. Default: 3072 (a cap, not a fixed length, greedy decoding stops at EOS, so short traces cost nothing). Raise it for unusually long reasoners. --kv-bits INTEGER: Uniform KV quantization (4 or 8) during generation-based benchmarks. KL and MMLU are forward-pass only, so this has no effect there. --kv-group-size INTEGER: Default: 64. --kv-config PATH: Per-layer mixed-precision KV from optiq kv-cache. Overrides --kv-bits. --output-json PATH: Dump the structured eval record (every metric + Capability Score) to JSON. --served URL: Score through a running optiq serve instead of loading the model into the eval process. Same questions, same greedy decode. What changes is that the answers come back over HTTP, through the batch generator, the KV cache, the server’s rendering of the chat template, and its own tool-call parser. MODEL_PATH is still required — it supplies the tokenizer and the model id sent on each request, and only metadata is fetched, never weights. Not available for --task kl, --task smoketest, or MMLU without --reasoning: those score by comparing raw logits, which no endpoint exposes. Flags that configure the process holding the weights (--kv-bits, --kv-config, --stream-experts) are refused rather than silently ignored; set them on optiq serve. Scoring the path your users are on A default run loads the weights into the eval process. Nobody consumes a model that way — they run optiq serve and talk to it over HTTP. The two paths agree on published quants, but they are not the same code: only the served one goes through the batch generator, the KV cache and the server’s family-specific tool-call parser. Function-calling numbers in particular are worth confirming served, because that parser is the one a client actually receives calls from. terminalbash # one terminal $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --host 127.0.0.1 --port 8080 # another $ optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit \ --task bfcl --served http://127.0.0.1:8080 # the whole suite over HTTP (MMLU needs --reasoning to be servable) $ optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit \ --task all --score --reasoning --served http://127.0.0.1:8080 Examples terminalbash # Fast triage $ optiq eval ./optiq_output/Qwen3.5-9B/optiq_mixed --task smoketest # Full benchmark suite + Capability Score, dump to JSON $ optiq eval ./optiq_mixed --task all --score \ --output-json ./bench.json # Single benchmark $ optiq eval ./optiq_mixed --task humaneval # GSM8K head-to-head against the uniform-4-bit baseline $ optiq eval ./optiq_mixed --task gsm8k \ --baseline ./optiq_output/Qwen3.5-9B/uniform_4bit # Eval the full suite with mixed-precision KV active during generation $ optiq eval ./optiq_mixed --task all \ --kv-config ./kv/qwen35_9b/kv_config.json optiq benchmark Quick-and-dirty perplexity + throughput on a converted model, with optional baseline side-by-side. For headline accuracy numbers, prefer optiq eval above. benchmark.shbash $ optiq benchmark MODEL_PATH [--baseline UNIFORM_PATH] [--n-samples N] --baseline PATH: Side-by-side comparison against a uniform-4-bit baseline (or any other model path). --n-samples INTEGER: Perplexity samples. Default: 50. optiq latency Predicts decode tok/s for a quantized model using the Apple Silicon roofline model: latency ≈ model_bytes / memory_bandwidth + per_layer_overhead. Detects your hardware tier and bandwidth automatically. latency.shbash # Bare prediction (linear-layer-only roofline; optimistic) $ optiq latency ./optiq_mixed # Calibrated: load + run the model once, fit the overhead constant $ optiq latency ./optiq_mixed --calibrate --calibrate: Run 8 warmup + 15 measured generations to fit the framework-overhead constant for this hardware. The bare prediction only counts weight-loading time and is optimistic; on M3 Max ~83% of decode latency is overhead (attention, norms, KV cache, framework), so calibration produces meaningfully better predictions for the model you measured and for similarly-sized models on the same hardware. Top-level options optiq --version: Print the installed version. optiq --help: Show top-level help. optiq COMMAND --help: Per-command help. Configuration Every setting resolves the same way, in this order: flag > environment > repo config > user config > default Config files are JSON objects keyed by setting name: /.optiq/optiq.json, checked in and shared by the team ~/.optiq/config.json, personal and applied everywhere Each setting also has an OPTIQ_ variable. An unknown key in a config file is reported rather than ignored, because a typo that silently does nothing is worse than a warning. terminalbash $ optiq config # every setting, its value, and which source won The coding agent keeps its own settings in ~/.optiq/code/config.json, listed by optiq code config. Settings Setting Default What it does home ~/.optiq Root for state on disk: the Lab database, saved chats, agent sessions. output_dir ./optiq_output Where converted artifacts land. adapter_cache ~/.cache/optiq/adapters Where optiq serve --adapter caches remote adapters. sandbox_container auto Container runtime for the code sandbox. A name forces it; 0 disables the container tier. See the sandbox matrix. sandbox_python_image python:3.11-slim Container image for the python tool. The default carries only the standard library, so point this at an image with numpy, pandas or matplotlib if your tasks need them. On macOS the tool runs your own interpreter and inherits its packages instead. sandbox_shell_image alpine:3.20 Container image for the terminal tool. lab_boot_timeout 900 Seconds to wait for a model server to come up. flash_attn auto Attention backward: auto, always or never. flash_attn_budget_gb 25% of the wired limit Memory budget routing stock against tiled backward. flash_block 128 Query-block size for the tiled FlashAttention backward. fused_ce auto Force fused cut-cross-entropy on or off. Auto gates on the size of the logit tensor. fused_ce_budget_mb 512 Logit-tensor size above which fused CE turns on. fused_dpo false Fused cross-entropy for DPO. Costs no time; leave it on for long context. lowbit_search true Pick each group's range by search when quantizing. Off reproduces a pre-0.4.14 artifact exactly. lowbit_search_max_bits 3 Widest bit-width the range search applies to. stream_prefetch false Prefetch the next layer's experts. Only helps when the model is far larger than RAM. stream_scales_budget_gb auto Cap on resident expert scales. stream_reference auto Force streaming of the reference model during sensitivity analysis. cluster_cwd the venv parent Working directory for cluster workers. cluster_headroom_gb 2 RAM left free per node. A comma list sets it per rank, as in 1.2,0.2. cluster_cache_mb unbounded Per-node MLX cache limit. cluster_gpu_reserve_gb 0.75 GPU memory reserved per node. no_think false Disable thinking blocks for models with a toggle. anthropic_no_think false Same, on the Anthropic-compatible shim. kv_debug / merge_debug false Diagnostics for KV-cache rotation and batched cache merges. Other environment variables HF_HOME: Hugging Face cache root. Default: ~/.cache/huggingface. HF_HUB_ENABLE_HF_TRANSFER=1: Enable accelerated HF downloads (requires pip install hf_transfer). ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY: Set these to point Claude Code (and any Anthropic-SDK consumer) at optiq serve. --- ## Quantize, serve, and fine-tune from the command line. Source: https://mlx-optiq.com/cli OptiQ CLI · optimizing compiler Quantize, serve, and fine-tune from the command line. The MLX-native core. Measure each layer's sensitivity, allocate mixed-precision bits, and ship a quant that keeps more quality than uniform 4-bit at the same size. Then serve it behind an OpenAI- or Anthropic-compatible API and fine-tune it with sensitivity-aware LoRA. No PyTorch, no cloud. Per-layer bit allocation · sample LLM 8-bit · sensitive layers 4-bit · robust layers $ pip install mlx-optiq copy 3.2× smaller than bf16 +15.6 best Capability gain vs U4 How it works Measure, allocate, convert. Which layers tolerate fewer bits is measured, not assumed. 01 · MEASURE Per-layer KL Simulate-quantize each layer at each bit-width, forward-pass calibration data, and measure the KL divergence from the bf16 reference. 02 · ALLOCATE Greedy knapsack Start every layer low, then upgrade the layer that buys the most quality per bit until the budget is met. lm_head and the first/last blocks stay high. 03 · CONVERT Standard MLX Hand the bit map to mlx_lm.convert. The output loads anywhere stock mlx-lm loads, with the sensitivity metadata stored alongside. What it does The same pass sets KV bits and adapter ranks. Weight bits, KV-cache bits and LoRA ranks all come out of that one per-layer measurement. i Mixed-precision weights Sensitive layers stay high-precision, the rest go low, at the same average size as uniform-4. Higher accuracy at the same disk size ii Mixed-precision KV cache A separate sensitivity pass on the cache. Layer 0 is often 56× more sensitive than average, so uniform 4-bit KV is catastrophic, while mixed-precision avoids it. Faster long-context decode, kept sharp iii LoRA, two ways Fine-tune with adapter rank scaled by each layer's bits, then keep N adapters mounted on one base and hot-swap per request. Sensitivity-aware rank · hot-swap iv Text and images Send pictures to vision-capable models. The vendored tower rides in a bf16 sidecar, so one repo loads text-only under mlx-lm or full image+text under OptiQ. VLM + LLM · one artifact v OpenAI + Anthropic serving optiq serve speaks both protocols from one process. Point Claude Code, Codex, or any client at a local quant. Dual-protocol local server vi Expert pruning Remove half a Mixture-of-Experts model's routed experts. Active parameters per token do not change, so it gets smaller without getting slower. REAP, in the quantized domain Quickstart From a model to a serving API. i Convert --reference auto picks bf16 when it fits and a uniform-4-bit baseline when it does not. terminalbash $ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8 ii Serve Dual-protocol server with mixed-precision KV and idle auto-unload. terminalbash $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --idle-timeout 300 iii Prune Halve a sparse MoE's expert bank. It reports the divergence from the unpruned model before writing anything. terminalbash $ optiq prune-experts mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit --target-retention 0.5 Full reference The CLI reference documents every command; How sensitivity works covers the algorithm, and Expert pruning covers what halving a MoE costs. The compiler that builds every OptiQ quant. Quantize, serve, and fine-tune LLMs natively on Apple Silicon. Read the CLI docsBrowse the models --- ## Your models, in a local web app. Source: https://mlx-optiq.com/lab OptiQ Lab · local workbench Your models, in a local web app. A local web app for the whole workflow: chat with your models, run deep research on them, quantize new ones, fine-tune with your data, split a big model across several Macs, and compare them side by side. It runs on your Mac; nothing is uploaded. localhost:8080 · OptiQ Lab $ optiq lab copy The workbench The full OptiQ workflow, in a browser. The same MLX-native engine as the CLI, wrapped in a local web app. Pick a model from the Hub, chat with it, quantize your own, fine-tune on your data, and compare two models side by side. Chat Chat with tools Talk to any served model with sandboxed web search, Python, and terminal tools, plus image upload and chat-with-files retrieval with citations. Arena Model Arena Run two models on the same prompt and compare speed and output. Hub Model Hub Browse the published OptiQ quants, search Hugging Face, and load a model with one click, straight into the Lab. Quantize Quantize wizard Point it at any Hugging Face model and get a sensitivity-driven mixed-precision quant, with a live view of the per-layer bit allocation. Fine-tune SFT & DPO Fine-tune with sensitivity-aware LoRA, SFT or DPO, on your own dataset, all local, with progress and loss curves in the browser. Dataset Dataset designer Build and shape a training set from files and templates before you fine-tune, without leaving the app. Deep research Research reports, from your own model. Ask a hard question and the Lab drafts a report, finds the gaps in it, searches the web to fill them, and revises, over several rounds. Every claim points to a page it actually read, and the whole thing runs on the local model you serve. 01 Draft, then dig. It writes a rough answer first, then treats its own gaps as the search plan, so later rounds chase what is missing instead of repeating what it already has. Iterative 02 Evidence first. Findings are tied to pages the model retrieved and read, and the report ends with the sources, so you can check the claim rather than trust it. Grounded 03 Nothing leaves the Mac. The model, the reasoning, and the report all stay local. Only the web searches go out, to the engine you choose. Private Cluster Run one model across several Macs. Link two or more Apple Silicon Macs over Thunderbolt and the Lab serves a single model pipeline-parallel across the ring, so a model too large for any one machine runs on the group. A live topology view shows the nodes, and the Lab chat points straight at the cluster endpoint. Discover Thunderbolt ring Nodes find each other over the Thunderbolt bridge and form a ring. The panel draws the live topology as peers join. Serve Pipeline-parallel The model's layers are split across the machines, so combined memory, not one Mac's ceiling, sets how big a model you can run. Use One endpoint The ring exposes a single server. Point the Lab chat, OptiQ Code, or any OpenAI-compatible client at it. How it connects The Lab drives the same optiq engine as the CLI and serves over the same dual-protocol server, so a model you serve in the Lab is the one OptiQ Code can drive. See the Lab docs. A local AI workbench in one install. Run, quantize, fine-tune, and compare models on your own Mac. Read the Lab docs pip install mlx-optiq ↗ --- ## A coding agent for your local model. Source: https://mlx-optiq.com/code OptiQ Code · terminal coding agent A coding agent for your local model. It drives whatever optiq serve is serving, a 4-bit quant on your MacBook or a 27B on a Mac Studio, through a read, edit, run-tests loop right in your terminal. It runs offline and stays on your machine. optiq code · ~/my-project $ optiq code copy How it works It runs the same loop as a cloud agent, locally. Describe a change. OptiQ Code plans, edits, and runs your tests, turn after turn, until the suite is green, pausing for your approval on anything that writes. 01 · READ Read the code Searches and reads the files it needs, so the change is grounded in your actual code. 02 · EDIT Make the change Writes a precise diff, or rewrites a whole file when that is cleaner. You approve it. 03 · TEST Run the suite Runs your tests, so it tracks the real pass/fail count. 04 · REPEAT Repeat Feeds the failures back and tries again, until the tests pass or you stop it. Built for local models It handles the failure modes small models hit. A small model often reasons its way to the right fix and then loses it on the mechanics. OptiQ Code targets exactly those failure modes, so the model you can actually run still lands the change. When the modelloses the diff on its final turn OptiQ Codesalvages the working git diff on every exit path, so it never returns an empty patch. When the modelbotches an exact-match edit, again and again OptiQ Codestops the retries and asks for a full-file rewrite instead of dead-ending on one file. When the modelreads in circles or repeats itself OptiQ Codetrips a stall nudge: make one small, testable change, then run the tests. When the modelwrites a tool call as text, not a structured call OptiQ Codeheals it into a real call, so a malformed call is recovered rather than wasting a turn. On real bugs Measured against other harnesses. We put three coding agents on two public benchmarks. In each one, all three drive the same local model, so the agent is the only thing that changes. OptiQ Code scores highest on both. It resolves the most on SWE-bench-Lite, and on openbench it beats opencode by a small margin while using far fewer tokens. Same local model across all three harnesses in each row · checker-graded Benchmark OptiQ Code mini-swe-agent opencode SWE-bench-Lite 4-bit 4B local · % resolved 36% 25% 0% openbench core 4-bit 35B local · 8 tasks · score 7.74 / 8 6.70 / 8 7.38 / 8 Same tasks, fewer tokens On openbench, OptiQ Code and opencode solve the same tasks, but OptiQ Code uses about 5× fewer tokens to get there: roughly 17k per task against 83k. On a Mac, that is your own compute and battery. Read the benchmark study → The interface You stay in the loop. A session banner that scrolls away, an open prompt, inline tool markers, and single-Enter approval before anything writes to your files. Or let it run unattended in auto mode. Session start Running a task Approval Quickstart Serve a model, then run Code in your repo. i Serve a model --idle-timeout frees the RAM when you step away and reloads on the next turn. terminalbash $ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit --idle-timeout 300 ii Launch in a repo Zero config, it discovers the served model. Describe the change; approve edits with a single Enter, or run auto. terminalbash $ cd my-project && optiq code iii Or run it headless Auto-approve, run to completion, print the diff, exit non-zero if the goal was not met. terminalbash $ optiq code -p "Fix the failing test in parser.py" $ optiq code -c # resume the last session $ optiq code export -o s.jsonl $ optiq code --preset plan # read-only: investigate, propose, change nothing $ optiq code -p "..." --preset minimal # benchmarking: two tools, no harness Where to next The OptiQ Code guide covers approval modes, sessions, the tool set, and headless use in full. Presets cover plan mode, which you toggle with Tab, and when to strip the harness back so a benchmark measures the model. Run a coding agent on your own model and machine. It runs locally, with no per-token billing. Read the guide pip install mlx-optiq ↗ --- ## OptiQ Code Source: https://mlx-optiq.com/docs/code OptiQ Code · Overview OptiQ Code OptiQ Code is a terminal coding agent that drives whatever model optiq serve is serving, on your Mac, fully offline. It is built to stay reliable on small local models. It always returns a working patch when one exists, and it recovers from failed edits and stalled turns rather than dead-ending. Every session is recorded so you can resume or export it. This guide continues in six more pages: Approval & permissions, Sessions & traces, Headless & CI, the Tools reference, Robustness, and Cloud & remote models. Install & launch textual and openai ship with the wheel, so there is nothing extra to install. OptiQ Code talks to optiq serve over the OpenAI Chat Completions API and discovers the served model automatically. terminalbash $ pip install mlx-optiq # serve a model once, freeing its RAM after 5 min idle $ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit --idle-timeout 300 # launch in a repo (or: optiq code PATH) $ cd my-project && optiq code If no server is running, OptiQ Code tells you how to start one. Point it at a server on another port or host with OPTIQ_BASE_URL, or pass --model to have it spawn one for you. How the loop works Each turn, the agent calls the model, runs whatever tools the model asks for, and feeds the results back. It grounds a change by reading the relevant files, makes an edit, runs your tests, and reads the pass/fail count, so both it and the stall detector track real progress rather than text. It keeps going until the model gives a final answer (a plain reply with no tool call ends the turn, like a chat), or the goal is done, or a limit is hit. The working git diff is the source of truth for what changed, salvaged on every exit path (scoped to the files the agent touched). The mechanics of that resilience are covered in Robustness. Commands & keys The interactive UI is deliberately familiar. Type a task and press Enter; the model works and answers, and the turn ends when it does. Slash commands and keys cover the rest. Type / or @ to open an autocomplete menu (slash commands, or repo files for an @-mention), and Tab completes it. Slash command Effect /help List the commands /init Analyze the repo and write an AGENTS.md guide (reads any existing AGENTS.md/CLAUDE.md first) /model Show the model, endpoint, and context window /resume Pick a past conversation in this repo (↑/↓ picker); /resume N loads directly /copy Copy the agent's last reply to the clipboard /compact Compact the context now (drop old tool output) /clear Clear the screen and start a fresh context /quit Exit (aliases /exit, /q) Key Effect Enter Send. Shift-Enter / Ctrl-J insert a newline; a paste keeps its newlines ↑ / ↓ Walk input history (moves between lines first in a multiline prompt) Esc Interrupt the running turn; Esc Esc opens a picker to rewind to an earlier turn Tab Cycle mode: standard ↔ plan (read-only). The status bar shows which one is live Ctrl-V Attach an image from the clipboard. Dragging a file onto the prompt attaches it too /mcp List connected MCP servers and the tools they expose Shift-Tab Cycle approval: approve each edit ↔ auto-approve all Ctrl-C Copy a selection if any, else clear the line, else press twice to exit Scrolling & selection. Like other full-screen TUI agents, OptiQ Code captures the mouse, so a two-finger trackpad swipe (or the wheel) scrolls the transcript; PageUp/PageDown and Ctrl-Home/Ctrl-End work too. To select text, hold your terminal's bypass key while dragging (Fn on macOS Terminal.app, Option on iTerm2), or use /copy. Prefer no-modifier drag-selection? Set "mouse": false in the config; you lose trackpad scroll but the scroll keys still work. Images If the model you are serving reads images, OptiQ Code can send it one. There are two ways in, and they match what a terminal can actually deliver. Drag a file onto the prompt. Your terminal responds to a dropped file by typing its full path, and OptiQ Code takes it from there: the path leaves the text, the picture is attached, and the status bar shows a count. Type your question alongside it and press Enter. Press Ctrl-V to paste one. A screenshot on your clipboard cannot arrive through a normal paste, because a terminal only carries text, so OptiQ Code reads the clipboard itself. Cmd-V still pastes text as usual. terminalbash # headless, one or more images $ optiq code -p "Why does this layout break?" --image ~/shot.png $ optiq code -p "Which renders correctly?" --image a.png --image b.png Only absolute paths attach on their own. A drop always produces one, while a sentence like "fix the logo in assets/logo.png" is talking about a file rather than attaching it, and quietly uploading it would be the wrong read. The image goes on the message you send it with, not on every message after. It stays in the model's context from that point, and re-sending it each turn would spend a local context window on something the model already has. PNG, JPEG, GIF, WebP and BMP, up to 10 MB each. The model has to support itText-only models reject an image, and the server says so rather than dropping it silently. The Vision page lists the families that read images, and any of them can be served with optiq serve. Configuration Flag / env Effect optiq code [PATH] Launch the interactive TUI in a repo (default: current dir) --model Serve and drive this model if no server is already running -p / --print Headless run toward a goal, then exit (see Headless) -c / --continue Resume the most recent session in this repo -r / --resume Resume a specific session by id OPTIQ_BASE_URL Point at a server on another port or host, or a cloud endpoint OPTIQ_API_KEY Bearer token for the server (default sk-optiq-local) These, plus per-model settings (context window, approval mode, max tokens, mouse capture), can live in a config file instead of flags: ~/.optiq/code/config.json for every repo, or /.optiq/code.json for one. Precedence is flag > env > repo config > user config > default; optiq code config prints the resolved values and where each came from. Prefer a cloud or third-party model over a local one? Point OptiQ Code at any OpenAI-compatible endpoint with your own key; see Cloud & remote models. Choosing a model Model recommendation OptiQ Code is model-agnostic. It drives whatever optiq serve is serving. For best results, use the OptiQ quant with the highest Capability Score your Mac can run; in general Qwen3.8-27B-OptiQ-4bit is the best pick. Serve a no-think variant when the model supports it, so the model answers directly. Track record An initial version of OptiQ Code, driving a 4B local model, resolved 36% of a SWE-bench-Lite subset, state of the art at that size, with a valid patch on every task, versus 40% empty for a comparable-budget baseline. A stronger local quant raises the resolve rate. The reliability features work the same regardless of model size. See the product overview. --- ## One hard turn shouldn’t cost you the session. Source: https://mlx-optiq.com/cloud OptiQ Cloud One hard turn shouldn’t cost you the session. Your local model handles the everyday work. When a turn starts going in circles, Boost hands that single episode to a frontier model and gives control straight back, in the same editor you were already using. Start with 20 free Boosts Read the docs How it works How a Boost happens Boost lives inside optiq serve, not in your editor. Anything already pointed at your local server gets it. 1 Connect the machine Sign in once. The token is written to your OptiQ config and never leaves the machine. $ optiq cloud login 2 Ask for a Boost Start any message with /boost. Clients that swallow slash commands, Claude Code among them, can use boost: instead. /boost work out why this test fails on CI but not locally 3 Carry on locally The moment the episode finishes, the next message goes back to your own model. A Boost is one episode, never a mode. What a Boost is One Boost covers the whole exchange A hard turn is rarely a single question. The cloud model reads files, runs commands and comes back several times before it answers, and all of that is one Boost. What happens Cost to you One question, one answer 1 Boost Six tool calls, then an answer 1 Boost A Boost that produced nothing refunded Every other turn in the session free, local Where it works It works in the editor you already use These are the integrations OptiQ tests against. Boost is implemented in the server, so anything else that speaks an OpenAI-compatible endpoint will work too, it is simply not something we verify. Claude Code Speaks the Anthropic API. Use boost: rather than /boost, since it handles slash commands itself. Codex Works on both wire formats it can be configured with, Chat Completions and Responses, streaming or not. OpenCode Point it at optiq serve as usual and start a message with /boost. OpenClaw Same endpoint, same trigger. Nothing to configure per project. Hermes Agent Boost applies to the model call, so it works inside whatever loop Hermes is running. Mistral Vibe Uses the OpenAI-compatible surface, so a Boost behaves exactly as it does everywhere else. Pricing What it costs You buy Boosts when you want them. There is nothing recurring to cancel, and nothing to pay on the days you do not use one. Free 20 Boosts Enough to find out whether you reach for it. No card needed. Boost pack 200 Boosts for $20 A one-off purchase, not a subscription. Turn on auto top-up if you would rather not think about it, with a monthly cap you set. Privacy What leaves your machine What leaves my machine? The messages in the one episode you Boosted, and nothing else. Ordinary turns never reach the network. Your repository is not uploaded, indexed or scanned. Is my code used for training? No. Prompts and replies are not stored after the episode completes. What is kept is token counts and timings, so you can see what a Boost cost. What if the cloud is unreachable? The turn continues on your local model and the message says why. You are not charged for a Boost that did not happen. Can I turn it off? It is off until you sign in, and after that nothing is sent unless a message asks for it. optiq cloud logout revokes the token rather than just forgetting it. Try it on the next turn that stalls Twenty Boosts to start. One command to connect, and no plugin for whichever editor you already use. Start with 20 free Boosts Read the docs --- ## OptiQ Cloud Source: https://mlx-optiq.com/docs/cloud OptiQ Cloud Boost hands one hard turn to a frontier model and gives control straight back. It lives inside optiq serve, so every client already pointed at your local server can use it without a plugin. Connect a machine Sign in once per machine. The browser approves the request; the token is written to your OptiQ config and never leaves the machine. $ optiq cloud login optiq cloud status shows the account, the Boosts remaining and the mode. optiq cloud logout revokes the token rather than only forgetting it, so a lost laptop cannot keep spending. Restart optiq serve after signing in. It reads the token at startup and prints a line confirming Boost is enabled. Asking for a Boost Start a message with /boost: /boost work out why this test passes locally and fails on CI Some clients handle slash commands themselves and never send them to the server. Claude Code is one. Use the plain-text form there: boost: work out why this test passes locally and fails on CI The trigger word is stripped before the request reaches the cloud model, so it never appears in what the model reads. Both runs below were given the same sentence. The local model read the file, ran the tests, grepped, then fetched a URL it had invented, and ran out of turns: The local model, stalling. The offer costs nothing. With /boost in front, the same sentence goes to the cloud model, which reads both files and answers: The same sentence, boosted. What one Boost covers A Boost is an episode, not an API call. A hard turn is rarely one question: the model reads files, runs commands and comes back several times before it answers. All of that is one Boost. The episode is matched on the tool-call ids the cloud minted, which the protocol obliges your client to echo back. That survives a client compacting its history mid-episode, and it means the client never has to know an episode exists. The episode ends on a reply with no tool call, on the per-episode cost ceiling, or on an error. The next message goes to your local model again. A Boost is one episode, never a mode. An episode that produced nothing is refunded. Modes Mode Behaviour manual Only a message that asks for a Boost gets one. suggest Offers a Boost after a turn that stalled. Never spends a credit on its own. The default. auto Retries a stalled turn on the cloud, once per request. A turn counts as stalled when it used every turn without finishing, made the same tool call repeatedly, or ended on a run of failing calls. Length alone is not a signal: a large refactor takes many turns and succeeds. In suggest, the stall is named and the decision is yours: suggest names the signal it saw, then waits. In auto, the same stall is retried without asking: auto retries once, and says why it did. Images Screenshots are resized and recompressed before upload, because an image is worth about a thousand input tokens and full-resolution pixels buy nothing. A payload still too large after that is refused with an explanation rather than pushed at the request limit. What leaves your machine The messages in the one episode you Boosted, and nothing else. Ordinary turns never reach the network. Your repository is not uploaded, indexed or scanned, and prompts are not retained after the episode completes. What is kept is token counts, timings and cost, so your dashboard can show what a Boost actually cost. When a Boost cannot happen The turn continues on your local model and the reply says why. You are not charged for a Boost that never reached the cloud. A request that provably never left the machine is retried once; a request that timed out is not, because it may already have been processed and billed. Asking for a Boost on a machine that was never connected says so, and says what to run: No account, no charge, no silent fallback. Boosts and billing Sign-up includes 20 Boosts. A pack is 200 Boosts for $20, bought when you want it rather than as a subscription. Auto top-up is optional and takes a monthly cap you set. The dashboard Your account lives at cloud.mlx-optiq.com. It shows four things and deliberately not much else: Boost balance, and a button to buy 200 more for $20. Connect a machine, with the optiq cloud login command to run. Auto top-up, off by default. See below. API tokens: one row per machine you have connected, showing the prefix, when it was last used, and a button to revoke it. Only the hash is stored, so a token is shown once and never again. Auto top-up A balance that runs out mid-task is the worst moment to discover it, so auto top-up buys another pack when you drop below a threshold you set. Setting Default What it does Enabled off Nothing is charged unless you turn it on. Threshold 20 Boosts Buys when the balance falls below this. Pack 200 for $20 What each top-up purchases. Monthly cap yours to set A hard ceiling. Once reached, top-ups stop until the month rolls. The cap is the point. Automatic spending without one is how a runaway loop becomes a surprising invoice, so the cap is checked in the same statement that claims the top-up: several Boosts arriving at once cannot each start their own purchase. Seeing what a Boost cost Every episode records its token counts, how many cloud calls it took and how long it ran. OptiQ Code prints the same thing at the end of a run: tokens: 21,480 in / 4,515 out = 25,995 · 4,003 reasoning (89% of out) · local + 1 Boost · 14 left Episodes are counted, not calls. A turn where the model read four files before answering is one Boost, and reads as one. Accounts, keys and billing Signing in, connecting a machine, how your coding agent gets the key (it does not), API tokens and auto top-up all live on Accounts and billing. Per-client notes See Integrations for setup of each supported client: Claude Code, Codex, OpenCode, OpenClaw, Hermes Agent and Mistral Vibe. --- ## MCP servers Source: https://mlx-optiq.com/docs/code/mcp OptiQ Code · MCP MCP servers MCP lets a model use tools somebody else wrote: a database, an issue tracker, your design system, a search index. OptiQ Code and the Lab both speak it over stdio, which is the transport nearly every MCP server ships with, and it reads the configuration you already have rather than asking you to write a new one. Bring the servers you already have If you use Claude Code, Claude Desktop, Codex, opencode, OpenClaw, Hermes Agent, Mistral Vibe or Cursor, your servers are already described on disk. Import them: terminalbash # see what would be imported, from every client it knows $ optiq code mcp import # write them $ optiq code mcp import --yes # or just one client $ optiq code mcp import claude-desktop --yes Nothing is written without --yes. Servers you already have keep their existing definition, so re-running it is safe. Adding one by hand terminalbash $ optiq code mcp add filesystem npx -y @modelcontextprotocol/server-filesystem ~/code $ optiq code mcp add gh docker run -i --rm ghcr.io/github/github-mcp-server --env GITHUB_TOKEN="$TOKEN" $ optiq code mcp list # what is configured $ optiq code mcp test # start them and show the tools $ optiq code mcp remove gh optiq code mcp test is the one worth running after any change. It actually spawns each server and lists what it exposes, so a typo in a path or a missing token shows up there rather than halfway through a task. The config file Servers live in ~/.optiq/code/mcp.json, or /.optiq/mcp.json for one project, where the repo wins on a name clash. The format is the mcpServers object every other client uses, so you can paste one in directly: ~/.optiq/code/mcp.jsonjson { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/me/code"], "env": {"API_KEY": "..."}, "enabled": true } } } enabled is the one field that is ours. Leave it out and the server is on; set it to false to park a server without deleting a definition that may have a token in it. Using them Connected tools appear to the model as mcp____, and the status bar shows how many are live. In the TUI, /mcp lists every server, its tools, and which of them are read-only. The prefix is not decoration. The reference filesystem server publishes tools called read_file, write_file and edit_file, which are the names of three of OptiQ Code's own. Without the prefix, connecting it would quietly replace the editor the agent depends on. Approval and modes An MCP tool is someone else's code, reached over a pipe, and its description is text that arrives in your model's context. So OptiQ Code treats it the way it treats an edit: MCP tools available Approval standard All of them Asked for, unless the server marks the tool read-only plan Read-only tools only Not needed minimal None — Read-only comes from the server's own readOnlyHint. A tool that says nothing is treated as not read-only: the failure worth avoiding is a plan-mode agent posting to a channel because a server forgot to annotate itself. Why minimal takes noneminimal is the preset a Capability Score is measured through. A score that moves depending on which MCP servers the person running the benchmark happened to have configured is not a measurement of the model, so that preset mounts no MCP tools at all. In the Lab The same servers back the Lab's chat. Configure them once and both OptiQ Code and the Lab can use them, which is why the config lives at ~/.optiq/mcp.json rather than under either one. Open Model & params on the Chat page and the MCP row lists each connected server with its tool count. Nothing else to switch on: if servers are configured, the model can use them. The servers start once for the Lab process, not once per conversation, so the cost of launching them is paid at first use and calls after that are immediate. Nothing is spawned at all if you have no servers configured. What a server can and cannot reach An MCP server is somebody else's program. A line like npx -y @someone/server fetches a package and runs it on your machine, and the tool descriptions it publishes are text that goes into your model's context. Connect servers you would be willing to run any other way. OptiQ narrows what one can reach: Not your credentials. A server gets a short list of variables it needs to run (PATH, HOME, locale, temp dir) and nothing else. Before this was fixed, a server could read every token in the environment. Anything a server genuinely needs goes in its own env block, where it is scoped to that server and visible in the config. Not your whole context. At most 64 tools per server, descriptions capped, and a single result capped at 100,000 characters. Anything dropped is reported rather than quietly cut. Not without you. Anything not marked read-only needs approval before it runs, and the prompt names the server. What OptiQ cannot do is make a description honest. A server can put anything in one, including text aimed at your model, and no amount of filtering fixes that. The defence is that the text is data rather than instruction, it is attributed to its server, it is bounded, and acting on it still needs your approval. What it costs Every connected tool is described to the model on every turn. One 14-tool server is about 2,300 tokens per request before you have typed anything, and a few servers is a real bite out of a local model's window. OptiQ warns past roughly 4,000 tokens. So connect what you use. optiq code mcp list shows what is on, and setting "enabled": false parks a server without deleting a definition that may have a token in it. Keys on diskServer entries hold API keys in env, so the config file is written owner-read-only. It is still plaintext: treat it the way you treat any dotfile with a token in it. What is not supported yet stdio only. Remote HTTP and SSE servers are recognised and reported as skipped rather than silently ignored, but OptiQ Code cannot connect to them yet. Tools are supported; MCP resources and prompts are not. A server that fails to start costs you its tools and nothing else. The error is shown, the other servers carry on, and the agent keeps working. --- ## Presets Source: https://mlx-optiq.com/docs/code/presets OptiQ Code · Presets Presets A preset decides how much harness runs around the model, and which tools it may reach for. standard is the full agent and the default. plan is read-only: it investigates and proposes, and cannot change anything. minimal strips the harness down to a floor, for when you want to measure the model rather than the tooling around it. standard plan minimal Tools All 11 read_file, search, git, done bash, edit_file Can write Yes No Yes System prompt Tuned, plus your AGENTS.md Asks for a plan, plus your AGENTS.md One fixed line, nothing appends Project context Repo listing in the first message Repo listing in the first message None Compaction On On Off Plan mode Plan mode is for the part of a task that happens before you want anything touched: reading the code, finding the callers, working out what the change should be. It has four tools, all read-only, and a system prompt that asks for a plan rather than a change. The standard prompt tells the model that if it can run something, it should, which is the wrong instinct here. terminalbash $ optiq code --preset plan $ optiq code -p "How would you add retries to the fetch path?" --preset plan In the TUI, Tab switches between plan and standard, and the mode shows in the status bar. Work out the approach in plan mode, press Tab, and carry the same conversation into the change. Shift-Tab is the approval cycle and is unaffected. Enforced, not requestedThe restriction lives where the tool is dispatched, not in the list of tools the model is shown. Hiding a tool only asks the model not to use it, and an early build of plan mode wrote to disk while the status bar said plan. A write in plan mode is refused before it runs. If the model keeps insisting, the run stops rather than spending your context arguing with itself. Plan mode reads better with a question than an order. Ask how you would do something and you get a plan. Tell it to do the thing and the model will try, get refused, and you have spent turns on that instead. Why a smaller harness A capable harness changes the score. The same model, run through two different harnesses on the same 25 SWE-bench tasks, solved 9 and 5. That gap is the tooling, not the weights. So when the question is how good a quantized model is, the agent around it is part of what you are measuring. minimal mounts the two tools every SWE benchmark settles on, a single line of system prompt, and nothing else. It is a worse agent on purpose. Using it terminalbash # benchmark a quant with the harness out of the way $ optiq code -p "Fix the failing test in parser.py" --preset minimal # the full agent, for getting work done (default) $ optiq code -p "Fix the failing test in parser.py" The preset is independent of headless. -p decides whether a human approves each tool call; --preset decides how much harness the model sees. Combine them as above, or run minimal interactively. Nothing appends Under minimal the system prompt is exactly one line, and that is enforced rather than intended. Your AGENTS.md is not read at all, and no repo listing is added to the first message. Every addition to a system prompt is reasonable on its own, which is how a minimal mode stops being minimal without anyone deciding to change it, so the test suite asserts that what standard injects cannot reach the model here. Compaction is off for the same reason. A benchmark run whose context was quietly reshaped partway through is not the run the number describes, so a run either fits the window or fails and says so. Which to use Use standard for real work: the extra tools, your project conventions, and compaction all make the agent more useful. Use plan when you want to understand a change before it happens, or when you are pointing a model at a repo you would rather it did not edit yet. Use minimal when you are comparing models or quants and want the number to be about them. standard and plan are the two you move between while working, which is why Tab cycles those and leaves minimal out. Landing in a two-tool harness mid-task is never what you meant. --- ## OptiQ Code vs opencode Source: https://mlx-optiq.com/blog/harness-matters Benchmark · July 25, 2026 OptiQ Code vs opencode Topic Local coding agents Reading time 8 min Related OptiQ Code We put two coding agents on the same local model, a 4-bit Qwen3.6-35B-A3B under optiq serve, and ran both through an open benchmark built to test coding agents. They solve the same tasks. The difference is cost: OptiQ Code gets the same work done with about five times fewer tokens. On a laptop those tokens are your own compute and your own battery, so the gap costs real time. We had wanted to test one claim properly for a while. The claim is that an agent built for local models does better than a general one when both run the same local model. The claim is easy to make. Testing it fairly is harder, because most agent benchmarks vary the model and hold the tool constant. openbench does the reverse. openbench is a benchmark for coding-agent harnesses. It holds the model and the tasks fixed and measures how much the scaffolding around the model changes the result. Each task is a small repo with an instruction and a grading script, and it only counts as solved when that script exits zero. The agent's own word does not count, so an agent that claims it fixed the bug but did not still scores a zero. The setup Every request in this test hits one model. We loaded mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit once under optiq serve: a 35B sparse MoE with about 3B active parameters per token, quantized to 4-bit, sitting resident on a 36 GB M3 Max. It exposes an OpenAI-compatible endpoint, and both agents point at it. The model, the quantization, and the KV cache do not change between runs. Held fixed Value Model Qwen3.6-35B-A3B-OptiQ-4bit (3B active) Server optiq serve · 64k context · mixed-precision KV Hardware M3 Max, 36 GB, no cloud Tasks openbench core set, 8 tasks, 1 trial Grading checker.sh exit code, not self-report The two agents are OptiQ Code, our terminal agent for local models, and opencode, a general-purpose coding agent that plenty of people use. OptiQ Code runs through openbench's bring-your-own-harness manifest, headless, approving its own edits. opencode runs through its own openbench adapter, pointed at the same endpoint. Measuring tokens the same way opencode reports its own token counts. OptiQ Code runs as a black-box CLI and reports none to openbench. Rather than compare one agent's self-reported numbers against nothing, we dropped a small proxy between both agents and optiq serve. It passes every request through untouched and reads the usage field back out of each response. Both agents end up counted the same way, at the server, with the token and turn counts coming from one place. Both solve the same tasks On correctness the two agents are hard to tell apart, which is roughly what openbench's own results lead you to expect once the model is good enough. Both go six for six on the standard tasks. Both also solve taskflow, a 7,800-line codebase. They only come apart on webcore, the other large task, where both hit the 600-second cap before finishing. When the clock ran out, OptiQ Code had gotten about twice as far, 0.74 against 0.38. Solve rate · openbench core set Tasks OptiQ Code opencode 6 standard tasks 6 / 6 6 / 6 taskflow (7.8k lines) 1.00 1.00 webcore (7.1k lines, capped) 0.74 0.38 Stop there and it is a tie. The question worth asking, once both agents finish the task, is what each one spent to do it. The cost of a solve Here are the six tasks both agents solve, with every figure read off the same proxy: prompt tokens sent, output tokens generated, round-trips to the model, and wall-clock time. Per-task · OptiQ Code vs opencode · same local model Task Prompt tok Output tok Turns Time fix-failing-test 7.2k vs 40.8k 250 vs 373 5 vs 5 10.9s vs 30.5s make-it-run 7.7k vs 40.5k 491 vs 628 5 vs 5 15.5s vs 36.7s misleading-error 12.3k vs 44.2k 457 vs 679 7 vs 5 15.6s vs 40.3s build-a-cli 13.5k vs 49.4k 719 vs 532 7 vs 6 21.0s vs 34.2s add-feature 28.5k vs 180.0k 1.2k vs 7.8k 8 vs 14 33.5s vs 268.8s make-ci-green 26.9k vs 127.7k 1.5k vs 2.9k 6 vs 9 40.4s vs 142.7s Bold is OptiQ Code. Dim is opencode. Both drove the same Qwen3.6-35B-A3B. Averaged over those six tasks: Mean per solved task Metric OptiQ Code opencode Advantage Prompt tokens 16.0k 80.4k 5.0× leaner Output tokens 770 2,140 2.8× fewer Total tokens 16.8k 82.6k 4.9× fewer Turns 6.3 7.3 fewer Wall time 23s 92s 4.0× faster Where the tokens go Almost all of the gap is in what each agent sends up. opencode ships a large general-purpose system prompt, around 7,600 tokens of instructions and tool schemas, and re-sends the growing conversation every turn. OptiQ Code is built for one job against one kind of model, so its prompts are shorter and it holds down how much context it resends. Same task, but opencode gives the model a lot more to read on every step. On a laptop those tokens are compute you pay for directly. There is no invoice, but every prompt token is the model re-reading context, and on local hardware that is real prefill time and real battery. Five times fewer tokens is about five times less of that work, and it is most of why the wall-clock gap is as wide as it is. What this means on a Mac So: a 4-bit 35B model, running entirely on a laptop, gets real coding tasks done, including a 7,800-line refactor, graded by an outside checker, with no cloud and no API key. Which agent you wrap around it decides whether a task costs 17k tokens or 83k. On a hosted model that overhead is someone else's server and a line on your invoice. On a local model it is your own hardware. A leaner agent finishes sooner and leaves more of the context window for the actual code. That is the argument for a local-first agent over a general one, at least on a Mac. Reproduce it All of this runs on hardware you already own. Serve the model, then point OptiQ Code at it. terminalbash # serve the model once optiq serve --model mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit --port 8099 # drive it with the local coding agent export OPTIQ_BASE_URL=http://localhost:8099/v1 optiq code launch . The openbench manifest, the proxy, and the raw per-task numbers are small enough to rerun on any 36 GB Apple Silicon Mac. Both agents reached the same result; OptiQ Code used far fewer tokens to get there. --- ## The best coding agent for local models on a Mac Source: https://mlx-optiq.com/blog/optiq-code Introducing OptiQ Code · July 16, 2026 The best coding agent for local models on a Mac Topic Coding agent Reading time 6 min Related OptiQ Code Point it at a repo, describe a change, and watch a model you are running yourself edit the code and get the tests green. No cloud, API key, or per-token billing. OptiQ Code is a coding agent that lives in your terminal and runs on your own model. It drives whatever optiq serve is serving, a 4-bit quant on your MacBook or a 27B on a Mac Studio, through the same read, edit, run-tests loop you would expect from a cloud agent. Everything stays on your machine. terminalbash $ pip install mlx-optiq $ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit --idle-timeout 300 $ cd my-project && optiq code Why run the model yourself Cloud coding agents are excellent. They also send your code to someone else's computer and bill you by the token. That trade is fine for a lot of work, but not for a private repo, an offline machine, or a long agent loop you would rather not meter. A local model has no metering or network round-trips. The catch has always been that local models are weaker, and weaker models fail in ways a strong one hides. Built for the model you actually have A small model often reasons its way to the right fix and then loses it on the mechanics: diffs that don't apply, wasted turns, empty patches. OptiQ Code puts its engineering there. Empty patches. Every way the loop can end salvages the working git diff, so a run that is out of turns, crashed, or stopped by the clock still hands back your change. Failed edits. When an exact-match edit keeps missing, the harness stops retrying and asks for a full-file rewrite instead of dead-ending. Stalls. Reading without editing, or repeating itself, trips a nudge to make one small, testable change and then run the tests. Messy output. ANSI is stripped, and a tool call the model wrote as text instead of a structured call still runs. Working through a task: inline tool markers, a live diff, and a test run. Watch it edit itself Here OptiQ Code, driven by a local 27B, adds a feature to its own tools.py and turns a planted test green. OptiQ Code editing OptiQ Code, on a local Qwen3.6-27B-OptiQ-4bit. SWE-bench-Lite results We measured this. An initial version of OptiQ Code, driving a 4B local model, resolved 36% of a SWE-bench-Lite subset, a strong result for a model that size, and produced a valid patch on every task. A comparable-budget baseline came up empty on 40%. A stronger local quant raises the resolve rate. The reliability features work the same regardless of model size. Approve every edit, or let it run Read-only tools always run. Anything that touches your files pauses for a single keystroke, so you stay in the loop by default. Trust the repo, or running unattended? Auto mode skips the prompts. Single-Enter approval before a write. Deny it and the model tries another way. Sessions persist per repo Every run is saved per repo. Pick up where you left off with optiq code -c and the model keeps its context. Export any session as a shareable trace with optiq code export, handy for a bug report or for building a dataset from your own work. Bring your best local model OptiQ Code drives whatever you serve, so it gets better as your model does. For the best results, use the OptiQ quant with the highest Capability Score your Mac can run. In general that is Qwen3.8-27B-OptiQ-4bit. Get started terminalbash $ pip install mlx-optiq $ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit --idle-timeout 300 $ cd my-project && optiq code # or headless, for scripts and CI $ optiq code -p "Fix the failing test in parser.py" The OptiQ Code guide covers approval modes, sessions, the tool set, and headless use. The product page has the full picture. --- ## OptiQ Lab Source: https://mlx-optiq.com/docs/lab Workflow · Lab OptiQ Lab A local web UI for everything you'd do with a model on your laptop: chat with it, compare two side by side, browse and load quants, quantize a new one, fine-tune a LoRA adapter, and build a dataset. One command boots the UI and an OpenAI-compatible API server together. Install + launch shellbash pip install mlx-optiq optiq lab # UI on :7860, API on :8080 First launch asks you to set a password (single user, Argon2-hashed, JWT session). The Lab binds to 127.0.0.1 unless you pass --host 0.0.0.0. What's inside Chat: streaming playground with local tools, self-healing tool calls, chat-with-files (RAG + citations), and a Canvas that renders HTML the model writes. Model Arena: two models, one prompt, side by side, with tokens/sec. Hub: browse published OptiQ quants, search Hugging Face, list local models, load any of them with one click. Quantize: the sensitivity + knapsack + convert wizard. Fine-tune: sensitivity-aware LoRA on any OptiQ quant. Build dataset: turn pairs / docs / code / seeds into training JSONL. Hugging Face token Save a write-scope token once in Settings → Hugging Face and the Quantize / Fine-tune / Dataset workflows reuse it for one-click HF push. The token is encrypted at rest with a Fernet key derived from your Lab password. How it's structured The Lab is a Flask app (optiq.lab.create_app) on its own port. optiq lab also boots mlx_lm.server in a daemon thread, so it's one process, two ports: UI on 7860, API on 8080. Background work (quantize, train, dataset generation) runs in multiprocessing.Process workers, progress streamed to the UI over SSE. Why localhost-only by defaultThe Lab binds to 127.0.0.1 unless you explicitly pass --host 0.0.0.0. Password protection is enforced either way, but please don't expose it to the public internet; this is a local development tool, not a hosted service. --- ## Chat Source: https://mlx-optiq.com/docs/lab/chat Lab · Chat Chat A streaming playground against whichever model the API is serving, with local tools, retrieval over your files, and live HTML rendering. Tools Three built-in tools the model can call locally: web_search: DuckDuckGo search plus a url mode to fetch a page as compact markdown. No API key. python: runs Python in a three-tier sandbox. Strongest first: a container with no network (Docker, Podman or Apple container), then macOS sandbox-exec, then a plain subprocess with an rlimit. AST checks block os.system, signal tampering, and direct network calls. Matplotlib charts the model saves render inline. terminal: a bash one-liner in the same sandbox. Dangerous commands (sudo, curl, rm) are rejected at command position with token-aware parsing. Local models often emit malformed tool calls. The Lab heals six common shapes (Hermes tags, fenced JSON, bare objects, trailing commas, fancy quotes, function-call style) before dispatching, flagged with a healed chip. On Linux and Windows, install Docker or Podman. Only macOS ships an OS sandbox, so without a container runtime these two tools fall back to running the model's code as you, in a temp directory, with the AST and blocked-command checks as the only guard. Installing a runtime switches on the container tier by itself. The active tier is shown in the Lab, and Installation has the full matrix. Self-healing tool calls The tool loop is server-side, capped at 25 turns. Consecutive identical successful calls de-duplicate to a nudge instead of re-running the sandbox. On top of that, a per-call retry budget stops the model re-running the same failing call forever: three attempts with the same arguments, then it's told to change course (different tool, different arguments, or answer with what it has), and the tool card shows a retry limit badge. If the model still won't finish after 25 turns, the orchestrator forces one final tools-disabled re-prompt so you get a text answer, not a hard error. Cloud & remote models Chat runs against the model your local optiq serve is hosting by default. To use a hosted model instead, set Model source to cloud API in Model & params and fill in the endpoint. It works with any OpenAI-compatible API, such as OpenRouter. Model source set to cloud API, with an OpenRouter base URL and key. API base URL: the provider's endpoint, for example https://openrouter.ai/api/v1. API key: your own key for that provider. It is kept in this browser and sent to your local Lab, which makes the request. The server never writes it to disk. Model: set the Model field to the provider's id, for example z-ai/glm-5.2. Tools, JSON mode, and chat-with-files behave the same against a cloud model as against a local one. The Lab proxies the request, so the key stays on your machine. This covers the Chat tab; quantizing and fine-tuning still work on local models. To drive a cloud model from the terminal instead, see OptiQ Code with a cloud endpoint. JSON mode Set JSON mode in Model & params to constrain the reply to valid JSON: any valid JSON, or match schema with a JSON Schema you paste into the box that appears. The decode is masked to only the tokens that keep the output valid (via lm-format-enforcer), so the reply always parses, no retries. Tools are off while JSON mode is on, since a single constrained response and the tool loop are mutually exclusive. The Lab's server installs the constraint automatically. See the structured output docs for the underlying response_format the Lab sends. Chat with files (RAG with citations) Attaching a non-image file indexes it instead of dumping the whole document into the prompt. A dependency-free BM25 retriever pulls the chunks most relevant to each question, prepends them with [n] citation markers, and the Lab renders a sources panel under the answer. Only the retrieved chunks enter the context, so a long PDF costs a few hundred tokens per turn instead of overflowing the context window. Artifacts An assistant message containing a fenced html block shows a Run card. Clicking it opens the page in a sandboxed allow-scripts iframe in a side panel, with restart, view-source, and close. Nothing the model writes executes until you ask for it, and the running page has room to run charts, small games, and interactive widgets. The iframe has no same-origin access, so the page stays isolated. A flight simulator written by a 122B sharded across two Macs, running in the artifact panel. Input + history Per-message controls for temperature, max tokens, and enable_thinking. Images attach as vision input on the multimodal quants. Multi-step tool runs collapse into a single "N tool calls" accordion. A Stop button cancels the run and SIGKILLs any sandbox subprocess. Saved chats persist to ~/.optiq/lab/chats/. --- ## Deep Research Source: https://mlx-optiq.com/docs/lab/deep-research Lab · Deep Research Deep Research Ask a real question and a local model plans it, searches the web, reads the sources, and writes a cited Markdown report. The research trail never leaves your machine, and nothing is paid to an external deep-research provider. Turn on the Deep Research toggle in the Chat composer and send a question. Instead of one answer the model runs a multi-step loop: it drafts, searches to fill the gaps in that draft, folds cited findings back in, then writes a final report. You get a short answer in the chat and the full report in the artifact panel. Everything runs against whatever model the Lab is serving, local or cloud. How it works The workflow is draft-centric, following TTD-DR (Test-Time Diffusion Deep Researcher). A naive pipeline searches once and summarizes. This one keeps a working draft and lets the draft drive the search, so each round targets what the report still lacks: Plan: the model outlines what a thorough report must cover. Noisy draft: a first pass from what the model already knows, with the uncertain parts marked for sourcing. Denoise loop: each round turns the draft's gaps into web searches, reads the top results, extracts citable evidence, and folds it back into the draft with inline [n] citations. Later rounds chase what is still weak or uncorroborated. Report: the cited draft is polished into a structured report: flowing analytical prose, a comparison table where it helps, and a Bottom line. Verify: a final pass grades every cited sentence against its sources and repairs the ones that overreach. The card in the transcript fills in as this runs: the plan, each source as it is read, the search count, and how many citations the verify pass repaired. A run on a local Gemma-4 12B: eight sources read, cited [1]–[8], in round two. Citations you can trust The hard part of research writing is keeping every claim tied to a source. Deep Research does this in three places rather than hoping the model gets it right: Verbatim evidence. When a source is read, extraction records a short quote copied word-for-word from the page alongside each claim, not a paraphrase. The drafter writes against those quotes and cites the one source a claim comes from. Verify and repair. After the report is written, each cited sentence is graded against its sources' quotes. A sentence that overreaches is rewritten to state only what the evidence supports; one nothing supports loses its citation. A generated reference list. The Sources section is built in code from the sources that were actually read, so the [n] markers always resolve, even when a small local model would otherwise garble its own list. On a fair citation-support check (each claim judged against the union of its cited sources, the way ALCE and DeepResearch-Bench measure it), this grounding roughly doubles citation accuracy over a plain draft-and-cite pipeline. The output: a response and a report Not every question wants a document. Deep Research always posts a short, direct answer in the chat, then attaches the full cited report to the artifact panel. Open it to read the rendered Markdown; the ↓ .md button downloads the report as a Markdown file. Model-authored HTML pages download the same way as .html. Running it Model. Deep Research uses whatever the Lab is serving. Point Model source at a local optiq serve or a cloud endpoint (any OpenAI-compatible API, key kept in your browser) exactly as in Chat. A capable local quant handles it; a run typically takes several minutes on-device. Search. Web search and page-fetch run locally: DuckDuckGo plus a fetcher that prefers a paper's full HTML over its PDF, so arXiv sources read cleanly. No search-API key. Budget. Rounds, queries per round, sources per query, and report length default lighter for local models so a run stays in the minutes, and can be tuned per request. Stop. The run is cancellable; it stops at the next step boundary. --- ## Model Arena Source: https://mlx-optiq.com/docs/lab/arena Lab · Arena Model Arena Compare two models on the same prompt, side by side, with tokens/sec for each. Model A runs on the Lab's main API server; model B runs in a second server on port + 1, started on demand. The obvious use is an OptiQ mixed-precision quant against a uniform 4-bit one, or a quant against its bf16 base, answering live in the same window so you can see the quality and speed difference at once. Pick a model in each column (from your published OptiQ quants or local converts), load them, type one prompt, and hit Compare. Both panes stream their answer and report tokens/sec and the token count. Reasoning models are asked to answer directly (thinking disabled) so the panes show a real answer rather than burning the budget on hidden chain-of-thought. Resident at onceBoth models stay loaded while the Arena is open, so it's best with small or fast models. On a tight machine, compare two quants of the same small base. --- ## Hub Source: https://mlx-optiq.com/docs/lab/hub Lab · Hub Hub Find a model and load it without copy-pasting ids into Settings. The Hub shows three things, each with one-click load-to-server and new-chat actions: Published OptiQ quants: auto-discovered from mlx-community, with family, size, bit profile, and download count. Hugging Face search: any MLX-compatible model on the Hub; filter to OptiQ quants with a toggle. Local models: anything you've converted locally (a directory with optiq/metadata.json), with a vision badge when it carries an optiq_vision sidecar. Loading a model reuses the same hot-swap path as Settings → Server: the running model is switched in place (about 5–30 seconds depending on size) without restarting the Lab. --- ## Quantize Source: https://mlx-optiq.com/docs/lab/quantize Lab · Quantize Quantize A four-step wizard around the OptiQ convert pipeline. Paste an HF model id → pick target BPW + reference mode → watch live progress on the sensitivity, knapsack, and convert phases → save locally and optionally push to your HF account with one click. The wizard detects supported architectures (Qwen3.5/3.6, Gemma-4) and warns on untested ones. The per-layer sensitivity pass and multi-tier bit allocation are exactly the same as the CLI optiq convert; the Lab just gives them a progress UI and a save/push button. --- ## Fine-tune Source: https://mlx-optiq.com/docs/lab/finetune Lab · Fine-tune Fine-tune LoRA on top of any OptiQ quant, MLX-native, no PyTorch. Sensitivity-aware rank scaling (by_bits default) is baked in, so layers kept at higher precision get more adapter capacity. A live train-loss sparkline streams from mlx-lm's TrainingCallback. Save the adapter and push it to HF when you're done. See the LoRA fine-tuning guide for the underlying method and the CLI equivalent. --- ## Build dataset Source: https://mlx-optiq.com/docs/lab/dataset Lab · Build dataset Build dataset Twelve templates to turn pairs / docs / code / seeds / target text / scenarios into JSONL the fine-tune workflow can read. LLM-driven templates call the Lab's own API server, so generation runs against the model you have loaded. Reasoning is auto-disabled on the generation calls so a thinking model doesn't burn its budget on the block. SFT from QA pairs, DPO from preference pairs, Style transfer, Code completion, Self-instruct expansion, Format conversion. Prompt reconstruction: work backwards from a target paragraph; the assistant target stays verbatim so facts and formatting are preserved. Multi-turn chat synthesis, Tool-use traces (messages-format with a top-level tools field), RAG Q/A from documents. Reasoning trace (CoT) synthesis and Verified code generation (asserts run in the sandbox; rows tagged verified: true/false). Datasets push to HF as repo_type="dataset". --- ## Integrations Source: https://mlx-optiq.com/docs/integrations Workflow · integrations Integrations The same optiq serve process exposes three API protocols on one port. Most coding agents and IDE plugins speak one of them, so you can run any of these tools against your local OptiQ-quantized model with a tiny config change. Endpoints optiq serve --model --port 8080 exposes: /v1/chat/completions: OpenAI Chat Completions (default, used by most tools) /v1/messages: Anthropic Messages (set ANTHROPIC_BASE_URL) /v1/responses: OpenAI Responses (required by Codex; used by Cursor, Continue, Cline) All three endpoints accept Bearer tokens that start with sk-optiq-. The suffix is anything you want; the prefix is checked. Authorization header may be omitted for local-dev curl calls. Coverage matrix Tool API protocol OptiQ endpoint Verified version Claude Code Anthropic Messages /v1/messages 2.1.201 Codex OpenAI Responses /v1/responses 0.130.0 OpenCode OpenAI Chat Completions /v1/chat/completions 1.15.4 OpenClaw Anthropic Messages /v1/messages 2026.5.12 Hermes Agent OpenAI Chat Completions /v1/chat/completions 0.14.0 Mistral Vibe OpenAI Chat Completions /v1/chat/completions 2.21.0 Cursor OpenAI Responses /v1/responses same config as Codex "Verified version" is the build of each agent we ran the full wire test against on macOS (Apple Silicon). Newer versions should keep working; if you hit a regression please file an issue. Quickstart (any tool) terminalbash # 1. Start the server (any OptiQ-quantized model) $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --port 8080 # 2. Hand the tool these settings: # Base URL: http://localhost:8080/v1 # API key: sk-optiq-local (any string prefixed sk-optiq-) # Protocol: see "Coverage matrix" above for which endpoint the tool uses MTP-aware serving Add --mtp to enable in-checkpoint MTP speculative decoding for ~1.4-1.8× decode tok/s on Qwen3.5 / 3.6 family. Works transparently for all three endpoints; tools don't need to know about it. terminalbash $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 --port 8080 Why one server, three protocols Tools have fragmented across three competing API standards. Rather than make you pick one (or run three servers), optiq serve speaks all three from the same process. Internally everything funnels into the same generation loop, the Responses and Anthropic endpoints translate to/from OpenAI Chat Completions and reuse the existing handler. MTP and KV-quant apply to all three transparently. What if my tool isn't listed? If the tool can be pointed at a custom OpenAI-compatible base_url, it works with mlx-optiq out of the box. The matrix above just covers the tools we've tested end-to-end. Common candidates that work but aren't documented here yet: aider, set --openai-api-base http://localhost:8080/v1 Open WebUI, add as an OpenAI-compatible connection in settings LangChain / LlamaIndex / DSPy, set the OpenAI client's base_url Anything using the openai Python SDK, instantiate with OpenAI(base_url=..., api_key="sk-optiq-...") --- ## Claude Code Source: https://mlx-optiq.com/docs/integrations/claude-code Integration · Claude Code Claude Code Anthropic's Claude Code is a terminal-based coding agent that uses the Anthropic Messages API. Point it at optiq serve via ANTHROPIC_BASE_URL and it'll talk to your local OptiQ model instead of Anthropic's hosted Claude. 1. Install Claude Code terminalbash $ npm install -g @anthropic-ai/claude-code 2. Start optiq serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --port 8080 The default --anthropic flag installs the /v1/messages endpoint. The --mtp flag enables ~1.4-1.8× decode speedup on Qwen3.5 / 3.6 family. 3. Point Claude Code at OptiQ terminalbash # In your shell rc, or per-session: export ANTHROPIC_BASE_URL=http://localhost:8080 export ANTHROPIC_API_KEY=sk-optiq-local export ANTHROPIC_MODEL=mlx-community/Qwen3.5-9B-OptiQ-4bit $ claude That's it. Claude Code will route every message through your local optiq serve. To go back to hosted Claude, unset the three env vars. 4. Right-size the context window (auto-compact) Claude Code decides when to auto-compact the conversation by comparing the token usage the server reports against the context window it assumes the model has (~200k for a Claude model). Point it at a smaller-context local model and it won't compact until far past the model's real limit. The model overflows and generation fails first. --context-scale FACTOR fixes the timing: it multiplies the token counts in the reported usage by FACTOR, so Claude Code's "compact at N% of the window" logic fires at the right real-token point. Only the reported usage is scaled; generation, the KV cache, and the prompt are untouched. Compute the factor as (window Claude Code assumes) / (your model's context): terminalbash # a model with a 32k context, behind Claude Code's ~200k assumption: # 200000 / 32000 ≈ 6.25 $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --context-scale 6.25 \ --port 8080 Now Claude Code auto-compacts as the conversation approaches your model's real ceiling instead of overrunning it. Leave it at the default 1.0 when your model's context already matches (or exceeds) what the client assumes. Notes Tool use: Qwen and Llama models that emit ... blocks are translated into Anthropic tool_use content blocks transparently. Models without native tool-call training (most small ones) won't drive Claude Code's full agentic loop; pick 9B+ for serious coding work. Streaming: works out of the box. Claude Code shows tokens as they arrive. Prompt caching: multi-turn reuse is automatic. After the first turn, the server reuses the KV of the shared conversation prefix and prefills only the new tokens, so every turn is near-instant to first token (a ~4× TTFT cut on a ~4k context, growing with model size). No configuration; see Prompt caching. Thinking on/off by name: reasoning models expose a thinking toggle as a model-id suffix. Set ANTHROPIC_MODEL=…-OptiQ-4bit:no-think for direct answers (faster, no rambling) or :think for full reasoning, with no extra request fields. Model id is forgiving: optiq serve is single-model by default, so if you forget to set ANTHROPIC_MODEL (Claude Code then sends claude-…) or send a basename, it's served the one local model rather than 404ing. Pass --allow-model-switch only if you want one server to hot-swap between cached quants. Auth token: any string starting with sk-optiq- works. Mirrors Unsloth's sk-unsloth-* convention. ?beta=true query string: Claude Code appends ?beta=true to /v1/messages for the prompt-caching beta. Our endpoint strips the query string and routes the request normally, so the beta header is a no-op on the server side without breaking the wire. Verified: tested against Claude Code 2.1.201 on macOS (Apple Silicon). Running Claude Code locally Claude Code is a capable terminal coding agent. Running it against a local model gives you the same UX locally with no API cost. The OptiQ + Qwen3.5-9B-MTP combination is fast enough for a fluid edit-and-run loop on M3 Pro and up. --- ## Codex Source: https://mlx-optiq.com/docs/integrations/codex Integration · Codex Codex OpenAI's Codex CLI uses the OpenAI Responses API exclusively (Chat Completions was deprecated for Codex in 2026). optiq serve exposes /v1/responses by default so Codex talks to your local OptiQ-quantized model with a one-block config addition. 1. Install Codex terminalbash $ npm install -g @openai/codex 2. Start optiq serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --port 8080 The default --responses flag installs /v1/responses; --mtp adds in-checkpoint MTP speculation for ~1.4-1.8× decode speedup on Qwen3.5 / 3.6 family. 3. Configure Codex Edit ~/.codex/config.toml and add: ~/.codex/config.tomltoml [model_providers.optiq] name = "OptiQ Local" base_url = "http://localhost:8080/v1" env_key = "OPTIQ_AUTH_TOKEN" wire_api = "responses" requires_openai_auth = false [profiles.optiq] model_provider = "optiq" model = "mlx-community/Qwen3.5-9B-OptiQ-4bit" Then export the auth token and launch Codex with this profile: terminalbash export OPTIQ_AUTH_TOKEN=sk-optiq-local $ codex -p optiq Notes wire_api = "responses" is required. Codex no longer accepts wire_api = "chat". Tool calls: Codex relies on function-calling for its edit / run / search loop. Models without robust function-calling training won't drive the full agent. Qwen3.5-9B-OptiQ and up handle this well; smaller models work for plain chat only. Built-in tools (web_search, file_search, computer_use): silently dropped by our shim. Codex's local-tool stack (apply_patch, shell, etc.) runs in the CLI itself and works fine. Streaming: works. Codex relies on response.output_text.delta and response.completed events; both are emitted in spec-compliant order. Verified: tested against Codex v0.130.0 on macOS (Apple Silicon). Codex + MTP on Apple Silicon Codex's edit-run-review loop is decode-heavy. The MTP speculation in --mtp stays at ~70% acceptance on Qwen3.5/3.6 family for typical code-edit prompts. Pairs well with longer max_tokens settings since the MTP head amortizes the per-token cost. --- ## OpenCode Source: https://mlx-optiq.com/docs/integrations/opencode Integration · OpenCode OpenCode OpenCode is an open-source AI coding agent that reads, edits, and runs code in your project. It speaks the OpenAI Chat Completions API, which optiq serve exposes by default. 1. Install OpenCode terminalbash # macOS / Linux: $ curl -fsSL https://opencode.ai/install | bash # or via Homebrew: $ brew install sst/tap/opencode # or, project-local via npm (handy for CI / pinned versions): $ npm install opencode-ai 2. Start optiq serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --port 8080 3. Configure OpenCode OpenCode reads custom providers from opencode.jsonc, global at ~/.config/opencode/opencode.jsonc, or per-project in the workspace root (overrides global). Drop in: opencode.jsoncjson { "$schema": "https://opencode.ai/config.json", "provider": { "optiq": { "npm": "@ai-sdk/openai-compatible", "name": "OptiQ Local", "options": { "baseURL": "http://localhost:8080/v1", "apiKey": "sk-optiq-local" }, "models": { "qwen-9b": { "name": "Qwen3.5-9B (OptiQ)" } } } }, "model": "optiq/qwen-9b" } The key under models (here qwen-9b) becomes the model id OpenCode sends in the request body. optiq serve ignores the body's model field and serves whichever model you started it with, so any short alias works. Do not put an absolute filesystem path in the model key (OpenCode would treat it as a literal directory and try to read files relative to it). Launch OpenCode in your project and start chatting: terminalbash # interactive TUI: $ opencode # or one-shot non-interactive: $ opencode run --model optiq/qwen-9b "list the python files here and tell me what main.py does" Notes Tool use: OpenCode uses function calling for file edits and shell commands. Qwen3.5-9B-OptiQ and up handle this reliably; smaller models will chat but call tools unreliably. Streaming: works. MTP works transparently: OpenCode sees a normal OpenAI endpoint; the speculation happens server-side. Verified: tested against opencode 1.15.4 on macOS (Apple Silicon). --- ## OpenClaw Source: https://mlx-optiq.com/docs/integrations/openclaw Integration · OpenClaw OpenClaw OpenClaw is an open-source AI agent interface that connects to a model server to run tasks across your project. It speaks the Anthropic Messages API, which optiq serve exposes via /v1/messages. 1. Install OpenClaw terminalbash # macOS / Linux / WSL: $ curl -fsSL https://openclaw.ai/install.sh | bash # Windows PowerShell: PS> iwr -useb https://openclaw.ai/install.ps1 | iex # or, project-local via npm: $ npm install openclaw 2. Start optiq serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --port 8080 3. Configure OpenClaw Drop this into ~/.openclaw/openclaw.json. If you already have other providers configured, add "mode": "merge" next to "providers" so OpenClaw splices OptiQ in without replacing them: ~/.openclaw/openclaw.jsonjson { "models": { "providers": { "optiq": { "baseUrl": "http://localhost:8080/v1", "apiKey": "sk-optiq-local", "api": "anthropic-messages", "models": [ { "id": "qwen-9b", "name": "Qwen3.5-9B (OptiQ)" } ] } } } } Keys are camelCase (baseUrl, apiKey), api is "anthropic-messages", and models is an array (not a dict). The id is a short alias OpenClaw uses to address the model. optiq serve ignores the request body's model field and serves whatever you started it with. Validate with openclaw config validate after saving. Sanity-check the wire with a one-shot: terminalbash $ openclaw capability model run --local \ --model "optiq/qwen-9b" \ --prompt "What is 2+2? Just the number." Notes Same endpoint as Claude Code: both tools use /v1/messages. You can run both against the same optiq serve simultaneously. Tool use: Qwen/Llama-style blocks are translated to Anthropic tool_use blocks; OpenClaw's agent loop works on 9B+ models. Verified: tested against OpenClaw 2026.5.12 on macOS (Apple Silicon). --- ## Hermes Agent Source: https://mlx-optiq.com/docs/integrations/hermes-agent Integration · Hermes Agent Hermes Agent NousResearch's Hermes Agent is an open-source autonomous agent that connects to a model endpoint, executes tasks, and improves over time via memory and learned skills. It speaks the OpenAI Chat Completions API, which optiq serve exposes by default. 1. Install Hermes Agent terminalbash # Official installer (macOS / Linux): $ curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash $ source ~/.zshrc # or ~/.bashrc $ hermes --version Prefer not to curl | bash? Clone manually: git clone https://github.com/NousResearch/hermes-agent && cd hermes-agent && pip install -e . 2. Start optiq serve terminalbash $ optiq serve \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --mtp --mtp-depth 2 \ --port 8080 3. Point Hermes at OptiQ Easiest path is the interactive wizard, which writes ~/.hermes/cli-config.yaml for you: terminalbash $ hermes setup # pick "custom" as provider, paste http://localhost:8080/v1 as base URL, # paste your model id, and sk-optiq-local as API key. $ hermes # start interactive chat Or skip the wizard with two env vars and a one-shot: terminalbash export CUSTOM_BASE_URL=http://localhost:8080/v1 export OPENAI_API_KEY=sk-optiq-local $ hermes chat -q "List the python files here." \ -m mlx-community/Qwen3.5-9B-OptiQ-4bit --provider custom --yolo Confirm the endpoint with hermes status; Provider: Custom endpoint should appear. Notes Tool use is core to Hermes: the agent's learn-from-skills loop depends on function-calling. Use a model trained with tool calling (Qwen3.5-9B-OptiQ and up, Hermes-3-Llama variants). Long-running sessions: Hermes accumulates memory in long context. Pair with --kv-config (see KV-quant serving) for decode speedup at 16k+. Streaming: works. --yolo: bypasses the per-tool approval prompt, required in non-interactive mode. Drop it for interactive sessions where you want to vet each tool call. Verified: tested against Hermes Agent v0.14.0 (2026.5.16) on macOS (Apple Silicon). --- ## Blog Source: https://mlx-optiq.com/blog Engineering · research · releases Blog Release notes, methodology dives, and benchmarking deep-dives. New posts land alongside major releases and research findings. 2026·08·18 Quantizing a mixture-of-experts model on MLX On gemma-4-26B-A4B the routed experts are 15.1 GB of a 17.6 GB model, fused 128 to a tensor. How sensitivity is measured on that, a flag that silently did nothing on every Gemma-4, and a prefetch that cost 8% of decode throughput. engineering 2026·08·07 A world model you can drive, hidden in mlx-optiq optiq game pulls 86 MB and opens a kart racer with no track, no physics and no renderer. A 131M diffusion model paints every frame from the last seven and the keys you are holding, at about seven frames a second on MLX. engineering 2026·08·03 Mage-VL: our first video model, running on a Mac Microsoft's 5B vision-language model reads images and video. Quantized by OptiQ, it runs on a Mac in 3.7 GB with no PyTorch. The language tower goes to mixed 4/8-bit; the vision tower stays bf16 in a sidecar. Release 2026·07·25 OptiQ Code vs opencode: a local coding-agent benchmark The same local model driven by two coding agents through one open benchmark. OptiQ Code and opencode solve the same openbench tasks, but OptiQ Code moves about 5x fewer tokens. Benchmark 2026·07·16 The best coding agent for local models on a Mac OptiQ Code is a terminal coding agent that drives whatever optiq serve is serving, offline, on your Mac. Engineered for weak local models: never an empty patch, edit-apply resilience, a stall-proof loop. 36% of a SWE-bench-Lite subset with a 4B, state of the art at that size, with a valid patch on every task. Product 2026·07·10 A 122B model, two Macs, and a flight simulator Qwen3.5-122B-A10B ships as 244 GB of bf16 weights. A 2-bit quant of it (42.8 GiB) now runs fully resident across a 36 GB M3 Max and a 24 GB M4 over Thunderbolt, at 20.5 tok/s, from one optiq cluster serve. Here it writes a flight simulator and flies it in the browser. Plus the memory rules that decide whether a ring runs at 20 tok/s or 0.1, and the four wrong explanations we chased to find them. Engineering 2026·06·20 Fine-tuning a vision model on a Mac OptiQ now fine-tunes the language tower of a quantized VLM on image+text data, locally on a 24 GB Mac. A LoRA on Qwen3.5-0.8B-OptiQ-4bit lifts ChartQA exact-match from 26 to 40%. The two walls we hit (memory ratcheting on variable image sizes, and mode collapse on short targets) and the fixes (uniform letterbox + gradient checkpointing, gradient clipping + lower LR), plus a Lab dataset template and Fine-tune wizard for the whole flow. engineering 2026·06·19 A 122-billion-parameter model, on a laptop mlx-optiq 0.2.5 runs Qwen3.5-122B-A10B on a 36 GB Mac by streaming its mixture-of-experts off SSD: 12 GB stays resident, the other 35 GB is read one expert at a time. The quant is a 2-bit static allocation (44 GB on disk), the new no-measurement method that assigns bits by architecture and matches the calibration method on small models at 125× the convert speed. The model wrote a working Flappy Bird and ran it. release 2026·06·06 mlx-optiq can see: image + text on Gemma-4 and Qwen3.5/3.6 Image+text on the Gemma-4 and Qwen3.5/3.6 families. One bf16 vision sidecar makes the same published repo load text-only under stock mlx-lm and full image+text under OptiQ, with no separate vision build and no mlx-vlm runtime dependency. Three vision architectures (SigLIP, the encoder-free Gemma-4 12B unified, and the Qwen3-VL tower) vendored and reproduced bit-exact (max|Δ|=0.0). Upload a picture in the Lab, or send an image_url to optiq serve: counting, shapes, colors, OCR, charts, and spatial questions all answer correctly. release 2026·06·01 A 1B humanizer that matches human writing on an AI detector Stacked SFT + DPO LoRAs on MiniCPM5-1B-OptiQ-4bit close 100% of the gap to human writing on the RADAR AI detector. P(AI) drops from 0.51 (source) to 0.37, exactly matching the human reference. The recipe uses OptiQ 0.1.4's --mount-adapter for textbook DPO continuation and per-request adapter stacking at serve time. Two 120 MB adapters on a 875 MB base, all local on a 24 GB Mac. engineering 2026·05·22 Gemma-4 spec decoding on Apple Silicon First MLX port of Google's Gemma-4 -assistant drafter. 1.18x decode geomean across five prompt categories on E4B with the 4-bit OptiQ quant, 31% acceptance, γ=1 greedy. Two RMSNorm-shaped formula bugs that took us from 0% to 33% acceptance, plus the bf16 multi-token verify artifact we cannot route around. Wired into OptiQ Lab Server as "Spec drafter". research 2026·05·21 Tools in OptiQ Lab chat: local web search, sandboxed Python, and a terminal v0.1.0 ships three tools the model can call locally: DuckDuckGo search, Python in an AST-checked sandbox, and a bash terminal in the same sandbox with token-aware command blocking. Healer for six malformed tool-call shapes. 25-turn budget with duplicate-call de-dup and a budget-exhausted re-prompt. Stop button that SIGKILLs the running subprocess. Matplotlib output renders inline. release 2026·05·21 When 4-bit KV cache uses more memory than fp16, and how OptiQ fixes it Stock mlx-lm 4-bit KV cache on a 24 GB Mac at 32k context actually peaks higher than fp16 (16.35 GB vs 11.51 GB). OptiQ's streaming converter and FlashAttention-2 N-tiling drops u4 peak to 7.60 GB, 34% below fp16. At ±2% fp16 speed parity. Mixed-precision KV scores 33% better than uniform 4-bit at the hash-hop hops=3 differentiation point. research 2026·05·19 Getting MTP to actually work on Apple Silicon Wiring Multi-Token Prediction speculative decoding into OptiQ took three fixes: a duplicated model load, a wrong probabilistic verify, and untruncated sampling distributions. We landed at 1.20x / 1.32x / 1.40x on Qwen 4B / 9B / 27B with greedy decoding on a 24 GB M4 Mac, within 5 percent of unsloth's 1.4x on the same model class on an RTX 6000. Plus the math for why depth 2 does not help on Metal. research 2026·04·28 The mlx-optiq eval framework: six benchmarks, one Capability Score GSM8K-50 alone misses tool-calling and long-context regressions. The two-stage eval (KL + GSM8K-50 for triage, MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop for headlines) drives every quant we ship. Plus the auto-resolved KL reference, sandboxed code execution, and a single Capability Score that's the unweighted mean of all six. methodology 2026·04·28 optiq.jsonl: a six-domain calibration mix for mixed-precision quantization WikiText-2 measures prose; modern LLMs do prose, reasoning, code, agent loops, tool-calling, and constraint-following instructions. We replaced the calibration set with 40 hand-curated samples across all six domains, bundled inside the package, fully reproducible. Calibration data decides what a quant protects. engineering 2026·04·25 Gemma-4 lands on mlx-optiq: four sizes, +32 pp on the small one Adding Google's full Gemma-4 instruct lineup: e2b, e4b, 26B-A4B sparse MoE, and 31B dense. The +32-point GSM8K recovery on gemma-4-e4b is the cleanest mixed-precision win we have. Plus the shared-KV caveat that means you'll want Qwen for quantized-KV serving. engineering 2026·04·17 TurboQuant: postmortem on a research path we didn't ship We built rotated-space KV attention with a custom Metal kernel. The benchmarks looked good: 100 % needle retrieval at 4-bit vs 73 % for affine. We still chose affine for the shipping path. Plain writeup of the technique, the numbers, and why the marginal win didn't justify a parallel serving stack. postmortem 2026·04·08 Sensitivity-aware LoRA: fine-tuning that respects the bit budget The same per-layer signal that drives mixed-precision quantization also drives adapter rank. 8-bit-quantized layers get 2× the adapter rank of 4-bit-quantized ones at the same parameter budget. Validation loss drops 12 % in head-to-head A/Bs. Plus the empirical training-ceiling map for a 36 GB Mac across all 10 supported models. engineering 2026·03·20 Not All Layers Are Equal: mixed-precision quantization for weights and KV cache on Apple Silicon The research foundation behind mlx-optiq. Some layers are 56× more sensitive than others. The KV cache becomes the dominant memory cost at long contexts. Mixed-precision recovers what uniform 4-bit drops; mixed-precision KV fixes the perplexity collapse uniform 4-bit causes. research --- ## Fine-tuning a vision model on a Mac. Source: https://mlx-optiq.com/blog/vision-lora-on-a-mac Engineering · June 20, 2026 Fine-tuning a vision model on a Mac. Topic Vision · Fine-tuning Reading time 7 min Related vision support OptiQ now fine-tunes the language tower of a quantized vision-language model on image+text data, entirely on a 24 GB Mac. A LoRA on Qwen3.5-0.8B-OptiQ-4bit trained on ChartQA lifts strict exact-match from 26% to 40% and output similarity from 0.39 to 0.60 on held-out charts. The vision tower stays frozen; only the language tower learns. It needs no GPU or cloud, and you can run the whole flow (build the dataset, train the LoRA) from the OptiQ Lab. The result 80 held-out ChartQA questions, base versus the LoRA, both with images letterboxed to a 512px canvas, scored three ways: ChartQA relaxed accuracy (substring or numeric-within-5%), strict exact-match, and a similarity ratio against the ground-truth answer string. Metric Base + LoRA Δ Relaxed accuracy 50.0% 55.0% +5.0 pp Exact match 26.2% 40.0% +13.8 pp Similarity 0.385 0.598 +0.21 You can see the difference in the samples. Asked for a value, the base answers "There are 10 food items shown in the bar graph" (right idea, wrong format, fails strict matching); the fine-tuned model answers "3". Exact-match nearly doubles. Relaxed accuracy moves a smaller +5 points: the bigger win is format and consistency, with a real gain in correctness on top. Why this matters OptiQ already runs images through quantized VLMs. This is the other half: adapting one to your data. The vision tower is frozen and the language tower trains, which is the common, low-risk VLM fine-tune (domain VQA, OCR, captioning), and it now fits on a 24 GB Mac. Build the dataset in the Lab The OptiQ Lab's dataset builder gained a VLM image+text template. Point it at any image+text dataset on the Hub (here, ChartQA), map the columns, and it standardizes and exports the JSONL the trainer reads. The VLM image+text template. Note Standardize to 512px square: every image is letterboxed to one fixed canvas. That single choice is what keeps training memory bounded, so the Lab makes it the default. It streams the dataset, letterboxes each image, and writes {image, prompt, completion} JSONL locally (optionally pushed to the Hub). Then fine-tune, also in the Lab The Fine-tune wizard gained a Vision objective. Pick it, point at the dataset you just built, and the vision-safe defaults are pre-filled: a 512px canvas, scale 8, gradient checkpointing on. Selecting Vision sets scale to 8 automatically (the Qwen3.5/3.6 hybrid family collapses at the text-SFT default of 20) and surfaces the image-canvas field. Start training, and the loss chart streams as it trains. Or three commands terminal · dataset, train, servebash $ pip install 'mlx-optiq' # 1. prep an image+text jsonl (one row per {image, prompt, completion}) # images letterboxed to a uniform canvas; see scripts/prep_chartqa.py $ optiq lora train mlx-community/Qwen3.5-0.8B-OptiQ-4bit \ --vision --data ./chartqa/train.jsonl \ --rank 8 --iters 800 --learning-rate 5e-5 \ --output ./chartqa-lora $ optiq serve --model mlx-community/Qwen3.5-0.8B-OptiQ-4bit \ --adapter ./chartqa-lora --vision auto-engages when the model ships an optiq_vision sidecar. The vision tower is frozen; LoRA trains the language tower's attention and MLP projections with gradient checkpointing on. How the training fits on a Mac Two defaults keep a VLM LoRA inside 24 GB, both on automatically. Every image is letterboxed to a uniform square canvas, which holds the per-step memory constant. Gradient checkpointing recomputes each decoder block's activations in the backward pass instead of storing them, which fits the Qwen3.5 hybrid (gated-delta) attention backward in a few gigabytes. The vision defaults also set gradient clipping and a 5e-5 learning rate, which keep training stable on the short answers chart datasets use. A full ChartQA run holds above 6 GB free throughout. Try it The VLM image+text dataset template and the Vision fine-tune objective ship in the OptiQ Lab, and optiq lora train --vision is in the CLI. Point either at a 0.8B VLM quant and your own image+text data, and you have a local vision fine-tune that fits on a Mac. --- ## A 122-billion-parameter model, on a laptop. Source: https://mlx-optiq.com/blog/stream-122b-on-a-mac Engineering · June 19, 2026 A 122-billion-parameter model, on a laptop. Topic SSD expert streaming Reading time 6 min Related quant methods Qwen3.5-122B-A10B ships as 244 GB of bf16 weights. Below, it is playing a Flappy Bird game it wrote, on a 36 GB MacBook, with no GPU and no cloud. The quant is 44 GB on disk. While it runs, 12 GB sits in RAM. The other 35 GB streams off the SSD, one expert at a time. This is the headline of mlx-optiq 0.2.5: large mixture-of-experts models that don't fit in memory now run anyway. Here is how, and why we report a game instead of a benchmark for quants this aggressive. The experts live on disk A mixture-of-experts model is mostly experts. Qwen3.5-122B-A10B has 256 experts per layer across 48 layers, and 98% of its parameters sit in those tensors. Only 8 of the 256 fire for any given token. The rest are loaded into RAM but never read for that token. mlx-optiq 0.2.5 keeps the experts on the SSD instead and reads only the active ones, per token, as the router selects them. Attention, the router, the embeddings, and the few sensitive blocks stay resident; the experts stream in on demand. Part of the model Size Where it lives Attention, embeddings, router, scales 10.7 GB resident in RAM 256 experts × 48 layers 35 GB streamed from SSD Peak while generating 12 GB on a 36 GB Mac The model on disk is six times larger than what it holds in memory. Decode runs at about 5 tokens a second. The active experts are read by byte range from the shards on each step, with the small bf16 scales kept resident, so the residency stays flat no matter how big the model on disk gets. The 2-bit quant Fitting 122 B in 44 GB means 2-bit weights for the experts. 2-bit is lossy. Spend it carelessly and the model breaks. mlx-optiq's usual method measures each layer's sensitivity with a calibration pass: perturb one layer, watch the output distribution move, allocate bits where it moves most. That works beautifully up to about 30 B. On a 122 B MoE it would run for days, and it needs the full-precision model resident as a reference, which defeats the purpose. So 0.2.5 adds a second method, static. It does no measurement at all. It assigns bits from architecture alone, using the priors the calibration pass keeps rediscovering: the embedding and output head, the first and last block, attention, and the MoE router get the high bits; the dense MLP and the routed experts stay low. We checked that skipping the measurement does not cost quality. On the smallest base, Qwen3.5-0.8B, static lands the exact same GSM8K as the full calibration method, 34.5%, while converting 125 times faster and at a lower bit-width. For a typical transformer, the structure carries most of the signal. The expensive measurement is insurance for the cases structure cannot predict, and on a model this large that insurance is not affordable. For the 122 B, static put 4-bit on the router, attention, and the protected blocks, and 2-bit on the experts. The average comes out at 2.5 bits per weight. Why we show a game instead of a benchmark Scoring a 2-bit quant of a huge model runs into a wall. A full capability suite is thousands of generations. At 5 tokens a second through SSD streaming, GSM8K alone is hours, and the code and long-context metrics are days each. You cannot run a six-metric suite on a streaming 122 B and call it a release. The field already settled on the alternative. When unsloth ships a 1.58-bit DeepSeek-R1, they do not post MMLU. They show the model writing a working Flappy Bird, or a spinning-heptagon physics sim, on hardware people actually own. The report is the size, the machine it runs on, tokens per second, and a coherence demo. So we asked the 2-bit 122 B for Flappy Bird in one HTML file. It returned 7.7 KB of self-contained markup: a canvas, a game loop, gravity, collision, a score counter. node --check passes on the script. The braces balance. An autopilot reading the game's own state threads the pipes and the score climbs, which a broken game cannot do. The gif at the top is that game, played. The demo is a 2-bit, 44 GB quant of a 122B model running on a 36 GB laptop, writing and running working code. Run it terminalbash $ pip install -U mlx-optiq # streaming turns on automatically for a MoE too big to fit resident $ optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit \ --stream-experts Open the Lab, ask for a game, and watch it render in the Canvas pane. The same playbook makes any large MoE laptop-runnable: convert with --method static --candidate-bits 2,4, smoke-test, ship the card with the demo. The two methods are laid out side by side in the sensitivity guide. --- ## Image + text on Gemma-4 Source: https://mlx-optiq.com/blog/vision-support Release · June 6, 2026 Image + text on Gemma-4 Topic Vision support Reading time 6 min Related vision guide mlx-optiq now adds image and text on the Gemma-4 and Qwen3.5/3.6 families. Hand it a photo, a chart, or a page of text, and the same OptiQ-quantized model that answers your text prompts now answers questions about the picture too. There is no separate model or build to manage, and no extra runtime dependency. Pre-built quants take image input, from a 0.7 GB Qwen3.5-0.8B that runs on any Mac up to the 27 B models. One published artifact The mechanism is a sidecar. OptiQ quantizes the language tower the way it always has; the vision and audio towers are written, at bf16, into a separate file that rides alongside the quantized shards: optiq_vision.safetensors. mlx-lm selects its weights with glob("model*.safetensors"), so it never matches the sidecar. Stock mlx-lm loads the repo as a text-only model; OptiQ loads it with the vision tower included. So there is exactly one artifact to publish and one to download. There is no text-versus-vision build to choose, and text-only requests skip the vision path entirely. Vision stays at bf16 on purpose: 4-bit vision tends to hurt OCR and fine detail, so the ecosystem keeps the encoder in higher precision. The language tower, which is where almost all of the size lives, is still fully OptiQ mixed-precision quantized. Loader Reads You get stock mlx-lm model*.safetensors Text-only model (sidecar ignored) OptiQ model*.safetensors + optiq_vision.safetensors Full image + text Vendored vision encoders mlx-optiq's value is in the language path: mixed-precision quantization, the MTP speculative decoder, mounted LoRA adapters, the mixed-precision KV cache. None of that lives in the vision libraries. So rather than take on mlx-vlm as a runtime dependency, we vendored just the vision encoders into mlx-optiq and kept decoding in mlx-lm. The vision front-end preprocesses the pixels, runs the vendored SigLIP tower, projects the result into the language model's hidden space, and scatters those soft tokens into the text-embedding sequence at the image-placeholder positions. From there mlx-lm decodes exactly as it does for text, with the same quantized weights, KV cache, and sampler. We checked the vendored path against mlx-vlm tensor for tensor. Feeding mlx-vlm's own pixel values through mlx-optiq's preprocessing, vision tower, and projection reproduces its outputs to a maximum absolute difference of zero (266 soft tokens on gemma-4-e2b). Two details had to be right for that to hold: mlx-lm's gemma4_text always rescales the incoming embeddings by embed_scale, so the vision features are pre-divided to compensate; and the per-layer inputs zero out the image-token positions before projection. See it in the Lab The OptiQ Lab's Chat tab takes images directly: attach a picture and ask a question about it. Everything below is gemma-4-e2b at 4-bit, the smallest model in the family, running locally on a Mac. Counting, shapes, and colors. A 4-bit model reads three red circles and one blue square, and gets both right. A natural-image description: two tabby cats on a bright pink surface. Reading a chart: how many bars, and which one is tallest. It handles the obvious vision tasks: counting objects, naming shapes and colors, describing a scene. It also reads text out of an image (we asked it to quote a sign and it returned the string exactly) and answers questions about a bar chart. All of it on the smallest Gemma-4, quantized to 4-bit. Two families, three vision architectures The same sidecar idea carries across families, but the vision towers do not look alike, and each one places the spatial understanding somewhere different. Gemma-4 e2b / e4b use a full SigLIP vision tower: a 16-layer transformer that encodes 16-pixel patches into soft tokens. The larger 26B and 31B use the same architecture at 27 layers and a wider hidden size. Both are self-contained, so the language backbone treats their soft tokens like any other embedding. Gemma-4 12B is different. It is a unified model with no separate vision tower at all: a light patch embedder feeds image tokens straight into the shared language backbone, which was trained to attend over them bidirectionally. Run it with the ordinary causal mask and OCR slips ("OPTIQ" becomes "OPTION"). Make the image-token span bidirectional, the way the model was trained, and the text comes back exactly right. The fix is a one-shot mask wrapper that leaves text and decode fully causal. Qwen3.5 and Qwen3.6 bring a third tower (the Qwen3-VL encoder, with its own 2D rotary positions and windowed attention) and a variable-resolution image processor. Both are vendored bit-exact against mlx-vlm. Here the spatial work happens inside the vision tower, so the visual tokens arrive already position-aware and the backbone needs nothing special. OCR is exact straight away. Family Models with image input Vision tower Gemma-4 e2b, e4b, 12B, 26B-A4B, 31B SigLIP, plus the encoder-free 12B unified Qwen3.5 0.8B, 2B, 4B, 9B, 27B, 35B-A3B Qwen3-VL encoder Qwen3.6 27B, 35B-A3B Qwen3-VL encoder Every preprocessing step and vision tower reproduces mlx-vlm's outputs to a maximum absolute difference of zero, so the soft tokens the language model sees are the same ones the reference implementation produces. And throughout, the text path is untouched: image code only runs when a request actually carries an image, so MTP speculation, LoRA, KV-cache quantization, and plain text generation behave exactly as before. How to use it Once a model carries the sidecar, optiq serve and the Lab turn on image support automatically. Point any OpenAI-compatible client at the server and send an image_url content part: terminalbash # serve a sidecar-equipped Gemma-4 quant; image support turns on by itself optiq serve --model mlx-community/gemma-4-e2b-it-OptiQ-4bit # then POST an image like any OpenAI chat request curl http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages":[{"role":"user","content":[ {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}, {"type":"text","text":"What is in this image?"}]}]}' The same command works for any sidecar-equipped quant, Gemma-4 or Qwen. Or skip the JSON and open the Lab: optiq lab, go to Chat, click attach, and drop in a picture. The full list of image-capable quants is on the models page, and the mechanism, supported families, and per-model details are documented in the vision guide. --- ## A 1B humanizer that matches human writing on an AI detector. Source: https://mlx-optiq.com/blog/humanizer-stacked-lora Engineering · June 1, 2026 A 1B humanizer that matches human writing on an AI detector. Topic Fine-tuning · Alignment Reading time 6 min Related sensitivity-aware LoRA Two LoRA adapters stacked on a 1B model close 100 % of the gap to human writing on the RADAR AI detector. Source AI drafts get 0.51 from RADAR. EditLens humans get 0.37. With the SFT + DPO LoRAs applied together on MiniCPM5-1B-OptiQ-4bit, the rewrites also come back as 0.37, on the same 200 held-out drafts. The whole stack runs on a 24 GB Mac. The result 200 held-out AI drafts from the EditLens ICLR 2026 corpus, rewritten by each system, then scored by RADAR-Vicuna-7B. Lower P(AI) means more human-like. Pipeline P(AI) ↓ Δ vs source Gap closed Source AI draft (Qwen3.5-4B + gemma-4-e4b) 0.51 n/a n/a MiniCPM5-1B + SFT humanizer LoRA 0.50 -0.01 7 % MiniCPM5-1B + SFT + DPO LoRAs stacked 0.37 -0.14 100 % Human reference (target) 0.37 -0.14 100 % Slop-phrase frequency (boilerplate patterns like "a testament to" and "underscores the importance of") drops from 0.6 per 1K tokens in the source to 0.0 in the stacked output. The human reference set has 0.1. The base model takes 875 MB on disk, each adapter another 120 MB. No 70B model and no API key required. The recipe in three commands OptiQ 0.1.4 ships every piece. The full pipeline is: Optional first stop: the Lab's dataset builder. The Style transfer + DPO from preference pairs templates produced the SFT and DPO datasets for this run. terminal · 1. quantizebash $ pip install 'mlx-optiq>=0.1.4' $ optiq convert openbmb/MiniCPM5-1B \ --target-bpw 5.0 --candidate-bits 4,8 \ --output ./optiq_mixed Sensitivity-aware mixed-precision quantization. Most layers land at 4-bit, the sensitive ones at 8-bit. Result is 875 MB and only 1.06 GB short of the bf16 base on Capability Score (eval framework). terminal · 2. train SFT, then DPO continuing from itbash $ optiq lora train ./optiq_mixed \ --data ./sft_dataset --method sft \ --preset large --iters 600 \ --output ./adapters/humanizer-sft $ optiq lora train ./optiq_mixed \ --data ./dpo_dataset --method dpo \ --preset large --iters 300 \ --mount-adapter ./adapters/humanizer-sft \ --output ./adapters/humanizer-dpo --mount-adapter is the standard SFT then DPO continuation recipe. It stacks a frozen SFT LoRA alongside a trainable DPO LoRA on every adapted layer. The DPO reference forward zeroes only the trainable scale, so the KL term in the loss is anchored against base + SFT, which is the SFT model. That matches how every modern alignment pipeline defines "DPO continuing from SFT". The saved adapter holds only the DPO delta. OptiQ Lab Fine-tune wizard, Hyperparameters step. Picking DPO swaps the learning rate to 5e-5 and surfaces the DPO defaults banner. terminal · 3. serve both, stacked per-requestbash $ optiq serve --model ./optiq_mixed \ --adapter ./adapters/humanizer-sft \ --adapter ./adapters/humanizer-dpo # request body activates both with the "+" operator: $ curl localhost:8080/v1/chat/completions \ -d '{"model":"...","messages":[...], "adapter":"humanizer-sft+humanizer-dpo"}' optiq serve mounts both adapters on the same base model. The request body's adapter field, given an a+b form, applies both LoRA residuals during a single forward pass instead of switching between them, and there is no model reload. The classic single-adapter syntax ("adapter": "humanizer-sft") still works. The sentinel "adapter": "base" bypasses adapter activation entirely. Use it for A/B comparisons from the same served process. The same multi-adapter mount, surfaced in the Lab. Settings → Server lists the registered adapters; once mounted they are picked per-request from the Chat surface. Why the stack beats either adapter alone The SFT adapter alone scores P(AI) = 0.50, barely better than the source. The DPO adapter on its own is meaningless. It was trained as a delta from SFT, not an absolute LoRA. Without SFT active, you're applying a small perturbation to the base model that doesn't recover the SFT distribution at all. The stack reproduces the training-time forward pass: StackedLoRALinear · trainingpython # during training (frozen SFT + trainable DPO): y = base(x) + sft_scale * (x @ sft_a @ sft_b) + dpo_scale * ((dropout(x) @ lora_a) @ lora_b) # at serve time (both mounted, "a+b" syntax): y = base(x) + sft_scale * (x @ sft_a @ sft_b) + dpo_scale * (x @ lora_a @ lora_b) So the math at training time and the math at inference time work out to the same expression with the same weights, which is why the held-out P(AI) lines up with what the training trajectory predicted. The same flow in OptiQ Lab If the CLI isn't your thing, the Lab UI walks through the same pipeline. After training finishes, the wizard's last step exposes adapter combine + model export + push-to-Hugging-Face as three optional checkboxes. Step 5 of the Fine-tune wizard. Combine merges two adapters, Bundle exports a model directory, Push ships it to the Hub. Try it Everything (base, both adapters, model card, held-out eval) is bundled into a single Hugging Face repo: mlx-community/humanizer-1B-OptiQ-4bit (~1.1 GB). Download once, serve with both adapters stacked: terminal · use the published artifactbash $ pip install 'mlx-optiq>=0.1.4' $ huggingface-cli download mlx-community/humanizer-1B-OptiQ-4bit \ --local-dir ./humanizer-1B-OptiQ-4bit $ optiq serve \ --model ./humanizer-1B-OptiQ-4bit \ --adapter ./humanizer-1B-OptiQ-4bit/adapters/humanizer-sft \ --adapter ./humanizer-1B-OptiQ-4bit/adapters/humanizer-dpo OpenAI-compatible endpoint on localhost:8080. Point any client (Open WebUI, Continue, your own scripts) at it, send "adapter": "humanizer-sft+humanizer-dpo" in the request body, and you have a local humanizer. An example, end to end One held-out rewrite from the eval set, pulled verbatim. The source is from the EditLens Qwen3.5-4B / gemma-4-e4b draft pool; the rewrite is what comes back when the stacked SFT + DPO LoRAs run on the same draft. source AI draftbefore The case of a prisoner at HMP Kilmarnock who was convicted of shoplifting and sought reintegration highlights the urgent need for dedicated throughcare support and collaborative efforts to ensure successful outcomes for individuals within the criminal justice system. Within this context, it is essential to address the complexities of reintegration processes and the importance of community-based programs. To accomplish this, a comprehensive approach must be adopted, focusing on both individual rehabilitation and community engagement. This involves creating structured programs that address the unique challenges faced by offenders and their families. Moreover, ongoing support and evaluation are critical to ensure that reintegration strategies are effective and tailored to the specific circumstances of each case. Ultimately, a holistic and patient-centered approach will be necessary to achieve the goal of successful reintegration and to reduce recidivism rates. stacked SFT + DPO rewriteafter A prisoner at HMP Kilmarnock who was convicted of shoplifting and sought to be reintegrated highlights the urgent need for dedicated throughcare support and collaborative efforts to ensure successful outcomes for individuals within the criminal justice system. Within this context, it is essential to address the complexities of reintegration processes and the importance of community-based programs. To accomplish this, a comprehensive approach must be adopted, focusing on both individual rehabilitation and community engagement. This involves creating structured programs that address the unique challenges faced by offenders and their families. Ongoing support and evaluation are critical to ensure that reintegration strategies are effective and tailored to the specific circumstances of each case. A holistic and patient-centered approach will be necessary to achieve the goal of successful reintegration and to reduce recidivism rates. The two slop-phrase openers (Moreover, Ultimately) are gone. The first sentence's nominalised opener (The case of a prisoner ... sought reintegration) collapses into the more direct A prisoner ... sought to be reintegrated. Every fact (HMP Kilmarnock, shoplifting, throughcare, recidivism) is preserved. Where to next The piece of OptiQ that unlocked this is --mount-adapter plus the multi-adapter serve, both shipped in v0.1.4. They also work for any other SFT then DPO continuation, not just humanization. If you have an SFT recipe and a preference dataset for any task (code style, brand voice, refusal behavior, anything), the same two commands give you a DPO LoRA that continues from your SFT instead of starting from scratch. Reference for the trainer, every --preset option, and the dataset format is in the LoRA fine-tuning guide. The multi-adapter serve and the "a+b" stacking syntax are in the serve docs. --- ## Gemma-4 spec decoding on Apple Silicon Source: https://mlx-optiq.com/blog/gemma-spec-decoding Research · May 22, 2026 Gemma-4 spec decoding on Apple Silicon Topic Speculative decoding Reading time 9 min Related MTP guide · MTP postmortem · Gemma-4 support post Qwen ships an MTP head inside every Qwen3.5 and Qwen3.6 checkpoint. OptiQ uses it to get 1.20x to 1.40x decode speedups (see last week's MTP post). Gemma-4 does not ship an MTP head. Google instead publishes a separate small model alongside the target, with the suffix -assistant, that you load as a draft model. The first time we tried to wire it up we got nowhere and parked it. This time we kept going. The end state is a generic spec runtime in optiq/runtime/spec/, a working -assistant drafter for Gemma-4 E4B, and a "Spec drafter" picker in the OptiQ Lab Server page that flips it on. Greedy γ=1, 1.18x decode geomean across five prompt categories, 31% acceptance. This is the writeup of how we got there, including the two RMSNorm-shaped bugs that took us from 0% to 33% acceptance and the bf16 numeric artifact we still cannot route around. Why the first attempt did not work When we first looked at Gemma spec decoding, we tried to make it fit the same shape as Qwen MTP. That meant treating the drafter as a head we could attach to the target and calling mlx-lm's built-in speculative path. Both assumptions were wrong. The Gemma -assistant drafter is not a head bolted onto the target. It is a 4-layer Q-only transformer with its own attention, MLP, norm, RoPE, and output layers. The thing that makes it Q-only is that those attention layers compute only the Q projection. K and V come from the target's KV cache, two specific donor layers, one sliding-window and one full-attention, picked by Google during training. The drafter is an EAGLE-family model in spirit, not an MTP one. This means three things had to be built fresh: A loader for the drafter weights that knows about the centroid-clustering output head, the per-block RoPE variants, and the Gemma-specific normalization. A KV viewer for the target that can produce typed K and V tensors in chronological order, accounting for the fact that one donor uses a RotatingKVCache (ring buffer, sliding window) and the other uses a plain KVCache (append-only, with stale tail). A spec loop that knows to feed the drafter the last emitted token's embedding scaled by sqrt(hidden_size), plus the target's last hidden state, plus the shared KV. mlx-lm does not provide any of these out of the box. We checked. There is an open issue tracking general drafter support but no implementation. Ollama merged a Gemma -assistant spec PR in February; their code is MIT-licensed and we used it as the algorithmic reference. Building the drafter The drafter weights load from the HuggingFace repo mlx-community/gemma-4-E4B-it-assistant-bf16 as a plain safetensors file. OptiQ implements the model class and reproduces the structure exactly: four blocks, three with sliding-window attention, one with full attention; an embedding-projection head that maps the target hidden into the drafter's smaller hidden; a centroid output head that splits the vocab into 2048 centroids of 128 tokens each and emits over top-K=32 centroids. The first version loaded the weights and emitted tokens. Acceptance was 0%: across thousands of drafted tokens, none matched the target's argmax. Bug one: RMSNorm formula Gemma 1, Gemma 2, and Gemma 3 use the "scale plus one" RMSNorm convention. The norm operator is x_normed * (1 + weight), where weight is a learned per-channel parameter that drifts from zero. This is what mlx-lm's older Gemma classes do. Gemma 4 changes this. The weight is stored as-is, no plus-one. The norm operator is x_normed * weight, full stop. We checked mlx-lm's gemma4_text.py after our forward passes were producing nonsense activations. The Gemma 4 weights are stored at typical scale, ranging from about 0.5 to 64, not the near-zero values that the plus-one convention assumes. Applying plus-one would multiply by 1.5 to 65 in places where the model expects 0.5 to 64. We instrumented the per-layer norm magnitudes and watched activations explode from 512 to 2181 across the four blocks. Replacing the formula with mx.fast.rms_norm(x, self.weight, self.eps) directly took us from 0% acceptance to 3%. Not great, but no longer zero, which meant the rest of the math was at least plausibly right. Bug two: layer scalar formula Gemma 4 has a per-block scalar applied as a residual gate. The block returns residual + h * layer_scalar. The drafter's checkpoint stores layer_scalar as a single learned scalar per block. We had it as h * (1 + layer_scalar), by analogy with the plus-one RMSNorm convention we just fixed. That was the second wrong assumption of the same kind. The actual operation in mlx-lm's gemma4_text.py:386-387 is plain h * scalar. Fixing this took us from 3% acceptance to 33%. Drafter output started looking like real Gemma output. Speedup landed at 1.21x on the math prompt. Bug three (still open): bf16 precision drift in multi-token verify Spec decoding's correctness check is byte-identity. If you turn off the drafter, the output of the spec loop has to be identical, token for token, to a normal greedy run. We benchmarked five prompts and reported the prefix length over which our spec output matched the baseline output: Prompt Speedup Acceptance Prefix match math 1.29x 37.5% 26 / 200 code 1.25x 34.0% 45 / 200 prose 1.18x 30.3% 24 / 200 dialogue 1.11x 29.5% 69 / 200 reasoning 1.06x 25.5% 200 / 200 Geomean 1.18x 31.4% The reasoning prompt matched all 200 tokens. The others drifted partway through. We sat on this for a while because if our spec loop was correct, the matching prefix should be 200 for every prompt, not just one. The root cause is in mlx-lm itself. When you feed two tokens to a Gemma 4 forward pass at once and look at the logits for the second position, you do not get exactly the same logits you would have gotten by feeding the two tokens one at a time. We instrumented this with a four-line probe: probe.pypython import mlx.core as mx from mlx_lm import load model, tok = load("mlx-community/gemma-4-E4B-it-4bit") prompt_ids = mx.array(tok.encode("Tell me about Apple Silicon."))[None] # Sequential: feed A, get logits at A; then feed B, get logits at B. cache_s = make_prompt_cache(model) _ = model(prompt_ids, cache=cache_s) log_A_s = model(mx.array([[100]]), cache=cache_s) log_B_s = model(mx.array([[200]]), cache=cache_s) # Together: feed [A, B] in one shot. cache_t = make_prompt_cache(model) _ = model(prompt_ids, cache=cache_t) log_AB_t = model(mx.array([[100, 200]]), cache=cache_t) print("diff at B:", mx.max(mx.abs(log_AB_t[0, 1] - log_B_s[0, 0]))) The output: diff at B: 0.679688. That is the maximum absolute difference between two logit vectors that should be identical. In bf16 magnitudes (top values around 51), 0.68 corresponds to about 1.3% relative drift. Most of the time the argmax still matches. Occasionally it does not. This is bf16 attention precision interacting with how mlx-lm blocks the multi-token forward pass. We have not traced exactly which intermediate is causing it (probably the softmax-over-attention numerator, but we did not confirm). What we know is that any greedy spec decoder built on mlx-lm's current Gemma 4 path will inherit this artifact. Our spec loop is algorithmically correct, as the reasoning prompt's 200-of-200 match demonstrates. The other four prompts hit a position where the verify's argmax disagreed with what the sequential baseline would have produced and from there they drift independently. This is a real correctness story to tell users honestly. Output stays effectively identical for tens of tokens, then can branch. For chat and code generation this is invisible. For exact-reproducibility workflows it is not. We label this greedy spec with bf16-precision drift, not lossless. What we shipped The runtime lives at optiq/runtime/spec/. It is independent of the existing optiq/runtime/mtp/ path, by design. Qwen MTP is mature and shipped; we did not want to refactor it under a new abstraction just to add Gemma support. The new module hosts the generic outer loop (draft K tokens, target verifies K+1, accept prefix, commit) plus per-architecture drafter adapters; for now the only adapter is GemmaAssistantDrafter. Future adapters can move in alongside. From the command line, the drafter routes through OptiQ Serve via the new install_assistant_drafter hook: serve.pypython from optiq.serve import install_assistant_drafter install_assistant_drafter( target_model_path="mlx-community/gemma-4-E4B-it-4bit", drafter_id="mlx-community/gemma-4-E4B-it-assistant-bf16", ) # then mlx_lm.server.main(); every /v1/chat/completions request now # routes through spec_generate transparently. From the OptiQ Lab Server page, the "Spec drafter (Gemma-4 family)" picker drops down with the published drafter. Pick the target, pick the drafter, Apply. The supervisor swaps the running model and the chat page picks up the spec path automatically. MTP and the spec drafter are mutually exclusive per loaded model; the UI greys out the alternate when one is selected. If you want to call it directly without going through Serve, here is the minimal Python path: direct.pypython from mlx_lm.utils import load_model, load_tokenizer from optiq.runtime.spec import GemmaAssistantDrafter, spec_generate, SpecConfig target, _ = load_model("mlx-community/gemma-4-E4B-it-4bit", lazy=False) tokenizer = load_tokenizer("mlx-community/gemma-4-E4B-it-4bit") drafter = GemmaAssistantDrafter.from_pretrained( "mlx-community/gemma-4-E4B-it-assistant-bf16") prompt = tokenizer.apply_chat_template( [{"role":"user", "content":"Explain spec decoding."}], tokenize=False, add_generation_prompt=True, ) for ev in spec_generate(target, drafter, tokenizer, prompt, SpecConfig(gamma=1)): if ev.kind == "token": print(ev.text, end="", flush=True) The 1.18x result The number is a geomean across five prompt categories on M4 Pro 24 GB with the 4-bit OptiQ quant. Math and code get the highest speedups, reasoning gets the lowest. The pattern matches what Qwen MTP shows: structured outputs with predictable continuations (operators, brackets, common code idioms) draft well; flowing prose drifts more. For a 30 token/s baseline (Gemma-4 E4B on M4 Pro), 1.18x lifts decoding to roughly 35 token/s. That is the difference between "barely faster than reading speed" and "comfortably faster than reading speed" for interactive chat. It also moves a 200-token completion from 6.6 seconds to 5.7 seconds, which compounds over an agentic loop with multiple tool calls. The gap to Ollama's reported 80% acceptance on the same drafter is mostly the bf16 verify artifact, plus the fact that they verify against the fp16 target while we verify against the 4-bit OptiQ target. Quantization adds its own small drift that lowers acceptance further. Where this leaves Gemma serving Today, on M4 Pro 24 GB, with the 4-bit OptiQ quant: Gemma-4 E4B serving runs at 1.18x decode geomean with the -assistant drafter enabled. The drafter loads in about 3 seconds and adds about 700 MB to the model footprint. Output is greedy and very nearly byte-identical for tens of tokens, then drifts within the bounds of bf16 attention precision in mlx-lm. The Server page picks it up automatically. γ>1 multi-token drafting is implemented (the runtime does the chained draft + batched verify + cache rollback) but ships defaulted to γ=1 because that is what wins on Metal. We measured γ=1 at 1.34x, γ=2 at 1.27x, γ=3 at 0.96x on the math prompt above. The K-token verify forward scales near-linearly with K on Apple Silicon, so the verify cost climbs faster than the accepted-tokens-per-cycle does, and γ=2's modest acceptance gain over γ=1 does not pay for the doubled verify. Same finding as Qwen MTP. The MTP guide has the full reference table, methodology, and the new compatibility matrix. --- ## Tools in OptiQ Lab chat Source: https://mlx-optiq.com/blog/lab-chat-tools Release · May 21, 2026 Tools in OptiQ Lab chat Topic Lab tools Reading time 7 min Related eval framework A small quantized model running locally is great for chat. It is much more useful if it can also search the web, run Python, and execute shell commands. OptiQ Lab v0.1.0 ships all three, gated behind a single sandbox layer, with a healer for the malformed tool calls that quantized open-weight models routinely emit. Local tools in a sandbox The chat surface offers three tools the model can call. None require an API key or make outbound calls beyond the tool's own request, and all three run behind the same isolation layer. web_search. Pass {"query": "..."} to get the top DuckDuckGo results back as title / URL / snippet triples, or {"url": "..."} to fetch a single page as compact markdown via html2text. We use the ddgs library (no API key, no rate limit ceremony), cap page bodies at 8 KB so a tool reply stays inside the context budget, and refuse non-http(s) schemes. python. Runs Python source in a sandbox with a 30 s wall clock and a 1 GB memory cap. The AST is pre-scanned for os.system, subprocess.*, signal tampering (signal.signal, signal.alarm, etc.), and direct network access (socket.socket, urllib.request.urlopen, requests.*). Anything that clears the AST check runs in the same sandbox the HumanEval evaluator uses. terminal. Runs a bash one-liner in the same sandbox, with a per-call temporary working directory. Dangerous commands (rm, dd, sudo, curl, ssh, plus the usual suspects) are blocked, but only when they appear in command position. echo "do not use sudo" still works, because the blocker walks tokens with shlex and tracks shell separators, command prefixes, and assignment-precedes-command patterns. The three-tier sandbox The Python and terminal tools share an isolation chain that picks the strongest layer available on the host: Tier Used when Isolation container apple/container is installed Full VM, no network, alpine:3.20 base sandbox-exec macOS host (default) (deny default) profile, file-write only to a temp dir subprocess Linux / fallback POSIX setrlimit + socket.socket patched out The chat UI shows which tier is active under Model & params → Sandbox. The same sandbox runs HumanEval programs during evaluation, so when you see sandbox: sandbox-exec rc=0 in a tool card, that is the same code path that scored the model's pass@1. Healing malformed tool calls Tool calling is theoretically a structured field in the assistant message: "tool_calls": [{"function": {"name": "...", "arguments": "{...}"}}]. Frontier models hit this every time. Small quantized open-weight models, in practice, often emit something else. We track six recurring broken shapes: Shape Looks like Hermes / Qwen tags {"name": "python", "arguments": {...}} Fenced JSON ```json\n{"name": "...", "arguments": {...}}\n``` Bare JSON in content {"name": "terminal", "arguments": {"command": "ls"}} Trailing commas {"name": "python", "arguments": {"code": "1",},} Function-call form python({"code": "..."}) Key-is-tool-name {"python": {"code": "..."}} The healer walks each variant, runs progressive JSON cleanups (trailing-comma stripping, fancy-quote replacement, embedded-object extraction), and synthesizes the standard tool_calls array. The model name is matched against the live tool registry, so the healer cannot hallucinate a tool that does not exist: {"name": "nuke", "arguments": {}} falls through to plain content and the user sees the literal JSON, rather than the orchestrator executing something undefined. Healed calls are flagged so the UI can show a small "healed" chip on the tool card. It is a hint to the user that the model's tool-call output is off-spec, not an error per se. The orchestration loop When tools are on, the browser hits a server-side SSE endpoint instead of streaming directly from mlx-lm. The orchestrator does the work the JS cannot safely do: POST the conversation to /v1/chat/completions with tools=[...] and stream=false. Local generation is fast enough that buffering one assistant turn is cheaper than streaming + parsing tool deltas. Heal the response. If tool_calls is non-empty after healing, execute each call via the tool registry, capturing stdout / stderr / sandbox kind / elapsed time. Append a role=tool message per result and loop. Cap at six turns so a model that refuses to stop calling tools gets cut off rather than burning the host's RAM. When the model finally produces a plain text reply, stream it back as a token SSE event. The UI renders it the same way it would the non-tools path. The wire format is small: event: session, event: tool_call, event: tool_result, event: token, event: assistant, event: cancelled, event: error, event: done. Each frame is one JSON line. The UI uses this to render tool cards in real time, with arguments and output collapsed by default and a per-call elapsed timer. Tuning for local-model failure modes A 6-turn cap was the first instinct: chats normally need one or two tool calls, six is plenty. It also turned out to be wrong. After watching Qwen3.5-9B hit the cap on a research task that needed five searches plus a Python summary, we audited what Unsloth Studio does in the same situation. Their default is 25 turns, paired with three small refinements we adopted: Budget-exhausted re-prompt. When the model hits the turn cap, the orchestrator doesn't return an error. It appends a user-role message that says "stop calling tools and answer now using only the information you have", removes the tools field from the request, and sends one more call. The model has to commit to a text reply, which beats failing the whole chat. Duplicate-call de-dup, but only on successful calls. Consecutive identical tool calls are common when a model gets stuck. We detect (name, arguments) matches and substitute a "you already called this" nudge so the sandbox doesn't burn a second execution. The "only on success" filter is the subtle bit: if a tool errored last turn, the same arguments are allowed to run again so the model can iterate on a fix. Without that filter, a model trying to recover from a blocked Python call would get permanently stuck. Recovery nudge on tool errors. When a tool result starts with a recognized error prefix (Error:, Blocked:, Exit code, Search failed, sandbox: rejected, etc.) the orchestrator appends a short instruction telling the model to try a different approach or different arguments. The UI shows an error chip on the card so the user knows what happened. Stop button A 25-turn budget is no good if you have to wait for it to elapse before getting your terminal back. The Lab supports cancellation end-to-end: clicking Stop in the composer hits a /api/chat/cancel endpoint, the orchestrator polls a threading.Event between turns, and any currently-running tool subprocess gets SIGKILL'd at its process group. The change inside the sandbox was the substantive part: we replaced the blocking subprocess.run with a Popen + watcher loop that polls the cancel event every 50 ms. A stuck sleep 30 in the terminal tool now dies inside half a second. Inline charts from the python tool The most useful thing a chat model with a Python tool can do is plot something. The challenge: the python sandbox is a transient subprocess in a temp directory, so any matplotlib output gets shredded when the workdir is cleaned up. We snapshot the workdir for image files (.png, .jpg, .svg, .webp) immediately after the subprocess exits, base64-encode the bytes, and surface them on a separate images field of the tool_result SSE event. The UI renders them as tags inside the tool card. The base64 payload never enters the model's context. The orchestrator strips an __IMAGES__: sentinel line out of the result string before the result is appended to the conversation; the model only sees a short "[1 image attached to the chat]" footnote. Otherwise a single 50 KB plot would consume 67 KB of context every turn, which is silly. The sandbox-exec profile needed two patches to make matplotlib work: realpath the temp workdir before formatting the SBPL policy (otherwise macOS's /private/var/folders symlink path slips past the subpath check), and set MPLCONFIGDIR=$workdir in the environment so matplotlib's config-dir lookup doesn't try to write to ~/.matplotlib. Grouping adjacent tool calls A model doing real work makes a lot of tool calls. A research session might be search → fetch → fetch → python summarize → answer. Five tool cards in a row, stacked vertically in the thread, drowns out the user's question and the assistant's eventual answer. The Lab groups consecutive role: tool messages into a single "N tool calls" accordion with little chips showing what got called. The most-recent group is auto-expanded so the user sees what's currently happening; older groups auto-collapse when the assistant starts replying with text. File attachments The composer accepts text formats with no extraction round-trip: the JS reads the file, wraps it in a fenced code block, and inserts it into the input. PDF and DOCX bounce through a server-side endpoint backed by pypdf and docx2txt. We cap extraction at 100 K characters and 200 PDF pages so a textbook does not blow the context window. Image and audio are out of scope for v0.1.0. The Lab's quantization pipeline is LLM-first; we would rather not pretend to handle VLM inputs that the rest of the stack does not yet support. Try it terminalbash $ pip install --upgrade "mlx-optiq[lab]" $ optiq lab --model mlx-community/Qwen3.5-9B-OptiQ-4bit Navigate to http://127.0.0.1:7860/chat, open Model & params, confirm Tools = enabled, and ask the model something that requires a calculation or a search. The Lab docs have the full reference. --- ## When 4-bit KV cache uses more memory than fp16 Source: https://mlx-optiq.com/blog/tight-ram-kv-quant Research · May 21, 2026 When 4-bit KV cache uses more memory than fp16 Topic KV cache quantization Reading time 7 min Related Not all layers are equal Stock mlx-lm's --kv-bits 4 on a 24 GB Mac at long context actually peaks higher than fp16 KV. Users who turn it on hoping to save memory get an OOM. The diagnosis Run granite-4.1-8b-4bit on a 24 GB M-class Mac at 32k context. Compare peak GPU memory: fp16 KV cache: 11.51 GB stock mlx-lm u4 KV cache: 16.35 GB ← worse than fp16 stock mlx-lm u4 at 48k: OOMs the process Two compounding spikes inside the mlx-lm path explain this. Conversion spike. maybe_quantize_kv_cache in mlx-lm 0.31.x enqueues every layer's c.to_quantized(...) as a lazy MLX op, then evals them all in a single batch. At the batch eval point MLX holds (fp16_all_layers + quantized_all_layers) co-resident in memory. On a 9 B model with 40 attention layers at 32k context, this transient is about 5 GB on its own. Prefill scores-matrix spike. mlx-lm's unfused quantized_scaled_dot_product_attention computes one big Q @ K^T scores matrix per call. With GQA expansion the shape is (B, n_kv_heads, n_repeats, L_q, N). At prefill chunk-size 2048 and N=16k on Granite-4.1-8b that is about 4.3 GB of fp16, plus the softmax output adds another 4.3 GB co-resident at the softmax step. Roughly 8 GB of transient just to compute attention scores once. Add these together and the "4-bit cache saves memory" story collapses on tight RAM. The persistent cache really is 4× smaller; the transient spikes exceed those savings. The fix: two pieces, both default-on OptiQ patches both spikes whenever you pass --kv-bits or --kv-config to optiq serve. No flags to set, no decisions for the user. Streaming per-layer converter (optiq.runtime.streaming_kv_quant). Replaces the batched maybe_quantize_kv_cache with a per-layer loop: quantize one layer's K, mx.eval, drop the fp16 reference, mx.clear_cache, same for V, repeat. The transient drops from "all layers worth of fp16 K and V" to roughly one layer's worth (~150 MB on a 9B at 32k). FlashAttention-2 N-tiled SDPA (optiq.runtime.fused_quant_sdpa). Replaces the unfused quantized_scaled_dot_product_attention with the standard online-softmax algorithm: tile the K/V N axis into chunks of 512, compute partial scores via mx.quantized_matmul (Apple's tuned Metal kernel), apply an online softmax update, accumulate into the output. The full (B, H_q, L_q, N) scores matrix is never materialized; the largest transient is a single tile of (B, H_q, L_q, 512), about 270 MB at our scale. Same FlashAttention-2 algorithm that fused fp16 SDPA uses; the inner matmul is Apple's existing kernel. The contribution is the orchestration that makes long-context KV-quant fit at all. Memory result granite-4.1-8b-4bit on M4 24 GB, NIAH retrieval at depth 0.5: Context fp16 peak Stock mlx-lm u4 OptiQ u4 16k 8.59 GB spike-prone 6.71 GB 32k 11.51 GB 16.35 GB 7.60 GB 48k 16+ GB OOM 8.5 GB At 32k the OptiQ path is 34% below fp16 and 53% below stock mlx-lm's broken u4. The same patches unlock contexts on Qwen3.5-9B where fp16 KV is tight (11.5 GB at 64k, 18.9 GB at 96k) but OptiQ u4 stays under 10 GB across the whole range. Speed result On Qwen3.5-9B (matched 25 trials, ctx=10k chars), the OptiQ path runs at fp16 KV cache speed within ±2%: Qwen3.5-9B, ctx=10k chars gen_tps vs fp16 fp16 KV 15.47 baseline OptiQ u4 KV 15.16 −2.0% OptiQ mixed-precision KV 15.29 −1.2% The speed lift comes from upstream mx.quantized_matmul; our FlashAttention orchestration is what carries it through to long-context inference without an OOM. Hash-hop is the right benchmark for KV precision NIAH (needle-in-a-haystack) is too easy a test for KV precision: models tolerate noise because the answer is a single phrase the model can recover approximately. Hash-hop forces exact 16-character hash retrieval through a multi-hop chain. Any KV noise that flips even one character of the final hash is a total failure. That makes it sensitive to KV quality in a way NIAH is not. Methodology All KV-mode comparisons below run on the same weight-quantized model: mlx-community/Qwen3.5-9B-OptiQ-4bit (OptiQ-quantized weights, ~4-bit average, with the most sensitive layers held at higher precision). The labels fp16, u4, and mixed in the tables refer to KV-cache precision only, not weight precision. The weights are constant; we're isolating the effect of KV-cache quantization. fp16: OptiQ-quantized weights + fp16 KV cache (the upper bound on quality at this weight quant) u4: OptiQ-quantized weights + uniform 4-bit KV cache (mlx-lm's --kv-bits 4 path) mixed: OptiQ-quantized weights + OptiQ mixed-precision KV cache (kv_config.json with the two most sensitive full-attention layers held at 8 bits, rest at 4 bits) For the "stock vs OptiQ stack" comparison further down, we also bench uniform 4-bit weights (mlx-community/Qwen3.5-9B-4bit) paired with uniform 4-bit KV (the typical "just turn 4-bit on for everything" user setup) against the OptiQ-weights + OptiQ-mixed-KV stack. Difficulty curve on Qwen3.5-9B at ctx=10k chars (25 trials per cell, matched conditions): Hops fp16 KV u4 uniform mixed-precision (OptiQ) 1 (easy) 25/25 (100%) 25/25 (100%) 25/25 (100%) 2 22/25 (88%) 25/25 (100%) 25/25 (100%) 3 (the differentiator) 9/25 (36%) 6/25 (24%) 8/25 (32%) Read across the difficulty curve: hops=1 and 2 are saturated. The model handles single-hop and 2-hop chains perfectly regardless of KV precision. Some sampling-path variation (all three modes at or above 88%) but no meaningful differentiation. hops=3 is the differentiation window. The model is partly capable but stressed by reasoning and KV precision together. Uniform 4-bit drops to 67% of fp16's accuracy. OptiQ mixed-precision retains 89% of fp16, 8 percentage points above u4 in absolute terms, 33% better in relative terms (32% vs 24%). The mixed-precision win comes from keeping the two most KV-sensitive full-attention layers (layer 3 and the last attention layer) at 8 bits while letting the rest run at 4 bits. Total KV memory is essentially the same as uniform u4 (avg ~5 bits), but the layers that matter most stay precise. That's the OptiQ-specific value: a single --kv-config file that closes most of the accuracy gap. Stacked: stock 4-bit everything vs the OptiQ stack The fairer end-to-end question: what does a user actually pick on a 24 GB Mac when they want long context with quantized memory? Two setups, both real shippable choices: Stock 4-bit-everything: mlx-community/Qwen3.5-9B-4bit (uniform 4-bit weights) + --kv-bits 4 uniform 4-bit KV cache. The minimal-memory option. OptiQ stack: mlx-community/Qwen3.5-9B-OptiQ-4bit (OptiQ mixed-precision weights, with the most sensitive linear layers held at higher precision) + --kv-config OptiQ mixed-precision KV (avg ~5 bits, layers 3 and 31 at 8 bits, rest at 4 bits). Larger memory, but spends those bits where the model can use them. Same hash-hop sweep, 25 trials per cell: Setup Active GB at load gen_tps hops=1 hops=2 hops=3 Stock 4-bit-everything 5.04 18.4 100% 84% 20% OptiQ stack (weights + KV) 7.10 13.9 100% 100% 32% At the differentiator (hops=3), the OptiQ stack scores 60% better than stock 4-bit-everything (32% vs 20%, or +12 percentage points absolute). The trade-off is +2 GB of weights memory and ~25% slower decode (14 tps vs 18 tps). Whether that's worth it depends on the workload: short prompts in a chat UI probably don't notice the accuracy difference at hops=3, but anything involving multi-hop retrieval, code symbol following, or long structured reasoning benefits substantially. For the kind of long-context serving where OptiQ is targeted, the OptiQ stack is the right pick. An intermediate choice some users will want: OptiQ weights + plain fp16 KV. That uses OptiQ's higher-quality weight quant without the extra KV-cache infrastructure. At hops=3 it lands at 36% (the highest of any 4-bit-weights setup we tested), but the persistent KV cache is 4× larger than the OptiQ stack and OOMs sooner at long context. The OptiQ stack is the right pick when long context is the goal; OptiQ-weights + fp16-KV is the right pick when short-prompt accuracy is the goal and you have headroom. Where on the model size axis this matters Hash-hop at ctx=10k chars across the Qwen3.5 sizes (25 trials per cell, hops=2 and 3): Size fp16 tps u4 tps mixed tps Speed gap hops=2 fp16/u4/mixed hops=3 fp16/u4/mixed 0.8B 121 107 108 −12% 0% / 0% / 4% 0% / 0% / 8% 2B 58.7 55.1 54.8 −7% 0% / 0% / 4% 12% / 8% / 0% 4B 27.5 26.2 26.2 −4.7% 44% / 48% / 44% 40% / 40% / 40% 9B 14.0 14.0 13.9 −0.6% 88% / 100% / 100% 36% / 24% / 32% Two observations. First, the speed gap shrinks as the model grows: 12% at 0.8B, 7% at 2B, 4.7% at 4B, and 0.6% at 9B. Below 4B the per-token quantize overhead is a bigger fraction of total cost. At 9B the cost is amortized, so you essentially get the memory savings free. Second, hash-hop has a minimum model size for the task itself. At 0.8B and 2B no mode answers reliably (the model is the bottleneck, not the KV cache). At 4B the model is at its capability cliff (~44% on hops=2, ~40% on hops=3 in all modes, no KV-mode signal above noise). Mixed-precision differentiation only appears once the model is reliably capable, which on Qwen3.5 is 9B+. How to use it # Default. Both patches install automatically with --kv-bits. $ optiq serve --kv-bits 4 --model mlx-community/Qwen3.5-9B-OptiQ-4bit # Mixed-precision KV (the differentiator). Generate a kv_config first. $ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit --target-bits 5.0 $ optiq serve --kv-config optiq_output/kv_cache/kv_config.json \ --model mlx-community/Qwen3.5-9B-OptiQ-4bit The serve docs cover request routing, adapter mounting, and the OpenAI/Anthropic/Responses endpoints that share this same KV-cache pipeline. --- ## Getting MTP to actually work on Apple Silicon Source: https://mlx-optiq.com/blog/mtp-on-apple-silicon Research · May 19, 2026 Getting MTP to actually work on Apple Silicon Topic Multi-token prediction Reading time 8 min Related MTP guide · TurboQuant postmortem Qwen3.5 and Qwen3.6 ship with a small extra prediction head bundled into the model. The literature calls it MTP, short for Multi-Token Prediction. The point of having it is to use the head as a draft model for speculative decoding: it spits out K candidate tokens fast, and the main model verifies all K in one parallel forward pass. When the head's guesses turn out to match what the main model would have produced anyway, you get those tokens essentially free. We expected this to be a straightforward feature to wire through OptiQ: hook into the serving stack's already-patched stream_generate path, route through our OptiqEngine, run a bench, and publish the speedup. We budgeted about an afternoon for it. It took longer. The wiring itself worked on the first try, but the surrounding pieces did not. We found three separate issues, fixed them in order, and ended up at the same 1.4x speedup unsloth publishes for the same model class on an RTX 6000. This is what each fix was. Memory OptiQ's install_mtp_speculation patched mlx-lm's stream_generate so that when --mtp was on, generation routed through our OptiqEngine. The engine, on first request, called mtplx_load(model_path), which calls mlx_lm.load internally. The catch: mlx-lm.server had already loaded the same model. So we were holding two full copies of the base weights in unified memory. For Qwen3.5-9B this meant peak memory went from 6.5 GB without MTP to 16.7 GB with it. The MTP head itself is only about 185 MB. The other 10 GB was the duplicate. On a 24 GB Mac this also explained why 27B refused to run with MTP at all. Two copies of a 17 GB model do not fit anywhere. The fix was an OptiqEngine.from_loaded(model, tokenizer, path) classmethod that takes the already-loaded model and injects MTP support into it in place. After the fix, 9B MTP peak dropped to 6.7 GB. 27B MTP fit at 17.6 GB peak, which is what actually made benching it possible at all. The verify was wrong at temperature Greedy decoding worked fine. The verify took the main model's argmax and compared it to the drafted token. If they matched, accept. Standard greedy spec decoding. At temp > 0 the same code path applied, just with sampled tokens instead of argmaxes. Sample from the main model's softmax, compare to the drafted token, accept on equality. This is wrong in a subtle way: both sides are now independent samples from their respective distributions. Two independent samples almost never match exactly. We measured this on 4B with temp=1.0, top_p=0.95, top_k=20 (the recipe Qwen ships in their generation_config.json). Acceptance came out at 28 percent. The greedy version on the same model was 67 percent. The correct probabilistic verify is rejection sampling, due to Leviathan et al and Chen et al. You accept draft dt with probability min(1, p_main(dt) / q_draft(dt)). If you reject, you sample a replacement from the residual distribution (p_main - q_draft)_+ rescaled. Rewriting the verify to do this raised 4B acceptance at the same sampler from 28 percent to 57 percent, and throughput went from 0.87x base (a regression) to 1.11x. Truncation has to apply on both sides Rejection sampling fixed the catastrophic case, but our temp=1.0 numbers were still below greedy. We were losing 8 to 17 percent across model sizes versus the greedy 1.36x at 27B. Unsloth's published numbers report around 83 percent acceptance at this model class. We were at 57 percent. The gap turned out to be sampling truncation. Unsloth's MTP guide specifies temp=1.0 with top_p=0.95 and top_k=20. Our verify was using the untruncated softmax even though the draft path was applying top_p and top_k. Truncating both the draft and verify distributions to the same support (top 20 tokens, top 0.95 cumulative mass) makes the L1 overlap between them much higher, and that overlap is exactly what rejection sampling acceptance measures. After applying the same truncation on both sides, acceptance on 9B jumped from 32 percent to 63 percent. Throughput jumped from 1.00x to 1.28x. We also added the model's recommended sampler from generation_config.json as a default in optiq serve and optiq lab. Without that, users who set just --temp 1.0 would see the same diffuse-distribution behavior we hit. The new default reads top_p and top_k from the model's published settings unless the user passes them explicitly. What about depth 2 or higher vLLM, HuggingFace Transformers, and unsloth's MTP guide all use K=2 or higher as defaults. We tested K=2 through K=4 on all three model sizes and found K=1 universally wins on Apple Silicon. The numbers, greedy: Model K=1 K=2 K=3 K=4 Qwen3.5-9B 1.28x 1.19x 0.89x 0.74x Qwen3.6-27B 1.36x 1.34x 0.94x 0.74x The reason is hardware. On CUDA the K-token verify forward is nearly free because Tensor Cores have spare throughput during single-token decode, so adding tokens to verify just uses idle compute. Apple Silicon's Metal backend has a compute-to-bandwidth ratio roughly 10x lower than H100. K=2 verify on Metal costs about 2x what K=1 verify costs, not the 1.1x that CUDA gets. Tokens-per-verify-cost works out like this: Outcome Tokens Cost Tokens/cost K=1 full accept 2 1x 2.00 K=2 full accept 3 2x 1.50 K=2 partial (1/2) 2 2x 1.00 K=1's best case is already higher than K=2's best case on Apple Silicon. No algorithmic trick beats this without a custom Metal kernel that reduces the K-token verify cost. We also tried HuggingFace's dynamic depth heuristic, where K rises after a fully accepted cycle and falls on partial accept. It lost 4 to 17 percent across all sizes versus fixed K=1. The adapter cannot find any working pattern of K=2 cycles that outweighs the cost when the cost ratio is 2x. Where we landed The OptiQ 0.1.0 MTP numbers, M4 Pro 24 GB, greedy decoding, 512-token generation: Model Base tok/s MTP tok/s Speedup Qwen3.5-4B 29.2 35.0 1.20x Qwen3.5-9B 19.5 25.8 1.32x Qwen3.6-27B 6.0 8.4 1.40x With Qwen's recommended sampler (temp=1.0, top_p=0.95, top_k=20), the same range becomes 1.09x / 1.17x / 1.30x. The 27B greedy number is within 5 percent of unsloth's published 1.4x on the same model class running on an RTX 6000. We will take that on a Mac. For 0.1.0 we ship K=1 with the numbers above. The MTP guide has the reference table, methodology, and the --mtp flag. --- ## The eval framework that drives every quant we ship. Source: https://mlx-optiq.com/blog/eval-framework Engineering · April 28, 2026 The eval framework that drives every quant we ship. Topic Evaluation methodology Reading time 8 min Related calibration mix The regression check on a fresh quant has to test the workload, not the cheapest thing to run. A 50-sample GSM8K pass is a quick gut-check that math still works. It doesn't catch a quant that quietly forgot how to emit valid function-call syntax, or one that lost the long-range KV precision that makes retrieval through 12k tokens work. By the time a user notices, the quant is already in their cache. So every optiq quant goes through a two-stage suite. A fast smoketest for triage, then a full benchmark run for the headline number. Six benchmarks roll up into one Capability Score on the model card. It runs sandboxed HumanEval execution and an auto-resolved KL reference, all from one CLI command. Two stages: a smoketest and the full benchmarks Quants pass through two checkpoints. The smoketest is fast and triages; the benchmark run is slow and sets the shipping number. Stage Time / model What it answers Tasks Smoketest ~5 min Did the convert work? Are we close to the reference distribution? KL on 64 prompts × 256 tokens · GSM8K-50 (chat-templated, thinking off) Benchmarks ~2 h How much capability did we keep across the workloads users actually run? MMLU-1k 5-shot · GSM8K-1k · IFEval (full) · BFCL-V3 simple (200) · HumanEval (164) · HashHop (25 × 4 hops) The smoketest is the gate. A quant that fails it doesn't get the full benchmarks. The benchmark numbers are what end up on the model card. The smoketest: KL + GSM8K-50 KL divergence between two language models, computed token-by-token over a small batch of held-out prompts, is a cheap signal that works well in practice. The reference is the highest-fidelity version of the model that fits on the box. The candidate is the OptiQ quant. We compute KL(reference ‖ candidate) per token, average across 64 prompts × 256 tokens, and report mean + p95. The auto-resolver picks the reference automatically: KL reference auto-resolverpython # Pick the highest-fidelity reference that fits on the box. bf16_gb = hf_repo_size_gb(strip_quant_suffix(model_id)) avail_gb = psutil.virtual_memory().available / 1024**3 if bf16_gb <= 0.70 * avail_gb: return "bf16", strip_quant_suffix(model_id) # bf16 doesn't fit; fall back to the uniform-4-bit MLX baseline. return "uniform_4bit", uniform_4bit_repo(model_id) Two flags fall out of the smoketest sweep that the GSM8K-50 number alone wouldn't have surfaced: KL 20× higher on the Gemma-4 26B-A4B MoE and 31B dense relative to Qwen3.5-27B at comparable size, despite both showing healthy GSM8K-50 numbers. KL catches calibration regressions that single-task accuracy misses. The smoketest is also how we decide when to invest two hours of compute in the full benchmark run. If a fresh convert fails KL, the bit allocation went bad and rerunning the full suite is wasted machine time. The benchmarks: six metrics The benchmark suite is what ends up on the model card. Each task targets a different capability slice, so a quant that quietly breaks one of them can't hide behind the others. MMLU: 5-shot, stratified across the 57 subjects, 1000 samples. Encyclopedic knowledge after instruction-tuning. The bf16 anchor. Scored by first-letter logit argmax (the answer token after "Answer:"), the standard cheap-and-stable method. GSM8K: 1000 samples, 3-shot CoT, chat-templated, enable_thinking=False for reasoning models with a thinking toggle. Multi-step arithmetic. IFEval: full Google IFEval set with all 25+ constraint verifiers. Measures whether the model can follow detailed format / length / capitalization / inclusion-exclusion instructions. We report strict (the standard, harder metric). BFCL-V3 simple: 200 single-turn function-calls with AST equivalence scoring. Whether the model can emit a syntactically valid call and pick the right tool from a small candidate set. HumanEval: all 164 problems, sandboxed Python execution, pass@1 only. HashHop Long-Context Evaluation: 25 instances at each of hops ∈ {1, 2, 3, 4} (100 total) at ~12 k context. Multi-hop key→value retrieval through a chain of N hash assignments. The model has to walk the chain through the long context and surface the terminal hash. Run from the CLI as a single task: terminalbash optiq eval ./optiq_mixed --task all --score Each individual task is also addressable (--task mmlu, --task ifeval, --task hashhop, etc.) for when you only need one number. Always-on reasoning models: --reasoning The setup above quietly assumes one of two things about a model: that it answers directly, or that it has an enable_thinking=False toggle we can flip. A growing class of models has neither. They emit a … block before every answer and can't be told to stop. On the stock harness those models score near-random, and not because they're bad: the generation tasks budget a few hundred tokens, which truncates a multi-hundred-token think trace before the answer ever appears; and MMLU's first-letter logit argmax collapses to chance, because a model trained to reason first doesn't put its answer-letter mass on the token right after "Answer:". We measured one strong math model at 27% MMLU (≈ chance) and 36% GSM8K on the stock harness. Both low scores came from the scoring format, not the model. optiq eval --reasoning scores these fairly. It lets the model think (no suppression), gives each question a large budget (--reasoning-max-tokens, default 3072, a cap, not a fixed length, since greedy decoding stops at EOS so short traces cost nothing), strips the block before extracting the answer, and scores MMLU generatively (generate with thinking, then parse the answer letter) instead of by logit argmax. The candidate quant and its baseline are scored identically, so the comparison stays apples-to-apples. On that same model the real numbers were 74% MMLU and 90% GSM8K. terminalbash $ optiq eval ./optiq_mixed --task all --score --reasoning Sandboxing HumanEval HumanEval requires actually executing the model's generated Python against a unit-test harness. Doing that on the user's machine with no isolation is a footgun. A model that emits os.system("rm -rf …") ruins someone's afternoon. The sandbox helper falls through three tiers: apple/container: when present, runs each candidate inside a fresh container with no network, no filesystem mount outside /tmp, and a wall-clock timeout. Hardest isolation, slowest start. sandbox-exec: macOS native, when /usr/bin/sandbox-exec is available. Subprocess with a tight seatbelt profile (no network, deny file-write outside /tmp). Fast. subprocess + rlimit: universal fallback. Spawn a Python child with RLIMIT_AS, RLIMIT_CPU, RLIMIT_FSIZE caps and a process-group timeout. No filesystem isolation; exists so the eval doesn't simply fail to run on Linux CI. The helper picks the strictest tier available at runtime. Reported pass@1 is identical across tiers because the test harness is deterministic. Only the blast radius of malicious code changes. HashHop Long-Context Evaluation The first five benchmarks are short-context. MMLU prompts are a few hundred tokens. BFCL function-call prompts are similar. None of them push the KV cache out past 2k tokens, so a quant that silently lost long-context attention precision can still post strong numbers across all five. HashHop is the one that catches it. Each instance is a dictionary of N hash assignments shaped like h0 = h1, h1 = h2, … h(N-1) = 'hN', scattered among thousands of unrelated chains and serialized into one large prompt. The model gets the starting hash and has to walk the chain through ~12k tokens of context, surface the terminal 16-character hash, and emit nothing else. Random guessing on a 16-character alphabetic hash is effectively zero (5216), so accuracy maps cleanly to how reliably the model is using its KV cache to retrieve the right key at each hop. We sample 25 instances at each of hops ∈ {1, 2, 3, 4}, 100 instances total. Easy hops (1) catch coarse retrieval breakage; deep hops (4) stress compounding attention error across many heads × many tokens. Why this matters specifically for mixed-precision: uniform 4-bit quantization of weights erodes long-range attention precision because small numerical errors compound across many heads × many tokens. Per-layer mixed precision (some layers 4-bit, sensitive ones 8-bit) preserves the layers retrieval depends on, so HashHop deltas vs uniform 4-bit are typically the largest single delta in the benchmark suite. On gemma-4-31B it's +22 pp over uniform 4-bit; on the 26B-A4B MoE it's +11 pp. The Capability Score Six percentages are hard to compare side-by-side. We want one number that answers which quant is more capable on average?. And we want a formula the reader can audit, not a hidden value judgement dressed up as math. The simplest one that meets that bar: Capability Scoreformula Capability_Score = mean(MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop) We tried a weighted formula first. Something like MMLU + 0.3 × IFEval + 0.5 × BFCL − 5 × disk_GB. It embedded our quality/disk tradeoff in a way users can't see, and it could turn a +1 pp capability win into a "loss" if the disk grew by half a gigabyte. That bakes in a recommendation rather than reporting a measurement. So we stripped it down. The six benchmarks each get an equal vote. disk_gb is reported next to the score as an unweighted second axis, and the reader picks their own tradeoff. Which model to prefer depends on available memory: a smaller quant suits an 8 GB Mac, a larger one a 64 GB Studio. The score doesn't decide that for you. (1) GSM8K and MMLU both vote, because in practice they disagree often enough on quants that letting both vote catches regressions either one alone would miss. (2) HumanEval votes, which means a quant that breaks code generation can't hide behind strong instruction-following. (3) HashHop votes, which means a quant that holds up at 2k context but breaks at 12k can't claim parity by averaging only short-context numbers. Picking the KL reference One technical note that took us a few iterations to get right. For models that fit in RAM (everything ≤ ~10 B at bf16 on a 36 GB Mac), the KL reference is unambiguous: it's the bf16 model itself. For 27 B+, bf16 doesn't fit, and you need a substitute reference that's still strictly higher-fidelity than the candidate. The community's uniform-4-bit MLX publish of the same model is exactly that: same architecture and weights modulo quantization noise, just at uniform 4-bit (no per-layer mixed precision). The auto-resolver picks bf16 if available, falls back to uniform-4-bit otherwise. The fall-back was originally driven by a crude params × 2 bytes size estimate, which under-counted gemma-4-26B-A4B's MoE expert tensors and tried to load 110 GB of bf16 into 36 GB of RAM. Now we hit HfApi.model_info() and sum the actual safetensors shard sizes. The resolver is exact and the OOM is gone. Reproducing Everything in this post runs from the CLI. No special setup beyond pip install mlx-optiq: terminalbash # Fast smoketest (KL + GSM8K-50) optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit --task smoketest # Full benchmarks (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop + Score) optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit --task all --score # Single tasks if you only need one number optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit --task bfcl optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit --task humaneval optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit --task hashhop # Custom reference for KL (skip auto-resolver) optiq eval ./my-quant --task kl --reference-model Qwen/Qwen3.5-9B --reference-mode bf16 Every task above is callable on its own. Pick the one you need with optiq eval --task . --- ## Calibration data decides what a quant protects Source: https://mlx-optiq.com/blog/calibration-mix Engineering · April 28, 2026 Calibration data decides what a quant protects Topic Calibration data Reading time 5 min Related eval framework The calibration data set is what mlx-optiq uses to measure per-layer quantization sensitivity. The bundled six-domain optiq.jsonl mix of 40 samples replaces WikiText-2 and protects tool-calling, agent loops, reasoning, and instruction-following alongside prose. mlx-optiq is a sensitivity-driven quantizer: for every layer in a model, it measures how much the output distribution shifts when that layer is quantized to k bits, then a knapsack solver hands out the bit budget to the layers that need it most. The calibration set decides one thing: what input data we run through the model to take that measurement. For most of mlx-optiq's life that calibration set was WikiText-2. This release replaces it with optiq.jsonl: 40 hand-curated samples spanning prose, multi-step reasoning, code, agent loops, function-calling, and constraint-bearing instructions. The file ships inside the Python package, so optiq convert never touches the network for calibration data. Why WikiText was the wrong floor WikiText-2 was the right calibration set for the GPT-2 era. It's clean, English, well-distributed, large enough to sample from, and the models you were quantizing were just next-token predictors over web prose. A small batch of WikiText sentences would activate roughly the same circuits the model used in production. Modern instruction-tuned and reasoning-tuned LLMs aren't that. Qwen3.5-9B-Instruct is asked to do six very different things in a typical day: explain a concept, work through a multi-step proof inside a block, write Python that compiles, execute an agent loop with tool feedback, emit valid JSON for a function call, and follow verifiable instruction constraints (response in three bullets, exactly fifty words, all caps, ending with a specific phrase). Each of those uses different layer subspaces. When the calibration set is exclusively prose, three failure modes happen at once: Tool-calling layers appear robust. They aren't activated by prose, so quantizing them produces ~zero KL divergence on the calibration probe. The optimizer drops them to the lowest bit-width. In production the model then hallucinates JSON brackets. Reasoning-critical layers register as noise. The sub-circuit is most active during long multi-step solutions; a 256-token WikiText snippet barely touches it. Constraint-following layers go unmeasured. The circuits that hold a "respond in exactly N words" or "all caps" rule across a long generation are not exercised under prose calibration, so the optimizer prunes them. Real IFEval scores drop double-digits. You can't see this from a perplexity number. WikiText perplexity stays flat. GSM8K and IFEval on the quantized artifact drop a few percent. The calibration set is accurate but doesn't exercise the behaviors that regress. The six domains We picked six capability slices that match what people actually run mlx-optiq quants for, and built a calibration sample from a public dataset for each. Domain Samples Source dataset What it activates prose 5 wikitext-2-raw-v1 Baseline next-token prediction. The thing WikiText was always good for. thought 6 open-r1/Mixture-of-Thoughts (math) R1-style reasoning with explicit blocks. Long, structured, self-correcting. code 6 nvidia/OpenCodeReasoning Programming problems with step-by-step reasoning into a working Python solution. agent 8 lambda/hermes-agent-reasoning-traces Multi-turn agent loops: system prompt → user → think+tool_call → tool_result → continuation. tool 7 NousResearch/hermes-function-calling-v1 Function-calling traces with tool schemas. Forces the model into JSON-emitting subspace. instruct 8 HuggingFaceH4/no_robots Human-written prompts carrying verifiable constraints (word counts, bullet lists, JSON, capitalization). Disjoint from google/IFEval, no test-set leakage. 40 samples total · ~127 K characters · ~32 K tokens. Small enough that even a 31 B-dense model finishes the per-layer KL pass in under an hour on a 36 GB Mac, dense enough that every major capability slice gets at least 5 K activated tokens of probe pressure. Schema Each line is a JSON object. Two shapes: raw text for prose, chat for everything else: optiq.jsonljsonl // raw text, used as-is {"domain": "prose", "text": "..."} // chat: runs through the model's tokenizer.apply_chat_template() {"domain": "thought", "messages": [{"role": "user", "content": "..."}, ...]} // chat with tools: schema is rendered into the system prompt by the tokenizer {"domain": "tool", "messages": [...], "tools": [{"name": "...", ...}]} The loader (optiq.calibration.datasets.load_llm_calibration) is opinionated about one thing: chat samples go through the target model's own chat template before tokenization. That means a sample we built from a Qwen-flavored agent trace gets re-rendered into Gemma chat tokens before being fed to a Gemma model. The activated subspace is the production subspace, not the donor model's. Per-domain design notes prose WikiText-2 still earns its keep at 15 % weight. It's the only domain that activates the model purely as a next-token predictor without instruction-tuned templating, and it keeps the calibration set honest about the underlying language model. Without it, mixed-precision quants started skewing too aggressively toward instruction layers. thought open-r1's Mixture-of-Thoughts math split, chosen because the chain-of-thought is long, structured, and self-correcting. The block exercises the residual stream in a way short-form QA never does, and the math domain forces step-by-step intermediate computation rather than free-form discussion. code nvidia/OpenCodeReasoning has the same shape as thought but in Python instead of math: competition-style problems with explicit reasoning into a working solution. Strong predictor of HumanEval pass@1 and code-generation BFCL slices. agent The hardest domain to source. Public agent-trace datasets are either short demos (don't exercise the loop) or production logs that pile up to ~86 K characters per trace. We use lambda/hermes-agent-reasoning-traces and aggressively truncate: keep the first 7 turns, cap each message at 600 characters. The result preserves the shape of an agent loop (system → user → think+tool_call → tool_result → continuation) without spending the entire token budget on one trace's tail. tool NousResearch/hermes-function-calling-v1 in func_calling config. We filter for traces that contain at least one explicit (or equivalent name+arguments JSON), so every sample exercises the JSON-emitting head. Tools schemas are extracted from the source's ... block and attached as a structured field, then re-rendered through the target model's tool template at calibration time. instruct HuggingFaceH4/no_robots is 10 K human-written prompts. We regex-filter for those carrying verifiable constraints: word counts, sentence counts, bullet/numbered list requirements, JSON-format clauses, capitalization rules, keyword-presence or keyword-absence rules, "starting with"/"ending with" phrases. The 326 matches across the 9.5 K-row train split are dense enough to give the optimizer real KL signal on the layers that hold a constraint alive across a long generation. We deliberately did not sample from google/IFEval itself. That's the test set, and using it as calibration would be circular. Why ship the JSONL inside the package We ship it inside the package for three reasons. Reproducibility. A bf16-source quantization run in 2026 should be byte-identical to one in 2027. If the calibration set is "stream from HuggingFace at convert time," you can't promise that. Datasets get re-uploaded, the source repos change, streaming order is non-deterministic. Offline. optiq convert already needs network for the bf16 source weights; calibration shouldn't add a second hop. The JSONL is 328 KB. It ships in the wheel. Auditable. One file, 40 lines, you can read all of them. Building your own The mix above is what we ship as the default. The CLI accepts a path to your own JSONL (same schema) for domain-specific quants: terminalbash # Default: ships in the package, no network. optiq convert Qwen/Qwen3.5-9B --calibration-mix optiq # Roll your own (same schema, point at any .jsonl on disk). optiq convert Qwen/Qwen3.5-9B --calibration-mix ./my-domain-mix.jsonl If you want to rebuild the default mix from scratch (different seed, different per-domain weighting, different sources), the script is checked into the repo: terminalbash # Re-derive optiq.jsonl from the 6 source datasets, deterministic seed=42. python scripts/build_calibration.py The builder streams from HuggingFace, applies the per-domain length filters and truncation rules, and writes the result back to optiq/calibration/data/optiq.jsonl. About 90 seconds end-to-end on a warm cache. What this gives you The interesting part is what changes downstream: which layers the optimizer protects, and what that does to held-out evals like IFEval (instruction-following), BFCL (tool-calling), and HumanEval (code) on the same target BPW. See the eval-framework writeup for the per-model deltas on Qwen3.5-27B, Qwen3.6-27B, and the Gemma-4 26B-A4B / 31B re-quants. --- ## Gemma-4 lands on mlx-optiq. Source: https://mlx-optiq.com/blog/gemma-4-support Engineering · April 25, 2026 Gemma-4 lands on mlx-optiq. Topic Family launch Reading time 4 min Related research foundation We added four mlx-optiq Gemma-4 instruct quants to Hugging Face, recovering up to 32 percentage points of GSM8K accuracy on the small model through mixed-precision quantization. gemma-4-e4b at uniform 4-bit collapses to 23.5% on GSM8K; mlx-optiq mixed-precision recovers it to 55.5%, a +32-point jump at the same 6 GB on disk. The full lineup spans the compact 4 GB e2b through the 18 GB 31B-dense. The lineup Model Size GSM8K vs uniform-4 gemma-4-e2b-it 4.0 GB 13.0 % +7.5 pp gemma-4-e4b-it 6.0 GB 55.5 % +32.0 pp gemma-4-26B-A4B-it 14.9 GB 94.0 % +2.0 pp gemma-4-31B-it 18.1 GB 96.0 % 0.0 pp Why e4b recovers so dramatically The pattern across all 10 quants we ship is consistent: the bigger the gap between bf16 quality and uniform-4 quality, the bigger mlx-optiq's win. Saturated benchmarks (gemma-4-31B already at 96 %) leave nothing to recover. Models where uniform-4 nearly breaks them (e4b at 23.5 %) have all the room to grow. e4b is right at the edge. It's a 4 B model with strong reasoning capability that is badly degraded by quantization noise on a few sensitive layers. The per-layer KL pass identifies those layers (lm_head, the first attention block, layer 0's KV-projection sibling, the last few transformer blocks) and protects them at 8-bit. Everything else stays 4-bit. Net result: same 6 GB on disk, 2.4× the GSM8K accuracy. The 26B-A4B sparse MoE Gemma-4-26B-A4B is the family's mixture-of-experts: 26 B total parameters, 4 B active per token. Different from the Qwen MoE layout, Gemma uses switch_glu with a fused gate-and-up projection, so each expert is a pair of fused tensors rather than the three-tensor split Qwen uses. mlx-optiq's MoE walker is arch-aware: it identifies the layout, walks the experts, and treats the fused expert tensor as a single layer for sensitivity purposes. Per-expert bit-widths come out of the same knapsack as everything else. The result: 26 B-A4B at 14.9 GB on disk runs faster than the dense 27 B variants because only 4 B of weights actually multiply per token. KV-quant serving on Gemma-4 (v0.1.3+) Earlier OptiQ versions failed on Gemma-4's KV-quant path because upstream mlx-lm raises NotImplementedError: RotatingKVCache Quantization NYI on the sliding-window cache used by Gemma's SWA layers. v0.1.3 ships optiq.runtime.kv.RotatingQuantizedKVCache as a drop-in subclass with quantized (packed, scales, biases) storage and the rotating-buffer mechanics preserved, plus a small SDPA dispatch patch for Gemma's KV-sharing layers (where one layer's quantized K/V tuples flow into a downstream layer whose own cache is None). The patch installs automatically when --kv-bits or --kv-config is set. Each Gemma-4 OptiQ-4bit repo on Hugging Face now ships a recommended kv_config.json from a real sensitivity-analysis pass, drop-in via optiq serve --kv-config kv_config.json. Get them terminalbash from mlx_lm import load, generate model, tok = load("mlx-community/gemma-4-e4b-it-OptiQ-4bit") # 6.0 GB on disk · 55.5 % GSM8K · runs on a 16 GB MacBook Air Full sampling defaults, training recipes and the shared-KV caveat are in the Gemma-4 family guide. --- ## TurboQuant: what we built, what we measured, and why we didn't ship it. Source: https://mlx-optiq.com/blog/turboquant-postmortem Postmortem · April 17, 2026 TurboQuant: what we built, what we measured, and why we didn't ship it. Topic Postmortem Reading time 7 min Related research foundation We built a KV-cache technique whose benchmarks looked good, and decided not to ship it. The reason isn't a flaw in the technique. The marginal win didn't justify the cost of a parallel serving path. The technique: rotated-space attention Affine quantization is the standard way to compress the KV cache: stretch each tensor to fit in a smaller integer range, store an offset and a scale per group. It works for storage. But attention is a dot-product operation, and affine quantization doesn't preserve dot products well. It preserves magnitudes. The error in q · k after quantizing k isn't isotropic; it correlates with semantic structure in ways that bias which tokens win the softmax. The fix is mathematically simple. Multiply both K and Q by the same random orthogonal matrix R. The attention score is preserved exactly: (R q) · (R k) = q · RTR · k = q · k. So you can quantize R k instead of k and recover the original score. Random rotation distributes a vector's mass roughly uniformly across all coordinates, so the per-coordinate distribution becomes nearly Gaussian (concentration of measure). That's exactly what scalar quantizers like best. The catch is the cost. Naive rotated quantization needs you to dequantize keys and rotate them back at attention time, which is O(seq_len × d²) per token, fatal at long context. Our trick was attending in rotated space: rotate the query once per step (cost O(d²) fixed), and dot it directly against the stored quantized rotated keys via a fused Metal kernel that does the dequant inline. That brings the overhead per step back to roughly the same as affine. The numbers we measured On Qwen3.5-9B with 4-bit KV at 64 k context, head-to-head against mlx-lm's affine QuantizedKVCache: Method Bits Needle retrieval Reasoning PPL drift Speed vs fp16 fp16 (reference) 16 100 % 32 % n/a 1.00 × Affine 4 73 % 30 % +0.48 0.96 × TurboQuant (rotated) 4 100 % 32 % +0.37 0.98 × The needle-in-a-haystack number was the most striking: 100 % vs 73 % at 4-bit, on the exact same model and exact same prompts. The retrieval failures with affine concentrated on long-distance matches, exactly where small inner-product errors push the wrong token over the softmax threshold. Reasoning quality and perplexity were close to fp16; speed was within 2 % of the affine path. Our first naive Python implementation of rotated-space attention was 47 % slower than affine. Three rounds of work (incremental dequantization, a custom Metal kernel, a fused SDPA pass that never materializes dequantized keys) closed the gap to 2 %. So why didn't we ship it Three things killed it in turn. 1. The 100 % vs 73 % needle test was synthetic Needle-in-a-haystack is a worst-case probe: single rare token, single position, against a vast distractor field. Real workloads almost never look like this. When we re-ran on more realistic long-context tasks (multi-fact retrieval, tool-result interpretation, multi-turn chat with file context), the gap shrunk dramatically. At 32 k context on a 9 B model, both quantizers landed within 1–2 percentage points of fp16 quality on the tasks our users actually run, and within noise of each other. The headline number was real, but it didn't predict user-visible quality. 2. Per-layer mixed-precision affine already captured most of the win The companion experiment, per-layer KV bit-width assignment, was the bigger lever. Once we measured per-layer KV sensitivity and protected layer 0 (often 56× more sensitive than the average) at 8-bit, the affine path's quality regressions on long context largely disappeared. Mixed-precision affine was already covering most of what TurboQuant fixed, and it integrates with mx.quantized_matmul's fused fast path on Apple Silicon. TurboQuant, with its custom kernel, does not. Net for users: mixed-precision affine gave roughly the same quality as TurboQuant on real workloads, and ~30–60 % faster decode at 64 k context. We can't ship a quality wash that's also slower. 3. Two parallel serving paths is one too many Even ignoring quality and speed, shipping TurboQuant in optiq serve meant maintaining a fork of mlx-lm's attention path. Every kernel update, every model class addition, every new attention variant (sliding window, hybrid, MoE-routed) would need to be ported into our rotated-space version. That's a structural ongoing cost the technique would have had to keep earning. That trade is acceptable in a research codebase, but not in the default serving stack we ship to users on PyPI. What we kept For a long time we kept the TurboQuant code in the package as an "import-it-yourself" library primitive: disabled by default, available for users who specifically wanted to play with rotated-space KV. As we got close to our first real release, we audited what was still in active use and what was just historical research that hadn't earned its place. TurboQuant was in the second bucket. We removed it from the package. What stayed: The conceptual framing that informed TurboQuant ("attention compression has different sensitivity than weight compression") is now embedded in the standard mixed-precision KV pipeline. What went away: optiq.core.turbo_kv_cache, optiq.core.turbo_quant, optiq.core.turbo_metal, optiq.core.turbo_state_cache: all deleted. The demo/demo_turbo_kv.py stress test. The TurboQuant-flavoured options that never made it into the public CLI anyway. Lessons we took with us The 100 % vs 73 % needle gap was real and reproducible, but it was a worst-case probe of one failure mode that doesn't dominate real usage. Synthetic benchmarks overstate marginal wins. We've since moved to multi-domain eval suites (5-shot MMLU, IFEval, BFCL, plus task-specific tests at multiple context lengths) so we don't get fooled the same way again. A feature that requires a separate serving path has to clear a much higher bar: the cost of maintaining parallel infrastructure compounds with every new model architecture. Anything that doesn't cleanly compose with stock mlx-lm needs to deliver a meaningful, durable win, not a marginal one that erodes as the upstream landscape changes. We cut the research demo rather than carry it in the library. mlx-optiq's focus is mixed-precision quantization, sensitivity-aware LoRA, and a dual-protocol serving stack. The TurboQuant code is gone from main. We won't be maintaining it. --- ## Sensitivity-aware LoRA: the same signal drives rank and bits. Source: https://mlx-optiq.com/blog/sensitivity-aware-lora Engineering · April 8, 2026 Sensitivity-aware LoRA: the same signal drives rank and bits. Topic Fine-tuning Reading time 5 min Related research foundation Sensitivity-aware LoRA matches constant rank-16 accuracy on a logical-puzzles reasoning task using 39% fewer trainable parameters by reusing the same per-layer bit signal mlx-optiq computes during quantization. Standard LoRA gives every adapted layer the same rank regardless of which layers matter most to the model's output. mlx-optiq already knows which layers matter: they got more bits during quantization. Reusing that signal at training time produces a measurable Pareto improvement. The mechanic When you quantize an mlx-optiq model, we record the per-layer bit assignments in optiq/metadata.json, shipped inside every quant on Hugging Face (at the root as optiq_metadata.json for quants published before 0.4.18; both are read). Every layer's quantization bit-width is right there. optiq lora train reads it and uses it to scale adapter rank: 4-bit layers (the model's robust ones) → base rank. 8-bit layers (the model's sensitive ones) → 2× base rank. At --rank 8 --rank-scaling by_bits, that's rank-8 for 4-bit layers and rank-16 for 8-bit layers. Same total parameter budget as constant rank-10, but capacity is moved toward the layers that demonstrably affect output more. Why this should work, intuitively LoRA's premise is that fine-tuning updates lie in a low-rank subspace, but it doesn't say anything about which layers need more capacity. In practice, the layers that need more capacity to fit a target task are not uniform. They tend to correlate with output-distribution sensitivity: a layer that, when perturbed, shifts the logits a lot is also a layer where small changes during fine-tuning produce large behavioral changes. Both phenomena are about the gain between a layer's weights and the model's output. So a layer that is fragile under quantization is also the layer most responsive to fine-tuning, which is why allocating more rank where there's more signal-amplitude makes sense. The empirical result Head-to-head on a 6-category logical-puzzles reasoning dataset where the base model produces correct chain-of-thought but never closes with the dataset's \boxed{...} answer format. Qwen3.5-4B-OptiQ-4bit, 1 epoch over 200 training samples, evaluated on 100 disjoint test samples by extracting the answer from the boxed expression. Config Trainable params Val loss Test accuracy Base (no LoRA) n/a n/a 0 / 100 = 0% Constant rank-8 11.58 M 0.328 27 / 100 = 27% by_bits (rank 8 / 16) 13.49 M (+16%) 0.319 35 / 100 = 35% Constant rank-16 22.20 M (+92%) 0.360 36 / 100 = 36% by_bits matches constant rank-16 on accuracy (35% vs 36%, within noise at n=100) using 39% fewer trainable parameters. Versus the rank-8 baseline at almost the same parameter budget, by_bits is +8 accuracy points (27 → 35) for +16% more params. Both directions land by_bits on the Pareto frontier. The base model scores 0/100 because it never emits the dataset's \boxed{...} format. It produces valid step-by-step reasoning but doesn't recognise the answer convention. All three LoRA configs learn the format from one epoch over 200 examples; the differences in accuracy are about how cleanly each preserves the underlying reasoning while bolting on the format. Per-category breakdown across the three configs: Category n rank-8 rank-16 by_bits Numeral Conversion 39 21 31 31 Unit Conversion 13 2 0 4 Equation Transformation 35 4 5 0 Bit Manipulation 8 0 0 0 Text Encryption 3 0 0 0 Gravitational Constant 2 0 0 0 by_bits wins on Numeral and Unit Conversion (the structured pattern-matching tasks) and ties the larger model on the overall total, but loses on Equation Transformation (0/35 vs 4 and 5 for the constant-rank configs). The categories where all three score 0 (Bit, Text, Gravitational) are dataset-coverage limits: only ~10–25 training samples each in 200 mixed-type total, not enough to learn from one epoch. The Equation gap on by_bits at the same data is a real category-specific trade-off worth flagging: concentrating extra rank on the high-bit (mostly attention) layers appears to help format-style tasks and disadvantage symbolic-manipulation ones. Real-world recipe: pair by_bits with the right dataset balance for your task. The training-ceiling map What follows is the empirical story we wish we'd had when we started fine-tuning mlx-optiq quants on a 36 GB Mac. All entries verified end-to-end against a real Hermes-traces dataset with the default config (q_proj, v_proj, num_layers=16, rank=8, rank_scaling=by_bits). Both iters stable, zero memory drift. Model Max seq len Peak mem Tokens / sec Time / iter Qwen3.5-0.8B 2,800 23.4 GB 29.2 96 s Qwen3.5-2B 2,400 19.3 GB 38.3 63 s Qwen3.5-4B 1,600 24.8 GB 19.1 84 s Qwen3.5-9B 1,400 25.4 GB 21.6 65 s Qwen3.5-27B / Qwen3.6-27B 512 27.7 GB 11.4 45 s gemma-4-26B-A4B 512 27.6 GB 22.2 32 s Qwen3.5/3.6-35B-A3B 128 25.3 GB 32.2 17 s gemma-4-31B-it 32 21.4 GB 30.9 11 s Two distinct failure modes Pushing past these ceilings hits two distinct failure modes that show up at different points: Memory cliff (~27–28 GB peak). When peak crosses the system-default GPU-wired cap, macOS absorbs the overflow via compressed memory. It works, but throughput drops 9–30 % depending on the activation/static-weight ratio. 9 B is most sensitive (−30 % at 28.0 GB); 27 B is least (−9 % at 29.2 GB) since most of its footprint is static weights. MTLResource-count cliff (independent of bytes). Apple Silicon GPUs cap simultaneously-bound MTLResources at 499 K, and per-iter resource count grows with both num_layers and seq_len. 2 B at T = 3,200 hits a hard kIOGPUCommandBufferCallbackErrorOutOfMemory at iter 1 even though peak memory is only 22 GB. Don't extrapolate "more headroom in GB" → "can push T further." The numbers in the table above are the conservative recipes: safe and reproducible. Pushing them further is possible if you know which cliff you're approaching. Output is PEFT-compatible Adapter output is standard adapter_config.json + adapters.safetensors, loadable by peft, mlx-lm, or any tool in the LoRA ecosystem. mlx-optiq adds one extra file: optiq_lora_config.json records the per-layer rank distribution so you can inspect what by_bits actually picked. terminalbash $ optiq lora info ./my_adapter # mlx-optiq LoRA adapter # base model: mlx-community/Qwen3.5-9B-OptiQ-4bit # total params: 2.9M trainable # rank distribution: # rank 8 (4-bit layers): 96 modules # rank 16 (8-bit layers): 16 modules Hot-swap at serve time Once you have an adapter, the mounted-LoRA primitive lets you keep N of them resident on a single base and switch per request via a ContextVar the server flips. ~50 MB per extra adapter on top of one base, vs ~5.6 GB per full model copy. Details in the serve docs. Full reference for the trainer, all rank-scaling modes, and the data format is in the LoRA fine-tuning guide. --- ## Not All Layers Are Equal:Mixed-precision quantization for weights and KV cache on Apple Silicon. Source: https://mlx-optiq.com/blog/not-all-layers-are-equal Research · March 20, 2026 Not All Layers Are Equal: Mixed-precision quantization for weights and KV cache on Apple Silicon. Topic Layer sensitivity Reading time 7 min Related sensitivity-aware LoRA Some transformer layers are up to 56× more sensitive to quantization than others. Per-layer KL-divergence measurement is the foundation for mixed-precision quantization on Apple Silicon. The standard approach is uniform 4-bit: every layer gets the same treatment. mlx-optiq instead measures which layers actually need precision and which don't, then allocates the bit budget where it matters. The KV cache, which gets less attention, becomes the dominant memory cost at long contexts, where uniform quantization breaks down. This is the sensitivity-driven quantization pipeline we built for MLX. Most of mlx-optiq grew out of this work. The setup We worked with Qwen3-0.6B-base on an M3 Max, using Apple's MLX framework. The approach is straightforward: for each layer, temporarily quantize it, run calibration data (WikiText-2) through the full model, and measure the KL divergence of the output logits against a reference. This gives a per-layer sensitivity score at each candidate bit-width. A greedy knapsack algorithm then assigns bits (more to sensitive layers, fewer to robust ones) to hit a target average bits-per-weight. We tested this on three model types: LLMs (Qwen3-0.6B), vision-language models (Qwen3-VL-2B), and speech recognition (Qwen3-ASR-0.6B). Result 1: weight quantization, +17 pp on math reasoning With per-layer 4 / 8-bit allocation at 4.5 average bits-per-weight, the mixed-precision model scores 51 % on GSM8K (100-question math reasoning) compared to 34 % for uniform 4-bit: a 17-percentage-point improvement at only 11 % more model size. Model Size BPW Perplexity GSM8K Uniform 4-bit 320 MB 4.0 19.2 34 % Mixed 4 / 8-bit 355 MB 4.5 17.4 51 % Mixed 3 / 4-bit 266 MB 3.5 32.7 9 % The sensitivity analysis shows a clear structure: lm_head has 8× the sensitivity of the median layer, the last transformer block layers go up to 6.8×, and early layers with downstream amplification reach 10.4×. Query / key projections sit at 1.0×, safely quantized to minimum bits. The mixed-precision allocation dominates the entire Pareto frontier between uniform 3-bit and uniform 8-bit. At every model size, it matches or beats the uniform baseline. Mixed-precision dominates uniform-bit at every model size on the quality-vs-size Pareto frontier. Result 2: KV cache, the hidden memory problem The KV cache stores key / value projections for all past tokens during generation. For Qwen3-0.6B at 4 K context, the FP16 KV cache is 448 MB: already 140 % of the model weights (320 MB). At 16 K context it's 1.8 GB, or 5.6× the weights. MLX supports uniform KV cache quantization, but when we measured per-layer sensitivity, we found Layer 0's KV cache is 56× more sensitive than average. Uniform 4-bit KV doesn't know this. The result is catastrophic: Uniform 4-bit KV sends perplexity from 21 → 507. Mixed-precision keeps the few sensitive layers at 8-bit and recovers most of the savings without breaking quality. KV Config Perplexity KV memory savings FP16 (reference) 21.2 n/a Uniform 8-bit 21.6 44 % Mixed 5-bit 31.1 62 % Uniform 4-bit 507.5 69 % Uniform 4-bit KV quantization sends perplexity from 21 to 507. The fix: keep 7 sensitive layers (out of 28) at 8-bit. This costs only 7 % more KV memory than uniform 4-bit but gives 16× better quality. The per-layer sensitivity chart shows why. Most layers are robust, but Layer 0 is an extreme outlier that must stay at higher precision. Per-layer KV sensitivity. Layer 0 is an extreme outlier; everything else is roughly equal. Result 3: the full stack, 57 % memory savings at 16 K context Combining mixed-precision weights and mixed-precision KV cache, the savings compound at long contexts: Context length Default (U4 + FP16 KV) Full mixed Savings 4 K tokens 768 MB 495 MB 35 % 16 K tokens 2,112 MB 915 MB 57 % 32 K tokens 3,904 MB 1,475 MB 62 % At 16 K context, mixed-precision saves 1.2 GB. On a base MacBook with 8 GB unified memory, this is the difference between fitting and not fitting. Memory footprint as a function of context length. The mixed-precision gap widens with context. The longer the prompt, the more KV-cache compression matters. KV cache sensitivity also depends on weight quantization. Different weight bit-widths produce different KV sensitivity profiles, and 10 of 28 layers get different KV allocations depending on the weight quantization. You can't just measure KV sensitivity on the FP16 model and call it done. Joint weight + KV sensitivity across configurations. The two are coupled; measuring them in isolation misses the interaction. Result 4: works across modalities The same approach works for vision-language and speech models: Qwen3-VL-2B: Mixed-precision recovers 32 % of the quantization accuracy gap on AI2D diagram understanding (41 % vs 35 % for uniform 4-bit) while being 25 % smaller. 85 of 104 vision encoder layers need 8-bit, while 195 of 197 language model layers are safe at 4-bit. Qwen3-ASR-0.6B: Mixed-precision reduces output divergence by 51.5 % vs uniform 4-bit on speech recognition, with WER improvement on LibriSpeech. What we learned The most sensitive layers are at the boundaries: first / last transformer blocks, lm_head, and embedding layers. Middle MLP layers are consistently robust. At long contexts, the KV cache dominates total memory. Quantizing weights while leaving the KV cache in FP16 misses the biggest cost. Uniform 4-bit KV fails badly. It looks like it saves memory (69 % reduction) until you check quality and find it's completely broken. Weight quantization and KV cache quantization aren't independent. The same model with different weight precision has different KV sensitivity profiles. Joint optimization over both gives better results than optimizing each in isolation. Static rules (like "keep first and last layer at high bits") get the direction right but miss the magnitude. Layer 0's KV cache is 56× more sensitive, not 2×. From research to production The per-layer sensitivity pass, the greedy knapsack, and the joint weight + KV optimization from this post are all in mlx-optiq now. Each is extended to the Qwen3.5 / 3.6 and Gemma-4 families and packaged for one-line install via pip install mlx-optiq. Browse the 12 pre-built quants or read the methodology docs for the productionized story. ---