Changelog

Fix: optiq serve --mtp crashed on tool-calling (and repo-id) requests

Two bugs made optiq serve --mtp unusable with a HuggingFace repo-id or with tool-calling clients. Both surfaced running the tool-eval-bench tool-calling suite against Qwen3.6-27B-OptiQ-4bit.

1. Repo-id not resolved to a local directory. --mtp with a repo-id (e.g. mlx-community/Qwen3.6-27B-OptiQ-4bit) raised FileNotFoundError: Missing config.json in mlx-community/Qwen3.6-27B-OptiQ-4bit, the MTP engine's attach() treated the repo-id string as a local path. It now resolves through a new resolve_model_dir(): a local dir with a config.json is used as-is, otherwise the string is treated as a repo-id and mapped to the cached snapshot via snapshot_download (a no-op fetch when already downloaded, which it always is under optiq serve, since mlx-lm loads the weights first). Serving with an explicit local snapshot path was the workaround.

2. logprobs=None crashed _serve_single. mlx-lm's server evaluates gen.logprobs[gen.token].item() on every yielded GenerationResponse. OptiQ's fast streaming loops (MTP, separate-drafter speculation, vision) don't surface a per-token logprob vector and were yielding logprobs=None, so every request through those paths, tool-calling requests always hit it, died with TypeError: 'NoneType' object is not subscriptable. A shared _NULL_LOGPROBS shim (returns 0.0 for any index) now backs all of those paths; the vision path already used it, and it's applied uniformly to the MTP and drafter paths too. The logprobs/top_logprobs response fields are placeholders on these paths (the default serving path never requests real logprobs).

With both fixed, --mtp serves tool-calling requests cleanly (verified: --mtp + repo-id + a tools= request returns proper tool_calls). MTP remains lossless speculative decoding, so this only affected the serve integration, not decoding quality.

Server-side tool-call healing in optiq serve

Quantized open-weight models often emit malformed tool calls, Hermes <tool_call> tags, fenced or bare JSON, trailing commas, fancy quotes, function-call form, the {"python": {...}} key-is-tool-name pattern, which mlx-lm's server leaves in content instead of parsing into OpenAI tool_calls. optiq serve now runs OptiQ's heal_tool_calls on the final completion message, recovering those into proper tool_calls and promoting finish_reason to tool_calls when a call is found, so any API client (agents, Claude Code via the OpenAI endpoint) gets clean calls. It's the same healer the Lab chat already used, now at the server layer so it applies to every client. Runs on non-streaming completions (where the full content is available); requests without tools are untouched.

Structured / JSON output in the Lab

The constrained-decoding from 0.2.7 (response_format / guided_json, via lm-format-enforcer) was serve-only; it's now wired into the Lab. The Lab server installs the structured-output path, and the Chat UI gains a JSON-mode control (off · any valid JSON · match schema, with a schema box). response_format is threaded through both the direct path and the tool-loop route. JSON mode and the tool loop are mutually exclusive (a single constrained response vs. an agentic loop), so tools are disabled while JSON mode is on.

Fix: vision sidecar shipped audio-tower convs in PyTorch layout

build_vision_sidecar copied the vision/audio towers raw out of the bf16 base (an HF/PyTorch checkpoint) without applying mlx-vlm's conv-layout sanitize(). So the published optiq_vision.safetensors stored the audio tower convolutions channel-first (PyTorch [out, in, kH, kW]) instead of MLX channel-last [out, kH, kW, in], unlike every stock mlx-community gemma-4 conversion (mlx_vlm.convert applies that transpose before saving). OptiQ's own vision/text path never touches the audio tower (and gemma-4's SigLIP patches are Linear, no conv), so it was unaffected, but third-party full-VLM loaders rejected the entire multimodal graph on the shape mismatch and silently fell back to text-only LLM (reported on gemma-4-e4b-it-qat-OptiQ-4bit).

The sidecar builder now transposes subsample_conv_projection.*.conv.weight (Conv2d) and *.depthwise_conv1d.weight (Conv1d) on store, idempotently, matching mlx-vlm exactly. In practice the wrong layout shipped on the QAT SigLIP models (gemma-4-e2b-it-qat-OptiQ-4bit, gemma-4-e4b-it-qat-OptiQ-4bit); their optiq_vision.safetensors were re-uploaded with the corrected layout (no language-tower re-quantization needed). The non-QAT e2b/e4b sidecars were already channel-last; the 12B (encoder-free gemma4_unified) and the 26B/31B sidecars carry no audio-tower convs, so they were unaffected.

Fair evaluation of reasoning models: optiq eval --reasoning

The Capability Score harness silently mis-scored always-on reasoning models (models that emit a <think>…</think> block before every answer and have no toggle to turn it off). Two failure modes: the generation tasks budget ~256 tokens, which truncates a multi-hundred-token think trace before the answer ever appears; and MMLU is scored by first-letter logit argmax (the answer token right after "Answer:"), which collapses to chance for a model trained to reason before answering. The result was a strong model looking near-random.

--reasoning fixes both. It lets the model think (no suppression), gives the generation tasks a large per-question budget (--reasoning-max-tokens, default 3072, a cap, not a fixed length, since greedy decoding stops at EOS so short traces cost nothing), strips the <think> block before extracting the answer, and scores MMLU generatively (generate with thinking, then parse the answer letter) instead of by logit argmax. Default eval behavior is unchanged, the flag is opt-in.

The 3072 default was calibrated on VibeThinker-3B (think-trace length median ~1160, p95 ~3050 tokens). On that model the difference is stark:

| Metric | Standard harness | --reasoning |

|---|---:|---:|

| MMLU | 26.8% (logit argmax ≈ chance) | 76.7% (generative) |

| GSM8K | 35.5% (thinking truncated) | 93.3% |

This makes the Capability Score usable for the whole class of thinking models going forward, not just VibeThinker.

Multi-turn agentic DPO

DPO now scores full {messages} trajectories: when chosen/rejected are message lists, the loss is taken over every assistant turn via the same multiturn.assistant_mask the multi-turn SFT path uses, instead of a single prompt→completion boundary (single-string pairs keep the old behavior). The gated-delta (qwen3_next linear-attention) and full-attention Metal kernels are routed through the DPO forward/backward, without them DPO's four passes (policy+ref × chosen+rejected), stacked with the SFT+DPO LoRA, blow the Metal 499k buffer-count cap. Same kernels the SFT trainer uses, scoped to compatible shapes.

Structured / JSON-constrained output in optiq serve

The OpenAI-compatible server now honors response_format (json_object and json_schema) plus the vLLM-style guided_json / guided_regex / guided_choice extensions. When a request carries one, OptiQ masks the model's logits each step so it can only emit tokens that keep the output valid for the schema, regex, or choice set. This makes the served model reliable as an agent/tool-call backend: the JSON it returns parses, the enum answers stay in range, the regex always matches.

It is deliberately built on lm-format-enforcer, not xgrammar: lm-format-enforcer is pure-Python (pydantic + interegular) with no PyTorch dependency, so OptiQ stays MLX-native. The constraining logit processor adds about 1 ms/token and is wired into mlx_lm.server's generation with a few small patches; free-form requests (no response_format) are untouched. For reasoning models, OptiQ auto-disables thinking when a spec is present (the grammar already forbids non-JSON text, so a <think> block would only mislabel the constrained output as reasoning instead of content).

```bash

curl localhost:8080/v1/chat/completions -d '{

"messages": [{"role":"user","content":"A person: Alice, 30."}],

"response_format": {"type":"json_schema","json_schema":{"schema":{

"type":"object","properties":{"name":{"type":"string"},"age":{"type":"integer"}},

"required":["name","age"]}}}}'

```

Model switching + local-quant listing for coding harnesses

optiq serve already inherits mlx-lm's behavior of hot-swapping to whatever model a request asks for and listing every MLX model in the HuggingFace cache via /v1/models, so a coding harness (Claude Code, Codex, Cline) pointed at the server can already enumerate downloaded models and switch between them per request.

New: --models-dir DIR additionally advertises locally-built quants (e.g. optiq_output/<name>) that were never pushed to the hub, so they show up in /v1/models and are switchable by passing their path as the request model. The served model, the local-dir quants, and the cache models are all listed together.

gpt-oss support: harmony-format evaluation

OpenAI's gpt-oss is now a supported family, and its quant evaluation needed real work because it's a reasoning model with two traits that break a naive eval harness. It emits the harmony response format, an analysis channel of chain-of-thought before a final channel with the answer, and it controls reasoning length through reasoning_effort, not the enable_thinking flag the Qwen path uses. The new optiq/eval/_harmony.py reduces channel-structured output to just the final answer before extraction (GSM8K number, MMLU letter, HumanEval code block), and concise_template_kwargs probes the chat template and applies reasoning_effort='low' so the model answers within the token budget instead of truncating mid-thought (GSM8K went from 50% to 80% on a 20-question smoke once this landed). gpt-oss's tool calls arrive in a commentary channel (to=functions.NAME … <|message|>{args}), now parsed by the BFCL extractor (0% → 70.5% on the full set), and HashHop's generation budget is raised for reasoning models so the 16-character hash isn't cut off mid-analysis (0% → 78%). The calibration loader also falls back to plain role-prefixed concatenation when a strict chat template rejects a sample, since gpt-oss's harmony template raises on a tool turn that isn't preceded by an assistant tool_call.

The published mlx-community/gpt-oss-20b-OptiQ-4bit posts a Capability Score of 71.84 vs 53.81 for a genuine uniform-4-bit baseline (+18.03, winning all six benchmarks, HashHop +59.0, MMLU +16.7, BFCL +11.0). The margin is unusually large because gpt-oss ships natively in MXFP4 (its experts are already 4-bit), so uniform-4 re-quantization compounds the loss while OptiQ keeps the retrieval-critical attention and router at 8-bit.

Re-quantizing natively-quantized (MXFP4) experts

mlx-lm loads gpt-oss's MoE experts as QuantizedSwitchLinear (native MXFP4), which nn.quantize skips because they're already quantized, so a sub-4-bit target would pass the experts straight through unchanged. optiq.backends.mlx_backend now dequantizes pre-quantized fused-expert tensors back to bf16 before re-quantizing them to the OptiQ-assigned bits, processing one expert slice at a time: a fused expert tensor is ~1 B elements on a 120 B model, which OOMs if dequantized whole and trips the Metal command-buffer GPU timeout if quantized in a single op. The path is a no-op for ordinary bf16 sources. (Converting a 120 B-class MXFP4 source this way still needs a Mac with more RAM than the source, ~60 GB exceeds a 36 GB machine at load time.)

SSD streaming covers SwitchGLU expert naming

is_streamable_moe and load_streaming now recognize fused routed-expert tensors stored under .mlp.experts. (Gemma / gpt-oss SwitchGLU) in addition to .switch_mlp. (Qwen3 / Nemotron SwitchMLP), so the SSD expert-streaming path extends to those architectures.

Quantization methods: optiq (default) and static

optiq convert --method now selects how the per-layer bit allocation is derived. optiq (default) is the exact calibration-driven logit-KL sensitivity, the gold-standard signal, but n_layers × n_bits × n_samples forward passes plus a uniform-4-bit baseline in RAM, which on a large model means hours of convert and tens of GB of memory (the cost that drove some users back to old 0.0.x releases). static is the fast path: it assigns bits by structural rules, embedding and output head, the first and last block, attention and the MoE router get the high bits; the dense MLP and routed experts stay low, feeding the same allocator at the requested candidate bits and target BPW. No calibration, no forward passes, and it loads the base lazily (names and shapes only, never resident), so it allocates a 100 B-class model on a 36 GB Mac in seconds.

Measured on Qwen/Qwen3.5-0.8B (GSM8K, 200 q, target 5.0 BPW): optiq 899 s / 34.5 %, static 7.2 s / 34.5 %, static matches optiq at 125× the convert speed and a lower BPW, because for a typical transformer architecture is most of the sensitivity signal. optiq still earns its cost when you want certainty or are characterising a new family; static is the practical path for the very large bases where exact KL is impractical, including 2/4-bit mixed quants of 100 B-class MoEs (e.g. Qwen3.5-122B-A10B) that run on a laptop through SSD expert streaming. See the methods comparison.

SSD expert streaming: run large MoE quants that don't fit in RAM

A mixture-of-experts quant holds every expert resident even though only top-k of N fire per token, so a model like Qwen3.6-35B-A3B-OptiQ-4bit (21 GB) hard-OOMs on a 24 GB Mac. optiq.runtime.moe_stream streams the active experts' weights off SSD instead: load_streaming(...) runs that model at ~4.5 GB resident, decode ~8.8 tok/s (with concurrent reads, below), output coherent.

How it works: lazy mmap slicing of the fused expert tensor doesn't bound memory (MLX materializes the whole operand), so we read only the active expert rows at the byte level with safetensors.get_slice. The uint32 expert weights (the bulk) stream; the small bf16 scales/biases stay resident. A StreamingQuantizedSwitchLinear reads the unique active experts, remaps the router indices, and runs mx.gather_qmm over that compact set, bit-identical to the resident op. Active-expert counts are bucketed to a few fixed sizes so MLX doesn't recompile the kernel every call (prefill is ~40x faster with this). Resident params are loaded by raw byte range, not mmap: MLX mmaps each shard as one Metal buffer, and the resident scales/biases are scattered across every shard, so an mmap load would pin all ~22 GB against the Metal budget; reading just each tensor's bytes (with a manual bf16 bitcast, since safetensors can't read bf16 directly) keeps the load footprint to the resident size.

Wired into optiq serve (--stream-experts/--no-stream-experts, default auto, streams only a MoE too big to fit resident) and the Lab (Settings → Server → "SSD expert streaming"). The streamed model is built on the main thread at startup and handed to the server, because mlx-lm's lazy model construction OOMs for a large MoE when run inside the request handler. optiq serve --stream-experts on Qwen3.6-35B-A3B answers requests at ~4.5 GB resident on a 24 GB Mac. Decode is slower than resident (experts read per token); only MoE models are affected.

Decode reads the active experts with concurrent pread (parallel positional reads off a shared fd, queue depth 24), which is the big throughput lever: on Qwen3.6-35B-A3B it takes decode from 3.3 to 8.8 tok/s and prefill from 0.8 to 26.5 tok/s, so a 400-token agentic generation finishes in ~49s at ~6 GB. Speculative prefetch (warm the previous token's experts during compute) is implemented but OFF by default (OPTIQ_STREAM_PREFETCH=1): when the model fits the OS page cache it adds redundant work and slightly regresses decode; it only helps when the model is far larger than RAM.

Approach validated against SharpAI's SwiftLM (MLX SSD streaming for 100B+ MoE): same residency split, same concurrent-pread design, and they independently found an in-app expert cache regresses throughput vs. the OS page cache (matching our cache-off default).

Convert: reuse a pre-placed uniform-4-bit baseline

optiq convert --reference uniform_4bit now reuses an existing _uniform_4bit_baseline in the output directory instead of always rebuilding it. Drop in a published mlx-community/<model>-4bit (or a prior run's baseline) and the convert skips the bf16-quantize step, the memory-heaviest part of the pipeline, which can swap-stall on large models. Faster and more reliable for 27B+ on a 36 GB Mac.

Live tokens/sec in chat

The Lab chat view shows a decode-speed readout while the model streams, and keeps it on the finished message, matching the Arena. The counter starts at the first token, so it reflects steady-state generation rather than prompt-processing latency.

Lab polish

• - The publish flow auto-stamps the mlx-optiq funnel banner onto quant-card READMEs at push, and the Hub surfaces each quant's Hugging Face download count.

OptiQ Lab: Arena, Hub, chat-with-files, Canvas, self-healing tools

Five additions to the local Lab UI, all reusing the existing serving plumbing.

Model Arena (/arena) loads two models and answers the same prompt side by side with tokens/sec for each. Model A runs on the main server; model B runs in a second ApiSupervisor on port + 1, started on demand. Ideal for seeing an OptiQ mixed-precision quant against a uniform one, or a quant against its bf16 base, in real time. Both models are resident at once, so it is best used with small/fast models.

Hub (/hub) browses published OptiQ quants (auto-discovered from mlx-community), searches Hugging Face for MLX models, and lists locally-converted models, each with one-click load-to-server and new-chat actions.

Chat with files (lightweight RAG with citations). Attaching a non-image file now indexes it instead of dumping the whole document into the prompt: a dependency-free BM25 retriever pulls the chunks most relevant to each question, prepends them with [n] citation markers, and the Lab renders a sources panel under the answer. Only the retrieved chunks enter the context.

Canvas. Assistant messages that contain a fenced html block render live in a sandboxed allow-scripts iframe next to the answer, with a view-source toggle.

Self-healing tool calling. On top of the existing malformed-call healing, a per-call retry budget stops the model from re-running the same failing tool call indefinitely (3 attempts, then it is told to change course). The tool card shows a retry limit badge when the budget is spent.

dhara-250m: the Diffusion LLM family's second model

OptiQ now ports, quantizes, and serves dhara-250m, a custom 250M tri-mode model and the second member of the Diffusion LLM family. dhara is not loadable by stock mlx-lm (it adds Canon depthwise-conv layers, QK-norm applied after RoPE, and a logit soft-cap); OptiQ ships a bit-exact mlx-native port (optiq/mlx_lm_patches/dhara_ar.py) that registers with mlx-lm, so the whole pipeline, convert, LoRA, eval, serve, KV-quant, works through the autoregressive path. The Canon convs, QK-norm, and soft-cap are not Linear modules, so they stay at bf16 automatically; only the attention and MLP projections are quantized.

It decodes three ways from one set of weights: autoregressive (the accuracy reference), self-speculation (--mtp, the recommended default), and block-diffusion. Self-speculation drafts a block in one forward and AR-verifies it (two forwards per round, prefix-cached, no commit pass), emitting output identical to autoregressive greedy decode while committing ~3–4 tokens per round, ~1.4× faster than token-by-token AR at the same accuracy on an M3 Max. Block-diffusion is prefix-cached too (O(block) per step, not O(sequence)).

dhara is built to be fine-tuned on a specific task (like Google's Gemma-270M). The full 6-benchmark Capability Score is preserved by quantization, bf16 8.34 / uniform-4 8.79 / OptiQ-mixed 8.54, all within run-to-run noise, so 4-bit is lossless here and the win is size: 460 MB → 170 MB. The model is overhead-bound, so 4-bit and bf16 decode at the same speed; the quant buys memory and disk, not throughput.

DiffusionGemma: OptiQ's first diffusion LLM

OptiQ now quantizes, runs, fine-tunes, and serves Google's DiffusionGemma-26B-A4B-it, a block/masked-diffusion image-text-to-text model that decodes by iteratively un-masking a fixed 256-token canvas instead of left-to-right. It is the founding member of a new Diffusion LLM family.

DiffusionGemma is not loadable by stock mlx-lm or mlx-vlm, so OptiQ vendors a trimmed, dependency-free decoder (a subset of mlx-vlm 0.6.3, MIT) under optiq/vlm/_mlxvlm/, core stays mlx-lm-only, the same principle as the vision sidecars. The OptiQ pipeline measures per-layer sensitivity on the denoising-canvas logits (KL(uniform-4 ∥ layer-at-8bit)) and the greedy knapsack moves the 8-bit budget off the dense-MLP (where mlx-vlm's hand-coded recipe puts it) and onto the early-layer attention + routers the measurement flags as more sensitive. At the same ~4.66 bpw, the OptiQ quant posts a 59.90 Capability Score vs the published 4-bit's 59.84 (MMLU +2.9, HumanEval +1.2) while being 0.5 GB smaller.

What works on DiffusionGemma:

• * Text + image generation, optiq.vlm.diffusion_gemma.{load, generate} (generate(..., images=[PIL.Image])). Image preprocessing reuses the Gemma-4 SigLIP path, bit-exact to mlx-vlm.
• * LoRA fine-tuning, train_diffusion_lora trains with the model's native denoising objective (corrupt the target to a random noise level, predict the clean tokens), not autoregressive cross-entropy. mlx-lm's tuner can't load or correctly train a diffusion model; this can.
• * optiq serve, auto-detects diffusion_gemma and routes mlx_lm.server through the vendored decode with the fast confidence-threshold sampler (4.6–5× faster than the model's entropy-bound default), retrying on the occasional empty canvas.

Not applicable (diffusion is non-autoregressive): MTP / speculative drafting and KV-cache quant, the parallel canvas un-masking is the native analog of MTP, and the fixed canvas means the KV cache only holds the prompt.

Also fixes a convert bug where quantize_model dropped vision_config from the saved config (the vision weights were present but never loaded, silently text-only); multimodal config keys are now preserved.

Published: mlx-community/diffusiongemma-26B-A4B-it-OptiQ-4bit.

Friendly error when the Lab extra is missing

optiq lab without the optional lab extra installed used to fail with a bare ModuleNotFoundError: No module named 'flask' (then jwt, then argon2 if you installed them one at a time) - a confusing dependency-chase that looked like a broken package. lab_cmd now catches the ImportError and prints exactly what to run: pip install 'mlx-optiq[lab]', noting that only the Lab web UI needs the extra and that serve / convert / eval / lora / kv-cache all run on the core install. Pure UX; no behavior change for anyone who already has the extra.

optiq serve no longer requires the Lab extra

A core install (pip install mlx-optiq, no [lab]) followed by optiq serve raised ModuleNotFoundError: No module named 'flask'. serve runs on mlx_lm.server and does not use flask, but it reaches optiq.lab.mlx_cleanup for RAM cleanup, and the optiq.lab package __init__ eagerly imported the flask app, so loading any optiq.lab.* submodule pulled flask in at import time. The Lab app import is now lazy (optiq/lab/__init__.py via __getattr__), so serve (and every other non-Lab command) runs on the core dependency set; only the Lab UI itself needs the [lab] extra. A subprocess-isolated regression test (tests/test_core_imports_without_extras.py) imports the full serve graph with flask and the other Lab-only deps blocked, so a core install can't regress again. Latent since v0.1.0; never caught because dev/release machines always have the Lab extras installed.

Vision (image) support on Gemma-4 and Qwen3.5/3.6

OptiQ answers image+text prompts on the Gemma-4 and Qwen3.5 / Qwen3.6 families. The language tower is still OptiQ mixed-precision quantized and decoded by mlx-lm; the vision tower runs from a vendored encoder (no mlx-vlm runtime dependency), kept at bf16 in a sidecar (optiq_vision.safetensors), and its features are spliced into mlx-lm's decode through the input_embeddings hook. Because mlx-lm selects weights with glob("model*.safetensors"), it never sees the sidecar: the same published repo loads as a text-only model under stock mlx-lm and as a full image+text model under OptiQ. No separate vision quant; vision and audio stay bf16 (the ecosystem norm, since int4 vision hurts OCR and fine detail).

Image input now works on every Gemma-4 size and on the Qwen3.5 / Qwen3.6 family. The Qwen path vendors the Qwen3-VL vision tower (Conv3d patch embed, windowed attention with 2D rope, patch merger) and the Qwen image processor (smart-resize + merge-grouped patchify) into optiq/vlm/qwen3_5/; both reproduce mlx-vlm's pixel values and vision features to max|Δ| = 0.0. Because Qwen's ViT carries its own 2D positional encoding, sequential backbone positions are sufficient (OCR comes out exact), so no mRoPE patch to mlx-lm is needed. Validated end-to-end on Qwen3.5-0.8B at 4-bit (OCR, counting, shapes, colors, spatial, scene description) with MTP, LoRA, KV-quant, and serving all unaffected.

Gemma-4 gained two more vision arches beyond the e2b/e4b SigLIP tower: the 26B/31B SigLIP variant (27 layers, no clip scalars), and the 12B gemma4_unified model, which is encoder-free (a light patch embedder feeding the shared backbone). The 12B needs bidirectional attention over the image-token span (use_bidirectional_attention=="vision"); a one-shot wrapper on gemma4_text._make_masks provides it (text/decode stay causal), which fixed OCR exactly. All five Gemma-4 quants and the small Qwen3.5 quants (0.8B/2B/4B/9B) now carry the optiq_vision sidecar on Hugging Face.

optiq.vlm resolves the right front-end per model_type (gemma4, gemma4_unified, qwen3_5); the sidecar build (build_vision_sidecar) downloads only the shards that hold vision/audio weights, so attaching a sidecar to a large quant doesn't pull the full bf16 base.

Gemma-4 SigLIP path

Numerically exact. Feeding mlx-vlm's own pixel values through the vendored path reproduces its vision_tower and embed_vision outputs to max|Δ| = 0.0 (266 soft tokens on gemma-4-e2b). Two correctness details the vendoring had to get right: mlx-lm's gemma4_text always rescales input_embeddings by embed_scale, so the vision features are pre-divided by it; and the per-layer inputs zero out the image-token positions before projection.

Serving and the Lab. optiq serve and the Lab's API server auto-enable image serving when the model ships an optiq_vision sidecar. Newer mlx-lm routes requests through a batch path that bypasses the old streaming hook and drops non-text content, so image requests are tagged on the per-request args at submission, forced onto the single-request path, and handed off inside the synchronous generation thread (no cross-request image leakage). The Lab Chat tab takes image uploads directly: attach an image, ask a question, get the analysis. On gemma-4-e2b at 4-bit, counting, shapes, colors, OCR, and chart reading all answer correctly.

OptiqEngine.generate / generate_stream take images= (or a full messages= list with image_url parts) for the multimodal prefill. MTP, LoRA, KV-quant, and the text path are unchanged: the vision path only runs when a request actually carries an image.

LoRA on Nemotron-H hybrid models

optiq lora train failed on NVIDIA Nemotron-3 (Mamba2 + attention hybrid) with "Could not locate transformer blocks": the trainer only looked for model.model.layers / model.language_model.model.layers, but Nemotron-H registers its blocks under model.backbone.layers. The block-discovery now also handles backbone.layers and a top-level model.layers list, and the per-layer rank resolver matches backbone.layers.* metadata keys. The apply loop already skips blocks without target modules, so the pure-Mamba layers are left alone and adapters attach only to the attention blocks. Verified end-to-end on mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit.

New-model smoke test

Added scripts/smoke_new_model.py, runs every basic OptiQ + Lab feature (load+generate, LoRA, serve, drafter speculative decoding, quantized KV cache, eval smoketest, Lab app + dataset template) against a model and reports PASS/FAIL/SKIP. Run it before publishing any new quant so arch-specific breakage (like the Nemotron-H LoRA gap above, or the Gemma-4-unified loader) is caught up front.

Unified Gemma-4 (gemma4_unified) support

Google's unified text+vision+audio Gemma-4 checkpoints (e.g. mlx-community/gemma-4-12B-it) ship under model_type: gemma4_unified / gemma4_unified_text. The language tower is architecturally mlx-lm's gemma4_text (MoE blocks, double-wide MLP, global_head_dim, attention_k_eq_v are all implemented there), and gemma4.py's sanitize() already drops the vision/audio towers. OptiQ now registers gemma4_unified → gemma4 and gemma4_unified_text → gemma4_text remappings in mlx_lm_patches, so optiq convert / optiq serve load, quantize, and serve the text path of these models. Vision/audio weights are dropped during conversion (text-inference artifact), consistent with how OptiQ already handles multimodal Gemma-4 bases.

mlx-lm requirement for gemma-4-12B: the unified gemma4_text path needs one mlx-lm commit (#1240, the shared-layer KV-projection sanitize fix) that postdates the 0.31.3 PyPI release. The PyPI wheel for mlx-optiq declares mlx-lm>=0.31.3 so it installs cleanly; for gemma-4-12B specifically, install mlx-lm main until upstream cuts a release with #1240: pip install "mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@df1d3f3c9a7aae402dcbb8f41d4c36bcc13a50ae". Every other model (Nemotron 4B/30B, Qwen3.5/3.6, Gemma-4 multimodal, MiniCPM5) works on stock mlx-lm 0.31.3.

mlx-lm pinned to main

The unified gemma4_text fields above postdate the 0.31.3 PyPI release, so the core mlx-lm dependency is pinned to a validated main commit. Bump the SHA in pyproject.toml when adopting newer upstream fixes.

Faster sensitivity sweep on the uniform-4-bit reference path

The --reference uniform_4bit sweep was probing every candidate bit-width per layer, including the one that equals the baseline's bit-width (4). That probe re-quantizes a layer to exactly what it already is, so KL(reference || current) is 0 by construction (observed as ~1e-18). OptiQ now records 0.0 for the baseline-equal bit directly and skips its forward passes, roughly halving convert time for the common candidate_bits=[4, 8] case. No effect on the bit assignment, the optimizer only needs the improvement from upgrading 4 → 8, which the 8-bit probe still measures.

optiq eval KL reference no longer 404s on new bases

The KL eval's auto/uniform_4bit reference picker constructed mlx-community/<stem>-4bit and assumed it existed on the Hub. For a brand-new base (e.g. gemma-4-12B-it) that repo doesn't exist, so --task all / --task kl crashed with a 404 mid-suite. It now checks repo_exists first and, when the published quant is missing, builds a local uniform-4-bit quant of the base once and caches it under ~/.cache/optiq/kl_refs/ for reuse. Published references are still used unchanged when they exist.

Serve: explicit ready banner after mlx-lm hand-off

optiq serve previously ended at mlx-lm's "not recommended for production" UserWarning, with no follow-up "server ready" line. mlx-lm's own Starting httpd at … port … notice is logging.info, but a root logging handler already exists by then (configured at import by huggingface_hub / transformers), so mlx-lm's basicConfig(INFO) is a no-op and the line is swallowed. The server is up, but the warning reads like the process died, especially on RAM-tight machines where the lazy model load then pauses the first request.

Now optiq serve forces the root logger to INFO so Starting httpd actually appears, and prints its own banner with the resolved URL plus a one-line callout that the model loads on the first request and the mlx-lm warning is harmless.

Eval: --baseline now works with --task all

optiq eval --task all --baseline X previously silently dropped the --baseline flag and ran the full 6-metric suite on a single model. The flag worked only with --task gsm8k. Anyone running the standard "OptiQ vs uniform-4-bit Capability Score comparison" before publishing to HF had to invoke optiq eval twice and stitch the table by hand, and the cost of finding out is a full 8-hour re-run, which is what happened on the v0.1.5 Nemotron 4B convert.

Now both modes accept --baseline. With --task all, the candidate runs through the full suite first, then the baseline runs through the same suite, and a two-row comparison table prints at the end:

```

Capability Score comparison (candidate vs baseline)

──────────────────────────────────────────────────────────────────────

metric candidate baseline Δ (cand-base)

MMLU 64.0% … +… pp

GSM8K 81.5% … +… pp

IFEval 56.2% … +… pp

BFCL 75.5% … +… pp

HumanEval 77.4% … +… pp

HashHop 27.0% … +… pp

Capability 63.60 … +…

Disk (GB) 2.94 … +…

```

With --output-json, the record nests both runs under candidate and baseline keys (single-model runs keep the flat schema).

Sensitivity: index already-fused switch_mlp routed experts (MoE bf16 sources)

The bf16-streaming sensitivity path (--reference uniform_4bit) recognized routed-expert tensors only in the HuggingFace raw layout, 3D *.experts.gate_up_proj / *.experts.down_proj, which it splits and renames to the mlx-lm switch_glu/switch_mlp wrapper. NVIDIA's Nemotron 3 Nano 30B-A3B (mlx-community/...-MLX-BF16) instead ships its MoE already fused in mlx-lm format: mixer.switch_mlp.fc1.weight [128, 1856, 2688] and mixer.switch_mlp.fc2.weight (a plain per-expert fc1→fc2 MLP, no gate/up split). Those names matched neither expert branch, so all 46 routed-expert tensors (23 MoE layers × 2) were skipped for "no bf16 source match," never probed, and never assigned a per-layer bit.

Consequence: mlx_lm.convert quantized the routed experts at its default precision instead of an OptiQ-optimized 4-bit. Since the fused experts dominate a 30B-A3B's parameter mass, the "mixed" model ballooned to 8.34 bits-per-weight / 31.4 GB, nearly double the uniform-4-bit baseline (4.50 bpw / 17.0 GB), while the optimizer's own achieved_bpw read a misleading 5.20 (computed only over the layers it had actually seen). Dense models (including Nemotron 3 Nano 4B) were never affected, they have no switch_mlp.

The fix adds a branch to _index_bf16_layers that indexes 3D tensors ending in .switch_mlp.fc1.weight / .switch_mlp.fc2.weight directly under their own module path (no split, no rename, the running QuantizedSwitchLinear is named identically). Routed experts now get sensitivity scores and proper mixed-precision bit assignment. The HF-raw .experts.* path is unchanged, so Qwen3.5/3.6 and Gemma-4 MoE bases convert exactly as before.

KV-cache: NemotronH hybrid model support

optiq kv-cache previously crashed with ZeroDivisionError on Mamba-attention hybrids like NVIDIA Nemotron 3 Nano (model_type: nemotron_h). Two problems compounded: (1) the code looked for layer.self_attn, but NemotronH names its attention submodule layer.mixer (same attribute holds Mamba2 mixers and MLP blocks too). (2) mlx_lm.utils.make_prompt_cache returns one cache slot per stateful layer, but NemotronH skips MLP layers in that list, so on a 42-layer model the cache list has 25 entries, and caches[i] does not correspond to model.layers[i]. The result was that the four KVCache slots (one per real attention layer) got mapped to Mamba indices and rejected by a self_attn attribute check; sensitivity analysis emitted zero results and the optimizer divided by zero.

The fix walks the layer list, classifies each as attention / Mamba (SSM) / MLP via attribute probes (q_proj / qkv_proj / wq for attention; in_proj + conv1d/x_proj/dt_proj/A_log/D for SSM), and builds an explicit cache_idx → layer_idx map by counting cache-bearing layers in order. The sensitivity loop then indexes the prompt cache by cache_idx (correct slot) while reading attention metadata from model.layers[layer_idx] via a _attention_module lookup that tries self_attn → attention → mixer in order. The memory-estimate helper in run_kv_benchmark got the same _attention_module lookup so the serve benchmark path stops assuming self_attn. Flat transformer architectures (Qwen3, Gemma-4, etc.) hit the identity mapping branch and behave exactly as before.

Concrete impact: on mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit the four attention layers (model indices 12, 17, 24, 32) now generate a valid kv_config.json with three layers at 4-bit and one at 8-bit, 5.0 average KV bits.

Optimizer: arch-aware per-block floor for Mamba / SSM / MLP-only blocks

_apply_block_aware_floor previously promoted n_floor_per_block=2 components on every block. That carve-out exists to keep late full-attention blocks from collapsing to all-low-bits, on Qwen3.5-4B v0.1.0, layers 23 and 31 had all four of q/k/v/o at 4-bit and IFEval regressed -12.9 pp. Transformer blocks have ~7 linears (q/k/v/o + gate/up/down), so flooring 2 of 7 is a small tax.

Mamba / NemotronH-style mixer blocks and MLP-only blocks have just 2 linears each (in_proj/out_proj for Mamba, up_proj/down_proj for MLP). Flooring 2 of 2 promotes the entire block, and over 42 blocks that's the whole model. Concrete case: NVIDIA-Nemotron-3-Nano-4B has 42 NemotronH blocks; with the old floor, 84 of 93 components were floored pre-knapsack and the achieved BPW landed at 7.48 against a 5.0 target, defeating mixed-precision.

The fix: blocks with <4 attention components (Mamba mixers, MLP-only blocks, linear-attention variants) cap their per-block floor at 1. Full-attention blocks (q/k/v/o all present) still honor the global n_floor of 2.

Result on Nemotron-3-Nano-4B:

| | before | after |

|---|---|---|

| Floor pre-upgrades | 84 / 93 | 46 / 93 |

| Achieved BPW (target 5.0) | 7.48 | 5.43 |

| Model size | 3807 MB | 2938 MB |

Transformer-only architectures (Qwen3, Gemma-4) are unaffected, their blocks already trip the full-attention branch. The patch lives in optiq/core/optimizer.py.

LoRA merge + model export: optiq lora merge and optiq lora export

Two new CLI commands that close the gap between "I trained adapters" and "I shipped a model."

optiq lora merge, rank-concat merges N LoRA adapters into a single composite adapter. The merge is mathematically exact, not a low-rank approximation: for sources with ranks r1, r2, … on a layer, the merged adapter on that layer has rank r1 + r2 + …, and its forward pass reproduces the sum of the original LoRA residuals exactly. Per-source scales are folded into the lora_b matrices so the merged adapter writes scale=1.0. Use case: after the standard SFT → DPO continuation (--mount-adapter), fold the two adapters into one drop-in artifact for shipping. Output is PEFT-compatible, any LoRA-aware runtime can load it.

```bash

optiq lora merge ./adapters/sft ./adapters/dpo -o ./adapters/merged

```

optiq lora export, bundles a base model + one or more LoRA adapters into a single self-contained directory ready for optiq serve --model <dir> or mlx_lm.generate --model <dir>. The layout puts the merged adapter at the top level (so the directory loads as a single model) and preserves the original adapters under source_adapters/<name>/ for reference. Writes an optiq_export.json audit trail recording the source adapters and the merge decision.

```bash

optiq lora export mlx-community/MiniCPM5-1B-OptiQ-4bit \

--adapter ./adapters/humanizer-sft \

--adapter ./adapters/humanizer-dpo \

-o ./humanizer-1B-OptiQ-4bit

```

The exported directory is HF-Hub-ready: huggingface-cli upload <repo_id> ./humanizer-1B-OptiQ-4bit ships the model, the merged adapter, and the unmerged source adapters together.

The Lab UI's Finetune wizard Step 5 now exposes the same flow without leaving the browser:

• - Combine with another adapter, checkbox + adapter picker that lists every local adapter (auto-discovered under ~/.optiq/lab/models, ./adapters/, and ~/.optiq/lab/adapters/). Calls the new POST /api/finetune/merge endpoint.
• - Bundle as a self-contained model, checkbox + base-model field that runs the export. Calls POST /api/finetune/export.
• - Push to Hugging Face, same button as before, now smart about which artifact to push (exported dir → merged adapter → bare trained adapter, in that preference order). New POST /api/finetune/list-adapters endpoint backs the picker.

DPO continuation from SFT: optiq lora train --mount-adapter

The standard alignment recipe is SFT first, then DPO continuing from the SFT weights, not "train DPO from a fresh zero-init LoRA on top of the base." The latter is what optiq lora train --method dpo did before this, which works in principle but typically can't outperform SFT on small models because the preference signal alone isn't strong enough to reach the SFT distribution from scratch.

New flag --mount-adapter PATH (and corresponding OptiqLoraConfig.mount_adapter field). For every layer that the SFT adapter covers, the trainer constructs a StackedLoRALinear carrying:

• * the base (quantized) Linear, frozen
• * the SFT (lora_a, lora_b) loaded from the mounted adapter, frozen
• * a fresh trainable DPO (lora_a, lora_b) at zero init

Forward becomes y = base(x) + sft_scale * (x @ sft_a @ sft_b) + dpo_scale * ((dropout(x) @ lora_a) @ lora_b). For the DPO reference forward, _set_lora_scale flips only the trainable self.scale to 0, the frozen SFT contribution stays active, so the KL term is anchored against base + SFT (= the SFT model), exactly the standard alignment-pipeline definition of "DPO continuing from SFT." Layers the SFT adapter didn't cover fall back to plain LoRALinear so the rest of the OptiQ adapted-layer set still gets a DPO LoRA.

The trainable / frozen split is enforced via MLX's model.freeze() + model.unfreeze(keys=["lora_a", "lora_b"], recurse=True), so the optimizer and _save_adapter only see the DPO delta. The saved adapter is the clean delta-from-SFT and composes with the SFT adapter at serving time via OptiQ's multi-LoRA registry (optiq serve --adapter ./sft --adapter ./dpo, request body "adapter": "sft+dpo").

MoE expert projections currently fall back to plain LoRA, stacked MoE is a separate refactor.

Mixed-precision KV cache now works on sliding-window models (Gemma-4, Cohere R2, OLMo 3, …)

Upstream mlx-lm ships RotatingKVCache.to_quantized() as raise NotImplementedError("RotatingKVCache Quantization NYI"). That single line blocked KV-cache quantization on 15 model families that use sliding-window attention: Gemma 3 / 3n / 4 (text), Cohere Command R 2, OLMo 3, EXAONE 4 / MoE, Ministral 3, Recurrent Gemma, Baichuan M1, AFMoE, MiMo v2 Flash, Step 3.5, GPT-OSS, and the Llama variants with SWA.

optiq.runtime.kv.RotatingQuantizedKVCache is a drop-in subclass with affine-quantized storage ((packed_uint32, scales, biases) 3-tuples, same layout as mlx-lm's plain QuantizedKVCache) plus the rotating-buffer trim / temporal-order logic preserved from the parent. update_and_fetch returns the tuple form so the standard quantized SDPA path runs unmodified, no custom Metal kernel.

patch_rotating_to_quantized() installs three patches at once: the to_quantized override, an SDPA dispatch fix for the Gemma-4 KV-sharing edge case (where one layer's quantized K/V tuples are passed to a downstream layer whose own cache=None, which the upstream dispatcher routed to the fp16 fast path because hasattr(None, "bits") is False), and a QuantizedKVCache.update_and_fetch wrapper that feeds a producer-cache registry so the dispatch can recover bits / group_size when needed.

optiq serve installs the patch automatically when --kv-bits or --kv-config is set, and the per-layer mixed-precision conversion now picks RotatingQuantizedKVCache vs QuantizedKVCache based on the source cache type. optiq kv-cache analysis also installs the patch and substitutes rotating quantized caches at probe time, so sensitivity sweeps run on Gemma-class models for the first time.

End-to-end verified on mlx-community/gemma-4-e2b-it-OptiQ-4bit: full 4-element stack (OptiQ-4bit weights + per-layer mixed-prec KV + Gemma-4 -assistant drafter spec decoding + OptiQ-trained LoRA adapter) returns correct trained and general responses at 37.7 tok/s on M3 Max. Per-element wall-clock breakdown:

| Stack | tok/s | speedup |

|---|---:|---:|

| OptiQ-4bit weights only (baseline) | 8.8 | 1.00× |

| + per-layer mixed-prec KV (--kv-config) | 13.3 | 1.51× |

| + Gemma-4 -assistant drafter spec decode (--drafter) | 16.0 | 1.82× |

Gemma-4 kv_config.json shipped with all four pre-built quants

Mirroring the Qwen-side bundling done in v0.1.2-followup, each Gemma-4 OptiQ-4bit repo now carries the recommended kv_config.json from a real sensitivity-analysis pass, ready to point optiq serve --kv-config at. Drop-in for the average user; the model still loads fine without the file.

| Repo | KV layers | Avg BPW |

|---|---:|---:|

| mlx-community/gemma-4-e2b-it-OptiQ-4bit | 15 | 5.07 |

| mlx-community/gemma-4-e4b-it-OptiQ-4bit | 24 | 5.00 |

| mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit | 30 | 5.07 |

| mlx-community/gemma-4-31B-it-OptiQ-4bit | 60 | 5.00 |

Notes for users on earlier OptiQ versions

The new RotatingQuantizedKVCache runtime is required to use the bundled kv_config.json on any Gemma-4 quant. On optiq <= 0.1.2 the per-layer KV-quant path will still fall over with NotImplementedError: RotatingKVCache Quantization NYI. Upgrade with pip install -U mlx-optiq to land the fix.

The 13 Qwen / MiniCPM kv_configs shipped via discussion-thread followup against v0.1.2 are unaffected, those families don't use sliding-window attention.

MiniCPM5 family quant (new)

Adds mlx-community/MiniCPM5-1B-OptiQ-4bit, a 1.08B-parameter Llama-architecture base from OpenBMB with hybrid enable_thinking reasoning. 875 MB on disk, Capability Score 30.28, +4.44 over uniform-4-bit. HumanEval jumps from 45.7% (uniform-4) to 57.9% (OptiQ mixed-precision); HashHop recovers from a 0% floor to 4.0% overall. Apache-2.0. Built with the same optiq convert --target-bpw 5.0 --candidate-bits 4,8 recipe as the rest of the family. Family guide at /docs/minicpm5.

New Labs dataset template: hf_dataset_import

Generic non-LLM template that pulls a public Hugging Face dataset by id, optionally filters rows by a column value, slices to a row cap, and emits the chosen output format (text / messages_user_only / prompt_completion). Closes a gap vs Unsloth Studio's Hub picker and NeMo Data Designer's HuggingFaceSeedSource. Use it as the first stage of any pipeline that starts from a published corpus: EditLens, no_robots, dolly, or your own dataset.

Multi-LoRA serving: "base" sentinel for adapter bypass

optiq serve --adapter X auto-applies the mounted adapter to every request when only one is registered. To opt out of that auto-activation on a per-request basis without restarting the serve, the adapter (or adapters) request-body field now accepts the literal values "base", "none", "off", or the empty string "", all of which generate from the underlying base model with no adapter active. Useful for A/B comparing adapter vs base from the same served process, e.g. when constructing DPO preference data from with-adapter (chosen) vs no-adapter (rejected) samples of the same prompt. Unknown adapter names still 400 as before.

Multi-LoRA serving (CLI + Labs)

optiq serve --adapter now accepts the flag multiple times. One value keeps the classic mlx-lm --adapter-path boot; two or more activate the OptiQ mounted-LoRA path so all adapters stay resident on one base and switch per request via a ContextVar in the forward pass (no model reload between switches). The mounted-LoRA primitive (optiq.adapters.registry + mount.py) existed since v0.0.x but had never been wired into either optiq serve or the Labs UI; this closes that gap.

The Labs Server settings page lists adapters auto-discovered under ~/.optiq/lab/models and lets you add arbitrary local paths. The Chat surface shows a dropdown that switches the active adapter per request once the running server has any mounted. A new GET /v1/adapters endpoint reports the registered names + per-layer mount counts for inspection.

DPO trainer memory fix

optiq lora train --method dpo used to peak at 75 GB on a 1B model at --max-seq-length 2048 on a 24 GB Mac, spending an hour thrashing compressed swap before producing the first iteration. Two real bugs, both fixed:

1. Reordered the four DPO forward passes so reference forwards (adapter scale=0) materialize their scalar log-probs via mx.eval and detach via mx.stop_gradient before policy forwards run. Previously all four forwards were part of the same nn.value_and_grad-traced graph and their intermediate (B, L, V) logits + per-layer activations sat in scope simultaneously. Now only the two policy forwards hold activations for backward.

2. Honored config.grad_checkpoint in the DPO path. The SFT trainer used gradient checkpointing for releases; the DPO trainer didn't reference the flag at all. Now mirrors mlx-lm's pattern by calling mlx_lm.tuner.trainer.grad_checkpoint on the first transformer block.

Verified on MiniCPM5-1B-OptiQ-4bit with the humanizer DPO data (767 train / 86 valid pairs, 24 GB M4 Mac): peak physical memory drops from 75 GB to 18 GB, 300 iters complete in ~42 minutes.

DPO defaults (CLI + Labs)

optiq lora train --method dpo and the Labs Finetune wizard now apply DPO-aware defaults when the user does not pass a learning rate or warmup:

• - Learning rate drops from the SFT default 2e-4 to 5e-5 (in line with Rafailov 2023 / TRL DPOTrainer). The CLI prints [optiq.lora] --method=dpo: using DPO learning rate 5e-05 when the default kicks in; the Labs UI auto-swaps the LR field unless you have edited it.
• - Warmup defaults to 10% of --iters (floor 10) with linear ramp 0 → peak.
• - Schedule decays cosine from peak to 10% of peak after warmup. Override with --dpo-lr-schedule constant or --dpo-warmup-iters N.

The trainer also adds two guardrails:

1. A loud WARNING if the resolved peak LR is > 1e-4 (the SFT default lands here, which collapses the policy within ~100 iters).

2. A one-shot collapse detector at the first val pass: if loss ≈ 0 and both chosen_r / rejected_r are drifting deeply negative with a near-zero margin, it points the user at the docstring's data-pairing requirement (chosen and rejected must both be plausible completions of the same prompt under the base distribution).

Default behavior for SFT runs is unchanged.

Fixes

• - HumanEval dataset id. huggingface-hub 1.x rejects the legacy unnamespaced openai_humaneval id; eval now uses openai/openai_humaneval. Same payload.
• - Family hints. optiq.lab.optiq_models._family_from_id and optiq.backends.mlx_backend._FAMILY_RECOMMENDED_SAMPLING recognize minicpm5 so the Lab picker shows the correct family label and the model-author sampling recipe is preserved through optiq convert.

Bugfix release. The v0.1.0 wheel and sdist shipped only .py files. The bundled calibration mix (optiq/calibration/data/optiq.jsonl), the Lab UI Jinja templates, the Lab static assets (CSS, JS, fonts, favicon), and the MTP benchmark prompt fixtures were all missing from the distribution, so a fresh pip install mlx-optiq failed on optiq convert with a FileNotFoundError on the calibration mix, and optiq lab failed on TemplateNotFound.

pyproject.toml now declares an explicit [tool.setuptools.package-data] block that pulls all data files into both the wheel and the sdist. Verified: wheel grows from 305 KB to 1.4 MB.

Also forwards the model-recommended sampling fix from 81da5ce. optiq serve no longer hands mlx_lm.server flags its argparse rejects (--repetition-penalty, --presence-penalty), which had silently turned the server into a --help dump on Qwen3.5 / 3.6 quants whose generation_config.json carried those keys.

Major simplification + sharper LLM focus. Breaking changes, no backwards compatibility shim.

Headline: 12 / 12 OptiQ-4bit quants beat uniform 4-bit on Capability Score

Every shipped quant in this release beats stock uniform 4-bit on the six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Range of Capability gains: +0.17 (Qwen3.5-27B) to +13.57 (gemma-4-e4b-it). HashHop is the new 6th benchmark; it exercises long-context multi-hop key→value retrieval and surfaces the layers OptiQ's sensitivity-aware allocation protects. See the eval-framework blog for the methodology and models page for the full 12-model table.

Speculative decoding for all Qwen + Gemma quants

• - Qwen3.5 / Qwen3.6: every OptiQ-4bit quant ships a bundled MTP head as mtp.safetensors. Enable with optiq serve --model … --mtp for ~1.4× decode (acceptance ~70 % at depth 2). MTP tensors are preserved automatically by optiq convert (the host-bit-width-matched sidecar is written next to the safetensors shards and registered in config.json).
• - Gemma-4: every OptiQ-4bit quant pairs with the matching mlx-community/<size>-it-assistant-bf16 drafter via optiq serve --model … --drafter …. The drafter runs γ=1 greedy speculation through optiq.runtime.spec; tokens are bit-identical to non-spec decode.
• - Both optiq serve and optiq lab expose --mtp (Qwen) and --drafter (Gemma) flags, mutually exclusive.

DPO fine-tuning

optiq lora train --method dpo --dpo-beta 0.1 adds Direct Preference Optimization to the LoRA trainer alongside SFT. Data shape: one {"prompt", "chosen", "rejected"} per line. The reference forward pass runs through the same adapter with scale=0, so there's no second model load. Standard DPO loss (Rafailov 2023) with diagnostic metrics (margin, chosen/rejected rewards, accuracy). Surfaced in the Lab Fine-tune wizard as a training-objective dropdown with a dpo_beta input.

Model-recommended sampling published in every quant

Every Qwen3.5 / 3.6 OptiQ quant on HuggingFace now ships a generation_config.json with the model author's "instruct / non-thinking general" recipe (temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, repetition_penalty=1.0, presence_penalty=1.5). Gemma-4 quants already shipped Google's "all use cases" recipe (1.0 / 0.95 / 64). optiq convert writes these for future quants automatically (existing upstream values always win the merge). The Lab Server-settings page surfaces them as pre-filled inputs the moment a model is selected, via a new POST /api/server/inspect-sampling endpoint that reads generation_config.json from local cache or pulls just that one small file from HF.

OptiQ Lab UI improvements

• - Drafter picker is now a free-form text input that auto-suggests the correct mlx-community/<name>-it-assistant-bf16 repo whenever a Gemma-4 target is selected. Hides entirely for non-Gemma targets (Qwen uses MTP). All four Gemma sizes mapped (was only e4b in the initial Lab build).
• - Sampler inputs pre-fill with the recommended values from generation_config.json the moment a model is picked, instead of staying blank until the model loads.
• - Local model picker filters drafters out, discover() and discover_hf_cache() skip any *-assistant* / *-drafter repos so users can't accidentally try to serve a drafter as the host model.
• - Dataset page header copy fixed, "Twelve templates" (was "Six").
• - Fine-tune hyperparams card adds a Training-objective dropdown (SFT default, DPO selectable) with a conditional dpo_beta input.

Site refresh

• - New 12-model headline table on the homepage and models page with Capability Score + Δ vs uniform-4-bit instead of single-benchmark GSM8K numbers.
• - Full rewrite of the eval-framework blog presenting the six-benchmark suite as the canonical Capability Score, with a dedicated HashHop Long-Context Evaluation section.
• - All --target-bpw examples now use the new 5.0 default (the prior 4.5 hint was leftover from earlier internal work).
• - Single-source-of-truth sidebar across all 19 docs pages via scripts/build_docs_sidebar.py, every page now exposes the same 24-link nav with a correctly-set is-active marker.
• - Six new Lab UI screenshots under site/assets/lab-screens/ referenced from the PyPI README so users see what the app looks like before installing.

Headline numbers (refreshed for v0.1.0)

Six-benchmark Capability Score (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop), OptiQ vs uniform 4-bit:

| Model | Uniform-4 | OptiQ | Δ |

|---|---:|---:|---:|

| Qwen3.5-0.8B | 31.73 | 36.00 | +4.27 |

| Qwen3.5-2B | 45.54 | 47.66 | +2.12 |

| Qwen3.5-4B | 63.86 | 65.76 | +1.90 |

| Qwen3.5-9B | 66.58 | 66.77 | +0.19 |

| Qwen3.5-27B | 78.88 | 79.05 | +0.17 |

| Qwen3.5-35B-A3B | 73.75 | 74.17 | +0.42 |

| Qwen3.6-27B | 82.50 | 82.96 | +0.46 |

| Qwen3.6-35B-A3B | 75.67 | 76.78 | +1.12 |

| gemma-4-e2b-it | 51.09 | 53.21 | +2.12 |

| gemma-4-e4b-it | 52.28 | 65.84 | +13.57 |

| gemma-4-26B-A4B-it | 69.62 | 72.68 | +3.06 |

| gemma-4-31B-it | 76.23 | 79.69 | +3.47 |

OptiQ Lab: local web UI

New optional component (pip install "mlx-optiq[lab]" → optiq lab). Local Flask app that opens four workflow surfaces in a browser, all using the same backend as the CLI:

• - Chat, streaming playground against the served model (matches Unsloth Studio's default landing). Three local tools the model can call:
• - web_search, DuckDuckGo via the ddgs library (no API key), plus a {"url": ...} mode that fetches a single page as compact markdown. Snippet replies end with a "fetch the URL for more" advisory so the model nudges itself toward full-page reads when needed.
• - python, runs code in a three-tier sandbox (apple/container → macOS sandbox-exec → subprocess+rlimit) with an AST safety check (os.system, subprocess.*, signal tampering, direct network calls are blocked). Per-call wall-clock + memory caps. Matplotlib output written to the sandbox workdir is captured, base64-inlined, and rendered as an <img> in the chat UI; the model never sees the base64 payload in its context.
• - terminal, runs a bash one-liner in the same sandbox. Token-aware command-position blocking, so echo "do not use sudo" works but sudo ls does not.
• - Tool-call healer recovers six common malformed shapes that quantized open-weight models emit instead of structured tool_calls: Hermes/Qwen <tool_call> tags, fenced JSON blocks, bare JSON, trailing commas, fancy quotes, function-call form, and the {"python": {...}} key-is-tool-name pattern. Unknown tool names are rejected so the model cannot hallucinate a tool. Healed calls are flagged in the UI with a healed chip.
• - Multi-turn orchestrator runs the tool loop server-side, capped at 25 turns (matches Unsloth Studio's default). On budget exhaustion the orchestrator appends a "stop calling tools and answer now" user message and re-prompts the model with tools disabled for one final reply rather than erroring out.
• - Duplicate-call de-dup detects consecutive identical tool calls and substitutes a nudge ("you already called this") rather than re-executing the sandbox. Failed calls are exempt from de-dup so the model can iterate on a fix.
• - Tool-error nudge appends a "try a different approach" instruction to any tool result whose body starts with a recognized error sigil (Error, Blocked:, Exit code, Search failed, etc.). Surfaced in the UI with an error chip.
• - Sandbox cancellation via a Stop button. The endpoint registers a per-stream threading.Event; the Stop button sends /api/chat/cancel, the orchestrator polls the event between turns, and any running tool subprocess gets SIGKILL'd at its process group. Replaces blocking subprocess.run with a Popen + watcher inside the sandbox.
• - Adjacent tool-call grouping in the UI: when the model emits N>1 tool calls in a row, they collapse into one "N tool calls" accordion with per-tool chips so the thread doesn't get visually swamped during multi-step runs. The most-recent group expands automatically; older groups auto-collapse when assistant text follows.
• - File attach: text + code formats inline, plus PDF (pypdf) and DOCX (docx2txt). Image / audio out of scope for v0.1.0.
• - Reasoning toggle: enable_thinking controllable per chat (default off for small quants).
• - Quantize, 4-step wizard: paste an HF model id, slide a target-BPW dial, watch live sensitivity + knapsack progress, one-click push the result to your HF account.
• - Fine-tune, 4-step wizard for sensitivity-aware LoRA training on any OptiQ quant. Live train-loss sparkline via SSE. Save + fuse + push to HF.
• - Build dataset, six templates (SFT from QA pairs, DPO, style transfer, code completion, self-instruct expansion, format conversion). Outputs JSONL the fine-tune wizard reads directly. Pushes to HF as repo_type="dataset".

Plus: sidebar with live API/MTP/model status, copy-paste configs for all five tested integrations (Claude Code, Codex, OpenCode, OpenClaw, Hermes Agent), and an HF settings page that stores a write-scope token encrypted at rest (Fernet key derived from the Lab password via PBKDF2-HMAC-SHA256).

Architecture: Flask + jinja2 + vendored htmx 2.0.4 + Alpine 3.14.7 (no Node build step). Background jobs run in multiprocessing.Process workers; progress streams via SSE. SQLite at ~/.optiq/lab/lab.db. Password auth on first run (argon2id + 24h JWT cookie). Localhost-only by default. The chat tool path uses a server-side SSE orchestrator that drives the multi-turn tool loop with a 25-turn cap.

Lab depends on a small extras set: flask, argon2-cffi, pyjwt, cryptography, data-designer, plus ddgs + html2text for web search and pypdf + docx2txt for file uploads. Apple Silicon only (matches the rest of mlx-optiq).

The Lab pins mlx-lm>=0.31 (needs the qwen3_5 arch module and the --prompt-cache-bytes flag, both landed upstream in mlx-lm 0.31.0).

KV-cache quantization on tight-RAM Macs

The diagnostic that started this cycle: mlx-lm's stock 4-bit KV cache on a 24 GB Mac actually peaks higher than fp16 KV at long context, because of (1) a conversion spike that holds both fp16 and quantized cache co-resident, and (2) a prefill scores-matrix spike inside quantized_scaled_dot_product_attention. On granite-4.1-8b-4bit at 32k, stock u4 peaks at 16.35 GB vs fp16's 11.51 GB. Users enabling --kv-bits 4 on tight RAM hit OOMs instead of memory savings.

Fix is two pieces, both default-on whenever KV-quant is enabled in optiq serve:

• - optiq.runtime.streaming_kv_quant, replaces mlx-lm's batched maybe_quantize_kv_cache with a per-layer streaming variant. Quantizes one layer's K, mx.eval, drops the fp16 reference, clears the buffer pool, then the same for V, repeat. Bounds the conversion transient to roughly one layer's worth (~150 MB) instead of all layers (~5 GB).
• - optiq.runtime.fused_quant_sdpa, replaces mlx-lm's unfused quantized_scaled_dot_product_attention with a FlashAttention-2 N-tiled variant using mx.quantized_matmul as the inner kernel. Per-tile scores shape is bounded by n_chunk=512, reducing the prefill transient ~10x. Algorithm matches what a fused Metal kernel would do; the matmul kernel is Apple's tuned Metal so we don't lose to a hand-rolled version.

CLI:

• - optiq serve --kv-bits 4 ... and optiq serve --kv-config ... ... install both patches automatically.
• - optiq serve --no-fused-kv ... opts out (for bit-exact comparison vs upstream mlx-lm).

Bench (granite-4.1-8b-4bit on M4 24 GB, NIAH retrieval at depth 0.5):

| context | fp16 peak | u4 stock mlx-lm | u4 + ours |

|---|---|---|---|

| 16k | 8.59 GB | (spike-prone) | 6.71 GB |

| 32k | 11.51 GB | 16.35 GB | 7.60 GB |

At 32k, u4 KV with our path uses 34% less peak memory than fp16 KV, and 53% less than stock mlx-lm's u4 path. The same fix unlocks 48k–96k contexts on the Qwen3.5-9B hybrid model where fp16 KV gets tight (11.51 GB at 64k, 18.86 GB at 96k) but our u4 stays under 10 GB peak across the whole range. Needle retrieval correct everywhere.

Speed on Qwen3.5-9B is at parity with fp16 KV (±2%). u4 = 15.16 gen_tps vs fp16 = 15.47 (−2.00%). Mixed-precision = 15.29 (−1.16%).

Accuracy (hash-hop differentiation on Qwen3.5-9B-OptiQ-4bit, 25 trials per cell, ctx=10k chars):

| Mode | hops=2 | hops=3 |

|---|---|---|

| fp16 | 23/25 (92%) | 10/25 (40%) |

| u4 (ours) | 23/25 (92%) | 5/25 (20%) |

| mixed (ours) | 25/25 (100%) | 7/25 (28%) |

At hops=2 (model's comfort zone) all three KV modes are within noise. At hops=3 (reasoning + KV precision stressed) uniform 4-bit halves accuracy vs fp16, while OptiQ's mixed-precision recovers ~40% of the lost ground. Hash-hop forces exact 16-character retrieval so any KV noise that flips one character is a total failure, which makes it sensitive to KV quality in a way that NIAH-style retrieval is not.

MTP speculative decoding (new in late-cycle v0.1.0 prep)

In-checkpoint MTP heads (DeepSeek-V3 style, found in Qwen3.5 / 3.6 family) are now preserved through optiq convert and usable at serve time.

• - Convert preserves MTP: optiq convert <model> automatically fetches and quantizes any mtp.* tensors from the source. Output gets a mtp.safetensors sidecar registered in config.json via the MTPLX-compatible mtp_file + mtplx_mtp_quantization keys. No-op for models without MTP. Implementation in optiq/runtime/mtp_convert.py; called from convert_llm_to_mlx and convert_llm_static_mixed post-quantize.
• - Serve runs MTP: optiq serve --mtp --mtp-depth N enables in-checkpoint MTP speculative decoding. Routes generation through OptiqEngine (built on top of vendored MTPLX runtime, Apache-2.0 attribution preserved under optiq/runtime/mtp/). MTP applies to all three endpoints transparently. Default off.
• - Engine streaming: OptiqEngine.generate_stream(...) yields {token, text, from_draft, done, decode_tps, ...} events per token. from_draft=True marks tokens accepted from the MTP head; False for AR steps, verified replacements, and bonus tokens.

Measured on Qwen3.5-9B-OptiQ-MTP-4bit (M4 24GB, depth=2, 3-run median): 1.36× / 1.56× / 1.78× on 2B / 4B / 9B, 1.42× floor on 27B (memory-bound). 0.8B hurts (below speculation inflection).

Three API protocols on one port

optiq serve now exposes three OpenAI-compatible endpoints from the same process:

• - /v1/chat/completions (OpenAI Chat Completions; default, used by most tools)
• - /v1/messages (Anthropic Messages; --anthropic, default on; used by Claude Code, OpenClaw)
• - /v1/responses (OpenAI Responses; --responses, default on; required by Codex which deprecated Chat Completions in 2026; also used by Cursor, Continue, Cline)

The Responses translator (new files optiq/responses_shim.py + optiq/responses_server.py) mirrors the existing Anthropic shim pattern: translate the Responses body to Chat Completions, reuse mlx-lm's existing handler, translate the response back to Responses output shape with proper response.created / response.output_text.delta / response.completed SSE events. Built-in Responses tools (web_search, file_search, mcp, computer_use) are dropped silently; only function tools are forwarded.

Stateful previous_response_id: Responses requests that carry previous_response_id resume from the prior turn. Each completed response (stream or non-stream) is stored in a process-local TTL cache (optiq/response_store.py, 1 hr TTL, 32 MiB LRU cap). On a follow-up the stored input + prior assistant output is spliced ahead of the new input. Unknown id returns 404. Prior reasoning items are dropped from the replayed history (clients don't want chain-of-thought fed back in).

Reasoning capture (both protocols): when a reasoning model (Qwen3.5, DeepSeek-R1, etc.) emits chain-of-thought via the chat template, both the non-streaming and the streaming paths surface it in the appropriate protocol-specific channel:

• - Anthropic non-stream: {"type": "thinking", "thinking": "..."} content block (Claude 3.7 extended-thinking format) followed by {"type": "text", "text": "..."} block.
• - Anthropic stream: content_block_start (type=thinking) → content_block_delta (type=thinking_delta) (many) → content_block_stop, then the text block.
• - Responses non-stream: {"type": "reasoning", "summary": [{"type": "summary_text", "text": "..."}]} output item followed by {"type": "message", ...} item.
• - Responses stream: response.output_item.added (reasoning) → response.reasoning_summary_text.delta (many) → response.reasoning_summary_text.done → response.output_item.done, then the message item.

Incremental SSE flush: both shims now expose AnthropicStreamTranslator / ResponsesStreamTranslator classes that the server proxy feeds chunk-by-chunk, so the client sees bytes flowing while the model is still generating. Reasoning models have first-token latency well past most SDK read timeouts; without per-chunk flushing the connection dropped before any byte arrived.

Auth: sk-optiq-* Bearer token

New --auth / --no-auth flag on optiq serve (default on). Bearer tokens that start with sk-optiq- are accepted; any suffix works. Missing Authorization header is allowed for local-dev curl convenience. Wrong prefix returns 401 with a clear error message. Mirrors Unsloth's sk-unsloth-* convention so integration configs feel familiar.

Integrations docs

site/docs/integrations/ adds per-tool guides with copy-pasteable configs:

• - Claude Code (via Anthropic endpoint + ANTHROPIC_BASE_URL)
• - Codex (via Responses endpoint + ~/.codex/config.toml)
• - OpenCode (via OpenAI Chat Completions)
• - OpenClaw (via Anthropic endpoint)
• - Hermes Agent (via OpenAI Chat Completions)

Bonus tools that work out of the box and are listed on the integrations index: Cursor, Continue, Cline, aider, Open WebUI, LangChain / LlamaIndex / DSPy via OpenAI SDK base_url override.

Previously documented v0.1.0 changes

Sensitivity analysis: 3 methods → 1. Removed `analyze_sensitivity_fast (empirical-Fisher) and analyze_sensitivity_stream (weight-space MSE). The only remaining sensitivity entry point is analyze_sensitivity_exact`, calibration-driven KL divergence on logits, the gold-standard signal that actually predicts downstream quality.

The exact path now supports two execution modes (selected by the new `--reference CLI flag, or the reference= kwarg in run_llm_pipeline`):

• * `--reference bf16`, load the bf16 base into RAM, swap each layer's weight in-place between bf16 and a simulate-quantized copy. Highest fidelity.
• * `--reference uniform_4bit`, first build a uniform-4-bit MLX baseline from the bf16 source, load that as the running model, and stream bf16 weights off disk one layer at a time to swap in for sensitivity probes. Lets 27 B+ models still get a calibration-driven signal on a 36 GB Mac.
• * `--reference auto (default), pre-checks available RAM and routes between the two automatically. Logs a warning when falling back to uniform_4bit`.

Whichever reference is used, the FINAL OptiQ output is the same mixed-precision artifact built freshly by `mlx_lm.convert` with the per-layer bit predicate (e.g. ~4.5 BPW = mix of 4-bit and 8-bit per layer).

YOLO support removed. Object detection didn't share any code with the LLM stack (separate sensitivity, separate eval, separate calibration data, separate `yolo-mlx runtime). The 5 published mlx-community/YOLO26{n,s,m,l,x}-OptiQ-6bit artifacts remain on HuggingFace as legacy from v0.0.x, they still work with yolo-mlx` directly, but the conversion path is no longer in mlx-optiq. Removed:

• * `optiq/models/yolo.py`
• * `optiq/eval/yolo_eval.py`
• * `[yolo] extras in pyproject.toml`
• * `yolo-mlx` dependency

Removed code: `analyze_sensitivity_fast, analyze_sensitivity_stream, analyze_sensitivity dispatcher, _lm_loss, _accumulate_grads, plus all the _iter_safetensors_* helpers that were stream-specific (the new uniform_4bit reference path uses different streaming logic, kept inside sensitivity.py`).

Kept: the `_chunked_quantize mlx-lm patch, still essential for the final mlx_lm.convert step on large MoE models (Gemma-4-26B-A4B, Qwen3.5/3.6-35B-A3B) where mlx-lm's standard nn.quantize` sweep blows the Metal GPU command-buffer timeout.

Library size: `optiq/core/sensitivity.py` shrank from 885 lines to ~520. Total deletion across all touchpoints: ~1,200 lines.

CLI changes (breaking):

• * `--sensitivity {exact,fast,stream}` → removed. Only one method now.
• * `--reference {auto,bf16,uniform_4bit} → new flag, default auto`.
• * `--calibration-mix {optiq,/path/to/your.jsonl} → new flag, default optiq`.
• * `--model-type yolo` → removed.

New: bundled six-domain calibration mix (`optiq/calibration/data/optiq.jsonl). 40 hand-curated samples across prose (5), reasoning (6), code (6), agent loops (8), function-calling (7), and constraint-bearing instructions (8, regex-filtered from HuggingFaceH4/no_robots and disjoint from google/IFEval). Chat samples are auto-rendered through the target model's tokenizer.apply_chat_template() before tokenization, so the activated subspace matches production. Replaces the prior WikiText-only calibration, which under-protected tool-call and instruction-following layers. Reproducible via python scripts/build_calibration.py. See site/blog/calibration-mix.html` for the methodology.

New: two-tier eval framework (`optiq.eval.*). optiq eval --task smoketest runs Tier-1 (KL on 64 prompts × 256 tokens + GSM8K-50, ~5 min/model). optiq eval --task all --score runs Tier-2 headline (MMLU-1k 5-shot, GSM8K-1k, IFEval full, BFCL-V3 simple-200, HumanEval-164, HashHop 25 × 4 hops) and emits a single Capability Score = mean(MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop), simple unweighted average across the six benchmarks, with disk_gb reported separately as an honest second axis (no hidden value judgement embedded in the formula). KL evaluator auto-resolves the reference (bf16 if it fits in RAM, else mlx-community uniform-4-bit baseline; uses HfApi.model_info for accurate per-shard sizes). HumanEval execution sandbox falls through apple/container → sandbox-exec → subprocess + rlimit. HashHop wraps the a hash-hop generator and exercises long-context multi-hop key→value retrieval at ~12 k tokens. New tasks individually addressable: --task {kl,gsm8k-50,mmlu,gsm8k,ifeval,bfcl,humaneval,hashhop}`.

Optimizer fix: KL-reduction sign-flip detection. Under `--reference uniform_4bit, the candidate's per-bit sensitivity to the uniform-4-bit baseline reverses sign relative to bf16-reference mode (KL(low) ≈ 0 because the layer is identical to baseline; KL(high) > 0 because the simulate-quantize at higher precision is *closer to bf16* than the baseline). The greedy knapsack's current.sensitivities[low] - current.sensitivities[high] term went negative, the priority heap rejected every upgrade, and 27 B+ models came out essentially uniform-4-bit instead of mixed. Fixed in _kl_reduction()`: detects sign-flip and inverts the subtraction. Verified against a 35B-A3B checkpoint where the planner went from 80 → 400 layers at 8-bit at the same target BPW.

Memory hygiene during sensitivity probes. Calibration sequences capped at `seq_len=512 (was 1024), the memory-safe sweet spot for 27 B+ models on a 36 GB Mac during bf16-streaming probes. Each probe explicitly gc.collect + mx.clear_cache` after use; reference logits cast to fp16 before per-probe KL math. Cuts peak resident memory by ~30 % during the layer sweep.

Description string updated to reflect LLM focus: "Mixed-precision quantization optimizer for LLMs on Apple Silicon (MLX)".

Removed: `_strip_unused_modalities (and --keep-unused-modalities flag, optiq.core.text_only, tools/strip_vision.py). The strip rewrote config.json (flipped architectures: [Qwen3_5ForConditionalGeneration] → [Qwen3_5ForCausalLM], flattened text_config, dropped vision/audio modality keys) under the assumption it was producing a cleaner text-only artifact. Empirically it provided **zero disk/memory savings** (mlx_lm.convert` already drops vision/audio weights from the safetensors output regardless of the strip flag) and broke instruction-following on Qwen3.5/3.6 hybrid-attention models by 14–17 pp on IFEval (versus uniform-4 baseline at the same target BPW). Bisection: bf16 IFEval = 71.9 %, uniform-4 = 68.4 %, OptiQ all-8-bit with strip = 55.8 %, OptiQ all-8-bit without strip = 72.3 %. The arch-class flip was routing mlx-lm to a different model class with subtly different long-generation handling. The fix is to leave the config.json intact; mlx-lm loads multimodal-arch quants via the VLM-arch class which delegates to the language path for text inference.

Sensitivity-side fix for MoE expert weights. The optimizer + planner now correctly assign per-layer bits to every expert in Gemma-4 / Qwen3.5-MoE / Qwen3.6-MoE base models. Conversion still requires more work for the largest MoE models, see "Known limit" below.

Fix: MoE expert weights were silently skipped during sensitivity analysis (optiq/core/sensitivity.py)

The Gemma-4 / Qwen3.5/3.6 MoE base models store their experts as 3-D fused tensors without a .weight suffix on disk:

• - *.experts.gate_up_proj of shape (num_experts, 2 * mid, hidden), gate and up packed back-to-back along axis -2
• - *.experts.down_proj of shape (num_experts, hidden, mid)

mlx-lm's sanitize() later splits these into the per-projection 3-D weights it actually loads (*.experts.switch_glu.{gate,up,down}_proj for Gemma-4, *.mlp.switch_mlp.{gate,up,down}_proj for Qwen3.5/3.6 MoE). _iter_safetensors_linears was both filtering by len(shape) != 2 and requiring key.endswith(".weight"), so every expert tensor was dropped from the sensitivity index. With no per-layer entry, make_quant_predicate fell back to default_bits (the max bit-width in the allocation, typically 8) for every expert weight, blowing up storage BPW well past the planning target.

The fix:

• - _iter_safetensors_linears now also accepts ≥3-D tensors whose last dim is divisible by group_size (mx.quantize handles these natively, it operates per-row over the last axis).
• - New _iter_safetensors_moe_experts(model_path, group_size) walker recognizes the on-disk MoE patterns above, applies the same axis-(-2) split mlx-lm's sanitize() does for gate_up_proj, and emits per-leaf entries with the sanitized mlx-lm-side names. The wrapper name (switch_glu for Gemma-4, switch_mlp for Qwen3) is picked up from config.json.
• - The shared loader in analyze_sensitivity_stream reads the on-disk fused tensor once per shard and applies the gate/up slice in-memory before evaluating each half independently. Param-count bookkeeping generalized from shape[0] * shape[1] to a product over all dims.

Empirical effect on google/gemma-4-26B-A4B-it (sensitivity index 400 → 490 entries; bit allocator now sees the experts):

| | v0.0.10 plan | v0.0.11 plan |

|---|---|---|

| Modules at 4-bit | 232 (no experts) | 111 |

| Modules at 8-bit | 168 (no experts) | 379 |

| Achieved planning BPW | 4.50 | 4.50 |

| Estimated model size | 1.5 GB (planner under-counted experts) | 13.8 GB (matches the actual quantization-accounting) |

Known limit: mlx-lm's nn.quantize graph times out on the largest MoE bases. With 30 layers × 250 M-parameter expert tensors, the Metal command buffer that mlx-lm builds when sweeping to_quantized across the full module tree exceeds the GPU's per-buffer timeout (kIOGPUCommandBufferCallbackErrorTimeout) on a 36 GB M3 Max, at any target BPW. mx.quantize itself runs in 0.01 s on a single 484 MB expert tensor, so the bottleneck is the queued pipeline, not the kernel. Until mlx-lm chunks the quantize sweep (or until OptiQ pre-quantizes the experts shard-by-shard before handing off), the largest MoE bases need a more memory-headroom-friendly machine to convert. The currently-published mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit was built under v0.0.10 (24.5 GB, experts at 8-bit by fallback) and remains usable; it will be re-uploaded once the convert path is unblocked. Smaller MoE bases should benefit from the v0.0.11 sensitivity fix immediately.

Two new 27 B HF artifacts and a small set of opt-in extensions that didn't make the v0.0.9 cut. No breaking changes; all v0.0.9 APIs remain.

New HF artifacts (text-only, language-stack-only, vision/audio metadata stripped at convert time):

• - mlx-community/Qwen3.5-27B-OptiQ-4bit, 343 layers @ 4-bit + 247 layers @ 8-bit, 4.50 BPW achieved, 15.7 GB on disk. Inference ~14 tok/s and ~10 GB resident on M3 Max 36 GB.
• - mlx-community/Qwen3.6-27B-OptiQ-4bit, same recipe, same shape. Both produced via optiq convert ... --sensitivity stream (mmap-streamed weight-space sensitivity, peak ~9 GB RAM during conversion, required for ≥ 27 B on a 36 GB Mac since mlx_lm.load of bf16 27 B OOMs).

New: optiq/ops/, opt-in long-context training kernels

• - flash_attention (tile-based forward) and flash_attention_metal + flash_attention_backward_metal (custom Metal kernels with explicit backward).
• - chunked_cross_entropy and chunked_ce_training for chunked LM-head loss + grad on long sequences.
• - fused_swiglu, attention_patch, sublayer_checkpoint for activation-memory experiments.
• - Honest positioning: vanilla mlx-lm + grad_checkpoint=True is the right default for ≤ T = 12 000 on a 36 GB M3 Max (~82 s/step at peak T). The custom kernels here are 2–4× slower but use slightly less memory; opt in only when the wired-limit + memory-pressure budget genuinely won't fit vanilla. See optiq.ops.attention_patch.enable_flash_attention_training for the activation switch.

New: optiq.anthropic_server, Anthropic-compatible /v1/messages endpoint

• - install_anthropic_endpoint() monkey-patches mlx_lm.server.APIHandler.do_POST to recognize Anthropic's /v1/messages path while leaving every OpenAI route untouched. Lets clients written against Claude's SDK (e.g. agentic tools that hard-code the Anthropic API shape) talk to a local mlx-optiq server with no code changes.
• - Idempotent installer (_INSTALLED guard), safe to call from optiq.serve startup or tests.
• - Streaming + tool-call shape conversion handled in optiq.anthropic_shim (request body normalization, response shape rewrite, Anthropic event-stream protocol).

New: optiq.core.text_only.strip_multimodal_metadata(), library API

• - Previously embedded in the optiq convert flow only. Now exposed as from optiq.core.text_only import strip_multimodal_metadata for users who want to clean up an existing model directory in place: rewrites architectures, flattens text_config, drops vision/audio token IDs, removes mrope sub-fields. Weight tensors are not touched.
• - Useful when post-processing third-party VLM-text-only checkpoints that load fine in mlx-lm but still publish the wrong pipeline_tag and architecture class.

Documented: empirical training-ceiling map at system-default iogpu.wired_limit_mb=0 (M3 Max 36 GB)

Conservative optiq lora train recipes (default config: q_proj, v_proj, NL=16, rank=8, rank_scaling=by_bits) verified end-to-end against a real Hermes traces fine-tune dataset:

| Model | T | Peak | Tok/s |

|---|---|---|---|

| Qwen3.5-0.8B-OptiQ-4bit | 2800 | 23.4 GB | 29.2 |

| Qwen3.5-2B-OptiQ-4bit | 2400 | 19.3 GB | 38.3 |

| Qwen3.5-4B-OptiQ-4bit | 1600 | 24.8 GB | 19.1 |

| Qwen3.5-9B-OptiQ-4bit | 1400 | 25.4 GB | 21.6 |

| Qwen3.6-27B-OptiQ-4bit | 512 | 27.7 GB | 11.4 |

All five sit at peak ≤ 27.7 GB with at least 0.3 GB of headroom under the system-default cap and zero observed memory drift across iters. Going higher in T trades throughput for context coverage and risks compressed-memory penalty, and at much higher T (e.g. 2 B at T = 3200) hits a sharp Apple-Silicon MTLResource-count cliff that fails before any byte ceiling. Don't extrapolate "more headroom in GB" → "can push T further", each model has its own throughput knee at some seq_len.

Major: sensitivity-aware LoRA and reversible hot-swap adapters go from v0.0.8 "works programmatically" to production, with end-to-end validation. Gemma-4 variants re-uploaded with corrected weights and the gemma4_text model type. Full website refresh at mlx-optiq.pages.dev. All 6 HF model cards rewritten with the v0.0.9 positioning.

New: reversible mounted LoRA + per-request hot-swap (optiq.adapters.mount)

• - MountedLoRALinear replaces the target linear modules with a wrapper that holds a dict {adapter_id: (A, B, scale)} and gates them via a ContextVar the server can flip per request.
• - AdapterActivation context manager for request-scoped activation.
• - prepare_model_for_mounted_lora / mount_adapter_on_model / unmount_adapter_from_model.
• - AdapterRegistry wired to use the mount path, activate / deactivate / unmount are now reversible without a model reload.
• - 5 rigorous tests verify: mount changes logits, activate→deactivate restores base bit-for-bit, two adapters switchable in the same process, ContextVar isolation across concurrent asyncio tasks, selective unmount preserves other adapters.

New: qwen3_5_text model type (optiq.mlx_lm_patches)

• - import optiq auto-registers a qwen3_5_text module into mlx_lm.models that wraps qwen3_5.TextModel directly (no VLM .language_model prefix). Facade only, follows upstream changes automatically.
• - Opt-in via tools/strip_vision.py --route-to-optiq-text for local conversions. HF-published models continue to use qwen3_5 (VLM wrapper) so they load cleanly with bare mlx-lm.

Fix: Gemma-4 weight-corruption bug in strip_vision.py

The v0.0.7 Gemma-4 strip wrote all-zero weights because mx.load returns lazy memory-mapped views into the file, and saving to the same path overwrote the file before the arrays were materialized. Fixed by forcing mx.eval(list(tensors.values())) before mx.save_safetensors to the same path.

Both Gemma-4 OptiQ variants (gemma-4-e2b-it-OptiQ-4bit, gemma-4-e4b-it-OptiQ-4bit) have been re-uploaded to HuggingFace with corrected weights and model_type: gemma4_text.

Housekeeping

• - All 6 HF model cards rewritten for v0.0.9: covers bare-mlx-lm usage, mlx-optiq unlocks (KV serving, LoRA, hot-swap), and benchmarks.
• - Website v0.0.6 → v0.0.9. New feature cards on the homepage (LoRA + hot-swap), new Thread 4 (sensitivity-aware LoRA) and Thread 5 (mounted hot-swap) on experiments.
• - README fully rewritten with "optimized deployment for MLX" positioning, unified story across weight quant, KV quant, TurboQuant, LoRA, hot-swap, VLM-strip, YOLO26, latency prediction.
• - Roadmap update: Gemma-4 KV serving blocker precisely diagnosed as gemma4_text shared-layer attention passing packed tuples to mx.fast.scaled_dot_product_attention (upstream mlx-lm fix needed).

Tests

• - tests/test_release.py, 7/7 passing (config strip on all 6, load+generate all 6, qwen3_5_text routing, gemma4_text routing + weight-corruption fix verified, TurboQuant untouched, LoRA metadata reads, LoRA roundtrip).
• - tests/test_hotswap.py, 5/5 passing.

Bug-fix release that makes the v0.0.7 LoRA / serve features actually work end-to-end. v0.0.7 shipped with three bugs that prevented optiq lora train and optiq serve --adapter from running to completion; none of them were caught because the release wasn't exercised beyond --help smoke tests. Fixed:

Fixes

• - optiq/lora/apply.py, removed a spurious DoRALinear import; mlx-lm 0.31.x only ships LoRALinear. DoRA is now an explicit NotImplementedError instead of ImportError.
• - optiq/lora/sensitivity_rank.py, read per_layer from the top level of optiq_metadata.json (where current OptiQ pipelines write it) in addition to the legacy optimization.per_layer path. Also supports reading metadata directly from a HuggingFace repo id via hf_hub_download when the model isn't on local disk.
• - optiq/lora/trainer.py, wrap mlx-lm's load_dataset output in CacheDataset before handing to the training loop; the naked TextDataset returns dicts and crashes iterate_batches.
• - optiq/lora/trainer.py, _write_peft_config now emits the mlx-lm-required fine_tune_type, num_layers, and lora_parameters fields alongside the PEFT-style keys, so mlx_lm.generate --adapter-path <optiq_adapter> and optiq serve --adapter <optiq_adapter> both load cleanly.

End-to-end validation (now part of the release checklist)

Qwen3.5-0.8B-OptiQ-4bit has 187 per-layer bit assignments (111 × 4-bit, 76 × 8-bit); 12 of those layers are q_proj/v_proj targets for LoRA.

Ran two 100-iter training runs on a 160/40 GSM8K subset (max_seq=512), same base, same optimizer, same data:

!sensitivity-aware vs constant rank LoRA

| | constant rank=4 | by_bits rank=4 |

|---|---:|---:|

| rank distribution | 12 × rank 4 | 5 × rank 4, 7 × rank 8 |

| trainable params | 0.605 M (0.080%) | 0.650 M (0.086%) |

| val loss @ iter 50 | 1.183 | 1.045 |

| val loss @ iter 100 | 0.893 | 0.990 |

| final train loss | 1.099 | 1.054 |

| adapter size | 1.45 MB | 1.63 MB |

| peak memory | 4.05 GB | 4.05 GB |

The sensitivity-aware configuration reaches a lower mid-training validation loss (1.045 vs 1.183 at iter 50), which is the expected pattern, giving more capacity to layers OptiQ identified as sensitive accelerates convergence. The gap closes by iter 100 on this small dataset; longer runs and larger datasets are needed for a definitive comparison. Full run metadata in article/assets/v007_lora/rank_scaling_ab.json.

Round-trip test

After training, the adapter saves into /tmp/optiq_lora_test/adapter/, is loadable by stock mlx_lm.generate --adapter-path, and plugs into optiq serve --adapter <path> with the OptiQ sidecar surfaced at startup. Verified with a generation of a held-out arithmetic question ("What is 9+9?" → "The answer is 18.").

Positioning shift: OptiQ is now framed as "the Apple Silicon deployment optimizer", mixed-precision quantization plus unused-component stripping plus sensitivity-aware LoRA fine-tuning, not just a quantizer.

New: sensitivity-aware LoRA fine-tuning (optiq lora train)

• - Reads optiq_metadata.json for per-layer bit assignments
• - Derives per-layer LoRA rank from OptiQ's KL-sensitivity measurements (--rank-scaling by_bits by default; by_kl and constant also supported)
• - Adapter output is PEFT-compatible (adapter_config.json + adapters.safetensors) plus an OptiQ sidecar (optiq_lora_config.json) recording the per-layer rank distribution
• - optiq lora info <adapter_dir> summarizes any adapter (OptiQ-trained or stock PEFT)
• - Monkey-patches out mx.compile from mlx-lm's trainer at runtime (required to avoid Metal OOM on Qwen3.5-9B 4-bit)

New: optiq serve --adapter <id_or_path>

• - Serve with a LoRA adapter applied at startup
• - Accepts HuggingFace repo ids (<owner>/<adapter-repo>) and auto-downloads into ~/.cache/optiq/adapters/ (override via OPTIQ_ADAPTER_CACHE)
• - Surfaces OptiQ sidecar info at startup when present (rank, per-layer rank distribution)

New: optiq convert strips unused multi-modal metadata by default

• - Multi-modal base models (Qwen3.5-VLM, Gemma-4) have their vision/audio weights dropped by mlx-lm during text-only quantization but the config.json keeps the VLM architecture class, text_config wrapper, and modality token IDs. OptiQ now cleans those up automatically so the quantized model advertises itself accurately as text-generation
• - Opt out with --keep-unused-modalities
• - The same cleanup is available as a standalone reusable function: optiq.core.text_only.strip_multimodal_metadata(model_dir)
• - tools/strip_vision.py script ships with the repo for retroactively cleaning previously-published OptiQ models

Housekeeping

• - All six published OptiQ LLM variants (Qwen3.5-0.8B/2B/4B/9B, Gemma-4-e2b/e4b) re-uploaded to mlx-community/* with cleaned metadata. Weights are unchanged and generation output is bit-identical
• - Version bump in pyproject.toml and optiq/cli.py

Docs-only release.

• - README + site point at https://mlx-optiq.pages.dev/ (was wrong URL in v0.0.5).
• - Added the article (Not All Layers Are Equal)

link in the site nav and footer.

New: optiq serve — OpenAI-compatible server with mixed-precision KV cache

• - optiq serve --kv-config path/to/kv_config.json wraps mlx_lm.server and injects

per-layer KV quantization at serving time. Uses mlx_lm.models.cache.QuantizedKVCache

+ mx.quantized_matmul (fused kernel — real memory savings, no fp16 materialization).

• - optiq serve --kv-bits 4 for uniform quantization.
• - optiq serve (no flags) forwards to fp16 mlx_lm.server unchanged.

Fix: optiq kv-cache supports hybrid-attention models

• - Previously assumed every layer had self_attn. Now detects layers with a quantizable

KV cache (via hasattr(c, "to_quantized")) and skips linear-attention layers.

Required for Qwen3.5 (GatedDeltaNet + self-attention mix).

Benchmarks (Apple M3 Max, 64k context, decode tok/s vs fp16):

| Model | fp16 | mixed KV | speedup |

|---|---|---|---|

| Qwen3.5-2B | 27.9 | 41.8 | +50% |

| Qwen3.5-4B | 8.1 | 13.1 | +62% |

| Qwen3.5-9B | 20.7 | 27.1 | +31% |

Known issues

• - Gemma-4 (e2b, e4b): mixed-precision KV path fails on Gemma-4's shared-KV attention

layers. Upstream mlx-lm limitation — tracked for a future release. Fp16 serving

works fine.

Previous release. Weight quantization pipeline (optiq convert), KV sensitivity

analysis (optiq kv-cache), latency/eval/benchmark commands, TurboQuant library

primitives (optiq.core.turbo_kv_cache).