CLI reference

The optiq command-line interface drives every mlx-optiq workflow on Apple Silicon: quantization, KV-cache config, LoRA fine-tuning, OpenAI- and Anthropic-compatible serving, evaluation, and latency benchmarking. Twelve top-level commands cover the full workflow:

• optiq convert: quantize an HF model with mixed-precision sensitivity
• optiq kv-cache: measure per-layer KV sensitivity, write a serving config
• optiq lora train / info: sensitivity-aware LoRA training
• optiq serve: dual-protocol inference server (OpenAI + Anthropic)
• optiq eval: smoketest + full benchmark suite (MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop)
• optiq benchmark: perplexity + throughput on a converted model
• optiq latency: Apple Silicon roofline throughput predictor
• optiq prune-experts: remove low-value routed experts from a quantized MoE
• optiq lab: local web UI for quantize / fine-tune / dataset / chat
• optiq code: terminal coding agent driven by your local model
• optiq cluster: serve one model across two Macs over Thunderbolt
• optiq config: show every setting and where its value came from

optiq prune-experts

Remove low-value routed experts from a quantized MoE checkpoint. Implements REAP (Cerebras Research, ICLR 2026) in the quantized domain: it runs directly on a quantized model, needs no BF16 parent, and copies retained experts bit-for-bit rather than dequantizing them.

$ optiq prune-experts MODEL --retain 128 --output ./pruned

Required

• MODEL: HF repo ID or local path to an MLX-format MoE checkpoint. OptiQ quants, stock mlx-community quants and unquantized MLX conversions all work. A raw Hugging Face checkpoint stores its experts unfused and is rejected rather than silently skipped.
• --output: directory for the pruned model. The source is never modified.

Options

• --retain N: experts to keep per layer. Must be at least the model’s top-k.
• --target-retention F: keep a fraction instead, e.g. 0.5.
• --profile PATH: reuse a previous profile instead of re-measuring.
• --n-samples N: calibration samples to profile on (default 8).
• --calibration-mix: which mix to rank experts on. Pruning is far more sensitive to this than quantization — an expert the mix never exercises is an expert that gets deleted.
• --profile-only: write the profile and stop.
• --keep-mtp / --drop-mtp: carry the MTP sidecar (default: keep).

What it costs

Experts are ranked by router weight times expert output norm, measured on calibration data; the lowest-ranked are removed and the router is sliced to match. Active parameters per token do not change, so the model gets smaller without getting slower.

How that score is aggregated depends on the router. Where the router softmaxes over every expert, as Qwen’s does, the routing weight is an expert’s share of the whole distribution and already expresses the router’s preference, so the score is a mean over the times the expert fired — a rarely-chosen but strong expert outranks a common weak one. Where the router softmaxes over only the selected top-k, as Gemma-4’s does, every chosen expert gets roughly 1/k whatever the router thought of it, and the preference survives only in how often it was chosen; there the score is weighted by selection frequency instead. Using the wrong one is not a small loss. On Gemma-4 at 50 % retention the frequency-weighted ranking gives a KL of 0.56 against the unpruned model where the conditional mean gives 1.22, against 1.93 for picking at random — and at that level the conditionally-ranked model generates degenerate text while the other does not. The rule is chosen per architecture; there is no flag to get wrong.

On Qwen3.6-35B-A3B-OptiQ-4bit at 50 % retention: 22.1 GB to 12 GB on disk, 24.5 GB to 11.5 GB peak memory, 47 to 54.7 tok/s, and a Capability Score of 80.03 to 76.57. Procedural ability holds — tool calling −1.0, HumanEval −1.3, GSM8K +2.6, IFEval +4.3 — while broad factual knowledge is the first thing to go, MMLU −21.4. Check MMLU against your own workload before relying on a pruned variant.

On gemma-4-26B-A4B-it-OptiQ-4bit at the same 50 % retention: 17.5 GB to 10.4 GB on disk, 16.4 GB to 9.5 GB peak memory.

Published OptiQ variants all target 50 % retention, the same way every quant targets 4-bit: one number across the lineup, chosen because that is where the saving is worth having rather than because it is free. What it costs varies sharply by model, and not in the way you would guess: it is not the expert count. Laguna-XS routes over 256 experts, exactly as many as Qwen3.6, and loses far more at the same fraction. The command measures it directly, reporting the KL divergence against the unpruned model before it writes anything — 0.07 and 0.13 for the two Qwens, 0.56 for Gemma-4, 1.9 for Laguna-XS — and warns when that number crosses 1.0. Every checkpoint measured above it wrote fluent, confident, factually wrong text while still handling code and arithmetic correctly, which is the kind of failure a quick read does not catch. If a model lands above the threshold, retain more experts: --retain takes any count.

optiq convert

Quantize a Hugging Face model with mixed-precision sensitivity-driven bit allocation.

$ optiq convert MODEL [OPTIONS]

Required

• MODEL: HF repo ID (e.g. Qwen/Qwen3.5-9B) or local path to a bf16 checkpoint.

Options

• --method [optiq|static]: Bit-allocation method. Default: optiq (exact calibration-driven KL sensitivity). static allocates by structural rules (embedding/head, first/last block, attention and the MoE router get the high bits) at the requested candidate bits and target BPW, with no calibration or forward passes. It matches optiq on typical architectures, loads lazily, and is the fast path for large bases where exact KL is impractical. See the methods comparison.
• --target-bpw FLOAT: Average bits per weight. Default: 5.0.
• --candidate-bits TEXT: Comma-separated bit-widths to choose from. Default: 4,8.
• --reference [auto|bf16|uniform_4bit]: Reference precision for sensitivity probes (optiq method only). Default: auto (bf16 if it fits in RAM, else uniform_4bit).
• --calibration-mix TEXT: Calibration set. Default: optiq (the bundled 6-domain mix). Pass a path to override with a custom JSONL file.
• --group-size INTEGER: Quantization group size. Default: 64.
• --n-calibration INTEGER: Calibration sequences for sensitivity (drawn from the 40-sample mix). Default: 24 (roughly 4 samples from each of the 6 domains). Higher = more stable per-layer KL ranking, linearly slower convert.
• --n-floor-per-block INTEGER: Block-aware floor: minimum components per transformer block kept above the lowest bit. Prevents lowest-bit assignments from concentrating in the middle. Default: 2; 0 disables.
• --skip-baselines: Don't also build the uniform-4-bit comparison artifact.
• -o, --output TEXT: Output directory. Default: optiq_output/<model_basename>.

Examples

# Standard 5.0 BPW mix on a 9B (auto-routes to bf16 reference)
$ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0

# 27B+ on a 36 GB Mac (auto-routes to uniform_4bit reference)
$ optiq convert Qwen/Qwen3.5-27B --reference uniform_4bit

# Custom bit set (3-bit / 6-bit mix at 4 BPW average)
$ optiq convert Qwen/Qwen3.5-4B \
    --target-bpw 4.0 --candidate-bits 3,4,6,8

# Fast structural allocation for a large MoE, mixed 2/4-bit
$ optiq convert mlx-community/Qwen3.5-122B-A10B-bf16 \
    --method static --candidate-bits 2,4 --target-bpw 2.5

optiq kv-cache

Measure per-layer KV-cache sensitivity and write a per-layer KV bit-width config that optiq serve --kv-config consumes.

$ optiq kv-cache MODEL [OPTIONS]

Options

• --target-bits FLOAT: Average KV bits across full-attention layers. Default: 5.0.
• --candidate-bits TEXT: Default: 4,8.
• --n-samples INTEGER: Calibration samples for KV sensitivity. Default: 5.
• --seq-len INTEGER: Calibration sequence length. Default: 512.
• --group-size INTEGER: Default: 64.
• -o, --output TEXT: Where to write kv_config.json and kv_sensitivity.json. Default: optiq_output/kv_cache.

optiq lora

Train and inspect sensitivity-aware LoRA adapters.

optiq lora train

$ optiq lora train MODEL [OPTIONS]

• --data PATH: Directory containing train.jsonl (and optional valid.jsonl). Required.
• --rank INTEGER: Base LoRA rank. Default: 8.
• --scale FLOAT: LoRA alpha scaling (alpha = rank * scale). Default: 20.0.
• --dropout FLOAT: LoRA dropout. Default: 0.0.
• --rank-scaling [constant|by_bits|by_kl]: How to scale rank across layers. by_bits uses the per-layer bit assignments OptiQ recorded at convert time; by_kl scales by raw KL sensitivity; constant is uniform. Default: by_bits.
• --num-layers INTEGER: Last N transformer blocks to adapt; -1 for all. Default: 16.
• --target-modules TEXT: Comma-separated module suffixes to adapt. Default: q_proj,v_proj.
• --use-dora: Use DoRA in place of LoRA (currently NotImplementedError on mlx-lm 0.31.x).
• --max-seq-length INTEGER: Tokens per training sample. Default: 1600 (M3 Max 36 GB safe ceiling for 9B). See the fine-tuning training-ceiling map for per-model recipes.
• --iters INTEGER: Training iterations. Default: 1000.
• --learning-rate / --lr FLOAT: Default: 1e-4.
• --batch-size INTEGER: Default: 1 (Mac UMA constraint).
• --grad-accumulation-steps INTEGER: Default: 1.
• --grad-checkpoint / --no-grad-checkpoint: Enable activation checkpointing. Default: on.
• --val-batches INTEGER: Default: 25.
• --steps-per-report INTEGER: Default: 10.
• --steps-per-eval INTEGER: Default: 200.
• --steps-per-save INTEGER: Default: 100.
• -o, --output PATH: Adapter output directory. Default: ./optiq_lora_adapters.

optiq lora info

$ optiq lora info ADAPTER_DIR

Prints the per-layer rank distribution and trainable parameter count.

optiq serve

Dual-protocol inference server. Wraps mlx_lm.server and exposes both the OpenAI /v1/chat/completions endpoint and the Anthropic /v1/messages endpoint from the same process. Drive it with the OpenAI SDK, the Anthropic SDK, plain curl, or Claude Code (via ANTHROPIC_BASE_URL). Adds OptiQ-aware mixed-precision KV cache and a mounted LoRA adapter on top.

$ optiq serve [OPTIONS] -- [mlx_lm.server flags]

OptiQ-specific options

• --kv-config PATH: JSON config from optiq kv-cache. Per-layer mixed-precision KV. Overrides --kv-bits.
• --kv-bits INTEGER: Uniform KV quantization (4 or 8). Omit both --kv-config and --kv-bits for fp16 KV.
• --kv-group-size INTEGER: KV quantization group size. Default: 64.
• --quantized-kv-start INTEGER: Token offset at which KV quantization kicks in. Default: 0.
• --adapter PATH-OR-REPO: Apply a LoRA adapter at startup. Accepts a HuggingFace repo id (auto-downloaded) or a local directory. OptiQ-trained adapters surface their optiq_lora_config.json sidecar in startup logs.
• --models-dir DIR: Advertise locally-built quants under DIR in /v1/models (alongside HF-cache models and the served one), switchable per request by path. The server already hot-swaps to any requested model.
• --anthropic / --no-anthropic: Expose the Anthropic /v1/messages endpoint alongside OpenAI's. Default: on. Pass --no-anthropic to disable.

Forwarded to mlx_lm.server

All other flags pass through unchanged. Common ones include --model, --host, --port, --max-tokens, --temp, --top-p, --top-k. See python -m mlx_lm.server --help for the full upstream list and current defaults.

Examples

# 1. Generate a per-layer KV config (once per model)
$ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit --target-bits 5.0

# 2. Serve with mixed-precision KV + Anthropic on by default
$ optiq serve \
    --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --kv-config optiq_output/kv_cache/kv_config.json \
    --port 8080

# 3. Drive Claude Code through it
export ANTHROPIC_BASE_URL="http://localhost:8080"
export ANTHROPIC_API_KEY="not-used"
$ claude

optiq eval

Two-stage evaluation harness. --task smoketest runs a fast triage (KL + GSM8K-50, ~5 min on 27B). --task all runs the full benchmark suite (MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop, ~2 h on 27B). Add --score for the unweighted-mean Capability Score. Individual tasks are addressable on their own. KL is not part of that suite: it is the one step that holds a reference model in memory alongside the candidate, which is how a large quant gets killed on a memory-tight Mac, and it does not enter the Capability Score. Ask for it with --with-kl, or run it alone as --task kl. The methodology is in the eval-framework write-up.

$ optiq eval MODEL_PATH --task [TASK] [OPTIONS]

Tasks

• smoketest: KL on 64 prompts × 256 tokens, plus GSM8K on 50 samples. Fast triage.
• all: full benchmark suite below, in one run.
• kl: KL divergence vs reference (default 64 prompts).
• mmlu: 5-shot MMLU on 1000 samples.
• gsm8k: GSM8K on 1000 samples (default), CoT 3-shot. Default task if none specified.
• gsm8k-50: GSM8K on 50 samples (smoketest tier).
• ifeval: full 540-prompt IFEval (instruction following).
• bfcl: BFCL-V3 simple, 200 questions (function calling).
• humaneval: full 164 problems, code execution sandboxed (container via Docker, Podman or Apple container → macOS sandbox-exec → subprocess + rlimit).
• hashhop: HashHop Long-Context Evaluation. 25 instances per hop count × 4 hop counts (~100 total) at ~12 k context. Multi-hop key→value retrieval.

Options

• --task TEXT: One of the tasks above. Default: gsm8k.
• --n-samples INTEGER: Override the per-task default sample count.
• --baseline PATH: Side-by-side comparison against another model (gsm8k task only).
• --reference-model PATH: Explicit reference for the KL eval. If unset, auto-resolved.
• --reference-mode [auto|bf16|uniform_4bit]: KL reference selection strategy. Default: auto (bf16 if it fits in RAM via HfApi.model_info per-shard sizes, else mlx-community uniform-4-bit baseline).
• --score: With --task all, also compute the Capability Score (unweighted mean of MMLU, GSM8K, IFEval, BFCL, HumanEval, HashHop).
• --reasoning: Score an always-on reasoning model fairly. Lets the model emit its <think> block (no suppression), gives the generation tasks a large per-question budget so the trace completes, strips the think block before extracting the answer, and scores MMLU generatively (parse the answer letter) instead of by first-letter logit argmax, which collapses to chance for a model trained to reason before answering. Opt-in; default eval behavior is unchanged.
• --reasoning-max-tokens INTEGER: Per-question budget in --reasoning mode. Default: 3072 (a cap, not a fixed length, greedy decoding stops at EOS, so short traces cost nothing). Raise it for unusually long reasoners.
• --kv-bits INTEGER: Uniform KV quantization (4 or 8) during generation-based benchmarks. KL and MMLU are forward-pass only, so this has no effect there.
• --kv-group-size INTEGER: Default: 64.
• --kv-config PATH: Per-layer mixed-precision KV from optiq kv-cache. Overrides --kv-bits.
• --output-json PATH: Dump the structured eval record (every metric + Capability Score) to JSON.
• --served URL: Score through a running optiq serve instead of loading the model into the eval process. Same questions, same greedy decode. What changes is that the answers come back over HTTP, through the batch generator, the KV cache, the server’s rendering of the chat template, and its own tool-call parser. MODEL_PATH is still required — it supplies the tokenizer and the model id sent on each request, and only metadata is fetched, never weights. Not available for --task kl, --task smoketest, or MMLU without --reasoning: those score by comparing raw logits, which no endpoint exposes. Flags that configure the process holding the weights (--kv-bits, --kv-config, --stream-experts) are refused rather than silently ignored; set them on optiq serve.

Scoring the path your users are on

A default run loads the weights into the eval process. Nobody consumes a model that way — they run optiq serve and talk to it over HTTP. The two paths agree on published quants, but they are not the same code: only the served one goes through the batch generator, the KV cache and the server’s family-specific tool-call parser. Function-calling numbers in particular are worth confirming served, because that parser is the one a client actually receives calls from.

# one terminal
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --host 127.0.0.1 --port 8080

# another
$ optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --task bfcl --served http://127.0.0.1:8080

# the whole suite over HTTP (MMLU needs --reasoning to be servable)
$ optiq eval mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --task all --score --reasoning --served http://127.0.0.1:8080

Examples

# Fast triage
$ optiq eval ./optiq_output/Qwen3.5-9B/optiq_mixed --task smoketest

# Full benchmark suite + Capability Score, dump to JSON
$ optiq eval ./optiq_mixed --task all --score \
    --output-json ./bench.json

# Single benchmark
$ optiq eval ./optiq_mixed --task humaneval

# GSM8K head-to-head against the uniform-4-bit baseline
$ optiq eval ./optiq_mixed --task gsm8k \
    --baseline ./optiq_output/Qwen3.5-9B/uniform_4bit

# Eval the full suite with mixed-precision KV active during generation
$ optiq eval ./optiq_mixed --task all \
    --kv-config ./kv/qwen35_9b/kv_config.json

optiq benchmark

Quick-and-dirty perplexity + throughput on a converted model, with optional baseline side-by-side. For headline accuracy numbers, prefer optiq eval above.

$ optiq benchmark MODEL_PATH [--baseline UNIFORM_PATH] [--n-samples N]

• --baseline PATH: Side-by-side comparison against a uniform-4-bit baseline (or any other model path).
• --n-samples INTEGER: Perplexity samples. Default: 50.

optiq latency

Predicts decode tok/s for a quantized model using the Apple Silicon roofline model: latency ≈ model_bytes / memory_bandwidth + per_layer_overhead. Detects your hardware tier and bandwidth automatically.

# Bare prediction (linear-layer-only roofline; optimistic)
$ optiq latency ./optiq_mixed

# Calibrated: load + run the model once, fit the overhead constant
$ optiq latency ./optiq_mixed --calibrate

• --calibrate: Run 8 warmup + 15 measured generations to fit the framework-overhead constant for this hardware. The bare prediction only counts weight-loading time and is optimistic; on M3 Max ~83% of decode latency is overhead (attention, norms, KV cache, framework), so calibration produces meaningfully better predictions for the model you measured and for similarly-sized models on the same hardware.

Top-level options

• optiq --version: Print the installed version.
• optiq --help: Show top-level help.
• optiq COMMAND --help: Per-command help.

Configuration

Every setting resolves the same way, in this order:

flag > environment > repo config > user config > default

Config files are JSON objects keyed by setting name:

• <repo>/.optiq/optiq.json, checked in and shared by the team
• ~/.optiq/config.json, personal and applied everywhere

Each setting also has an OPTIQ_<NAME> variable. An unknown key in a config file is reported rather than ignored, because a typo that silently does nothing is worse than a warning.

$ optiq config          # every setting, its value, and which source won

The coding agent keeps its own settings in ~/.optiq/code/config.json, listed by optiq code config.

Settings

Other environment variables

• HF_HOME: Hugging Face cache root. Default: ~/.cache/huggingface.
• HF_HUB_ENABLE_HF_TRANSFER=1: Enable accelerated HF downloads (requires pip install hf_transfer).
• ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY: Set these to point Claude Code (and any Anthropic-SDK consumer) at optiq serve.