mlx-optiq
OptiQ CLI · optimizing compiler

Quantize, serve, and fine-tune from the command line.

The MLX-native core. Measure each layer's sensitivity, allocate mixed-precision bits, and ship a quant that keeps more quality than uniform 4-bit at the same size. Then serve it behind an OpenAI- or Anthropic-compatible API and fine-tune it with sensitivity-aware LoRA. No PyTorch, no cloud.

Per-layer bit allocation · sample LLM
Per-layer bit allocation across a 32-layer transformer: tall emerald bars are 8-bit protected layers, short warm-grey bars are 4-bit.
8-bit · sensitive layers 4-bit · robust layers
$ pip install mlx-optiq
3.1×
smaller than bf16
+13.6
best Capability gain vs U4
How it works

One sensitivity signal. Three steps.

Uniform 4-bit treats every layer the same, but layers are not the same. OptiQ measures, then allocates.

01 · MEASURE

Per-layer KL

Simulate-quantize each layer at each bit-width, forward-pass calibration data, and measure the KL divergence from the bf16 reference.

02 · ALLOCATE

Greedy knapsack

Start every layer low, then upgrade the layer that buys the most quality per bit until the budget is met. lm_head and the first/last blocks stay high.

03 · CONVERT

Standard MLX

Hand the bit map to mlx_lm.convert. The output loads anywhere stock mlx-lm loads, with the sensitivity metadata stored alongside.

What it does

A whole toolkit around one signal.

The same per-layer sensitivity pass drives weight, KV-cache, and LoRA-rank allocation. The rest of the toolkit sits around that core.

i

Mixed-precision weights

Sensitive layers stay high-precision, the rest go low, at the same average size as uniform-4.

Higher accuracy at the same disk size
ii

Mixed-precision KV cache

A separate sensitivity pass on the cache. Layer 0 is often 56× more sensitive than average, so uniform 4-bit KV is catastrophic; mixed-precision is not.

Faster long-context decode, kept sharp
iii

LoRA, two ways

Fine-tune with adapter rank scaled by each layer's bits, then keep N adapters mounted on one base and hot-swap per request.

Sensitivity-aware rank · hot-swap
iv

Text and images

Send pictures to vision-capable models. The vendored tower rides in a bf16 sidecar, so one repo loads text-only under mlx-lm or full image+text under OptiQ.

VLM + LLM · one artifact
v

OpenAI + Anthropic serving

optiq serve speaks both protocols from one process. Point Claude Code, Codex, or any client at a local quant.

Dual-protocol local server
vi

Evaluation

A two-stage harness: a fast smoketest for triage, and a full six-metric suite that produces the Capability Score on every model card.

Smoketest + full benchmark suite
Quickstart

From a model to a serving API.

i

Convert

--reference auto picks bf16 when it fits and a uniform-4-bit baseline when it does not.

terminalbash
$ optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8
ii

Serve

Dual-protocol server with mixed-precision KV and idle auto-unload.

terminalbash
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --idle-timeout 300
Full reference The CLI reference documents every command; How sensitivity works covers the algorithm.

The compiler behind every OptiQ quant.

Quantize, serve, and fine-tune LLMs natively on Apple Silicon.

Read the CLI docsBrowse the models