KV-quant serving

optiq serve is a drop-in replacement for mlx_lm.server. It exposes both the OpenAI /v1/chat/completions endpoint and the Anthropic /v1/messages endpoint from the same process. Point Claude Code, the OpenAI SDK, the Anthropic SDK, or plain curl at the same local URL. On top of that: sensitivity-aware quantized KV cache for long-context throughput, automatic prompt caching (multi-turn prefix reuse), structured JSON/regex output, model switching per request, and mounted LoRA adapters that swap per request.

Quickstart

# Stock fp16 KV serving. Works for any mlx-optiq quant.
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --port 8080

Then call it like any OpenAI endpoint:

$ curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mlx-community/Qwen3.5-9B-OptiQ-4bit",
      "messages": [{"role": "user", "content": "What is RoPE?"}],
      "max_tokens": 300,
      "stream": false
    }'

Mixed-precision KV cache

Default optiq serve uses fp16 KV. Pass --kv-bits 4 for uniform 4-bit KV or --kv-config kv_config.json for OptiQ mixed-precision KV. Both automatically install the streaming-converter and FlashAttention-2 patches that prevent stock mlx-lm's OOM at long context: on a 24 GB Mac at 32k context, peak drops from 16.35 GB (stock u4) to 7.60 GB (ours), 34% below the fp16 KV path. Speed is within ±2% of fp16 on Qwen3.5-9B-class models. At hash-hop hops=3 on Qwen3.5-9B, OptiQ mixed-precision scores 33% better than uniform 4-bit (32% vs 24% retention of fp16's 36%). For the engineering write-up see Why u4 KV cache OOMs harder than fp16; the underlying research on per-layer KV sensitivity is in Not All Layers Are Equal.

Step 1: measure

# 1-2 min. Once per model.
$ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --target-bits 5.0 \
    --candidate-bits 4,8 \
    -o ./kv/qwen35_9b

# writes ./kv/qwen35_9b/kv_config.json
# [{"layer_idx": 3, "bits": 8, "group_size": 64}, ...]

Step 2: serve

$ optiq serve \
    --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --kv-config ./kv/qwen35_9b/kv_config.json \
    --max-tokens 32768 --temp 0.6 --top-p 0.95

Prompt caching: automatic prefix reuse

Multi-turn conversations reuse the KV cache of their shared prefix automatically. No flag. When a request extends a conversation the server has already seen (the agentic pattern, where each turn appends to the history), the server matches the longest cached prefix and prefills only the new tokens. So the time-to-first-token of turn N stops growing with conversation length.

Measured on Qwen3.5-0.8B with a ~4.3k-token context: turn 1 (cold) prefills in ~0.97s; turn 2, extending it, reuses 4306 of 4331 tokens and drops to ~0.24s, a ~4× TTFT cut that grows with model size (bigger models have slower prefill). This is what makes Claude Code and other agents feel responsive against a local model: every turn after the first is near-instant to first token.

The cache is an LRU across conversations, held in RAM under a byte budget set by --prompt-cache-bytes. It's inherited from mlx-lm and on by default. Nothing to configure for the common case; raise the budget to keep more concurrent conversations warm:

$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --prompt-cache-bytes 8000000000 --port 8080

Each response reports how many prompt tokens were served from cache in usage.prompt_tokens_details.cached_tokens.

LoRA adapter at serve time

The CLI accepts one adapter per server process via --adapter. The argument is either a HuggingFace repo id (auto-downloaded into the OptiQ adapter cache) or a local directory:

# Local path
$ optiq serve \
    --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --adapter ./my_adapter

# HF repo id (downloaded on first use)
$ optiq serve \
    --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --adapter your-org/my-adapter

OptiQ-trained adapters surface their optiq_lora_config.json sidecar (rank, rank distribution, scaling mode) in the startup log.

Multi-adapter, hot-swap at the Python layer

The mounted-LoRA primitive supports multiple adapters resident at once, switched per call via a ContextVar. This is a programmatic API, not a CLI flag, embed it in your own server or notebook to serve N adapters from one base:

from mlx_lm import load, generate
from optiq.adapters.mount import (
    prepare_model_for_mounted_lora,
    mount_adapter_on_model,
    AdapterActivation,
)

model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
prepare_model_for_mounted_lora(model)
mount_adapter_on_model(model, "agent-A", "./adapter_a")
mount_adapter_on_model(model, "agent-B", "./adapter_b")

with AdapterActivation("agent-A"):
    out_a = generate(model, tok, prompt=p, max_tokens=100)

with AdapterActivation("agent-B"):
    out_b = generate(model, tok, prompt=p, max_tokens=100)

Mounted adapters stay separate from the base (unlike mlx-lm's load_adapters, which merges weights). The ContextVar means concurrent asyncio tasks or threads with different active adapters don't step on each other.

Embedding the server in your own process

If you want to install the OptiQ KV-cache hooks and Anthropic endpoint into a self-managed mlx_lm.server launch:

from optiq.serve import _load_kv_config, install_mixed_kv
from optiq.anthropic_server import install_anthropic_endpoint
import sys

# Per-layer mixed-precision KV cache
configs = _load_kv_config("./kv/qwen35_9b/kv_config.json")
install_mixed_kv(kv_configs=configs, quantized_kv_start=0)

# Anthropic /v1/messages alongside OpenAI /v1/chat/completions
install_anthropic_endpoint()

# Hand off to mlx_lm.server
sys.argv = ["mlx_lm.server",
            "--model", "mlx-community/Qwen3.5-9B-OptiQ-4bit",
            "--port", "8080"]
from mlx_lm.server import main
main()

OpenAI client compatibility

Use the official openai Python client by pointing it at your local server:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-used",  # local server, but key is required
)

resp = client.chat.completions.create(
    model="mlx-community/Qwen3.5-9B-OptiQ-4bit",
    messages=[{"role": "user", "content": "hi"}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Structured / JSON output

Pass an OpenAI response_format and the server constrains generation so the model can only emit tokens that keep the output valid. Both json_object (any valid JSON) and json_schema (a specific shape) are supported, plus the vLLM-style guided_regex and guided_choice extensions. Output is constrained to stay valid, so it parses without retries.

$ curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "messages": [{"role": "user", "content": "A person: Alice, 30."}],
      "response_format": {
        "type": "json_schema",
        "json_schema": {"schema": {
          "type": "object",
          "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
          "required": ["name", "age"]
        }}
      }
    }'
# -> {"name": "Alice", "age": 30}

The vLLM extensions take a raw regex or a fixed choice set:

# constrain to a regex
$ curl ... -d '{"messages": [...], "guided_regex": "[0-9]{3}-[0-9]{3}-[0-9]{4}"}'
# -> 123-456-7890

# constrain to a fixed set of answers
$ curl ... -d '{"messages": [...], "guided_choice": ["yes", "no"]}'
# -> yes

In the OptiQ Lab

The Lab Chat surface exposes the same constraint without curl. In Model & params, set JSON mode to any valid JSON or match schema (paste a JSON Schema in the box that appears), then send a message. The reply is guaranteed-valid JSON, parsed the same way the curl examples above are constrained. Tools are turned off while JSON mode is on, since a single constrained response and a tool-call loop are mutually exclusive. The Lab's server installs the constraint automatically, so any model you load supports it.

Tool-call healing

Quantized open-weight models often emit a malformed tool call instead of the clean structured form: a Hermes <tool_call> tag, a fenced JSON block, a bare object, trailing commas, fancy quotes, or function-call syntax. mlx-lm's parser only recognizes the canonical format, so a malformed call leaks into the message content as raw text and the client never gets a tool_calls array.

OptiQ heals these server-side. On a non-streaming completion that carried tools, it scans the output for the six common malformed shapes, recovers them into proper OpenAI tool_calls, strips the leftover "I'll call X" preamble from content, and sets finish_reason to tool_calls. Unknown tool names are rejected so the model cannot invent a tool. Always on; requests without tools are untouched. It is the same healer the OptiQ Lab uses, lifted to the server layer so any client (an agent, or Claude Code through the OpenAI endpoint) gets clean calls, not just the Lab UI.

The model field: single-model by default

optiq serve --model X hosts one model, but the OpenAI/Anthropic protocol requires a model field on every request. By default the server treats it as a label: every request is served by --model X, whatever the client sends. So a client sending the model's basename, a friendly alias, or a wrong default (Claude Code's claude-…) is served the one model instead of the server trying to download that id and returning a 404. This matches how the OptiQ Lab serves. One model per process, switched by restarting the server.

To let one running server hot-swap between cached models per request, pass --allow-model-switch (auto-enabled by --models-dir). Then the model field selects the model to load; an unknown id errors rather than being pinned. /v1/models lists every MLX model in your HuggingFace cache plus the served one.

# opt into per-request switching
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --allow-model-switch --port 8080

# now naming a different (cached) model switches to it
$ curl http://localhost:8080/v1/chat/completions -d '{
      "model": "mlx-community/Qwen3.6-27B-OptiQ-4bit",
      "messages": [{"role": "user", "content": "hi"}]
    }'

Locally-built quants that were never pushed to the hub won't show up in the cache scan. Point --models-dir at the directory holding them and they're advertised in /v1/models too, switchable by passing their path as the request model:

$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --models-dir ./optiq_output --port 8080

Model variants: thinking & sampling presets by name

Append a variant suffix to the model id and the server strips it before load (it maps to the real model at zero extra memory) and applies the preset to that request. So any OpenAI client selects it by name, without the non-standard chat_template_kwargs/sampler fields many clients don't expose. Two kinds of variant:

• Thinking: :no-think (direct answers) / :think (full reasoning), the reasoning-model toggle.
• Sampling presets: :precise (temp 0, deterministic), :creative (temp 0.8, top_p 0.95), :balanced (temp 0.4, top_p 0.9).

The served model's variants are listed in /v1/models.

# direct answer, no reasoning trace (faster, no rambling)
$ curl http://localhost:8080/v1/chat/completions -d '{
      "model": "mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit:no-think",
      "messages": [{"role": "user", "content": "Write a bubble sort in Python."}]
    }'

On families whose template has no thinking toggle (e.g. Gemma-4) the suffix is a harmless no-op; an unknown suffix (or a repo-id / path that legitimately contains a colon) passes through untouched. Works identically in optiq serve and the OptiQ Lab's API server.

Anthropic API: point Claude Code at your local quant

The same server simultaneously answers Anthropic's /v1/messages endpoint with the exact response shape Claude clients expect. This means you can drive a local mlx-optiq quant from any tool that speaks the Anthropic API: Claude Code, the official anthropic Python SDK, or your own integrations.

# Same optiq serve invocation. No extra flag needed.
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --port 8080

Anthropic SDK against your local quant:

from anthropic import Anthropic

client = Anthropic(
    base_url="http://localhost:8080",
    api_key="not-used",
)
resp = client.messages.create(
    model="mlx-community/Qwen3.5-9B-OptiQ-4bit",
    max_tokens=300,
    messages=[{"role": "user", "content": "hi"}],
)
print(resp.content[0].text)

Claude Code via env var (one line):

export ANTHROPIC_BASE_URL="http://localhost:8080"
export ANTHROPIC_API_KEY="not-used"
$ claude    # now driven by your local quant

Context scaling for smaller-context models. Claude Code auto-compacts based on the token usage the server reports vs the context window it assumes (~200k). A smaller-context local model would overflow before it compacts. --context-scale FACTOR multiplies the reported usage so compaction fires at the right real-token point, set it to (window the client assumes) / (your model's context), e.g. 6.25 for a 32k model behind a 200k assumption. Only the reported usage is scaled; generation is untouched. See the Claude Code guide.

Memory & resilience on Apple Silicon

Three flags keep a server healthy on unified memory, where the model, KV cache, and everything else share one RAM pool.

Idle auto-unload

A served model sits in unified memory even while idle, so the machine can't use that RAM for anything else. --idle-timeout SECONDS drops the model after that many seconds with no requests and reloads it lazily on the next call, the same lazy first-request load, re-triggered. A generation already in flight finishes safely (it holds its own reference to the model); only the next request pays the reload. Set the timeout longer than your longest single generation so a normal long decode is never interrupted. Off by default.

optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --idle-timeout 600

Memory-aware context cap

A request that runs to a big model's full native context (128k–256k) can allocate more KV cache than unified memory holds and kill the whole server mid-generation. --max-context auto (the default) reads the model's KV geometry and free RAM and engages a cap only when the full native context wouldn't fit, otherwise it's a no-op, so a machine with enough RAM sees no change. Once a prompt exceeds the cap, the KV window rotates (mlx-lm's RotatingKVCache) instead of crashing; batching, prompt-cache reuse, and KV quantization all keep working. Pass an integer for a hard token cap, or off to disable. Sliding-window models (Gemma-4, Qwen3-Next) manage their own KV and are untouched.

Resilient downloads, Xet → HTTPS failover

HuggingFace's Xet high-performance transfer is fast when it works but can stall partway through a large shard on corporate proxies, TLS-inspection middleboxes, or flaky links. OptiQ retries the normal path, then, as a last resort, forces the plain HTTPS path (HF_HUB_DISABLE_XET=1) for one final attempt, resuming from cache, so completed shards are never re-fetched. On by default for every optiq serve / optiq convert / adapter download; nothing to configure.

Production tips

• Bind to 127.0.0.1 for local-only use, or behind a reverse proxy. Don't expose the raw 0.0.0.0 binding to the public internet; there's no auth.
• Cap concurrency with --max-concurrent (default 8). Concurrent requests are batched, but each in-flight request holds its own KV cache, so mlx-lm's datacenter default of 32 can OOM-crash unified memory under a burst. OptiQ injects a Mac-safe cap; lower it further for big models / long contexts (excess requests queue), raise it if you have RAM to spare.
• Cap context with --max-context (default auto). Keeps a runaway-length prompt from OOM-crashing the server; a no-op unless the model's full native context wouldn't fit RAM.
• Reclaim idle RAM with --idle-timeout. On a shared machine, free the model when the server goes quiet; it reloads on the next request.
• Tune --max-tokens conservatively. Each in-flight request keeps a KV cache resident; long contexts dominate memory.
• Pre-load adapters. Loading a new adapter mid-flight stalls all in-flight requests. Mount everything you need at startup.