Speculative decoding on Apple Silicon

A language model writes one token per forward pass. Speculative decoding guesses several tokens ahead, then checks all of them in a single pass of the full model. Every guess the model agrees with is a token it did not have to generate on its own; the first guess it disagrees with is thrown away and the model’s own token is used. The output stays the model’s output. What changes is how many passes it takes.

OptiQ has three places to get the guesses from. They share the same verify loop and differ in what does the guessing, which decides where each one pays off.

One method runs per server. Use n-gram lookup when the work repeats its context: coding agents reading and editing files, tool calls that name paths from earlier turns, answers that quote code. It needs nothing extra and works on models the other two don’t cover. On the families that have one, the MTP head or the drafter also speeds up new prose, where there is little to copy.

N-gram lookup: any model

Agents repeat themselves. A file edit copies most of the file it just read, a tool call names paths already in the context, a final answer quotes the code it wrote. --ngram-draft turns that into speed: when the last few tokens match something earlier in the conversation, the server drafts the tokens that followed it and checks them all in one forward pass. Accepted drafts cost a fraction of a normal step each. A rejected draft is discarded, and the model’s own token is used instead.

$ optiq serve --model mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit-REAP-19B \
    --ngram-draft 8 --port 8080

There is no draft model to download or keep in memory, so it works with every model the server loads: dense, MoE and hybrid linear-attention alike. The draft length is not fixed. The server times one verification pass per model when it first loads (a few seconds), then picks how many tokens to draft on each step from how often recent drafts were accepted. It drafts long while text is being copied and stops drafting during new prose, where a wrong guess would only add work. On Qwen3.5 and Qwen3.6 hybrids, a rejected draft rolls the recurrent state back to the last accepted token directly, with no second pass.

Measured by replaying real OptiQ Code and OptiQ Lab conversations, where the model regenerates a turn it once wrote (methodology below):

Live on an M3 Max with Qwen3.6-35B-A3B-REAP-19B, coding turns decoded at 84.8 tok/s against 64.7 plain (1.31×), and rewriting a file went from 59 to 105 tok/s. Short replies and prose that copies nothing gain little. The gate keeps those close to plain speed.

Every emitted token is the model’s own choice at that position. In float32 the output is identical to plain decoding. On quantized weights, checking several tokens in one pass uses different kernels than one token at a time, so a near-tie between two tokens can occasionally resolve the other way. Plain decoding does the same when a prompt is prefilled in different chunk sizes.

• --ngram-draft N: most tokens to draft per step. 8 is a good value; 0 (default) turns it off.
• --no-ngram-gate: always draft N tokens when a match is found. For measurement; the gate is faster.
• --ngram-min N: shortest match that triggers a draft. Default 3.

MTP head: Qwen3.5 and Qwen3.6

Qwen3.5 and Qwen3.6 ship an extra small prediction head in their weights, which the literature calls MTP, for Multi-Token Prediction. OptiQ quants keep it, and OptiQ uses it as the guesser. The base model still produces every output token; on the steps where the head’s guess matches, that token comes for free.

$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

The server reads the model’s recommended sampling settings from generation_config.json and applies them unless you pass your own via --temp, --top-p and --top-k.

We report greedy here because it isolates the speedup measurement from sampling noise and matches how unsloth and the upstream MTP literature publish their headlines.

With Qwen's recommended production sampler (temp=1.0, top_p=0.95, top_k=20), the speedup is smaller but still positive everywhere:

Acceptance is the literature definition, drafts_accepted / drafts_attempted, read straight from the engine.

Where MTP does not pay off

For Qwen3.5 0.8B the base model is already at 130 tokens per second, and the speculation overhead per cycle eats more than the head can give back. We measured a regression to about 0.7x. The 2B model lands close to break-even. Skip MTP at these sizes. 4B and up consistently win.

About depth

The default is depth 1, meaning one drafted token per cycle. Depth 2 and above does not help on Apple Silicon. The reason is that Metal's K-token verify forward scales close to linearly with K, while on CUDA the same forward is nearly free due to spare matmul throughput on Tensor Cores. We measured depth 2 through 4 and depth 1 wins every single configuration. We also tried HuggingFace's adaptive depth heuristic (raise K after a clean cycle, lower it on partial accept). It lost 4 to 17 percent depending on the model and sampler. So we ship a fixed depth 1.

Assistant drafter: Gemma-4

Gemma-4 has no MTP head. Google instead publishes a separate small drafter, the -assistant variant: a 4-layer model that shares K and V with two layers of the target. OptiQ loads it next to the target and runs the same verify loop. The plumbing differs (typed K/V sharing, target-hidden conditioning, two kinds of cache layer), but from the serving side it is just another source of guesses. This path is greedy only for now.

$ optiq serve --model mlx-community/gemma-4-E4B-it-4bit \
    --drafter mlx-community/gemma-4-E4B-it-assistant-bf16

Greedy, γ=1, 200-token generation, five prompt categories, median of three runs each on M4 Pro 24 GB:

Greedy, γ=1, 200-token generation, five prompt categories, median of three runs each on M4 Pro 24 GB:

Acceptance is lower than Qwen MTP for two reasons. First, the Gemma drafter is a separately trained Q-only model that has to predict the target's distribution from a few shared cache layers, not a head that was co-trained on the target's loss. Second, mlx-lm's multi-token verify forward is not bit-identical to the equivalent sequence of single-token forwards due to bf16 attention precision; the largest diff we measured was 0.68 in logit magnitude at the second of two positions. This means a draft the target would have accepted in a single-token verify can be rejected in a multi-token verify. Greedy outputs still match a baseline greedy run for a long prefix (200 tokens identical on our reasoning prompt), then drift on longer sequences.

γ-sweep on the math prompt above (200 tokens, median of three):

γ=1 is optimal on Metal for the same reason it is optimal for Qwen MTP: the K-token verify forward scales near-linearly with K, while acceptance stays roughly constant per draft slot. The math is in Getting MTP to actually work on Apple Silicon, “What about depth 2 or higher”. γ>1 is implemented and lossless within the same bf16 precision bound as γ=1, but ships defaulted to γ=1.

Serving and the Lab

All three work behind every serving endpoint (OpenAI, Anthropic, Responses), so clients need no knowledge that speculation is happening. Chat templates work the same, and enable_thinking=False still applies. Weight quantization is unaffected: the MTP head ships as a 4-bit projection with a bf16 final layer, matching the host model.

The flags are mutually exclusive; pick one per loaded model. In OptiQ Lab, the Server page has a checkbox for the Qwen MTP path and a “Spec drafter” picker for the Gemma path; n-gram lookup is a server flag. With n-gram lookup on, requests are decoded one at a time, which matches how one person runs a local model.

Compatibility

Methodology

Each Qwen measurement runs in its own subprocess for clean memory state. The prompt is a 166-token Python question. We generate 512 tokens. Decode tokens per second comes from mlx-lm's GenerationResponse.generation_tps, which is measured after prefill so it captures only the decoding phase.

For Gemma-4 we run five chat-templated prompts (one each from math, code, prose, dialogue, reasoning), 200-token generations, median of three runs per prompt, all in a single Python process. Decode tokens per second is wall-clock n_tokens / elapsed for both baseline and spec, with a warm-up generation discarded.

Qwen acceptance comes from the engine’s drafts_accepted and drafts_attempted counters. Gemma acceptance comes from the drafter loop’s equivalent counters. Both follow the standard literature definition, so the comparisons are apples to apples with unsloth and llama.cpp numbers.

Hardware for MTP and the drafter: Apple M4 Pro, 24 GB unified memory, 19.1 GB Apple-recommended working set.

For n-gram lookup, 120 OptiQ Code and 120 OptiQ Lab conversations are cut before an assistant turn and that turn is replayed through the real drafting and gate code. The cost of each verification pass comes from timings measured on Qwen3.6-35B-A3B-REAP-19B, with and without the hybrid rollback, so the replay is free of machine-load noise. The live check ran the same model on an M3 Max, 36 GB, over 12 conversations, plain and speculative interleaved from identical prefilled caches, best of two runs each.

For the longer story of how we got MTP working correctly on this stack, see the blog post on Apple Silicon MTP. For the Gemma -assistant path, see Gemma-4 spec decoding on Apple Silicon.