Qwen on Apple Silicon

Three Qwen generations run on OptiQ: Qwen3.8, Qwen3.6 and Qwen3.5. They share a chat template, a hybrid linear+full attention architecture, and the same <think>...</think> reasoning channel, so most of what follows applies to all three. Where they differ, the section says so. Newest first.

Qwen3.8

The newest of the three, and the first Qwen we ship with image input. Qwen3.8-27B is a hybrid: three quarters of its 64 blocks use linear attention and the rest full attention, which is why the sensitivity sweep scores 498 tensors rather than the ~450 a dense model this size would have.

261 of its 498 quantized tensors sit at 8-bit and 237 at 4-bit. The vision tower stays at bf16 in a sidecar under optiq/, so the same repo loads text-only under stock mlx-lm and image+text under OptiQ.

Qwen3.6

Qwen3.6 is the successor to Qwen3.5: strong reasoning at sizes that fit on consumer Apple Silicon. We ship two quants: a dense 27 B and a 256-expert MoE with 3 B active per token. Both load with stock mlx_lm.load and reach 89–95 % on GSM8K.

Available quants

Hello world

from mlx_lm import load, generate

model, tok = load("mlx-community/Qwen3.6-27B-OptiQ-4bit")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Compare REINFORCE and PPO in two paragraphs."}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=600))

Recommended sampling

from mlx_lm.sample_utils import make_sampler

# Strong reasoning baseline (Qwen3.6 supports thinking mode)
sampler = make_sampler(temp=0.6, top_p=0.95, top_k=20)

# Conversational
sampler = make_sampler(temp=0.7, top_p=0.9)

The 35B-A3B MoE model

Qwen3.6-35B-A3B is a 256-expert sparse mixture-of-experts: 35 B total parameters, only 3 B active per token. The fused expert tensor (switch_mlp in MLX terminology) gets quantized as a single layer in mlx-optiq's sensitivity pass, but each expert independently uses the assigned bit-width.

Expect MoE inference to be faster than the dense 27 B at the same memory footprint, because only 3 B of weights actually multiply per token. The sensitivity pass is also faster because there are fewer "layers" (the experts collapse into single switch_mlp tensors).

Long-context serving

# Sensitivity pass (1-2 min, once per model)
$ optiq kv-cache mlx-community/Qwen3.6-27B-OptiQ-4bit \
    --target-bits 5.0 --candidate-bits 4,8 \
    -o ./kv/qwen36_27b

# Mixed-precision KV serving
$ optiq serve --model mlx-community/Qwen3.6-27B-OptiQ-4bit \
    --kv-config ./kv/qwen36_27b/kv_config.json \
    --max-tokens 32768 --temp 0.6 --top-p 0.95

Fine-tuning

27B fits at max-seq-length=512 on a 36 GB Mac with default rank=8 LoRA on q_proj/v_proj. The 35B-A3B MoE caps at max-seq-length=128 due to per-expert memory overhead but trains 3× faster per iteration than the dense 27B because of sparse activation. The rank-scaling story is in the sensitivity-aware LoRA blog.

# Qwen3.6-27B at T=512, peak ~27.7 GB
$ optiq lora train mlx-community/Qwen3.6-27B-OptiQ-4bit \
    --data ./my_data \
    --max-seq-length 512 \
    --rank 8 --rank-scaling by_bits \
    --iters 1000 -o ./my_adapter

# Qwen3.6-35B-A3B at T=128, peak ~25.3 GB, 32 tok/s
$ optiq lora train mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit \
    --data ./my_data \
    --max-seq-length 128 \
    --rank 8 --rank-scaling by_bits \
    --iters 2000 -o ./my_moe_adapter

Next: see how sensitivity works, or read the fine-tuning guide.

Qwen3.5

Qwen3.5 is Alibaba's late-2025 release: a six-model dense lineup plus one sparse MoE. We ship mlx-optiq-4-bit quants for all six. They share the same chat template, hybrid linear+full attention architecture, and the same <think>...</think> reasoning channel.

Available quants

Hello world

from mlx_lm import load, generate

model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is the capital of Australia?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=200))

Recommended sampling

Qwen3.5-Instruct variants behave well at:

from mlx_lm.sample_utils import make_sampler

# Reasoning / math / code
sampler = make_sampler(temp=0.6, top_p=0.95, top_k=20)

# Conversational chat
sampler = make_sampler(temp=0.7, top_p=0.9)

# Deterministic / classification
sampler = make_sampler(temp=0.0)

Reasoning channel

Qwen3.5 has built-in chain-of-thought via <think>...</think> tags. Default-on for instruct variants:

# Default: thinking enabled (slower, more accurate on math/logic)
prompt = tok.apply_chat_template(messages,
    tokenize=False, add_generation_prompt=True)

# Disable thinking for snappier replies (chat, classification)
prompt = tok.apply_chat_template(messages,
    tokenize=False, add_generation_prompt=True,
    enable_thinking=False)

To programmatically strip the <think> block before showing output to the user:

import re
out = generate(model, tok, prompt=prompt, max_tokens=800)
final = re.sub(r"<think>.*?</think>", "", out, flags=re.DOTALL).strip()

Hybrid attention

Qwen3.5 uses a hybrid architecture: most layers are linear-attention (Gated DeltaNet), and a sparse subset are full-attention. Layer indices [3, 7, 11, 15, 19, 23, ...] (every 4th) are full-attention; the rest are linear. This matters for KV-cache serving. Only the full-attention layers carry a KV cache that needs sensitivity-aware quantization.

Long-context serving

For 64 k-token contexts, run a one-time KV sensitivity pass and serve with the resulting config:

# 1-2 min, once per model
$ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --target-bits 5.0 --candidate-bits 4,8 \
    -o ./kv/qwen35_9b

# Serve at :8080 with mixed-precision KV
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --kv-config ./kv/qwen35_9b/kv_config.json \
    --max-tokens 32768 --temp 0.6 --top-p 0.95

Fine-tuning recipes

Empirical training-ceiling map at iogpu.wired_limit_mb=0 on a 36 GB Mac (default config: q_proj, v_proj, num_layers=16, rank=8, rank_scaling=by_bits). The rank-scaling rationale lives in the sensitivity-aware LoRA blog.

# 9B at T=1400, proven sweet spot, peak 25.4 GB
$ optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --data ./my_training_data \
    --max-seq-length 1400 \
    --rank 8 --rank-scaling by_bits \
    --num-layers 16 --iters 1000 \
    -o ./my_adapter

See the full LoRA fine-tuning guide for the algorithm, rank-scaling explanation and PEFT-compat output format.