Family guide · DeepSeek-V4

DeepSeek-V4 on Apple Silicon

DeepSeek-V4-Flash is a sparse-attention mixture of experts. Across its 43 layers it runs three different attention mechanisms, chosen per layer by a compress_ratios entry in the config: five layers use sliding-window local attention, twenty compress the KV, and twenty-one compress it and then select a top-k subset to attend over. Almost all of its weight sits in the routed experts, which is what makes it quantize well.

Needs OptiQ 0.4.12+ Stock mlx-lm has no deepseek_v4 class. Three separate pull requests tried to add one and none were merged, so OptiQ vendors the working implementation and registers it on import optiq. Run pip install "mlx-optiq>=0.4.12", then import optiq before mlx_lm.load.

Running it on a small machine

The 2-bit quant is 92.5 GB on disk, which is more than most Macs have in RAM. OptiQ streams the routed experts off SSD instead of holding them resident: the scales and biases stay in memory, the packed expert weights are read as a token routes to them. The model loads in a few seconds at 6.5 GB resident. Peak memory climbs with context rather than with model size, so a short prompt stays near 8.5 GB and a 900-token one reaches about 12.7 GB.

Nothing needs configuring. optiq serve and the Python API detect a MoE quant too large to sit resident and switch to streaming on their own.

The quant

Model	bf16 size	OptiQ size	Resident	Context
DeepSeek-V4-Flash-0731-OptiQ-2bit	608 GB	92.5 GB	6.5 GB	1M

Hello world

hello.pypython

import optiq  # registers the deepseek_v4 arch with mlx-lm
from optiq.runtime import moe_stream
from mlx_lm import generate

model, tok = moe_stream.load_streaming("mlx-community/DeepSeek-V4-Flash-0731-2.4bit-mixed")
print(generate(model, tok, prompt="What is 17 * 23?", max_tokens=64))

Expect it to be slow Streaming experts from SSD trades throughput for fitting at all. Decode runs at a few tokens per second on a 24 GB machine, which is fine for a batch job and painful for a chat. With enough RAM to hold the model resident it runs at normal speed.

How the bits are allocated

The routed experts hold the overwhelming majority of the parameters and tolerate 2-bit; attention, the shared experts, the embeddings and the output head do not, and stay at 4-bit or higher. Quantizing everything uniformly collapses the model into repeated fragments, which is the same result other projects report on this architecture.

This is what OptiQ's per-layer allocation does everywhere, but the effect is starker here than on a dense model, because the split between what matters and what does not lines up so cleanly with the expert boundary.

Prompt format

DeepSeek-V4 ships no chat template, so the prompt is assembled by hand:

prompttext

<|begin_of_sentence|><|User|>{your message}<|Assistant|></think>

The trailing </think> closes the reasoning block so the model answers directly. Leave it off and it thinks first.