Nemotron on Apple Silicon

NVIDIA's Nemotron 3 is a hybrid: it interleaves Mamba2 state-space blocks with a handful of full-attention layers, and the MoE variants add a sparse expert mixture. In the 4B (dense), only four of the 42 backbone blocks are true attention; the rest are linear-attention SSM or MLP. The 30B-A3B routes through 128 experts at ≈3 B active parameters per token. The Super 120B-A12B goes further: a 512-expert MoE (22 active) that, at 2-bit, runs on a 36 GB Mac by streaming its experts off SSD. All load through mlx-lm's built-in nemotron_h class.

Nemotron 3.5 Lightning

Lightning-30B-A3B is the newer generation: the same Mamba-attention hybrid, now with a 128-expert MoE and 6 experts active per token. 22.8 GB on disk, down from 65.8 GB, and the MTP speculative head is preserved in the sidecar.

The routed experts carry most of the weight, so that is where the bits come from: every 4-bit assignment in the model is an expert tensor, with the rest of the stack at 8-bit. Code generation is its strength; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers predicts.

At 22.8 GB it will not sit resident on a 24 GB machine. Stream the experts from SSD instead:

$ optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts

Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory.

The quants

Per-benchmark breakdown, 30B-A3B

Per-benchmark breakdown, 4B

Hello world

from mlx_lm import load, generate

model, tok = load("mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain why hybrid Mamba+attention models scale to long contexts."}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=300))

Hybrid KV cache

Only the four full-attention layers carry a KV cache, the Mamba2 blocks keep recurrent state instead, which is what gives the architecture its flat long-context memory profile. The repo ships a kv_config.json from a real sensitivity pass that covers just those attention layers: three at 4-bit, one at 8-bit, 5.0 average KV bits. Point optiq serve at it for mixed-precision KV.

Serving

$ optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit \
      --kv-config kv_config.json --port 8000

# From any OpenAI-compatible client:
$ curl -s http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit",
         "messages":[{"role":"user","content":"What is 17 * 23?"}]}'