mlx-optiq
Benchmark · October 15, 2026

The best coding agent for local models, and Boost for nearly 4x the solves

We ran three coding agents on the same local model, a Qwen3.6-35B-A3B on a 24 GB Mac mini, across 30 tasks from DeepSWE-mini and Terminal-Bench. OptiQ Code passed 313 of the 771 DeepSWE target tests, against 179 for mini-swe-agent and 42 for opencode, and solved 4 of the 14 Terminal-Bench tasks against 3 and 2. Then we turned on Boost, which hands the task to optiq-boost, the OptiQ Cloud model, and keeps the local model as its helper. Solved tasks went from 4 to 15 and passing DeepSWE tests from 313 to 706. All 30 tasks cost $2.80 at Boost's per-token price.

The setup

The 30 tasks are the 16 in DeepSWE-mini and the 14 in Terminal-Bench mini, both public on Hugging Face. They are the set a cloud model has to clear before we offer it through Boost, chosen to cover different languages, repo sizes and kinds of work. DeepSWE-mini tasks are feature requests on real open-source repos (Go, TypeScript, Python), graded by hidden tests. Terminal-Bench tasks are system jobs in a container: build an old C program, recover a database from its write-ahead log, crack a 7z hash, fit a Bayesian network.

Held fixedValue
Local modelQwen3.6-35B-A3B, OptiQ 4-bit
HardwareMac mini M4, 24 GB, served by optiq serve
TasksDeepSWE-mini 16 + Terminal-Bench mini 14
Limits2 hours and 1,000 turns per task, one attempt each
Gradingthe benchmark's own hidden tests

The local model is Qwen3.6-35B-A3B, the latest Qwen 3.6 mixture-of-experts model, quantized with OptiQ so it fits on a 24 GB Mac with room for long context. Every agent talked to the same optiq serve endpoint, so the model, its quantization and its KV cache were the same in every run. One attempt per task is a small sample. With 30 tasks spread across two benchmarks, the totals are steadier than any single task, which can swing by many tests from one run to the next.

Three agents, one local model

DeepSWE-mini is hard for a model this size. None of the three agents fully solved a task, so the useful number is how many of the hidden target tests each one got passing. OptiQ Code passed 313 of 771, which is 1.75 times mini-swe-agent's 179 and more than seven times opencode's 42. On Terminal-Bench, where a task is pass or fail, OptiQ Code solved 4, mini-swe-agent 3 and opencode 2.

DeepSWE-mini target tests passing · same local model
Local model only · all 30 tasks
OptiQ Codemini-swe-agentopencode
DeepSWE-mini target tests313 / 771179 / 77142 / 771
Terminal-Bench tasks solved4 / 143 / 142 / 14

Most of the gap comes from what a small model does over a long task. It loses track of a requirement, repeats a command that already failed, or declares the work done while a test is still red. OptiQ Code is built around those habits. It notices when the model has stopped making progress and tells it what it has been repeating, checks the build after edits, keeps tool output to the part that matters, and before accepting "done" makes the model point to where each stated requirement is implemented and tested. A general agent leaves all of that to the model, and a 3B-active model does not manage it on its own.

So with a local model, OptiQ Code was the strongest of the three agents we tested. It still solved none of the DeepSWE tasks outright, and at that point the limit is the model rather than the agent. Boost is for those tasks.

What Boost does

Boost runs a task with optiq-boost, the OptiQ Cloud model, in charge. The cloud model reads the task, plans, decides what to change and writes the important edits. Your local model becomes one of its tools: the cloud model can hand it a self-contained job, such as reading a set of files and answering a question, running the tests and reporting only the failures, or making an edit it has fully specified. The local model does that job in its own context on your Mac and sends back a short answer.

Earlier versions of Boost worked the other way round. The local model led, and OptiQ Code called the cloud model in when the local model got stuck, either for advice or to take over for a few turns. We measured both designs on these tasks. Letting the local model lead and calling for help on a stall did worse every time we compared them, so as of 0.6.0 Boost means the cloud model leads.

Boost on the same 30 tasks

Same tasks, same limits, same local model on the same Mac, with optiq-boost leading.

Tasks solved · 30 tasks
OptiQ Code, local model alone vs with Boost
Local aloneWith Boost
DeepSWE-mini tasks solved0 / 167 / 16
DeepSWE-mini target tests313 / 771 (40.6%)706 / 771 (91.6%)
Terminal-Bench tasks solved4 / 148 / 14
Terminal-Bench tests26 / 6052 / 60
All tasks solved4 / 3015 / 30

The seven DeepSWE solves are in koota, textual, yaegi, valibot, aiomonitor, ofetch and narwhals, across TypeScript, Python and Go. On Terminal-Bench, Boost picked up five tasks the local model had missed (qemu-alpine-ssh, build-pmars, mcmc-sampling-stan, rstan-to-pystan and vulnerable-secret) and lost one it had solved, regex-log.

Several of the misses were close. Bandit finished with all 69 of its target tests passing and one existing test broken, and termenv had 32 of 35. We reran both with the cloud model set to think harder. Bandit came out at 67 of 69 and termenv at 34 of 35, so neither was solved, and the extra thinking did not help on these two.

Who did the work

Before reading the table as a measure of the local model's contribution, here is how often the cloud model actually used it. On Terminal-Bench it never called the local tool. On DeepSWE it called it on 5 of the 16 tasks, twice each. So these results mostly measure the cloud model working inside OptiQ Code, with the local model as a minor helper. Getting the cloud model to delegate more of the reading and test runs, which is where the local model saves cloud tokens, is the next thing we are working on.

Tests the agents broke

We also counted the existing tests each run broke, since a patch can pass every new test and still break old ones. With Boost, two tasks each lost one existing test (bandit and prometheus) because the model did not run the full suite before finishing. A third, dynamodb-toolbox, lost 953, and that one was our bug. The agent's own suite had passed with 1,293 tests. Then the end-of-run rollback mistook TypeScript type errors for a broken build and restored half of the changed files, leaving an import that no longer resolved. 0.6.0 fixes this: type errors alone no longer trigger a rollback, and a rollback that does not fix the build is undone. Running alone, the local model broke 564 existing tests in expr and 82 in prometheus.

What it cost

Boost bills per token. optiq-boost costs $0.15 per million input tokens, $0.03 per million cached input tokens and $0.50 per million output tokens, charged from your OptiQ Cloud balance. All 30 tasks used 60.4 million input tokens, 54.7 million of them served from cache, and 0.6 million output tokens. At those rates the whole run cost $2.80.

At Boost's priceCost
All 30 tasks$2.80
Median task$0.056
Most expensive task (koota, 124 turns)$0.33
Per solved task$0.19

For a sense of scale, here is the same token count priced at the listed rates of other models people use for coding, if you called them directly. This prices our workload, not theirs: a different model would take a different number of turns and tokens on the same tasks, and might solve more of them. The comparison shows what the per-token price does to a long agent run, where nearly all of the input is the conversation being re-read.

The same 61 million tokens, Boost vs other models at their listed rates

Other models at their listed per-token rates on October 15, 2026, applied to the input, cached-input and output tokens of the Boost run.

Try it

OptiQ Code works with no account and no cloud. Boost needs an OptiQ Cloud account, which comes with $2 of free credit, enough for around 35 median tasks from this run.

terminalbash
# the local agent
pip install -U mlx-optiq
optiq code

# connect OptiQ Cloud, then type /boost in OptiQ Code before a hard task
optiq cloud login

With Boost on, the run summary shows what the task cost and what is left on your balance. Everything except the cloud model's own calls runs on your Mac: the files, the tests, and every job the cloud model hands to your local model.