OptiQ Lab beat Unsloth Studio on hard research questions

We gave OptiQ Lab and Unsloth Studio the same job: answer 15 BrowseComp-Plus questions by searching a fixed set of documents, with the same local model on the same 24 GB Mac mini. Both apps talked to one optiq serve process, so the model, its quantization and its server were identical, and only the app around it changed. OptiQ Lab answered 9 of the 15. Unsloth Studio answered 5 in its best search mode.

Why BrowseComp-Plus, and why a mini corpus

BrowseComp-Plus is a research benchmark. Each question is a puzzle of several clues, and the answer must come from a fixed corpus of 100,195 web documents. The answer is one short fact, graded exactly. That makes it a good test of how well an app helps a model search, read and decide. Building a meaning index for the full corpus takes about six hours on a 24 GB Mac. Comparing several apps and search modes that way is impractical. So we used a smaller version that keeps what makes the benchmark hard.

The mini corpus is built from the benchmark's own labels. For each of the 15 questions in BrowseComp-Plus mini, BrowseComp-Plus lists the gold documents that hold the answer, the evidence documents that support it, and hard negatives: documents picked to look relevant without being so. We kept exactly those, 1,349 documents and 26 MB of text. Every answer still has to be pieced together across several documents, and the hard negatives are still there to mislead. The search is easier than over the full corpus. But it is the same search for every app, so the comparison measures the apps and not the corpus size. Anyone can rebuild it from the dataset's query ids and BrowseComp-Plus's document lists.

The setup

Both apps can search documents three ways: by keyword (BM25), by meaning (embeddings), or both together. We ran each app in all three modes. Thinking was off in both, the model's own sampling settings were used in both, and each question was graded from the answer line the benchmark asks for.

The results

One attempt per question. The Lab's keyword and meaning runs used its earlier default of 5 results; the hybrid run used 0.6.6's 10.

OptiQ Lab's best mode was hybrid search, with 9 of 15. Keyword search alone got 8. Meaning search alone got 3. That is consistent with what we saw in every test. BrowseComp questions are built from specific names, dates and places, and keyword matching finds those far more reliably than a small embedding model does. Hybrid search helps only if it never pushes a good keyword match out of the results. So in OptiQ 0.6.6 the Lab lists keyword matches first and gives the last two of ten places to the best meaning matches keyword search missed.

Unsloth Studio's best mode was keyword search, with 5 of 15. Hybrid got 2 and meaning search 1. Its answers rarely drew on the documents: across the keyword and hybrid runs, one answer cited a document. In those two runs, 6 and 7 of the 15 replies ran to the 16,384-token limit without finishing.

What it took to run Unsloth Studio fairly

Getting Unsloth Studio to run this comparison fairly took more work than we expected. Everything below was needed to give it the same model, documents and settings as the Lab. Each has a workaround, and each would affect anyone who connects Studio to their own model server.

• Choosing Qwen3-Embedding broke indexing. 737 of the first 783 documents failed with a compute error from the embedding server. Studio cuts chunks at 500 tokens, sized for its default bge-small, and Qwen3's tokenizer makes them longer than the embedder's 512-token batch. Setting RAG_CHUNK_TOKENS=400 fixed it. With bge-small, every document indexed.
• Thinking stayed on. Studio does not pass "thinking off" to a custom model server until the connection's reasoning setting is changed to chat_template_kwargs.enable_thinking. Before that, 14 of 15 replies reasoned to the token limit.
• No thinking budget for your own server. Studio's reasoning budget applies to models it runs itself. With thinking on, 41 of 45 answers came back empty across the three modes.
• Studio's sampling replaces the model's. It sends temperature 0.6 and presence penalty 0 on every request. With the model's own presence penalty gone, 9 of 13 replies in one run looped on a guess until the limit. We sent the model's values explicitly.
• Document search only runs on streamed requests. Without streaming, the model never got the search tool and invented document titles.
• The embedder starved the chat model. With Studio's embedder on the GPU next to a 19B model on a 24 GB Mac, the model server ran out of memory on a three-word request. RAG_EMBED_DEVICE=cpu fixed it.

A note on deep research

Both apps also have a deep research mode. On these questions it did worse than chat in both: the Lab answered 3 and Studio 4. BrowseComp-style questions reward a long search loop that follows each clue to the next, while both deep research modes plan their searches up front and write a report, so this benchmark is not the right one for comparing them.

Try it

OptiQ Lab ships with mlx-optiq. Hybrid search over your own document collections needs an embedding model next to the chat model. The optiq serve command loads it with one flag.

pip install -U mlx-optiq
optiq serve --model mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit-REAP-19B \
  --embedding-model Qwen/Qwen3-Embedding-0.6B
optiq lab