Qwen 3.6 27B: the local model that actually works

Qwen 3.6 27B hit the front page of Hacker News on June 29 with 822 points and a title that sounds like clickbait but is not: "Qwen 3.6 27B is the sweet spot for local development." I have been burned by local models before. They are slow, they hallucinate protocols that do not exist, and they suggest npm packages that were deprecated in 2023. So I was skeptical. Then I ran it myself.

It is good. Annoyingly good. Good enough that I have been reaching for it instead of Claude for a lot of day-to-day coding tasks, which is not something I expected to say about a model that fits in 42GB of RAM.

Two variants, pick the right one

Qwen 3.6 comes in two sizes. There is a mixture-of-experts model called Qwen 3.6 35B A3B, which activates only 3 billion of its 35 billion parameters per token. It is fast. And there is a dense Qwen 3.6 27B, which runs every parameter on every token. It is slower, but it is smarter.

The consensus from people actually using both is that 27B is the one you want. The 35B MoE is roughly three times faster, but the output quality is noticeably worse for anything that requires sustained reasoning. If you are doing code generation, technical writing, or anything where the model needs to hold a complex idea together for more than a few hundred tokens, 27B wins. I would rather wait an extra second per token and get code that compiles on the first try.

The author of the Quesma blog post that went viral on HN put it plainly: "I'd rather generate a third as much code, but of higher quality." That tracks with my experience. Speed matters, but wrong code that arrives fast is not actually fast. You spend the saved time debugging.

How good is it, in numbers

Artificial Analysis scores are a rough but useful yardstick. Their composite benchmarks put Qwen 3.6 27B at a score of 37. For context, here is how that compares to models you know:

A score of 37 from a 27B parameter model is wild. That is a model smaller than Llama 3 70B, running on a MacBook, scoring comparably to frontier models that cost millions to serve. The era equivalences are approximate and you should not read too much into them, but the point is clear: open models have caught up faster than most people expected.

One caveat on the DeepSeek V4 Flash numbers: that was tested at Q2-Q4 quantization, which is aggressive. The full model is better. But even accounting for that, Qwen 3.6 27B at 8-bit quantization appears to be in the same neighborhood, which is remarkable given the size difference.

Running it yourself

Setting up Qwen 3.6 27B with llama.cpp is surprisingly simple. One command:

llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 \
  --spec-type draft-mtp -ngl 999 -fa on -c 65536 --port 8080

That downloads an 8-bit quantized model from HuggingFace on first run, caches it locally, and starts an OpenAI-compatible API server on port 8080. The -hf flag handles the download. The draft-mtp flag enables multi-token prediction, which is Qwen 3.6's built-in speculative decoding mechanism. The -ngl 999 puts all layers on the GPU. -fa on turns on flash attention. Context size is set to 64K tokens (the model supports up to 256K, but 64K is more practical for most tasks on consumer hardware).

You do not need Ollama, Docker, or a web UI. Just llama.cpp and a machine with enough RAM.

What hardware you need

This is where things get real. The 8-bit quantized 27B model uses about 42GB of RAM with llama.cpp. On Apple Silicon, that means you want at least 48GB of unified memory. On an M-series Mac with 64GB or more, it runs comfortably. On a 32GB machine, you need to drop to a lower quantization (Q4 or Q5), which costs some quality.

A few things jump out. On the MacBook, MTP doubles the throughput of the 27B model, from 18 to 32 tokens per second. That is the difference between "usable but you wait" and "comfortable for interactive coding." The 35B MoE model is faster in absolute terms, but again, the output quality trade-off makes it a worse deal for most coding tasks.

On an RTX 5090 with Q6_K quantization, someone on HN reported a consistent 50 tokens per second at 123K context, using about 28 of 32GB VRAM. That is fast. Faster than most cloud APIs when you factor in network latency.

For CPU-only setups, I would be honest: 27B is going to be slow on pure CPU. You are looking at maybe 3-8 tokens per second depending on your hardware. It is usable for batch tasks but frustrating for interactive coding. If you are CPU-bound, the 35B A3B MoE variant is the better choice because it only activates 3B parameters per token, making it much faster on low-end hardware. Note that OpenVINO optimizations only apply to Intel CPUs; AMD processors fall back to stock PyTorch with no acceleration.

Why local models matter now more than a year ago

The recent history of cloud model APIs is not comforting. Claude Fable 5 was taken down. Other frontier models run at what is almost certainly a massive subsidy; that $100/month subscription gives you thousands of dollars worth of tokens. That cannot last forever. When the subsidy ends, pricing will adjust, and the calculus for local inference changes.

There is also the privacy question. I do not want my company's codebase, my medical questions, or my personal writing sent to servers in another country. I do not care how many SOC 2 reports they have. The data leaves my machine, and I lose control of it.

And there is the fragility. Cloud APIs go down, get rate-limited, change their terms, or suddenly stop serving your use case. A model on your machine does none of those things. It is there when the internet is out. It is there when the provider decides your use case violates some policy they updated at 2am. It is yours.

Qwen 3.6 27B is the first model that makes this argument practically rather than theoretically. Previous local models were a compromise. You ran them because you could not afford the cloud, not because you wanted to. This one, you might actually choose.

Connecting it to your coding tools

Once llama-server is running on port 8080, it speaks the OpenAI-compatible API. That means most coding tools can connect to it with a config change. OpenCode, Continue, Aider, and similar tools all support custom OpenAI-compatible endpoints.

For OpenCode, the config looks like this:

{
  "provider": {
    "llama": {
      "name": "llama.cpp (local)",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "qwen3.6-27b": {
          "name": "Qwen3.6-27B Q8 +MTP"
        }
      }
    }
  },
  "model": "llama/qwen3.6-27b"
}

That is it. No API key. No billing. No rate limits. Just a local model answering requests as fast as your hardware allows.

The open model catching-up moment

I keep coming back to the Artificial Analysis score of 37 for a 27B model. A year ago, that score required a model two or three times larger, served from a data center. Now it runs on a laptop with 48GB of RAM. With GLM 5.2 also available as open weight (though requiring much more hardware), the open model ecosystem has clearly crossed a threshold.

There is a reasonable argument that the gap between open and closed models is now small enough that for many practical purposes, it does not matter. If you are doing routine coding, writing documentation, or building prototypes, Qwen 3.6 27B is good enough. Not as good as Claude Opus 5 on genuinely hard problems. But good enough for the work most people do most of the time.

I do not think cloud APIs are going away. The best proprietary models will probably always be slightly ahead. But "slightly ahead" is a very different thing from "the only option," which is where we were a year and a half ago. The fact that I can run a model on my laptop that performs comparably to mid-2025 frontier models is a shift I did not expect to happen this fast.

Try it. The instructions above will get you running in under ten minutes if you have the hardware. If you do not, the 35B A3B MoE variant works on machines with less RAM and is still surprisingly capable. Just pick 27B if you can.