Bonsai 27B: a 1-bit model that runs on your phone, and the benchmarks are real
A 27-billion-parameter model that fits on a phone. That sentence alone used to sound like a press release waiting to be debunked. You can quantize a model to 4-bit with llama.cpp and squeeze it onto a decent laptop, but a phone? A phone gives you maybe 6 GB of usable memory once the OS takes its cut, and the model has to share that with its own KV cache and activations. A 27B model at 4-bit is 18 GB. The math does not work.
Bonsai 27B, released yesterday by PrismML, is the first model I have seen that clears that bar. Not through a marketing trick. Through 1-bit and ternary quantization that actually preserves reasoning ability. The 1-bit variant is 3.9 GB. It fits on an iPhone 17 Pro. The ternary variant is 5.9 GB and targets laptops. Both are Apache 2.0, both are multimodal, and both ship with benchmarks that are specific enough to argue with.
I have been watching the 1-bit and ternary model space since Microsoft's BitNet papers. The idea has always been compelling: if you can get weights down to two or three possible values instead of 65,536, you save enormous space and the math becomes simple additions instead of multiplications. The problem was that early 1-bit models were small (under 4B parameters) and the quality drop was noticeable in anything beyond simple chat. Bonsai 27B is the first release that takes this approach to a model large enough to do real agentic work, and the benchmark numbers suggest the quality gap has narrowed to something you might actually accept.
How the compression works
There are two variants, and the difference between them matters more than the marketing lets on.
The ternary version uses weights constrained to three values: -1, 0, and +1. Each weight uses 1.71 effective bits after FP16 group-wise scaling. The total footprint is 5.9 GB. This is the one you run if you care about quality and have a laptop or a desktop GPU.
The 1-bit version goes further: weights are either -1 or +1, 1.125 effective bits per weight, 3.9 GB total. This is the phone variant. It is the one that makes the headline work.
Both variants compress the entire model end to end. Language network, embeddings, attention, MLPs, the LM head. No high-precision escape hatches where the model quietly keeps some layers in FP16 to make the benchmarks look better. The vision tower ships in 4-bit because image encoders are less sensitive to precision and the space savings matter more there. That is a pragmatic choice, not a cheat. The text and reasoning path is fully 1-bit or ternary.
There is a 262K-token context window and speculative decoding support. That speculative decoding matters more than it sounds: on low-bit models, the draft model and verifier can both be cheap, and the speedup compounds with the smaller memory footprint.
The benchmarks, which are the part that matters
PrismML evaluated both variants against the full-precision Qwen 3.6 27B baseline across 15 benchmarks in thinking mode (where the model's full reasoning chain is exercised, not just the final answer). The overall retention is 95% for ternary and 90% for 1-bit. Those averages hide some interesting per-category spread:
1-bit loses 3.6 points. For math, that is remarkably tight.
Coding takes a bigger hit. 6.8 points lost at 1-bit.
This is where 1-bit hurts most. 14 points lost. Tool calling is fragile.
Vision takes the biggest relative hit. Makes sense: the vision tower is 4-bit, not 1-bit, so it is already partially degraded vs. the text path.
Read the table by capability and the story is more useful than the averages. Math barely moves. Coding takes a modest hit that most people would accept. But agentic tool calling at 1-bit drops to 66 from 80, and vision drops 13 points. If your use case is "reason about this text and give me an answer," the 1-bit model is surprisingly close to the full-precision version. If your use case is "call this API, parse the response, decide what to do next, and also look at this screenshot," the degradation is real and you will feel it.
PrismML makes one comparison that is worth holding onto: the most aggressive conventional 4-bit build of the same base model scores lower than 1-bit Bonsai 27B while using 2.5x more memory. If that holds up under independent testing, it is a genuine Pareto improvement. You get better quality at less than half the memory cost. That is the actual headline, not "runs on a phone."
Throughput on real hardware
Desktop GPU. Fast enough for interactive agent loops.
Laptop-class Apple silicon. Unified memory makes this work cleanly.
The phone numbers are not in these tables. The demo video shows agentic workflows on an iPhone 17 Pro Max, but it is labeled "Demo Mode: Cached and Prefilled Image Context." That label is doing a lot of work. Cached and prefilled context means the heavy prefill computation was done ahead of time, and the demo is showing decode speed, not cold-start performance. Real on-device inference from a cold start will be slower, possibly much slower depending on how much of the 6 GB budget is spent on the KV cache versus activations.
I am not calling the demo dishonest. It is clearly labeled. But anyone thinking "I will run a 27B model on my phone at 80 tok/s" should adjust expectations. The phone use case that actually works today is: load the model, prefill a system prompt, and then chat. Decode-only throughput on Apple's Neural Engine with custom MLX kernels can be fast. Cold starts with a long context are where you will wait.
Why memory budget is the real constraint
The press material says "12 GB iPhone offers about 6 GB for the model to use on-device." That number is more important than it looks. iOS does not hand apps the full RAM. The OS, background services, and the app's own overhead all eat into it. The model weights are one thing, but you also need room for the KV cache (which grows with context length) and the activation tensors during forward passes.
At 3.9 GB for weights, the 1-bit variant leaves roughly 2 GB for KV cache and activations on a 6 GB budget. A 262K context window would eat far more than 2 GB of KV cache at standard precision. The speculative decoding draft model also needs space. So in practice, you are not going to use the full 262K context on a phone. You will use maybe 8K to 32K tokens depending on batch size and how the KV cache is managed. That is still useful for most conversations and document Q/A. It is not useful for feeding the model an entire codebase and asking it to reason across all files.
On a laptop with 16 or 32 GB of RAM, the ternary variant at 5.9 GB has plenty of room for a large KV cache. That is where this model is practically interesting for agentic work. A laptop that can run a 27B model with tool-calling capability, locally, with no network round-trips, for free. If the agentic benchmarks were closer to full-precision, I would call this a watershed moment for local agents. At 66/80 on tool calling, it is a promising step, not the finish line.
The architecture-agnostic claim
One thing I want to flag: PrismML says their compression methodology is "architecture-agnostic" and that larger models and new architectures are already in progress. Architecture-agnostic is a strong claim. It means the technique does not depend on specific properties of the Qwen architecture (attention patterns, MLP structure, normalization scheme). If it is truly agnostic, it should transfer to Llama-family models, Mistral, and whatever comes next.
The whitepaper is on GitHub and I would read it before betting on that claim. Architecture-agnostic compression is the kind of thing that works until you hit an architecture where the weight distribution is fundamentally different and the group-wise scaling assumptions break. Microsoft's BitNet work suggested that ternary weights work best when the model is trained from scratch with the ternaryization constraint in mind, not applied post-hoc. If Bonsai is applying this to a pretrained Qwen model without retraining, the quality retention is more impressive than if they fine-tuned. The whitepaper should clarify this. If it does not, that is the question to ask.
The thing that actually matters
PrismML frames this as an "intelligence density" play: how much reasoning ability per gigabyte. By that metric, 1-bit Bonsai 27B delivers 0.53 intelligence units per GB, roughly 10x the full-precision baseline and 2.7x the best conventional low-bit alternative. Whether that metric is useful depends on whether your bottleneck is memory or quality. For phone deployment, memory is the bottleneck. For agentic work on a laptop, quality is the bottleneck, and the ternary variant is the better trade.
What this means for local AI
The most interesting argument in the release is not about phones. It is about the economics of agentic workloads. An agent does not make one model call. It makes hundreds. Each step carries context, produces structured output, and feeds the next step. If every step is a cloud API call, the per-token cost compounds across the entire loop, and every intermediate result crosses the network with the user's private data attached.
Local execution changes that math. A hundred-step loop on a local model costs nothing per step beyond electricity. The user's files, screen, and data never leave the machine. PrismML is explicitly targeting this use case: a capable local model that handles non-frontier and privacy-sensitive tasks, with cloud models reserved for the hardest steps.
This hybrid architecture is not new as an idea. What has been missing is a model small enough to fit on consumer hardware and capable enough to trust with multi-step work. Bonsai 27B is closer to that than anything I have seen at this size. The ternary variant on a laptop with 16 GB of RAM could plausibly run a useful agent loop for document Q/A, code review, or research synthesis. The 1-bit variant on a phone is more of a "smart chatbot that works offline" than a "phone agent," given the tool-calling quality drop and the context limits.
There is also a developer preview API, which I am cautiously interested in. Free, limited-time, so you can try the model without running it locally. Useful for benchmarking against workflows before committing to the local deployment.
Where I land
The headline ("27B model on a phone") is real but it is the least interesting part of this release. The interesting part is that 1-bit and ternary quantization, which spent years looking like a research curiosity, has produced a model where the math and coding benchmarks are within single digits of full precision. The agentic and vision benchmarks show real degradation, and that is where the honest limitation sits. Anyone claiming "no quality loss" is not reading the table.
I would run the ternary variant on a laptop for text reasoning and coding questions. I would think twice before trusting the 1-bit variant with anything that requires reliable tool calling or precise image understanding. The phone use case is a technical achievement, but today it is more proof-of-concept than daily driver. The real promise is the trajectory: if the next release closes the agentic gap the way this one closed the math gap, local agents stop being a hobbyist experiment and start being a practical default.
The weights are on GitHub under Apache 2.0. The whitepaper is there too. Read the per-benchmark table before you decide which variant to use. The averages flatter the 1-bit model more than the category breakdown does.