Needle 2 fits an entire agentic LLM into 14 megabytes
Most "small model" news this year has meant something like "only 3 billion parameters, fits on one GPU." That is small relative to the 400B monsters, sure, but it still assumes real compute. A release called Needle 2 from Cactus Compute, posted to Hacker News this week and trending at around 290 points, wants a different definition of small. The whole model is 45 million parameters. The entire artifact is one 14 megabyte binary. It runs a full session in 28 megabytes of RAM.
And it is not a toy. It is an agentic tool-calling model already running in production in a shipping consumer product, the Pebble Index smart ring. I keep wanting to treat claims like that with suspicion, and then I read the benchmark page.
What Needle 2 actually is
Needle 2 is an open source, Apache 2.0-licensed model built for one job: taking a spoken or typed instruction and mapping it onto function calls on a device. Turn on a light. Set the thermostat. Call the weather API. The pitch is that turning on a light does not need a frontier model. The function is right there, typed parameters and all. The only hard part is figuring out which function the user meant and filling in the arguments from their messy sentence. That is a much narrower problem than general chat, and it is the reason 45 million parameters can do it at all.
It does three things. Tool calling: you declare your functions with Python type hints and docstrings, the model picks the right one and fills the arguments. Structured extraction: you pass a Pydantic schema and get typed data back from a paragraph of text. And confidence gating: every response carries a score from a learned head, so the device can act on high-confidence answers and punt the rest to the cloud. The "empty call" is how Needle says no. If the request does not map cleanly to an available tool, it returns nothing, not a confident hallucination.
The numbers
These are from Cactus Compute's announcement page and the repo README. I am repeating them because the scale of them is the story.
For context: 500 tokens per second on a Raspberry Pi 5 is fast. Faster than you can read. Faster than the UI can keep up with streaming. On a $150 phone it is still 300 to 700 tokens per second. The bottleneck for a voice command is not going to be the model.
How 45 million parameters trades blows with bigger models
The headline comparison is against FunctionGemma 270M (270 million parameters), LiquidAI's LFM2.5 230M (230 million), and Apple FM (around 3 billion). Needle is 5 to 70 times smaller than those. It also runs at 2 bits of weights while the baselines stay at f16. Cactus explicitly notes both asymmetries: the baselines are deliberately left at full precision because naive post-training quantization to 2 bits "collapses models that were never trained for aggressive compression," and Needle is trained for exactly that from the ground up. The scope is also different. The baselines are general language models carrying chat and prose alongside their tool calling. Needle is trained for tool calling and nothing else. Neither asymmetry is hidden. The point is narrow: which model executes tool calls correctly within an on-device budget.
The reason Needle trades wins and losses becomes visible in the detail. On Mobile Actions it loses to LFM2.5 by about 5 points overall, but it has the highest "name accuracy" of any model. 98.3% versus 93% for LFM2.5. It knows which function to call. Where it loses is filling in argument values, which makes sense for a model with a fifth the parameters. On Seal-Tools out-of-domain, where entire tool categories are held out at training time, Needle wins outright. 28.7% versus 17% for LFM2.5. That tells me the architecture benefits from generalizing across schemas it has not seen, even if the raw knowledge budget is tiny.
The BFCL v4 general-purpose benchmark is where it struggles. On Java and JavaScript function calls it falls behind both baselines, and the writing acknowledges that: Needle's training corpus is consumer device actions plus some general API surfaces, and the Java/JS SDK categories "sit entirely outside that distribution." I respect that kind of upfront honesty. The numbers are not flattering and they are published anyway.
Why the size is not the most interesting part
The 14MB number is what gets attention. What I kept thinking about while reading the architecture section is the energy argument underneath it. On device silicon, moving a byte from flash or DRAM costs orders of magnitude more than a multiply-accumulate. So the budget that matters is both FLOPs per token and bytes per token, not just one. Cactus publishes a compute-per-token table that makes this concrete.
Needle spends 70 MFLOPs per decoded token. A same-shape dense transformer of the same width and depth spends 164. LFM2.5 spends 460. FunctionGemma 540. The difference comes from two places. The Hadamard MLP replaces the usual dense up-projection and down-projection with a fixed Walsh-Hadamard transform and learned diagonals, so the channel mixing that usually dominates a small model's weight reads costs almost no parameters. And the "engram" memory moves world knowledge into hashed n-gram tables that are read a few rows per token, so 8 million of the 45 million parameters cost no matmul arithmetic at all. They are read by gather, not by multiply.
The engine side is just as deliberate. Weights never decompress into RAM. The 2-bit codes get expanded inside vector registers, fused into integer dot products, so resident memory stays at blob size and the whole arithmetic path is int8 end to end, including activations, KV cache, and the routing tables. The grammar decoder is also a speedup. Because the matcher knows which tokens are legal before the logits exist, the engine computes output scores only for candidate rows, skipping up to 98% of the vocabulary projection on structural tokens. One universal binary probes the CPU at startup and picks its kernel tier, which includes SDOT, NEON, AVX2, RISC-V vectors, WASM SIMD, or scalar. The thread pool spins through short serial sections instead of sleeping, which the writeup says alone nearly doubled decode speed.
I genuinely do not know how to feel about the fact that "the grammar is an optimization, not just a guarantee" is an efficient design choice. It is clever, it checks out, and it is also exactly the kind of coupling that makes future debugging painful if the grammar and the logits ever disagree. Time will tell.
Production already
This is the part I find hardest to argue with. Pebble, the company behind the Index 01 smart ring, runs Needle locally in their companion app. The ring has no screen. You speak to it, and something needs to turn that into an action immediately, without depending on a network connection, because the ring is worn 24 hours a day and the user is not going to tolerate a "connecting..." spinner every time they say something. The Pebble quote in the announcement is blunt about it: "The model's footprint is tiny and the performance never lets us down."
That is a real constraint that a 14MB model solves well and a 3B model solves badly. Most of the industry is racing for more general intelligence, and a release like this is a bet that there is a whole layer of device control where general intelligence is overkill. The contested part is how wide the layer is. I think it is wider than people assume, specifically because the privacy and latency wins are structural for a watch, a hearing aid, a doorbell, or anything worn on a body. But it is not unlimited, and the HN thread picks at exactly that line.
What people on Hacker News actually noticed
The thread hit 106 comments and the things people tested are the right things to test. Someone tried "make it warmer in here" and got back a thermostat call set to 65 degrees in cool mode, with reasoning that read: "'warmer' implies need for cooling. set thermostat with temperature 65, typical warmth, and mode cool." That is wrong. 65 is not warm and "warmer" did not mean cool. It is the kind of mistake a 45M model is going to make because it has seen a lot of "thermostat" plus "mode" examples but not enough "warmer means heat" signal to disambiguate consistently. The author would probably say fine-tune on your device's actual calls, and they are probably right, but the web demo is the first impression and it does fail in ways a general model would not.
Another person tried a simple math tool, add(a, b), and asked "calculate 1+1." Needle answered "No calculator or math tool available." They declared one tool and it did not match it to a prompt phrased as a question. The commenter's verdict was "completely useless." That is harsh but also, again, a real signal. Tool-calling success is extremely dependent on how well you describe the tool, and the model has very little room to absorb ambiguity.
The thread I found most illuminating, though, is someone pointing out the asymmetry between "intelligence" and "form." Most of the field is chasing the frontier of function. A release like this is chasing the frontier of form, how small a useful artifact can get. Their argument is that both are needed if you ever want Opus-level capability running locally on commodity hardware. I think that is right, and also a useful way to frame the tradeoff. A 14MB binary that trades wins with a 270M model is not threatening frontier labs. But the existence proof, that a 2-bit model can do real work and fit on a microcontroller, is what makes the next round of optimization possible.
Where the tradeoffs land
I want to be careful here because small model releases have a tendency to get oversold. The constraints are real.
The KV window is 256 tokens. Tools and system prompt get pinned as permanent sinks inside that window, so the user gets the remaining context. That is fine for "turn on the bedroom light" and a short follow-up or two. It is not fine for a long conversation with memory of earlier context. The design is explicitly fixed-RAM, and that choice buys you the microcontroller target. You do not get both.
The training corpus is consumer device actions plus BFCL general API surfaces. That means smart home, mobile, wearables, TV, automotive, and extraction tasks. It means you should not expect it to understand a Python data analysis tool, or a complex code search API, without fine-tuning. The fine-tune story is nice on paper, because the model is tiny enough to retrain on a laptop in minutes to hours, and the Python package supports LoRA. Whether "fine-tune on your Mac" is enough in practice for a production product is a separate question, and one the Pebble deployment partially answers.
The 2-bit precision works because it is baked into training from the start. This is not a tool you can take a HuggingFace f16 model of your own and quantize down to match. You either use Needle as is, or you spend comparable effort retraining. That is a wider caveat than "this model is Apache 2.0 so you are free to modify it" implies.
And the web demo is there to show off speed, not judgment. People in the thread noted that it tends to call a tool even on prompts where a general assistant would say "I do not have a tool for that." The empty-call mechanism is supposed to handle this, and sometimes it does, but it is not consistent in the demo. If you are building on Needle you want to test on your own tool catalog, not the canned one.
Why it matters more than the size
The headline is 14 megabytes. The thing I keep returning to is the energy argument. Most edge hardware runs on batteries. Every MFLOP is milliwatt-hours. A model that spends 7 to 85 times fewer of them per token than the models it is benchmarked against is not just faster, it is what makes "always on" possible at all. The Pi 5 number is for enthusiasts. The Samsung A series number, 300 to 700 tokens per second on a sub-$200 phone, is for the actually-large market.
Cactus cites the rough breakdown: about 21 billion connected IoT devices against roughly 1.5 billion PCs. Add budget phones, Raspberry Pis, microcontrollers, wearables, small robots, connected home devices, and about four in five edge devices cost under $200. That is the hardware Needle targets. No GPU, no NPU, a few hundred megabytes of RAM at the high end and 32MB of external SRAM at the low.
The repo is at github.com/cactus-compute/needle. The announcement with the full benchmark tables and the architecture diagrams is at cactuscompute.com/needle. Both are worth reading. The README is unusually honest about what the model does and does not do, and the announcement goes deep on the architecture in a way most model writeups do not bother to.
This is not the next frontier model. It is a bet that a lot of the world's compute is going to happen on devices that are too cheap, too battery-constrained, and too privacy-sensitive to ever call a cloud model. Whether the bet pays off is mostly about whether device makers believe a 14MB binary is enough, and the Pebble shipment is at least one piece of evidence that it is.