Reame: the CPU-first LLM inference server that gets faster the longer it runs

Ollama is the default answer when someone asks how to run a local model. It is good at what it does: one command, a model downloads, and you are chatting. The assumption baked into that experience is that every request is brand new. You compute, you discard, you compute again.

That assumption is fine on a GPU, where compute is cheap. On a $5 VPS or a free-tier ARM box, compute is the most expensive thing you have, and throwing it away on identical work is a problem. Reame is a new open source LLM inference server, MIT licensed, built on llama.cpp, that starts from the opposite assumption. On a CPU, never compute the same thing twice.

The result is an inference server with a property I have not seen in any other project: it gets faster the longer it runs. The hundredth request costs a fraction of the first. The system prompt is paid for once by the first user. Similar answers draft themselves from an on-disk archive. Formatting tokens are speculated for free. None of this exists in Ollama, and none of it is hypothetical. The project shipped this week on Hacker News and the benchmarks are in the README.

What it actually is

Reame is a single-binary LLM server built on llama.cpp for all tensor kernels. CPU-only. One model per process. It exposes an OpenAI-compatible REST API so you can point any existing client at it. /v1/completions, /v1/chat/completions, SSE streaming, bearer auth, and a /metrics endpoint with cache and speculation stats. The zero-config path is as easy as Ollama: reame run qwen2.5-1.5b downloads the model, autoconfigures threads and cache, and drops you into a chat.

Where it diverges is in what happens after the first request. Six mechanisms stack on top of llama.cpp to stop the CPU from doing redundant work. The design is worth understanding before you look at the numbers, because the numbers only make sense once you see what is being cached where.

The persistent disk KV cache

This is the idea that ties everything else together. LLM inference has two phases: prefill, where the model processes the entire prompt and builds the KV cache, and decode, where it generates tokens one at a time. Prefill is expensive. On a CPU it can dominate the wall time for short generations.

GPU inference servers like vLLM keep prefix caches in GPU memory. If two requests share a system prompt, the second one reuses the first one's KV state. But that cache dies when the process restarts. Reame snapshots the KV cache to NVMe at fixed token-block boundaries, keyed by a chain hash of the prompt prefix. A different request that shares a prefix restores the longest matching boundary and decodes only its own tail. The cache survives restarts. It survives across processes. It is LRU-budgeted so it does not eat your disk.

The practical effect: a system prompt of 2,000 tokens is computed once. Every subsequent request that uses the same system prompt skips the prefill for those 2,000 tokens. On a free-tier ARM box running Qwen2.5-7B at 3.3 tokens per second, that is the difference between your server feeling responsive and feeling like it is thinking each time someone pings it.

Palimpsest: the server remembers what it generated

This one caught me off guard. Every completed generation gets fed into an on-disk n-gram archive. Future requests are drafted from that archive at zero cost. The project calls this Palimpsest, which is a layering metaphor I am not going to explain, but the mechanic is straightforward: if your workload involves asking variations of similar questions, the server builds up a statistical model of what it has already said and proposes those tokens as speculative draft candidates.

The README reports a 2.3x speedup on repeated requests on an Apple M3 Pro with Qwen2.5-1.5B, going from 22 tokens per second to 51. That is not a synthetic benchmark designed to look good. That is a server that has been running the same kind of workload for a while and has learned the shape of its own output.

Il Suggeritore: grammar as a draft source

Constrained decoding is a well-known technique where you forbid the model from producing tokens that violate a grammar. Reame inverts this. It uses the grammar to propose tokens. List numbering, bullet markers, JSON keys with their colons and quotes, formatting tokens. These are predictable from structure alone, and on content nobody has ever generated before, the server can speculate them correctly for free.

The measured effect on the M3 Pro is a 2.1x speedup on fresh list generation tasks, from 4.4 seconds to 2.1 seconds. The interesting part is that this works on novel content. Traditional speculative decoding needs a draft model that has seen something similar. Il Suggeritore just needs the structure to be predictable, which in practice it almost always is, because format is more repetitive than substance.

Self-regulating speculative decoding

Speculative decoding, introduced by Leviathan et al. in 2023, uses a small draft model to propose tokens that the target model verifies in a single batched pass. When the draft model guesses right, you skip decode steps. When it guesses wrong, you resample from the residual distribution so the output is exactly the target model's distribution.

Reame adds two CPU-specific twists. First, the draft source can be free n-gram lookup mined from the prompt itself, which means you do not need a second model in memory for extraction and rewrite workloads. Second, and this is the part I respect, it has a feedback controller that measures whether speculation is actually paying off on your hardware. If the draft model is running as slowly as the target because the vCPUs are oversubscribed, Reame turns speculation off. Automatically. The README reports a 3.2x speedup with speculative decoding on a shared Contabo VPS with an 87% acceptance rate. But it also reports that on heavily oversubscribed vCPUs, speculation is counterproductive, and Reame detects this and disables it at runtime.

That kind of honest negative result in a project README is rare. Most inference servers will happily run speculation at 30% acceptance and tell you it works. Reame measured the break-even point and built a controller around it.

Interleaved multi-user serving

On a GPU, you serve multiple users by batching their requests together so the model weights are read once per batch. On a CPU, memory bandwidth dominates, so the same principle applies but the gains are smaller. Reame interleaves N concurrent generations inside a single multi-sequence batch so every read of the model weights is shared. The README reports a 1.6x throughput improvement on an M3 Pro with 3 concurrent users compared to serialized serving.

1.6x is modest by GPU batch standards, but on a free-tier ARM box where you have 2 cores and 12 GB of RAM, every read you can share is one you do not have to pay for twice.

The Conclave: consensus from the model you already have

This is the most creative feature in Reame. --best-of N generates N candidate answers to the same prompt in one interleaved batch. The scheduler notices the identical prompts and clones the prompt KV instead of prefilling N times. Every candidate shares the same weight reads during decode. Election is an exact-majority vote on each candidate's final result, with a Jaccard text-medoid fallback for prose. The moment a majority agrees, the stragglers are stopped mid-generation.

The measured effect on a Qwen2.5-1.5B with 5 candidates on 3 arithmetic quizzes: plus 0.5 to plus 2 correct answers, at roughly 2.5x wall time, not 5x. The README is explicit about what this does and does not do. Majority voting corrects random slips. It does not correct systematic misunderstanding. A 1.5B model running 5 candidates lands between the 1.5B and a 3B. It never lands above the 3B. Consensus fixes variance, not bias.

The framing in the README is the sentence I want every benchmark table to include: "Benchmarks that only show wins are advertising; these are engineering."

The benchmarks

Every number below is from the Reame README, produced by the shipped binary on the hardware named. The project lists negative results alongside positive ones, which is how you know the positive ones are probably honest.

The OLMoE result is worth pausing on. OLMoE 7B-A1B is a mixture-of-experts model with 7B total parameters and about 1B active per token. On the same 8-needle long-context extraction test, it scored 100% accuracy at 17.8 tok/s. A dense 7B scored 100% at 3.3 tok/s. That is a 5.4x speedup from the same accuracy, on the same free hardware, just from choosing a sparse model that fits the workload. Reame is the first server I have seen that explicitly calls this out and recommends it.

What Reame is not for

The README is unusually clear about this, and I think the honesty is the point. Reame is not for general-purpose chat, agentic coding, or creative long-form writing. If your task needs a 100B-class model, the recommendation is to buy one. Reame is for narrow, repetitive workloads over your own data on hardware you already pay for: document extraction, batch pipelines, privacy-bound processing, private code autocomplete on the laptop you already own.

That framing matters because it tells you something about the target user. This is not a tool for someone who wants to run a frontier model. This is a tool for someone with a 5 euro VPS, a narrow workload, and a desire to stop paying per-token API bills for work that is structurally repetitive. If you have 10,000 product descriptions to tag overnight, Reame will draft most of them from its own archive and charge you 0 in API costs to do it.

Why Reame and not Ollama

The one-line version from the README is good enough to quote directly: "Ollama runs models. Reame remembers having run them."

Ollama optimizes for running many models casually. Reame optimizes for serving one workload seriously on hardware that costs nothing. On a fresh request with no shared prefix, they are in the same neighborhood. On the hundredth request with the same system prompt and similar content, Reame pulls ahead because it has been building memory the whole time. The disk cache, the archive, the form speculation, the self-regulating speculative pipeline. None of it exists in Ollama.

This is not a knock on Ollama. Ollama is the right tool for the person who wants to try out a model this afternoon. Reame is the right tool for the person who has been running the same extraction pipeline for a month and wants the hundredth job to be cheaper than the first. Different tools for different shapes of work.

The energy angle

The README has a section on energy that I want to flag because it is interesting and slightly evasive. Reame's footprint is watt-scale, not kilowatt-scale. It targets machines that already exist and are already powered on. No new silicon is racked to serve your model. The claim is not that Reame gets better joules-per-token than a saturated datacenter GPU. The claim is that you do not need one.

That is a fair claim. The question it does not answer is whether running inference on a 2-core ARM box at 3.3 tok/s is more or less energy-efficient overall than batching the same request on a GPU that handles thousands of requests simultaneously. On a pure joules-per-token basis, the GPU probably wins at scale. But the GPU requires a data center, a network link, and someone else's infrastructure. Reame requires a box you already have. The right comparison is not joules per token. It is total cost of ownership per workload, and on that axis, a free-tier ARM box beats anything with a recurring bill.

Getting started

The fastest path is Homebrew on macOS or Linux:

brew tap swellweb/reame
brew install reame
reame run qwen2.5-1.5b

That downloads the model, autoconfigures threads and cache, and drops into a chat. For serving, add --serve and you get the OpenAI-compatible API on port 8080. Prebuilt binaries are available for Linux x64, Linux arm64, and macOS arm64. An npm package (npx reame) is planned.

The build from source path is standard CMake. Clone the repo, init the llama.cpp submodule, run ./build.sh. The test suite has 210 isolated cases and every layer is mockable without a model, which is more test coverage than I usually see in hobby inference servers.

OpenVINO note for Intel users

One thing the README does not mention: Reame is built on llama.cpp, which means it does not use OpenVINO. If you are on an Intel CPU, OpenVINO can give you meaningful speedups for some models, and that path is not available here. AMD CPUs get the same treatment as Intel with no special optimization. If your workload is on Intel hardware and raw throughput matters more than the caching features, OpenVINO-backed servers are worth comparing. But the caching and memory features in Reame are the actual differentiator, and OpenVINO does not give you those.

Where I land

I like this project. Not because it is the fastest inference server, because it is not. I like it because it has a thesis about CPU inference that nobody else has committed to: that the right move on cheap hardware is to stop recomputing things you have already computed, and that a server should accumulate state across requests the way a database accumulates indexes.

The Conclave is a genuinely new idea. Auto-regulating speculative decoding that measures itself and turns off when it is not helping is the kind of engineering detail that separates a project that was built for a specific workload from one that was built to look good in a release post. The negative results in the benchmark table are the strongest signal of all. A project that tells you when its own features do not work, and on which hardware, is a project you can actually trust when it tells you they do.

If you have a narrow, repetitive workload and a spare box, Reame is worth a week of your time. Not because it will replace your frontier model, but because it might replace the API bill you are paying for work that turns out to be structurally predictable once you stop throwing away every computation after you use it.

The repo is at github.com/swellweb/reame. MIT licensed. The README is long and honest and worth reading in full before you install anything.