Gemma 4 26B at reading speed on a 13-year-old Xeon, no GPU, AVX1 only
There is a server in someone's basement that has no business running a modern language model. It is a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google's Gemma 4, a 26-billion-parameter open-weights mixture-of-experts model, at about five tokens per second. Reading speed.
Anyone can rent a GPU. It is harder to take a modern MoE model and a dead enterprise box and make them meet in the middle. That gap is the whole reason this is interesting. "Good with AI" has quietly come to mean "pays for a subscription." The real skill is different. Knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is correct.
These numbers come from a writeup by the person who did it. The author is clear about one thing up front: they are not a C++ programmer. They can read a stack trace and navigate a build system, but they did not hand-write kernel fallbacks for a quantized matmul engine. What they did was drive. They ran the experiments, read the output, asked the next question, and knew what "correct" had to look like. The diagnosis and the patch came from a Claude instance running on the server itself.
The post that started it
A couple of weeks before, a piece called "A 10 year old Xeon is all you need" circulated on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik_llama.cpp and about 25 carefully chosen flags. It leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering.
"I have a Xeon too," the author of this new piece thought. Several, in fact. So they tried it. It didn't run.
What an AI agent turned out to be good for
The build died on startup. They handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The 2016 chip in the original post is a Broadwell part. The Ivy Bridge chips getting repurposed here are the generation Intel calls "v2." The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn't ship until Haswell, the "v3" generation, in 2014. The CPUs are older than the instructions the code was written against. The optimized paths weren't there to execute.
So the obvious follow-up: can we make it run anyway? They had already taken a first swing with a free model that got close but couldn't land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off. Reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren't there.
This is the part worth pausing on. This didn't come from typing "fix it" once and getting a working patch back. Somebody had to read another person's performance-critical C++, work out why a kernel wasn't valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. The human's job was narrower: run the right experiments and recognize when the output was finally correct.
The result
Gemma 4's 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model's architecture existed. The original writeup never published a tokens-per-second figure, just "reading speed." The concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free.
The patch is up as ikawrakow/ik_llama.cpp#2138, still open and awaiting maintainer review at the time of writing, so the recommendation is to run it from the branch for now. The hope is that anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn't make sense.
For the people who want the actual bug
Full disclosure from the author: they did not hand-write kernel fallbacks for a quantized matmul engine, and they won't pretend they did. They drove. The diagnosis and the patch came from the Claude instance running on the server itself. What follows is that summary, lightly edited.
What was actually broken
The engine needed was ik_llama.cpp, ikawrakow's fork of llama.cpp that adds the optimizations Gemma 4's MoE inference depends on. It assumes AVX2 as its floor. The Xeon E5-2690 v2 has AVX1 but not AVX2. Turn GGML_USE_IQK_MULMAT off at build time and most of the codebase respects it: the fast paths compile out, and the model falls back to plain scalar/SSE math. That's fine for a normal Q8_0 matmul.
Two graph ops are the exception. The Gemma 4 MoE feed-forward network emits MOE_FUSED_UP_GATE (a per-expert gate+up matmul fused with SwiGLU) and FUSED_UP_GATE (its dense analog). Both are #if-gated on GGML_USE_IQK_MULMAT inside the compute dispatcher, but the graph builder still emits them unconditionally. On this build the dispatcher's switch had no case for those op enums, so they fell through to the default, and the destination tensors for every expert FFN silently never got computed. Gemma 4 26B has 30 layers by 8 active experts per token, so every forward pass consumed roughly 240 tensors of whatever happened to be sitting in that memory buffer already.
The symptom was fluent-looking multilingual gibberish. Token IDs spread uniformly across the 262K vocabulary, the model equally happy to emit Thai script, Korean,
That determinism is what cracked it.
Instrumenting the raw logits before sampling, printing the top-5 tokens plus range, mean, and NaN count, the numbers gave it away: a mean logit of +16 for the first predicted token when it should sit near zero, and about 80% of the vocabulary at positive logits. Random corruption doesn't look like that. A bias that clean only happens when a big chunk of the hidden state is uninitialized memory that happens to hold small positive floats.
Reading the code kept clearing the obvious suspects. The RMSNorm helpers looked correct. The AVX1 fallback in ggml_vec_dot_q8_0_q8_0 looked correct. A bit-identical single-thread run ruled out threading. Only after instrumenting the logits, and seeing the mean pinned at +16 with every long-tail token roughly tied, did the search narrow to "a big chunk of the residual stream is uninitialized." Grepping for #if GGML_USE_IQK_MULMAT in the dispatcher turned up the two missing cases about a minute later.
The fix
Three commits on top of the fork's main.
First, compile fixes. The scalar #else branches for quantize_row_q8_0_x4 and quantize_row_q8_1_x4_T in iqk_quantize.cpp weren't actually scalar. They still referenced hsum_i32_8 and other AVX2 helpers. Those got rewritten as portable scalar loops, with #if GGML_USE_IQK_MULMAT guards added around a handful of stray IQK calls leaking through ggml.c and ggml-quants.c, plus a missing include so iqk_cpu_ops.cpp compiles standalone. Without these, the fork won't build at all on non-AVX2 hardware.
Second, the runtime bug. Rather than touch the dispatcher, the fix makes the graph builder emit ops that do have compute paths on this build. In ggml_moe_up_gate, when GGML_USE_IQK_MULMAT is off: if the weight is the combined up_gate_exps tensor (shape [n_embd, 2*n_ff, n_experts], gate in the first half, up in the second), split it into two ggml_view_3d slices, run two separate ggml_mul_mat_id calls, and combine them with ggml_fused_mul_unary(gate, up, SILU). If gate and up are already separate weights, skip the split and do the same two mul-mat-IDs plus the fused mul-unary. The dense version used in non-MoE layers gets the same treatment. Every op involved already has a working non-IQK implementation. The whole change sits behind #if !GGML_USE_IQK_MULMAT, so an AVX2 build stays bit-identical to what it was before.
Third, CI stubs. The #else stub sections of the iqk sources had drifted out of sync with iqk_mul_mat.h, so ci/run.sh couldn't even build on non-AVX2 hardware: a missing sinks there), and no stubs at all for a couple of functions, which meant undefined references at link time. Boring work, but without it nobody on this hardware can run the test suite.
The fallback costs something. Two separate matmul-IDs instead of one fused kernel. But this CPU is memory-bandwidth-bound anyway, and the fused kernel was AVX2-only, so nothing real was given up. End to end: about 5.2 tok/s decode and ~16 tok/s prompt-eval on a 26B-A4B MoE.
One more gotcha. --run-time-repack reorders quantized weights into an AVX2-only interleaved layout (Q8_0_R8) at startup, which garbles output on AVX1 the same way. That's a separate bug, and the patch doesn't try to fix it. The run script just drops the flag.
Reproduce it
If you have a pre-AVX2 box and want to try this, the recipe from the writeup:
- Hardware: dual Xeon E5-2690 v2 (Ivy Bridge, AVX1, no AVX2), DDR3, no GPU.
- Build: compile ik_llama.cpp from the PR branch with
GGML_USE_IQK_MULMAToff. The compile fixes are what let it build at all on non-AVX2 hardware. - Model: Gemma 4 26B-A4B,
Q8_0. - Run: use the usual ik_llama.cpp CPU flags, but drop
--run-time-repack(it reorders weights into an AVX2-only layout and re-garbles the output on AVX1).
The whole recipe: about 5 tok/s decode, CPU-only, no GPU anywhere in the box.
Why this is the interesting part
The subscription is the easy part. The rest is the willingness to open the hood, read a stranger's code, and keep asking until a thirteen-year-old CPU does something it was never meant to. That's the same work a fifteen-year-old Rails app needs, or a database nobody left on the team still understands: someone who'll dig until they find where the leverage is, and what the tool won't tell you on its own.
The bug is instructive for a reason that has nothing to do with AVX. The dispatcher's switch had no case for the op enums, and instead of throwing or logging, it fell through to the default. The destination tensors silently never got computed. The model didn't crash. It didn't produce NaNs. It produced fluent, confident, multilingual gibberish. That is the worst kind of failure: the kind that looks like success.
If you have pre-AVX2 iron gathering dust and you try the branch, the PR thread is the right place for bug reports. The honest question the writeup ends on is how far down the CPU generations this goes. How old can the silicon get before AVX1 itself isn't enough? That's the kind of answer you only get from people who try it on hardware the rest of us threw out.
The source writeup is at neomindlabs.com. The patch is at ik_llama.cpp PR #2138.