GLM 5.2 on AMD MI355X hits 2626 tok/s and Nvidia's software moat is leaking

Inference demand is outrunning supply and Blackwell prices keep climbing. So a small team at Wafer did the obvious-but-hard thing: they got GLM 5.2 running on AMD Instinct MI355X, hit 2626 tokens per second per node at saturation, and got there at over 2x lower cost per GPU than B300. The Hacker News thread on it sat at 114 points with engineers arguing over whether this counts as "SOTA" or "almost SOTA for cheap." Both are kind of true, which is the interesting part.

The headline numbers, from Wafer's post on July 3, 2026:

• 2626 tok/s/node aggregate throughput at 2.4 rps, on a 20k in / 1k out workload with 60% cache hit rate.
• 213 tok/s single stream decode on the Artificial Analysis workload shape (10k in / 1.5k out).
• ~2.75x cheaper per GPU than a B300, with "comparable hardware specs."
• 80% of the throughput measured on a B200 (3192 tok/s/node at 3.0 rps). Not faster than Blackwell. Cheaper per unit of work.

"Comparable hardware specs" is doing a lot of work in that sentence. AMD's MI350 series genuinely competes at the silicon level. The actual problem is everything above the silicon.

The boring friction that ate weeks

Wafer quantized GLM 5.2 from bf16 to MXFP4 using AMD Quark. They compared it against z-ai's official FP8 quantization on three evals: GSM8K, GPQA-Diamond, and tau2 macro. MXFP4 came out essentially lossless. GSM8K dropped 1.0 point, GPQA-Diamond dropped 1.9 points, tau2 actually went up 1.5 points. If you have been around quantization long enough you know that "loses less than 2 points and sometimes gains" is a win.

Then they had to pick a framework. Three options: vLLM, ATOM, sglang. vLLM had no working MXFP4 plus GlmMoeDsa path, so MXFP4 weights did not actually help. ATOM's output "degraded at long context." sglang was the only one that could take advantage of the quantization while staying coherent. They went with sglang. The version of sglang they needed, in theory the most mature option on ROCm, still needed two patches before speculative decode worked.

I am going to quote the post directly here because the second bug is funny in a "of course it is" way:

The fused multi-step metadata kernel needed for draft depth ≥4 writes #include with no ROCm guard. Fix: one #ifdef USE_ROCM guard.

One preprocessor directive. That was the difference between speculative decode at depth 5 (the config z-ai recommends) and not. The other patch was a layer-name mismatch in the MTP head that made the loader try to stuff a bf16 weight into a 4-bit slot and crash on a shape error. Two trivial changes for what the post describes as "close to a 3x gain in single stream throughput." When people say "the AMD software stack is rough," this is what it actually looks like on the ground. Not architectural problems. Preprocessor guard omissions.

Quant table, eased

The eval comparison from Wafer's post, for people who like seeing the actual numbers instead of being told they are fine:

You can argue about whether 4-bit should be called "lossless." For the three evals people actually report, it basically is. A 1.9-point drop on GPQA-Diamond is inside the run-to-run noise for most teams. I have shipped worse. I would not call this lossless in a paper, but for inference serving it is the right call.

Why the prefill number was bad until someone tuned a kernel

At TP8 (the configuration tuned for single stream decode), MI355X could only hit 1461 tok/s/node on the aggregate throughput workload. They switched to TP4xDual and got to 1944 tok/s at 2.0 RPS. Better but slow. The reference B200 was already at 3192.

The fix was MoE kernel tuning. GLM-5.2's fp4 MoE was silently running on a slow FlyDSL heuristic fallback because aiter (AIT Enhancement Routines, AMD's tuned kernel library) only shipped tuned configs for the a8w8/fp8 path. Nothing in the image told anyone this was happening. Wafer tuned the kernel selection themselves on GLM's fp4 shapes: model_dim 6144, moe_inter 2048, E=256, topk=8. That got them from 1944 to 2626 tok/s/node.

Read that again. The diff between "MI355X is a bit slow" and "MI355X is competitive with B200 on perf per dollar" was one team tuning MoE kernel selection for one model's fp4 shapes. The tuned configs were never shipped in the AMD image. They are now in Wafer's repo, not AMD's.

This is why people have historically given up on AMD for inference. Not because the silicon is bad. Because the people who tune the kernels have other priorities, and the people who need tuned kernels usually have Nvidia dollars to spend. The gap is not about FLOPS. It is about whether someone bothered to write the kernel config you need.

What this changes, and what it does not

Wafer's own framing is the right one: "SOTA on AMD is becoming more a matter of support, not software." They did not write any custom kernels this time. Their earlier Qwen3.5 397B work required custom kernels. This run was patches and kernel config tuning. The fact that the same team can do both, and that the second run needed far less exotic work than the first, is the actual signal.

The HN comments had two camps. One said "80% of Blackwell is not winning." The other said "2x cheaper per GPU means you can buy three MI355X nodes for the price of two B300 nodes and still come out ahead on perf per dollar." Both are correct, which is exactly the situation where I would buy the cheaper hardware if my workload fit the cache profile and TTFT envelope Wafer measured.

The narrow caveat: the 2626 tok/s number is specific to a 20k in / 1k out workload at 60% cache hit. Change the workload and the picture changes. Prefill-heavy jobs with low cache hit rate will look worse. Single stream decode at 213 tok/s is good but not top of the Artificial Analysis leaderboard. They win on perf/dollar, not raw perf. Read the workload before quoting the number.

The broader caveat: single-node. The post says this study does not take multi-node into account. Single-node deployments are still common in practice, but at production scale you eventually need multi-node, and that is where the AMD networking story still has more rough edges than Blackwell's NVLink setup.

The CUDA moat, status update

For roughly a decade the line you would hear is "amd has fine hardware, the software is the problem." That is now an inaccurate sentence. AMD's ROCm has working paths for frontier models. The roughness is in shipping the right defaults, not in capability. Two preprocessor guards and an unshipped MoE config should not be the things standing between you and 3x single-stream speedup. But they were, this time, and a small inference team fixed them in a week.

The CUDA moat is still real. It is just shallower than it was a year ago, and it is leaking through preprocessor guards. Nvidia's actual durable advantage at this point is that the defaults ship tuned. AMD's advantage is that the hardware is 2.75x cheaper per GPU. If you have kernel engineers and your workload is cache-friendly, the math has started to favor AMD for frontier open models. If you do not have kernel engineers, you should still buy Nvidia and not feel bad about it.

For CPU-only readers doing the math at home: this is GPU. None of it applies to you. OpenVINO does not run on AMD GPUs (Intel-only), PyTorch on ROCm is the AMD path, and AMD's CPU inference story is not the same conversation. The interesting open question for the CPU crowd is whether kernel engineers start porting the same kind of MoE tuning work to consumer x86 and Arm, because the same "default config silently slower than tuned" problem lives there too.

I am going to keep watching this. If a year ago I would have told you it takes a custom kernel team to make AMD competitive on a frontier model, and now it takes half a week of patching, the question is what next year looks like. Probably defaults that ship tuned. Probably a smaller gap. Probably still Nvidia outselling AMD on developer trust. But the dollar gap, at the inference margin, is closing.