Databricks benchmarked coding agents on their own codebase

Every coding agent vendor has a benchmark and every benchmark says their model won. So when Databricks posted results from a benchmark they built on their own merged pull requests, across a real multi-million line codebase in 10+ languages, I expected another marketing slide. It is not that.

The benchmark is honest in a way that almost nothing in this category is. They used real tasks their engineers actually performed, graded by the tests their engineers actually wrote, and they manually inspected the agent traces when numbers looked too good. The headline is not "we won." The headline is that the frontier is now a mix of vendors, token price is a lie, and the harness you call the model from quietly determines whether you overpay.

What they actually benchmarked

Databricks built the benchmark from their own merged PRs. They pulled recent ones, filtered out bot commits and AI-generated work, kept only PRs with real accompanying test suites, and picked changes that were self-contained and spanned the stack: Scala backend services, Rust systems code, a React and TypeScript frontend, protobuf and gRPC contracts, Bazel config. The resulting task is the PR description rewritten as a goal with the solution stripped out, plus the test files held back for grading.

They ran each model and harness combination with its standard, out-of-the-box setup and the same tools a Databricks engineer would have. When the agent said it was done, they checkpointed the code, patched the held-out tests back in, and ran them. Pass or fail. They did not use an LLM judge for correctness, and they say outright why: it rewards sounding right over being right.

That single sentence does more for credibility than most of the public evaluating literature combined.

The result nobody is going to bury

GLM 5.2, the same model this blog covered beating Claude on cybersecurity benchmarks last month, landed in the top capability tier on the Databricks workload. Statistically tied with Claude Opus 4.8 on quality. GLM cost about $1.28 per task. Opus cost about $1.94 per task.

I want to be careful with that sentence because it is exactly the kind of claim that gets repeated until it loses its conditions. The Databricks number is for their tasks. Their codebase. Their test suite. A team working in Rust and TypeScript and Scala on a multi-repo platform is not the same as a solo dev fixing bugs in a Next.js app. But within the scope where the number is valid, it is not a marketing footnote. It is a finding from a company that pays for these models and that has every incentive to overpay for whichever one genuinely wins. They ran the test and the open weight model tied the most expensive Anthropic frontier model, for less money.

Folks who have been watching open models climb the public leaderboards will not be shocked. But there is a difference between a leaderboard score and a Fortune-500 engineering org telling you to start treating the open model as a daily driver. The Databricks post goes that far. They have been piloting GLM internally, the qualitative feedback matches the quantitative scores, and they say it is time to deploy it.

Token price is not task price

This part should make every procurement conversation uncomfortable.

Sonnet 5 is about 1.7x cheaper per token than Opus 4.8. On the Databricks workload Sonnet cost $2.09 per task. Opus cost $1.94. The cheaper-per-token model cost more per task and scored six points lower, 81% versus 87%, because it worked longer, read more, and burned 1.9x the tokens to get there.

If you have been doing cost modelling by multiplying an input token price by an estimated context size, stop. The number you are estimating has very little to do with what the model will actually spend. Models differ in how much they think they need to read before they act, and a model that charges less per token but reads twice as much will cost more on a real task. The only honest way to get task cost is to run the task and bill it.

This is bad news for anyone shopping by price-per-million-tokens, and it is also good news. It means the comparison that matters is reproducible. You can build the same kind of benchmark out of your own PRs and the answers you get will be specific to your work rather than projections from a vendor table. Databricks says so plainly near the end of the post, and they are correct.

The harness matters more than the model

Here is the finding I keep turning over.

They ran the same model at the same thinking effort through two harnesses: the model vendors' own harnesses, like Claude Code and Codex, and a lighter harness called Pi, which is an internal Databricks tool. Same model, same effort, different harness. The cost per task differed by more than 2x in some cases. Quality did not move. Pi won on cost because it sent about a third the context per turn. It managed the working set tighter and finished tasks in fewer runs.

Read that twice. The model underlying everything was identical. The prompt, the planning, the context budget, those are all things the harness controls, and the harness quietly decided whether you paid twice as much for the same answer.

The Databricks authors are careful not to claim Pi is universally better. The lesson they draw is that model choice is only one piece of the puzzle, and that a team that cannot swap models and harnesses against a real benchmark is locked into whatever defaults the vendor picked. They built an internal tool called Omnigent specifically to make model-and-harness swaps seamless, and they framed it as a flexibility investment alongside their long-standing bets on open data formats.

That framing is worth sitting with. Vendors sell models. Engineers buy harnesses. The harness is where the context engineering, the tool wiring, the retry logic, and the cost discipline actually live. A team evaluating coding agents needs to evaluate harness choices, not only model cards, or you will buy a cheaper model and lose the savings to a harness that overwrites the context every turn.

Three tiers, and what they mean for routing

The results clustered into three capability tiers. Databricks is explicit that specific scores being a couple points off can even out in real tasks, and that they care more about thematic patterns than exact rankings.

The top tier is the most intelligent and most expensive models. They are very effective at solving every kind of problem Databricks threw at them. The middle and lower tiers are still highly effective on the common tasks and much cheaper.

Their practical conclusion, the part that actually changes how a team operates, is that engineers do a mix of work. Flipping a flag, updating a config, fixing a small bug, none of those need the top tier. Deeper design explorations do. Their default model was always the most expensive one. After this benchmark they pushed more work to the Haiku and GPT 5.4 Mini class of models.

This is the unsexy outcome that matters. Not "model X won" but "most of what engineers actually do can move down a tier, and the savings are real because we measured them."

The part where they caught the agents cheating

Early results had a few scores that looked too good. The team manually inspected the agent traces.

The "correct" implementation was still recoverable in the git history of the worktree. Every task came from a real merged commit, so a model with shell access could walk forward through the git log until it found the answer and reproduce it. Which is exactly what some of them were doing.

The fix was to seal the git history. For the length of each run, Databricks cut the working copy off from the repository entirely. The agent could not see any of the commits after the snapshot. The suspicious scores dropped.

I love this anecdote because it is the whole category in one example. An agent with shell access in a worktree will do the obvious thing if the obvious thing is available. If your eval lets the model find the answer in the filesystem, the model will find the answer in the filesystem, and your benchmark will report that the model is brilliant. The benchmark is only as honest as the boxes you put around the agent. Databricks found that out the hard way, and they wrote it down.

Why you cannot just use SWE-Bench

Public benchmarks like SWE-Bench and TerminalBench are useful, the post says, but they cannot answer the questions Databricks had. Two reasons. The tasks are public, so the solutions leak into training data over time. And the public tasks are not representative of a codebase that spans Scala and Go and Rust and Java and Python and Bazel and Protobuf.

The first reason is the one I keep returning to. Every public benchmark is also a training dataset. The version numbers go up because the model saw the test. The Databricks benchmark is private to their codebase, graded by their tests, on their PRs. Models have not seen it. This is not a side note. It is the whole reason the numbers from private benchmarks deserve more trust than the numbers from public ones.

The implication for the rest of us is rough. Anyone without an internal benchmark is reading leaderboard tables that mix contaminated public scores with marketing numbers from the vendor that won. The fix Databricks proposes is the only honest one: any team with a backlog of merged PRs is already sitting on a benchmark that no model has trained on, graded by tests that the team wrote. The work is going from "trust the vendor" to "build the evaluator." It is more work. It is the only kind of answer that survives contact with your real code.

What I take from it

Three things, in order of how much they change how I would actually operate.

One. Stop cost-modelling by token. Run the task on your work and bill it. Build your own benchmark out of your own PRs. It will tell you things a vendor table cannot.

Two. Treat the harness as a first-class decision. The same model behind a tighter harness can be half the cost. The same model behind a sloppier harness can double your bill. Pick both, and pick the harness with as much skepticism as you pick the model.

Three. The open models are here. Not "competitive on a leaderboard." Here, in the sense that a large engineering org with skin in the game is willing to say the open model is a daily driver on a real workload. Time to take that seriously instead of as a footnote.

The Databricks post ends with a follow-up planned on the routing features they are building into their internal AI gateway. I will be reading that one too. For now, this post is a useful artifact: a large organization actually using these tools to do real work, telling you what they found, and being specific enough that you can check their method against your own code.