Harness showdown

Six harnesses, all driving one local DeepSeek V4 Flash. Off-the-shelf, by tool count lightest to heaviest: Pi (0.80.10), a bare 4-tool harness; OpenCode (1.17.10), a mid-weight framework; Claude Code (2.1.186), a heavy one that does not speak the local server's API, so it reaches the model through a small proxy (CLIProxyAPI); and nanocoder (1.29.0), a community, local-first harness. Plus two compact Rust agents, jcode (0.72.0) and claurst (0.1.7), the latter a clean-room reimplementation of Claude Code. The same model sits underneath all of them, served the same way throughout, the same set of real bug fixes, the same grading, and only the wrapper moves.

One measurement note, put here once. Metering the two Rust agents took more work than the other four. jcode and claurst report only a final-turn token count, not the session total, so a first pass left them off the cost charts entirely. Reading jcode's full event stream and metering claurst against the server's own token counter recovered the real numbers, and all six now sit on the same token and time axes. The correction earned its keep: it is what showed the two lean-looking Rust agents are nothing of the sort.

The grades come out the same

Across the tasks no grade difference held up among the harnesses that commit their fixes. Pi, OpenCode, jcode, Claude Code and nanocoder all land in the same overlapping place, because the run-to-run noise is wide enough to swallow the gaps between them whole.

On the hardest tasks the same harness scores a basic attempt on one run and a solid fix on the next, a full grade apart, with nothing changed between them but the random seed.

The swing is that wide, so with only 2 or 3 runs the noise decides which harness looks best, and a ranking that changes every time you rerun it is no ranking at all. One of the six read a grade worse at first, jcode, until I noticed it was not being given the resume-nudge OpenCode gets when a harness stops after planning without editing; nineteen of its first sixty-four runs were empty diffs for that reason alone. With the identical nudge it came straight back into the band, an apparent quality gap that was really a difference in how the harness was driven, not in the fixes it wrote.

claurst is the one that does not come back. It sits a full grade below the others, and not because its fixes are bad, when it commits an edit the fix is often right, and on the hardest search bug it diagnosed the exact one-line cause. The trouble is that on solvable tasks roughly half its runs never produce an edit at all, about 46% empty against none for Pi, OpenCode or jcode, and why that happens is the interesting part.

claurst's intent is different

The reason is what claurst is built to do when it is unsure. It is an interactive pair-programmer, a TUI tool meant to sit beside a person, and its own system prompt tells it to ask the user rather than guess and to describe a blocker so the user can assist, backed by a set of tools for exactly that, a question tool, a notify tool, a native approval dialog for every edit. Run headless, with no user to answer, that caution has nowhere to go, so instead of editing it stops to ask a question or flag a blocker, and an unanswered question grades as nothing. A completion contract compounds it, claurst is told not to declare a task finished without an audit and evidence, so when it cannot fully verify it simply does not finish. The nudge that rescued jcode barely moves it, because its resume opens a fresh session with no memory of the plan it just made, so telling it to apply the fix it described lands on nothing. Benched this way, claurst measures the gap between a supervised tool and an unsupervised harness at least as much as the model beneath it.

From claurst's own system prompt. On getting stuck: "If stuck, ask the user rather than guessing," and "If you cannot find a concrete path, ask the user." Its question tool, as described to the model: "Ask the user a question and wait for their response. Use this when you need clarification, confirmation, or additional information from the user." And its rule for finishing: calling the completion tool "without a real audit is a goal contract violation."

So the band is real, and it has a floor. Among harnesses that actually commit their fixes, Pi, OpenCode, jcode, Claude Code and nanocoder, the grade is a wash and only the cost differs. But a harness can be bad enough to drop out of the band, and when it does the failure does not look like worse code, it looks like no code, edits that never land. The noise swallows the difference between good harnesses; it does not rescue a harness that under-commits.

The difference is time and tokens

With all six on the same axes, the lean end turns out to be small: only Pi and OpenCode reach the fix cheaply, around 13 to 20 thousand tokens. The other four are heavy explorers that generate several times as much to land the same edit, and the two Rust agents belong squarely in that group, not with the lean pair they superficially resemble. jcode spends about what Claude Code does; claurst is the slowest harness of the six. The cost is all on the output side, in what the model generates and the tools it runs. They read very differently too, the heavy group exploring far more, but on this model that barely registers, because DeepSeek reads input at nearly the same speed at 10K tokens or 200K, so heavy context is close to free here and would only bite as billed input on a metered API. What costs is generation.

Wall-clock is not really a separate axis from that. Across the clean runs, time tracks output tokens at a per-harness decode rate that barely moves, a correlation around 0.9. Every harness's minutes below are measured directly, each run taken one at a time on a quiet server so concurrency never distorts the clock. Claude Code's measured time even runs above what its tokens alone would predict, because on the hardest tasks it balloons its context past a million tokens and decode slows to a crawl, a cost that never shows up in the word count.

Pi reasons, OpenCode delegates

The two lean harnesses spend the clock differently. Pi does the whole task in one context, verifying by thinking, so almost all of its roughly twenty thousand tokens are reasoning in one long stream. OpenCode is leaner still, near thirteen thousand: it hands chunks of work to subagents, each a separate sub-run, but on this model those sub-runs stay short, so counting them back in barely moves its total. What separates the two is pace, the same model decodes at one rate for both, but Pi spends its clock generating in long reasoning bursts while OpenCode shuttles work out to subagents and waits on the hand-offs, which is how it still comes in the fastest of all six despite the round-trips. And none of it moves the grade, because the 8 bugs are subtle logic bugs, a guard testing the wrong condition, a state change firing a step too early, that a subagent catches no better than a careful line of reasoning, and a capable model lands either way.

Claude Code and nanocoder explore most, and edit only when certain

These two are the heavy explorers, each burning several times the tokens of a lean harness to reach the same fix and each holding off on edits until late in the run. Claude Code runs the most tool calls of the six by a wide margin, around 70 to a task where Pi needs 39 and OpenCode 30, with jcode and claurst exploring hard too at roughly 50 and 43, and its first edit lands about 80% of the way through; nanocoder makes fewer calls but re-reads the same files more than anything else, and commits nearer 70%. Nearly all of that front-loaded work is looking rather than touching, and the edit each finally makes is the same handful of lines the lean harnesses landed long before. That thoroughness is usually just time, but not always: on the one bug every lean harness fails, leaving an empty diff, Claude Code ground through 140K tokens and came away with partial credit, the rare case where heavy exploration actually buys something the lean harnesses miss.

The six, side by side

Laid out on one set of axes, all six separate cleanly on everything above the grade, and five of them land together on the grade itself. The heavy four, Claude Code and nanocoder plus both Rust agents, spend multiples of Pi and OpenCode in steps taken, tokens generated and minutes on the clock to reach the same edit. claurst is the exception on quality too, sitting below the rest for the empty diffs it leaves when run headless.

The scorecard

The whole thing, one harness to a line: where it landed on quality, what it spent to get there, and the one-line read. On a single 0-to-3 scale the shape is plain, five harnesses stacked together and claurst alone below them.

Which one to reach for

This is DeepSeek V4 Flash on eight focused bug fixes, so read it scoped. For a capable model, and a harness that reliably commits its edits, the harness is largely an efficiency choice rather than a quality one, at least in my agentic workloads in a large codebase. The one thing that does move the grade is a harness that under-commits, and that is a different failure, it shows up as empty diffs rather than worse fixes, and no amount of nudging fully closes it. And the next time a post claims that some harness, some prompt, or some clever context file made a coding model measurably better, the first thing to ask is how many times they ran it, because under about 10 runs a condition there is a good chance they mostly measured the weather.