The data
Every model graded on the same 8 coding tasks across 1437 runs, each attempt scored 0-3 against the fix that actually shipped. All of it generated straight from the benchmark data.
Every model
Ranked by mean tier across all runs, with how each one is served. Setup shows exactly how.
Scorecard
The shape behind each score: how every model's attempts spread across the four grades, with gaps where a grade never happened.
Time per task
Spread of wall-clock time to finish a task, in real minutes, with each task weighted equally so uneven run counts don't skew it; dashed line is the median, fastest on top.
Speed & taper
Decode speed as context fills, on the two-card box. Sliding-window and MoE models stay flat, dense models taper hard. Toggle to prefill (top-right) for how fast each reads context in, the TTFT side of the story.
Working style
How much context each model ingests versus how many tokens it generates to solve a task. Up-and-right = reads and writes a lot.
VRAM
What each model needs while serving, split into fixed weights (solid) and reserved KV cache (faint) that scales with how much context you keep live. Reference lines mark common total capacities, so it maps onto whatever cards you have rather than any one setup.
The tasks
Each task is a real bug from a production TypeScript/GraphQL codebase, reverted to just before its fix and generalized to an archetype.
The aim was tasks that separate models rather than flatter them, so each one had to clear a few bars:
- Small diff, big search: the actual fix is only a handful of lines, but finding it takes real digging through the codebase, so it rewards understanding over typing.
- More than one way to do it, not all equal: several fixes will pass, but only some are clean and complete, which is exactly what separates Solid from Complete.
- An objective fix that actually shipped: each is a real bug with a known-good merged fix, so attempts get graded against ground truth, not taste.
- Contamination-controlled: a clean just-before-fix checkout with history stripped, so nothing about the solution leaks into the prompt.
The set spans difficulty on purpose, from near-floor tasks almost everyone completes to discriminators only the strongest models get right.
A save that changes nothing still fires an expensive downstream state transition; separately, a class of legacy records silently skips an audit step. Two orthogonal defects, graded independently - and a blanket guard that also suppresses legitimate updates doesn't count as a fix.
Re-saving a parent record re-runs a costly re-processing side effect on a child field that didn't actually change. The guard has to compare on normalized values (whitespace / line-ending-only differences shouldn't count as a change) and run before the side effect fires.
Terms with internal punctuation (initials, separators) return nothing, because the query and the index tokenize punctuation differently. The real fix aligns the two at the tokenizer; patching the single failing query string is brittle.
A status filter is scoped too broadly and surfaces records that should be excluded - re-scope the underlying query condition. Straightforward; every model solves it, so it's kept only as a floor (tiers barely separate).
A safety assertion is too strict and fires on a legitimate edge case, blocking an operation that should succeed. The fix has to narrow the assertion to what it's actually meant to guard - without deleting the safety check.
Two related writes run in separate transactions; if the second fails (timeout, contention) they're left permanently out of sync. It compiles and passes the happy path - the desync only manifests under failure, so you have to reason about the failure path, not the common case.
An unclamped boundary value produces wrong behaviour right at the limit (a rounding / comparison-direction bug). A one-line boundary fix - every model localizes it, so it's kept only as a floor (core is already near-complete, no real split).
The change must land on the correct one of two similar internal surfaces (a routing trap), then trigger a follow-up recompute + refresh - a second step that's easy to miss. Putting it on the wrong surface compiles fine but is wrong.
About
Every model solved the same 8 tasks, run autonomously as an agent, and each resulting diff was graded 0-3 against the fix that actually shipped by a strong LLM judge, with a small penalty for avoidable inefficiency. Local models run under the opencode agent on vLLM or llama.cpp; the hosted ones run under Claude Code. Different scaffolds on purpose: the question is what you get when each model is set up the way people actually run it, not which raw model wins in a vacuum.
Coverage is deliberately uneven. The discriminator tasks, the ones that actually separate models, are run hardest, up to 19 times on a single model; the easy floor tasks and the slowest local models get as few as 1, so read the thin cells as directional rather than precise. Scores are the mean tier across a model's runs, never best-of-N. Task time is wall-clock with each task weighted equally, so no single slow task dominates. Every chart on this page is drawn from that same benchmark data, 1437 graded runs exported as JSON.
How the speed is measured. Decode speed is measured on a diverse, realistic code corpus, never on repetitive filler, and that distinction matters because several models here decode speculatively (DeepSeek's DSpark, Qwen's MTP): a small draft proposes tokens and the model keeps the ones it would have written anyway. Repetitive padding is trivially predictable, so the draft's acceptance climbs as the filler grows and the meter inflates into a rising, physically impossible curve, decode getting faster deeper into context. Measured on real code, a speculative model's speed is content-dependent and either flat or gently tapering, which is what these charts show; a filler measurement once had DeepSeek reading 183 to 347 tok/s before it was corrected. The taper tooling now refuses to measure a speculative serve on filler, so it cannot creep back.
scorecard.json · ~139 KB · the aggregated data behind every chart on this page