For OpenCode 2.0
Benchmarks test single model. Developers often pair them.
A benchmark is an exam: a clean task, a hidden answer key, one model, nobody steering. Real work is the job: a messy repo, your tools, your steering, and increasingly one model planning while another builds. Pragmatikos scores the job, not the exam: every planner → builder pairing, on real sessions, by what actually shipped.
Rankings below are pooled from real developer sessions.
claude-fable → muse-spark · 59 / 100
the rankings
Models and pairings, ranked by what ships
Did the work land in a commit, how many nudges did it take, how clean was it, what did it cost. Scored on real sessions, not lab tasks. Scores are expected to change as more data is contributed.
1,473 Cycles* · 2 contributors
join in
This is a first answer, not the final one. It's built from the sessions developers have shared so far. The more histories in the pool, the harder the ranking gets to argue with.
why it exists
Three questions. Three instruments.
The first two are useful and stay useful. Developers choosing a model and a workflow are missing the third answer.
question 01 · the lab
Benchmarks ask: how capable is this model?
SWE-bench, Terminal-bench, Aider, LMArena. Controlled tasks, hidden tests, preference votes — the best way to compare models on equal footing and know each model’s ceiling.
- signal
- pass rate · votes
- blind spot
- One model, a clean task, nobody steering.
question 02 · the crowd
Usage rankings ask: what is the market using?
OpenCode Data, OpenRouter Ranking. Tokens, users, retention, dollars per session — the best way to see adoption and where the mix is shifting.
- signal
- tokens · users · $/session
- blind spot
- Popular is not productive — and still one model at a time.
question 03 · the recorder
Pragmatikos asks: what ships in real work?
Turns, phases, edits and errors from real sessions, correlated with what shipped. The one instrument that scores the setups developers actually run.
- signal
- session record × shipped outcomes
- blind spot
- Anything it never saw. Observational, not a benchmark.
Capable. Popular. Effective.
Only one of them has a row for the setup you actually run.
the blind spot both miss
Same builder. Different planner. Opposite results.
The build model is identical. One planner got it to a commit in a few turns; the other burned dozens and shipped nothing. A model ranking can't see this. A pairing ranking can.
claude-fable-5 → grok-4.6
0 turns
grok-4.6 → grok-4.6
0 turns
Real sessions. Unshipped work counts against the model.
how the score works
Six tiers, ten axes, one weighted mean
Outcome
weight 30
Ship rate and one-shot rate (prompt plus approval).
Cost of a ship
weight 20
Turns, hours and dollars per shipped cycle.
Precision
weight 15
Tool errors and human interventions.
Discipline
weight 15
Share of cycles verified after the last edit.
Efficiency
weight 10
Edits per human turn.
Latency
weight 10
Median seconds per assistant step.
Pool-relative
Every axis is a log distance to the pooled average — log odds for rates — on a fixed ×4 span, so ×2 and ÷2 sit the same distance from the line.
Evidence-weighted
Small groups shrink toward the pool with k = 10 pseudo-cycles. A two-cycle fluke nearly vanishes.
Failure lowers the rate
Unshipped cycles stay in the ship-rate denominator, so few-turn dead ends can never look good. Turns, hours and dollars are priced per shipped cycle.
two records, correlated
The session record comes first
Every turn, plan and build phase, edit, tool error and dollar is read from the agent's own record — that is where every axis is measured. Commits only decide which cycles count as shipped: a window from the previous commit, file overlap decides credit, active-but-absent sessions advised only.
where it falls short
A hypothesis to try, not a verdict
Pragmatikos reads sessions after the fact, based on voluntarily shared sessions. Nobody assigned pairings to developers or tasks, so the ranking says which setups shipped for the people who ran them — not which setup would ship for you. Here is where that bites, and what we intend to do about it.
Observational, not controlled
The developers who run one pairing are not the developers who run another, and they are not working the same tasks. Skill, repo and task difficulty can move a score as much as the models do. “Ranked by what ships” is a correlation. Read it as one.
Limited data
The sample is still small, and some setups have been used more than others. Rankings may shift as more people contribute. A slightly higher score doesn’t necessarily mean a better setup.
Shipped is a floor, not a grade
Following work through review and merge varies across projects. A git commit is simpler to track, so we use it as the bare minimum for “shipped.” It doesn’t mean the work was reviewed, merged, or released.
More histories
Every new contributor puts the same pairings in front of different developers, repos and tasks. That is what lets a pairing’s effect separate from the developer’s. The scale and the shrinkage were built for a bigger pool; the pool is what is missing.
Share your sessions →If that is not enough
The pool has one thing going for it: everyone in it is a developer running OpenCode on real work — the population this ranking is for. That does not fix the task and skill mix, but a different approach can start from it. If more histories do not settle the picture, the method changes, and this page will say what changed.
contribute
Add your sessions in two minutes
For OpenCode v2 — other coding agents coming soon.
Install the ocInsights plugin for OpenCode: add "@pfoundation/ocinsights" to the "plugin" list in my global opencode.json config. Then tell me to restart OpenCode, and afterwards verify the plugin loaded and the insights deck answers at http://127.0.0.1:4173/. Also report whether insight contribution is on, without changing that setting.
- 1
Paste the prompt into your OpenCode agent and let it edit your config.
- 2
Restart OpenCode when it tells you to, then open the deck it verifies.
- 3
Preview with Contribute in the deck header — sharing is on by default.
Twenty-five fields per cycle — day, models, effort, harness, turns, edits, cost, shipping. No paths, prompts, session ids or projects. Preview in the deck's Contribute panel before anything leaves your machine.