For OpenCode 2.0
Benchmarks pass tests. You ship code.
A benchmark is a clean task, a hidden test and a single model. Real work is a messy repo, your tools, your steering — and increasingly one model planning while another builds. Pragmatikos scores that: every model and every planner → builder pairing, on real sessions, by what actually shipped — landed in a git commit.
Rankings below are pooled from real developer sessions.
claude-fable → grok · 62 / 100
the rankings
Models and pairings, ranked by what ships
Did the work land in a commit, how many nudges did it take, how clean was it, what did it cost. Scored on real sessions, not lab tasks. Scores are expected to change as more data is contributed.
1,202 Cycles*
join in
This is a first answer, not the final one. It's built from the sessions developers have shared so far. The more histories in the pool, the harder the ranking gets to argue with.
why it exists
Three questions. Three instruments.
The first two are useful and stay useful. Developers choosing a model and a workflow are missing the third answer.
question 01 · the lab
Benchmarks ask: how capable is this model?
SWE-bench, Terminal-bench, Aider, LMArena. Controlled tasks, hidden tests, preference votes — the best way to compare models on equal footing and know each model’s ceiling.
- signal
- pass rate · votes
- blind spot
- One model, a clean task, nobody steering.
question 02 · the crowd
Usage rankings ask: what is the market using?
OpenCode Data, OpenRouter Ranking. Tokens, users, retention, dollars per session — the best way to see adoption and where the mix is shifting.
- signal
- tokens · users · $/session
- blind spot
- Popular is not productive — and still one model at a time.
question 03 · the recorder
Pragmatikos asks: what ships in real work?
Turns, phases, edits and errors from real sessions, correlated with what shipped. The one instrument that scores the setups developers actually run.
- signal
- session record × shipped outcomes
- blind spot
- Anything it never saw. Observational, not a benchmark.
Capable. Popular. Effective.
Only one of them has a row for the setup you actually run.
the blind spot both miss
Same builder. Different planner. Opposite results.
The build model is identical. One planner got it to a commit in a few turns; the other burned dozens and shipped nothing. A model ranking can't see this. A pairing ranking can.
claude-fable-5.1 → grok-4.6
0 turns
grok-4.6 → grok-4.6
0 turns
Real sessions. Unshipped work counts against the model.
how the score works
Six tiers, ten axes, one weighted mean
Outcome
weight 30
Ship rate and one-shot rate (prompt plus approval).
Cost of a ship
weight 20
Turns, hours and dollars per shipped cycle.
Precision
weight 15
Tool errors and human interventions.
Discipline
weight 15
Share of cycles verified after the last edit.
Efficiency
weight 10
Edits per human turn.
Latency
weight 10
Median seconds per assistant step.
Pool-relative
Every axis is a log distance to the pooled average — log odds for rates — on a fixed ×4 span, so ×2 and ÷2 sit the same distance from the line.
Evidence-weighted
Small groups shrink toward the pool with k = 10 pseudo-cycles. A two-cycle fluke nearly vanishes.
Failure lowers the rate
Unshipped cycles stay in the ship-rate denominator, so few-turn dead ends can never look good. Turns, hours and dollars are priced per shipped cycle.
two records, correlated
The session record comes first
Every turn, plan and build phase, edit, tool error and dollar is read from the agent's own record — that is where every axis is measured. Commits only decide which cycles count as shipped: a window from the previous commit, file overlap decides credit, active-but-absent sessions advised only.
contribute
Add your sessions in two minutes
For OpenCode v2 — other coding agents coming soon.
Install the ocInsights plugin for OpenCode: add "@pfoundation/ocinsight" to the "plugin" list in my global opencode.json config. Then tell me to restart OpenCode, and afterwards verify the plugin loaded and the insights deck answers at http://127.0.0.1:4173/. Also report whether insight contribution is on, without changing that setting.
- 1
Paste the prompt into your OpenCode agent and let it edit your config.
- 2
Restart OpenCode when it tells you to, then open the deck it verifies.
- 3
Preview with Contribute in the deck header — sharing is on by default.
Twenty fields per cycle — day, models, turns, edits, cost, shipping. No paths, prompts, session ids or projects. Preview in the deck's Contribute panel before anything leaves your machine.