$ replaybook

Versioned evidence

Benchmark history

Older results still explain model behavior. They remain separate when a stronger verifier or harness changes what a pass means.

Ling 3.2 Flash Free Core via Claux / OpenRouter

Benchmark 20261115.0.0 · Core tier

Latest published

22/33 durable repairs, 11:34 median, $0.0000 known cost.

Inspect cohort 20261115.0.0

Gemini 4 Hydrogen Core via Claux / OpenRouter

Benchmark 20261113.0.0 · Core tier

Archived cohort

30/33 durable repairs, 1:24 median, $5.8412 known cost.

Inspect cohort 20261113.0.0

Nex N2.6 Core via Claux / OpenRouter

Benchmark 20261112.0.0 · Core tier

Archived cohort

33/33 durable repairs, 1:49 median, $0.3920 known cost.

Inspect cohort 20261112.0.0

MiMo-V2.6-Flash Core via OpenCode Go

Benchmark 20261110.0.2 · Core tier

Archived cohort

23/33 durable repairs, 10:24 median, n/a known cost.

Inspect cohort 20261110.0.2

GLM-5.3-Flash Core via OpenCode Go

Benchmark 20261110.0.1 · Core tier

Archived cohort

32/33 durable repairs, 1:45 median, n/a known cost.

Inspect cohort 20261110.0.1

Big Pickle Core via OpenCode Zen — quota-constrained

Benchmark 20261110.0.0 · Core tier

Archived cohort

18/21 durable repairs, 2:12 median, n/a known cost.

Inspect cohort 20261110.0.0

Frozen Core: Grok 4.7

Benchmark 20261106.0.1 · Core tier

Archived cohort

32/33 durable repairs, 3:40 median, $21.2807 known cost.

Inspect cohort 20261106.0.1

Frozen Core: Ripple-65B-Preview

Benchmark 20261106.0.0 · Core tier

Archived cohort

18/31 durable repairs, 12:39 median, $2.3439+ known cost.

Inspect cohort 20261106.0.0

Frozen Core: Crouton Alpha

Benchmark 20261103.0.0 · Core tier

Archived cohort

33/33 durable repairs, 1:01 median, $0.0000 known cost.

Inspect cohort 20261103.0.0

Experimental Core Hard: Particle-1, Sonnet 5.5, and Pareto 26.10

Benchmark 20261029.0.0 · Frontier tier

Archived cohort

27/27 durable repairs, 0:48 median, $1.5906 known cost.

Inspect cohort 20261029.0.0

Frozen Core: Particle-1

Benchmark 20261028.0.0 · Core tier

Archived cohort

33/33 durable repairs, 0:54 median, $0.9534 known cost.

Inspect cohort 20261028.0.0

Frozen Core: Fireworks Ember 1

Benchmark 20261027.0.1 · Core tier

Archived cohort

32/33 durable repairs, 1:20 median, $4.7909 known cost.

Inspect cohort 20261027.0.1

Frozen Core: MiniMax, Mistral Large 4, and Reflection Beam

Benchmark 20261027.0.0 · Core tier

Archived cohort

94/99 durable repairs, 3:03 median, $11.2293+ known cost.

Inspect cohort 20261027.0.0

Frozen Core: Seed Turbo, MiMo Pro, and GLM 5.3

Benchmark 20261026.0.0 · Core tier

Archived cohort

91/99 durable repairs, 2:43 median, $9.0408 known cost.

Inspect cohort 20261026.0.0

Frozen Core: MiMo Flash, Qwen Flash, and Tencent Hy4

Benchmark 20261025.0.2 · Core tier

Archived cohort

95/99 durable repairs, 3:42 median, $3.4274 known cost.

Inspect cohort 20261025.0.2

Frozen Core: GLM Flash, Step 5 Preview, and Nemotron Ultra

Benchmark 20261025.0.1 · Core tier

Archived cohort

85/99 durable repairs, 1:24 median, $5.9719 known cost.

Inspect cohort 20261025.0.1

Frozen Core flagship evaluation: Opus, GPT-6.1 Sol, and Gemini

Benchmark 20261025.0.0 · Core tier

Archived cohort

92/99 durable repairs, 1:28 median, $19.3888+ known cost.

Inspect cohort 20261025.0.0

Frozen Core: Kimi K3.1 Pro, DeepSeek 1017, and Qwen Max Prime

Benchmark 20261024.0.0 · Core tier

Archived cohort

97/99 durable repairs, 2:05 median, $18.5488 known cost.

Inspect cohort 20261024.0.0

Claux + OpenRouter October Core evaluation

Benchmark 20261020.0.0 · Core tier

Archived cohort

66/72 durable repairs, 1:20 median, $1.4792 known cost.

Inspect cohort 20261020.0.0

OpenCode Go October core infrastructure evaluation

Benchmark 20261018.0.0 · Core tier

Archived cohort

65/72 durable repairs, 1:55 median, n/a known cost.

Inspect cohort 20261018.0.0

OpenRouter Space Bunny Alpha core cohort

Benchmark 20260924.0.0 · Core tier

Archived cohort

16/24 durable repairs, 1:54 median, $0.0000+ known cost.

Inspect cohort 20260924.0.0

OpenRouter core infrastructure baseline

Benchmark 20260906.0.0 · Core tier

Archived cohort

125/144 durable repairs, 2:36 median, $22.3951 known cost.

Inspect cohort 20260906.0.0

OpenRouter Fable 5.1 and Gemini 3.8 mixed infrastructure cohort

Benchmark 20260905.0.0 · Unclassified tier

Archived cohort

56/66 durable repairs, 2:20 median, $29.6143+ known cost.

Inspect cohort 20260905.0.0

OpenCode Go core infrastructure cohort

Benchmark 20260904.0.0 · Core tier

Archived cohort

102/115 durable repairs, 3:19 median, n/a known cost.

Inspect cohort 20260904.0.0

OpenCode Go GPT-5.6 Luna core infrastructure cohort

Benchmark 20260830.0.1 · Core tier

Companion cohort

22/24 durable repairs, 1:54 median, n/a known cost.

Inspect cohort 20260830.0.1

OpenRouter GPT-5.6 Luna core infrastructure cohort

Benchmark 20260830.0.0 · Core tier

Archived cohort

22/24 durable repairs, 1:52 median, $0.8045+ known cost.

Inspect cohort 20260830.0.0

Vercel AI Gateway funded infrastructure cohort

Benchmark 20260827.0.1 · Unclassified tier

Archived cohort

27/27 durable repairs, 2:07 median, $2.2074 known cost.

Inspect cohort 20260827.0.1

OpenCode Go visual infrastructure cohort

Benchmark 20260827.0.0 · Unclassified tier

Archived cohort

34/36 durable repairs, 2:30 median, n/a known cost.

Inspect cohort 20260827.0.0

OpenCode Go visual infrastructure cohort

Benchmark 20260826.0.1 · Unclassified tier

Archived cohort

20/20 durable repairs, 2:16 median, n/a known cost.

Inspect cohort 20260826.0.1

Reasoning effort and convergence: three OpenAI models

Benchmark 20260826.0.0 · Unclassified tier

Archived cohort

8/12 durable repairs, 1:58 median, $1.5424 known cost.

Inspect cohort 20260826.0.0

Reasoning effort experiment: Luna on interrupted deploy

Benchmark 20260825.0.2 · Unclassified tier

Archived cohort

4/6 durable repairs, 1:34 median, $0.1987 known cost.

Inspect cohort 20260825.0.2

Visual infrastructure benchmark: deployment timeline

Benchmark 20260825.0.1 · Unclassified tier

Archived cohort

19/25 durable repairs, 2:25 median, $3.1130+ known cost.

Inspect cohort 20260825.0.1

Visual infrastructure benchmark: deployment timeline

Benchmark 20260825.0.0 · Unclassified tier

Archived cohort

16/22 durable repairs, 2:30 median, $2.5716+ known cost.

Inspect cohort 20260825.0.0

Visual infrastructure benchmark: topology and metrics

Benchmark 20260822.0.4 · Unclassified tier

Archived cohort

32/36 durable repairs, 2:22 median, $2.3708+ known cost.

Inspect cohort 20260822.0.4

Four-model visual infrastructure cohort

Benchmark 20260822.0.3 · Unclassified tier

Archived cohort

12/12 durable repairs, 2:20 median, $0.7839 known cost.

Inspect cohort 20260822.0.3

Kimi and Gemini visual infrastructure cohort

Benchmark 20260822.0.2 · Unclassified tier

Archived cohort

6/6 durable repairs, 2:00 median, $0.8096+ known cost.

Inspect cohort 20260822.0.2

Ox Alpha core infrastructure cohort

Benchmark 20260822.0.1 · Core tier

Archived cohort

22/24 durable repairs, 4:06 median, $0.0000 known cost.

Inspect cohort 20260822.0.1

Five-model core infrastructure cohort

Benchmark 20260822.0.0 · Core tier

Archived cohort

110/120 durable repairs, 2:42 median, $12.1587 known cost.

Inspect cohort 20260822.0.0

Qwen 3.8 and GLM 5.3 core infrastructure cohort

Benchmark 20260821.0.1 · Core tier

Archived cohort

47/48 durable repairs, 2:19 median, $7.5341 known cost.

Inspect cohort 20260821.0.1

Ox Alpha full infrastructure cohort

Benchmark 20260821.0.0 · Full tier

Archived cohort

42/48 durable repairs, 2:50 median, $0.0000 known cost.

Inspect cohort 20260821.0.0

OpenCode Go five-scenario infrastructure cohort

Benchmark 20260820.0.1 · Unclassified tier

Archived cohort

80/86 durable repairs, 3:22 median, n/a known cost.

Inspect cohort 20260820.0.1

OpenCode Go infrastructure smoke cohort

Benchmark 20260820.0.0 · Unclassified tier

Companion cohort

14/18 durable repairs, 3:32 median, n/a known cost.

Inspect cohort 20260820.0.0

Five models across the full infrastructure suite

Benchmark 20260819.0.0 · Full tier

Archived cohort

214/240 durable repairs, 2:34 median, $22.4110+ known cost.

Inspect cohort 20260819.0.0

GLM 5.3 across the full infrastructure suite

Benchmark 20260818.0.0 · Full tier

Archived cohort

47/48 durable repairs, 1:38 median, $4.8792 known cost.

Inspect cohort 20260818.0.0

Five agents across thirteen infrastructure incidents

Benchmark 20260817.0.3 · Unclassified tier

Archived cohort

59/65 durable repairs, 2:43 median, $6.2799+ known cost.

Inspect cohort 20260817.0.3

Five agents repair Nix store disk pressure

Benchmark 20260817.0.2 · Unclassified tier

Archived cohort

14/15 durable repairs, 3:46 median, $2.3316+ known cost.

Inspect cohort 20260817.0.2

Five agents complete a partial service rollout

Benchmark 20260817.0.1 · Unclassified tier

Archived cohort

15/15 durable repairs, 2:37 median, $1.3191 known cost.

Inspect cohort 20260817.0.1

Five agents face a Discourse plugin boot loop

Benchmark 20260817.0.0 · Unclassified tier

Archived cohort

12/15 durable repairs, 1:46 median, $0.7299 known cost.

Inspect cohort 20260817.0.0

Five agents deploy real Discourse

Benchmark 20260816.0.0 · Frontier tier

Archived cohort

5/5 durable repairs, 11:30 median, $0.7927 known cost.

Inspect cohort 20260816.0.0

Five agents across 13 infrastructure incidents

Benchmark 20260815.0.1 · Unclassified tier

Archived cohort

59/65 durable repairs, 2:31 median, $13.4734+ known cost.

Inspect cohort 20260815.0.1

Four agents repair an interrupted Discourse deployment

Benchmark 20260815.0.0 · Unclassified tier

Archived cohort

7/12 durable repairs, 4:36 median, $0.9276 known cost.

Inspect cohort 20260815.0.0

Eleven incidents across four infrastructure agents

Benchmark 20260812.0.0 · Full tier

Archived cohort

122/131 durable repairs, 3:04 median, $6.6771+ known cost.

Inspect cohort 20260812.0.0

Durable upload recovery on an isolated host

Benchmark 20260811.0.0 · Unclassified tier

Archived cohort

8/12 durable repairs, 5:54 median, $0.4651 known cost.

Inspect cohort 20260811.0.0

Six agent configurations with execution recording

Benchmark 20260810.0.2 · Unclassified tier

Archived cohort

82/88 durable repairs, 3:24 median, $3.1828+ known cost.

Inspect cohort 20260810.0.2

Six agent configurations with execution recording

Benchmark 20260810.0.1 · Unclassified tier

Archived cohort

82/88 durable repairs, 3:24 median, $3.1828+ known cost.

Inspect cohort 20260810.0.1

Three models with execution recording

Benchmark 20260810.0.0 · Unclassified tier

Archived cohort

39/44 durable repairs, 2:42 median, $2.4652+ known cost.

Inspect cohort 20260810.0.0

Four models on the declarative host suite

Benchmark 20260809.0.3 · Unclassified tier

Archived cohort

53/60 durable repairs, 1:56 median, $0.7964 known cost.

Inspect cohort 20260809.0.3

DeepSeek on the declarative host suite

Benchmark 20260809.0.2 · Unclassified tier

Archived cohort

13/15 durable repairs, 2:28 median, $0.2908 known cost.

Inspect cohort 20260809.0.2

Eight models across five stateful incidents

Benchmark 20260809.0.1 · Unclassified tier

Archived cohort

109/120 durable repairs, 2:16 median, $12.2351+ known cost.

Inspect cohort 20260809.0.1

Seven models across five stateful incidents

Benchmark 20260809.0.0 · Unclassified tier

Archived cohort

94/105 durable repairs, 2:34 median, $5.4983+ known cost.

Inspect cohort 20260809.0.0

Harness v5 initial smoke matrix

Harness v5 discounted-model smoke matrix

One attempt per model and scenario, August 8, 2026

Superseded

DeepSeek V4 Pro, Tencent HY3 Preview, and GLM 5.2 each repaired all five scenarios once. The 15/15 result established cost and latency profiles, but the repeated 45-trial baseline exposed reliability differences that a perfect smoke run could not. Total reported cost was $0.2740 and the overall median was 3:26.

Host harness v2 baseline

DeepSeek V4 Flash 0731

Three attempts per scenario, August 8, 2026

Verified
ScenarioVersionRepairsMedianCost
Nginx 502v13/30:55$0.0093
Sidekiq wrong Redis databasev23/32:02$0.0146
Missing Rails migrationv23/34:24$0.0353
Total9/92:02$0.0592

All nine trials survived service restart and host reboot verification. They used 1,703,055 input tokens and 44,979 output tokens. This was the first baseline with the reference repair and verifier absent from the model VM.

Controlled DeepSeek revision comparison

Original V4 Flash and the 0731 revision used the same Replaybook commit, harness, scenarios, Claux release, timeout, concurrency, and attempt count. Only the model ID changed.

MetricOriginal0731Change
Durable repairs8/99/9+1 pass
Median trial time2:132:0211s faster
Input tokens3,488,4671,703,05551% fewer
Output tokens64,44844,97930% fewer
Reported cost$0.0811$0.059227% lower
The failed repair was meaningful. Original V4 Flash added the missing database column manually but did not apply or record the deployed migration. The verifier returned migration_not_applied. The 0731 revision completed the full migration repair in all three attempts.

Host harness v1 archive

Sidekiq wrong Redis database

Scenario v2, three attempts per model, August 7, 2026

Archived

These runs tested backlog preservation, but harness v1 exposed the reference repair inside the VM. They remain useful development records and must not be presented as a controlled model ranking.

ModelRepairsMedianCost
DeepSeek V4 Flash3/31:31$0.0187
Poolside Laguna S 2.13/30:57$0.0133
MiniMax M33/31:49$0.1348
Qwen3.8 Max3/34:52$0.5816
GPT-5.6 Luna2/31:12$0.0236

Archiving does not mean every agent used the leaked repair. It means the harness did not guarantee isolation, so its scores are not comparable to later isolated-host results.

Why results move here

A verifier change can make yesterday's pass invalid under today's rules. Replaybook keeps the old result, its version, and the behavior it exposed. It does not silently recalculate history or mix incompatible denominators. The full development record lives in benchmarks.md.