$ replaybook

Benchmarks / end-to-end success

One score. Full context.

A score belongs to a model, provider, harness, and benchmark contract. Not to a model name alone.

Score = 100 × average scenario success rate. Each scenario has equal weight. All recorded attempts count, including unavailable attempts. Only complete coverage of the pinned contract gets a headline score. This differs from Evidence's conditional repair rate, which excludes unavailable trials. Different benchmark suites have separate scores and are not directly comparable. Small samples are not definitive rankings; completeness is not statistical certainty. Speed and cost are not blended into the score.

Sort within each provider. Incomplete or unknown costs sort last; partial costs are lower bounds.

Core 2026.10.24.2

3 attempts per scenario · 900s deadline · matching controller and scenario fingerprints

Compare configurations, not model names alone. Each card retains its own release; attempts are never pooled across releases. Harness and provider differences can affect results.

Provider: openrouter

100.0 / 100

Reflection Beam 1.0 501B A23B

openrouter · Claux custom build b959ef677ff7

Release 20261027.0.0

High reasoning

33/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: $1.1666 · per repair: $0.0354 · median: 63s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityreflection/beam-1.0-501b-a23b

View attempts, timing and cost ↗

97.0 / 100

DeepSeek V4.1 Flash 1017

openrouter · Claux custom build b959ef677ff7

Release 20261024.0.0

High reasoning

32/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: $0.5362 · per repair: $0.0168 · median: 74s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identitydeepseek/deepseek-v4.1-flash-1017

View attempts, timing and cost ↗

93.9 / 100

Gemini 4 Argon

openrouter · Claux custom build b959ef677ff7

Release 20261025.0.0

High reasoning

31/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: ≥$9.5315 · per repair: ≥$0.3075 · median: 227s

Cost reported for 32/33 trials.

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identitygoogle/gemini-4-argon

View attempts, timing and cost ↗

90.9 / 100

GPT-6.1 Sol

openrouter · Claux custom build b959ef677ff7

Release 20261025.0.0

High reasoning

30/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: ≥$3.5375 · per repair: ≥$0.1179 · median: 81s

Cost reported for 32/33 trials.

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityopenai/gpt-6.1-sol

View attempts, timing and cost ↗

90.9 / 100

Gemini 4 Hydrogen

openrouter · Claux bounded-search 25e7b9c (binary 7a7268af30cf)

Release 20261113.0.0

Default reasoning

30/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: $5.8412 · per repair: $0.1947 · median: 84s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identitygoogle/gemini-4-hydrogen

View attempts, timing and cost ↗

84.8 / 100

MiniMax M3.1 Flash

openrouter · Claux custom build b959ef677ff7

Release 20261027.0.0

High reasoning

28/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: ≥$1.4007 · per repair: ≥$0.0500 · median: 540s

Cost reported for 32/33 trials.

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityminimax/minimax-m3.1-flash

View attempts, timing and cost ↗

72.7 / 100

Nemotron 3 Ultra 550B A55B

openrouter · Claux custom build b959ef677ff7

Release 20261025.0.1

High reasoning

24/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: $3.3572 · per repair: $0.1399 · median: 88s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identitynvidia/nemotron-3-ultra-550b-a55b

View attempts, timing and cost ↗

66.7 / 100

Ling 3.2 Flash Free

openrouter · Claux bounded-search 25e7b9c (binary 7a7268af30cf)

Release 20261115.0.0

Default reasoning

22/33 repairs · 11/11 scenarios

0 unavailable · 100.0% evaluated

Cost: $0.0000 · per repair: $0.0000 · median: 694s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityinclusionai/ling-3.2-flash:free

View attempts, timing and cost ↗

54.5 / 100

Ripple-65B-Preview

openrouter · Claux custom build b959ef677ff7

Release 20261106.0.0

Default reasoning

18/33 repairs · 11/11 scenarios

2 unavailable · 93.9% evaluated

Cost: ≥$2.3439 · per repair: ≥$0.1302 · median: 759s

Cost reported for 31/33 trials.

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityricursiveai/ripple-65b-preview

View attempts, timing and cost ↗

Observed failure rate by scenario

For these benchmark conditions and the selected harness. This describes the selected configurations, not intrinsic difficulty. Small samples, provider behavior, and execution failures affect the rates. Headline scores remain equally weighted.

26 complete configuration entries · each entry belongs to one release, model, provider, harness, and reasoning setting. Repeated configurations in different selected releases count separately.

Unsuccessful / recorded attempts, including unavailable trials. Rates combine only the complete configuration entries shown for these conditions; no model score is pooled or reweighted.
ScenarioObserved failure rateUnsuccessful / recordedConfigurationsUnavailable
discourse shared uploads021-discourse-shared-uploads · v1 16.7% 13 / 78260
nix store disk pressure028-nix-store-disk-pressure · v1 15.4% 12 / 78260
discourse interrupted deploy024-discourse-interrupted-deploy · v1 11.5% 9 / 78260
rails pool exhaustion016-rails-pool-exhaustion · v1 10.3% 8 / 78262
sidekiq poison pill015-sidekiq-poison-pill · v1 9.0% 7 / 78260
auth secret rollout023-auth-secret-rollout · v1 7.7% 6 / 78260
TLS trust rotation033-tls-trust-rotation · v2 6.4% 5 / 78260
partial rails rollout017-partial-rails-rollout · v1 5.1% 4 / 78260
rust FD leak019-rust-fd-leak · v1 5.1% 4 / 78260
nginx 502 host001-nginx-502-host · v1 2.6% 2 / 78260
sidekiq wrong redis013-sidekiq-wrong-redis · v3 1.3% 1 / 78260

Core 2026.10.15

3 attempts per scenario · 900s deadline · matching controller and scenario fingerprints

Compare configurations, not model names alone. Each card retains its own release; attempts are never pooled across releases. Harness and provider differences can affect results.

Provider: openrouter

Provider: opencode-go

Observed failure rate by scenario

For these benchmark conditions and the selected harness. This describes the selected configurations, not intrinsic difficulty. Small samples, provider behavior, and execution failures affect the rates. Headline scores remain equally weighted.

6 complete configuration entries · each entry belongs to one release, model, provider, harness, and reasoning setting. Repeated configurations in different selected releases count separately.

Unsuccessful / recorded attempts, including unavailable trials. Rates combine only the complete configuration entries shown for these conditions; no model score is pooled or reweighted.
ScenarioObserved failure rateUnsuccessful / recordedConfigurationsUnavailable
discourse shared uploads021-discourse-shared-uploads · v1 27.8% 5 / 1860
discourse interrupted deploy024-discourse-interrupted-deploy · v1 27.8% 5 / 1860
nix store disk pressure028-nix-store-disk-pressure · v1 11.1% 2 / 1860
nginx 502 host001-nginx-502-host · v1 5.6% 1 / 1860
sidekiq wrong redis013-sidekiq-wrong-redis · v3 0.0% 0 / 1860
sidekiq poison pill015-sidekiq-poison-pill · v1 0.0% 0 / 1860
partial rails rollout017-partial-rails-rollout · v1 0.0% 0 / 1860
rust FD leak019-rust-fd-leak · v1 0.0% 0 / 1860

Core Hard 2026.10.28.1 · Experimental

Experimental suite · measured separately from Core. These scenarios have not yet been established as harder; scores apply only to this pinned contract.

3 attempts per scenario · 900s deadline · matching controller and scenario fingerprints

Compare configurations, not model names alone. Each card retains its own release; attempts are never pooled across releases. Harness and provider differences can affect results.

Provider: openrouter

100.0 / 100

Claude Sonnet 5.5

openrouter · Claux custom build b959ef677ff7

Release 20261029.0.0

High reasoning

9/9 repairs · 3/3 scenarios

0 unavailable · 100.0% evaluated

Cost: $0.6679 · per repair: $0.0742 · median: 37s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityanthropic/claude-sonnet-5.5

View attempts, timing and cost ↗

100.0 / 100

Pareto 26.10

openrouter · Claux custom build b959ef677ff7

Release 20261029.0.0

Default reasoning

9/9 repairs · 3/3 scenarios

0 unavailable · 100.0% evaluated

Cost: $0.6310 · per repair: $0.0701 · median: 116s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityunbiased/pareto-26.10

View attempts, timing and cost ↗

100.0 / 100

Particle-1

openrouter · Claux custom build b959ef677ff7

Release 20261029.0.0

High reasoning

9/9 repairs · 3/3 scenarios

0 unavailable · 100.0% evaluated

Cost: $0.2917 · per repair: $0.0324 · median: 48s

Durable repairs by scenario

Scenario ID · passed / recorded. Select a cell to compare outcomes.

Exact model identityparticle-ai/particle-1

View attempts, timing and cost ↗

Observed failure rate by scenario

For these benchmark conditions and the selected harness. This describes the selected configurations, not intrinsic difficulty. Small samples, provider behavior, and execution failures affect the rates. Headline scores remain equally weighted.

3 complete configuration entries · each entry belongs to one release, model, provider, harness, and reasoning setting. Repeated configurations in different selected releases count separately.

Unsuccessful / recorded attempts, including unavailable trials. Rates combine only the complete configuration entries shown for these conditions; no model score is pooled or reweighted.
ScenarioObserved failure rateUnsuccessful / recordedConfigurationsUnavailable
Checkout dependency rollout034-checkout-dependency-rollout · v1 0.0% 0 / 930
Notification outbox rollout035-notification-outbox-rollout · v1 0.0% 0 / 930
Reservation write contention036-reservation-write-contention · v1 0.0% 0 / 930

Experimental / agent cooperation

Scenario / durable repair

Observed failure rate
Unsuccessful / recorded
Configuration entries

Compare configurations

Passed / recorded attempts. Select a configuration to inspect its run.

These are observations from the selected configurations, not intrinsic difficulty. Small samples, provider behavior, and execution failures affect the rate. Headline scores remain equally weighted.

Explore scenario history ↗

Cohorts are explicitly selected in the score configuration. No automatic newest-run or best-attempt selection.