Methodology

Replaybook evaluates the state of a running system after an agent leaves, not the confidence of its final answer.

Repair the live system

The user-facing behavior must recover without replacing the host.

Preserve existing work

Queued jobs and controller-owned state cannot be discarded.

Survive service restart

Temporary processes, shell state, and in-memory workarounds disappear.

Survive host reboot

Only changes made durably to the deployed system remain.

Who owns what

The controller builds a disposable NixOS VM and owns the incident instruction, timeout, lifecycle, restart, reboot, oracle, and verifier. The agent adapter invokes a harness inside the VM and returns normalized output. An adapter can change how an agent runs. It cannot redefine what counts as success.

The oracle and verifier remain outside the model VM. Replaybook checks for answer-key leaks before starting the agent and preserves the complete tool transcript for later inspection.

Evaluated, failed, and unavailable

An evaluated trial reaches the agent and the external verifier. A failed repair counts against the model and carries a category such as migration_not_applied, backlog_not_recovered, or agent_timeout.

Provider HTTP errors, authentication failures, and harness runtime errors are unavailable attempts. They remain visible in trial and cost totals, but are excluded from the model's pass-rate denominator. A provider refusing a request is not evidence that the model failed a repair.

Price per durable repair

Reported cost includes every evaluated attempt, including failed repairs. Replaybook divides that spend by durable repairs, not by invocations. A cheap attempt with poor reliability can therefore cost more per repair than a more expensive attempt that succeeds consistently.

price per durable repair = evaluated trial spend / verified repairs

Subscription-backed harnesses may report tokens without a dollar cost. Replaybook records that cost as unavailable, never as zero.

Comparable results

A leaderboard-style comparison requires:

the same frozen scenario and verifier versions;
the same host harness version and agent timeout;
the same agent harness, prompt, tools, and permissions;
at least three attempts per model and scenario;
complete cost, duration, and failure provenance;
sanitized artifacts suitable for independent inspection.

One-attempt smoke runs select candidates and expose gross failures. They do not establish reliability. Results from different verifier versions belong in History, not in the same pass-rate table.

Harness versions

Version	Change
v2	Removed the reference repair and verifier from model VMs.
v3	Added typed scenario manifests and structured verifier failures.
v4	Added the harness adapter contract and normalized artifacts.
v5	Separated unavailable attempts from evaluated model failures.
v6	Moved host incidents to declarative scenario definitions executed by the shared controller.
v7	Recorded independently versioned scenario-pack provenance as part of benchmark compatibility.
v8	Snapshotted runner, scenario packs, and agent artifacts before launching workers.
v9	Recorded Claux reasoning effort as a distinct matrix dimension.
v10	Kept provider credentials outside incident VMs behind a host-side forwarding proxy.
v11	Recorded provider-round and tool-call timings for adapters that expose them.

Inspect the host-native runner or read the full benchmark record.