$ replaybook

Stateful infrastructure evaluation

Replay real incidents. Verify real repairs.

Practice production failures yourself, or put an AI agent inside a fresh Linux host and measure what survives after it leaves.

PLAY

Practice incidents

Enter a broken Docker environment with real services, an SLA, hints, and a terminal transcript. Diagnose the failure and repair it.

EVALUATE

Test infrastructure agents

Run Claux, OpenCode, Codex, or another harness on a disposable NixOS VM. An external verifier checks the repair after restart and reboot.

Replaybook began as a way to turn postmortems into playable incidents. The same idea now supports repeatable agent evaluation: reproduce a stateful failure, preserve the work already in flight, hide the answer from the agent, and verify the deployed system instead of its final explanation.

Latest published cohort: 22 durable repairs in 33 trials.

Ling 3.2 Flash Free Core via Claux / OpenRouter (20261115.0.0).

33 evaluated, 11 failed, 0 unavailable. Ling 3.2 Flash Free passed 22 of 33 Core trials (66.7%), with a 694-second median agent duration and $0 in provider-reported inference cost. Eleven attempts exceeded the deadline; seven left repairs that passed subsequent lifecycle verification but remain failures.

Explore Core scores · Inspect this cohort · All published releases

Practice an incident

cargo install replaybook
replaybook add ducks/replaybook-scenarios
replaybook list
replaybook run 001-nginx-502

Playable scenarios require Docker. Replaybook opens a workstation container on the incident network and records the complete terminal session. Prebuilt binaries for Linux and macOS are available on the releases page.

Evaluate an agent

python integrations/host/run_host_matrix.py \
  --scenario 014-missing-rails-migration \
  --models deepseek/deepseek-v4-pro \
  --attempts 1

Host-native evaluation runs from a Linux controller with Nix and QEMU. Each trial gets a new NixOS VM with real systemd services and no Docker socket. Claux is the default adapter; the normalized adapter contract can run another harness without changing the verifier.

Next: read the usage guide, explore the scenario catalog, or inspect the benchmark guide.