Replay real incidents. Verify real repairs.

Practice production failures yourself, or put an AI agent inside a fresh Linux host and measure what survives after it leaves.

PLAY

Practice incidents

Enter a broken Docker environment with real services, an SLA, hints, and a terminal transcript. Diagnose the failure and repair it.

EVALUATE

Test infrastructure agents

Run Claux, Codex, or another harness on a disposable NixOS VM. An external verifier checks the repair after restart and reboot.

Replaybook began as a way to turn postmortems into playable incidents. The same idea now supports repeatable agent evaluation: reproduce a stateful failure, preserve the work already in flight, hide the answer from the agent, and verify the deployed system instead of its final explanation.

Current isolated-host matrix: 7 durable repairs in 12 trials. Four models attempted the same interrupted Discourse deployment for $0.9276 in total reported cost. See the current results.

Practice an incident

cargo install replaybook
replaybook add ducks/replaybook-scenarios
replaybook list
replaybook run 001-nginx-502

Playable scenarios require Docker. Replaybook opens a workstation container on the incident network and records the complete terminal session. Prebuilt binaries for Linux and macOS are available on the releases page.

Evaluate an agent

python integrations/host/run_host_matrix.py \
  --scenario 014-missing-rails-migration \
  --models deepseek/deepseek-v4-pro \
  --attempts 1

Host-native evaluation runs from a Linux controller with Nix and QEMU. Each trial gets a new NixOS VM with real systemd services and no Docker socket. Claux is the default adapter; the normalized adapter contract can run another harness without changing the verifier.

Next: read the usage guide, explore the two scenario formats, or inspect the benchmark methodology.