$ replaybook

Scenarios

Replaybook has two scenario formats: playable Docker incidents and host-native NixOS evaluations with external durability verification.

Docker scenario packs

Playable scenarios live in separate repositories and install under ~/.local/share/replaybook/scenarios/. Add the official pack with replaybook add ducks/replaybook-scenarios.

ID	Incident	Difficulty
`001-nginx-502`	502 Bad Gateway	1
`002-postgres-rejecting-connections`	Postgres rejecting connections	2
`003-missing-env-var`	Application crashing on boot	1
`004-disk-full`	Health checks failing	2
`005-oom-kill`	Application repeatedly dying	2
`006-sidekiq-cant-connect`	Jobs not processing	2
`007-packet-loss`	Intermittent request failures	3
`008-connection-pool-exhaustion`	Checkout unavailable	3
`009-phantom-backend`	Backend not receiving traffic	3

A Docker scenario contains:

my-scenario/
  meta.json            # incident page, hints, tags, checks, fault variants
  docker-compose.yml   # deployed environment
  break.sh             # inject the fault, or use declarative break steps
  check.sh             # optional exit-zero success check
  solve.sh             # reference repair used only by replaybook test

meta.json can use an HTTP success target or an exit-zero script. Faults can be shell scripts or ordered cp, exec, and restart steps. A faults list gives one incident several hidden root causes with per-fault hints and repairs.

Scenario packs are trusted code. Their break, check, and repair logic runs with your privileges, and the workstation receives the Docker socket. Only install packs you trust.

Host-native evaluation scenarios

Replaybook includes a default host scenario pack under integrations/host/scenarios and accepts independently versioned external packs through --scenario-pack. Each trial boots a disposable NixOS VM and gives the agent normal Linux tools such as systemctl, journalctl, ps, and ss. The model VM never receives the oracle or verifier.

A pack declares a stable ID and version in replaybook-pack.toml. Matrix summaries retain that identity, and the publisher will not combine results produced from different pack revisions. Supplying any explicit pack replaces the bundled default; repeat the option to combine packs with distinct scenario IDs.

A matrix copies its selected packs and host runner into an immutable execution snapshot before launching workers. Content hashes become part of benchmark compatibility, so edits to a checkout cannot change an active matrix or be silently mixed into another published result.

ID	Version	Incident	Durable requirement
`001-nginx-502-host`	v1	Wrong Nginx upstream port	Repair the deployed proxy configuration
`013-sidekiq-wrong-redis`	v2	Web and Sidekiq use different Redis databases	Recover exact queued jobs and future work
`014-missing-rails-migration`	v2	Shipped migration was never applied	Record the migration and recover retries exactly once
`015-sidekiq-poison-pill`	v1	Poison payload blocks the only worker	Quarantine poison and preserve valid backlog
`016-rails-pool-exhaustion`	v1	ActiveRecord pool is smaller than Puma concurrency	Recover failed checkouts under concurrent load

Every repair must pass the user-facing verifier immediately, after the affected services restart, and after the VM reboots. Scenarios can also preserve opaque controller-owned IDs so an agent cannot replace customer work with its own test data and pass on a matching count.

Declarative lifecycle

New host scenarios use scenario.toml to declare topology, files, preflight steps, verification steps, state, and failure categories. Supported phases can wait for HTTP assertions, generate concurrent controller-owned requests, and replay exact failed IDs.

[scenario]
version = 1
nixos_config = "nixos.nix"
instruction = "instruction.md"
oracle = "oracle.sh"
required_services = ["postgresql.service", "checkout-web.service"]
restart_services = ["checkout-web.service"]

[[verify.steps]]
type = "wait_http"
path = "/pool"
body_integer_min = 4
failure_category = "database_pool_exhausted"

All bundled host-native scenarios use the typed manifest. Legacy scenario.conf plus shell preflight and verifier hooks remain supported for external scenarios that have not migrated yet.

Build your own

Playable Docker scenario

replaybook new checkout-db-exhaustion --pack ./company-incidents
replaybook validate ./company-incidents/checkout-db-exhaustion
replaybook test ./company-incidents/checkout-db-exhaustion

The interactive author asks for the incident page, difficulty, tags, hints, learning objectives, fault, repair, success check, and optional provenance. Replace the starter service with sanitized behavior from the real incident, then run replaybook test --all in pack CI.

Host-native evaluation scenario

Replaybook includes a Codex skill that scaffolds the NixOS topology, instruction, oracle, declarative verifier, failure categories, leak checks, and deterministic validation.

mkdir -p ~/.codex/skills
ln -s "$(pwd)/skills/replaybook-build-scenario" \
  ~/.codex/skills/replaybook-build-scenario

Then ask Codex:

Use $replaybook-build-scenario to build a scenario for <incident>.

Run the oracle before spending model tokens. A good verifier asks what state must survive, which shortcuts should fail, whether an empty queue is being mistaken for recovered work, whether reboot succeeds, and whether any agent-visible artifact leaks the answer.