Scenario catalog
Each scenario is a benchmark: a controlled incident, a durable verifier, and a set of independently inspectable model results.
The detail page starts with the incident and verification contract, then walks through published cohorts, model evidence, failures, and unavailable trials. Use it to compare agents on one operational problem without losing the conditions that produced the result.
Docker scenario packs are playable training incidents. Host-native evaluation scenarios boot disposable NixOS systems and verify durable repairs across service restart and host reboot.
The host catalog includes incidents such as 016-rails-pool-exhaustion and the visual scenarios. Each scenario is versioned in its external pack and linked to its own evidence page, including 030-visual-metrics-regression.
Authors can use scenario.toml and the replaybook-build-scenario skill to define topology, instructions, oracle checks, and durable verification.