the replay
one hash-chained event log, replayed event by event: every task attempt, the cost of each arm against its solve rate, and the trail of events behind any point.
no data loaded
could not fetch the data file (browsers block fetch from file://).
open an export made with scripts/export_ui_data.py, serve this folder over http, or build a standalone page with --html.
the question
when a cheap model fails, does escalating to a stronger model beat simply trying the cheap model again?
when a worker fails, the log forks and the task re-runs twice from the same point: once with the same cheap model (the contraprova, the control) and once with a stronger one. the gap between them is what escalation actually buys.
method
a strong model plans and reviews; claude haiku 5.5 does the narrow jobs. the orchestrator reads outlines only and writes one brief with a file scope and a test command. a haiku worker does the job inside that scope, and a gate checks the changed files, refuses edits to tests, and runs the visible and private tests. a pass ships; a block forks the log into the control and the escalation. every step is an event in an append-only, hash-chained log, and cost is never stored, only tokens, re-priced on demand.
the main study ran fresh swe-bench pro tasks, paired by task: arm A is a single opus 5.5 at low effort; arm D is the governed cascade; arm B is derived from the first try of D's haiku worker. this page replays that log.
pre-registration
the main study was pre-registered: PREREG.md was written and committed (f43ad38) before any paid request of the run, on 2026-11-12. deviations and incidents are disclosed in the study's report.
what this study found
cost per task against solve rate, the whole study
what this rests on
| claim | rests on |
|---|
the log, event by event
- event
- 0
- attempts shown
- 0
- gate blocks so far
- 0
- verdict block, any arm
- fork re-runs so far
- 0
- control and treatment
what am i looking at?
every figure is a projection of one hash-chained event log, computed in python and exported as json. the timeline replays the log: everything shows the state as of event n.
attempts: one point per arm × task, including fork re-runs, labelled from each fork's role (D:control = the same cheap model again, D:treatment = escalated; a second control at the same fork point is D:control-2). fork re-runs are fork events and fork outcomes in the log, never arms. position is a deterministic pca layout of token, request and gate counts, not semantic meaning. colour = one logged fact; size = cost. click a point for its event trail.
cost against solve rate: per arm, with 95% wilson intervals as stippled bands; the ink step is the pareto frontier over arms A–D. hollow = fork re-runs, which cover only failed-once tasks and are not comparable with the arms. switch price tables to re-price the same history.
attempts
cost per task against solve rate
the timeline
- gate block
- escalation to a stronger model
- fork re-run started
- outcome: solved
- the scrubber