Frontier results from lesser models. A calibrated critic in the loop.
The question TypeSafe's launch post asks — can a System One model stand in for a frontier LLM on a real decision — measured on a decision I actually pay for: will this coding agent's edit pass its test suite? 1,554 graded Pillbox runs, four deciders, one question, every number recomputable from the published data.
Start a five-minute session.
A per-visitor Durable Object binds the actor, the spend budget, and an ordered event log before the first model token is spent. Then pick a held-out agent run and watch the cascade decide — and get revealed.
Four deciders, one question, calibration first.
Accuracy tells you if a decider is right. Brier and ECE tell you whether its confidence means anything — the number a cascade actually routes on. Jev returns a probability natively; the LLM tiers verbalize one, which is the thing TypeSafe says not to trust. Latency and cost are measured per call from the vendor's own usage report.
Spend the frontier model only where the cheap one hesitates.
Walk the tiers in order; accept the first answer at least τ away from a coin flip; the last tier always accepts. Slide τ and the whole frontier recomputes in the browser from the published per-item scores. If the cheap tier is calibrated, the curve hugs frontier accuracy at a fraction of the cost. If it isn't, you can see exactly where it lies.
Accuracy vs $ per 1,000 decisions
Ghost with a cheap critic: pick the winner before paying the verifier.
Several tasks have dozens to hundreds of independent attempts from the same cheap coding model. Draw N of them, let each critic pick its argmax, and count whether the pick passed — against a random pick (the base rate) and an oracle (any passer in the draw). This is Pillbox's dispatch --critic-policy select, simulated on runs already paid for.
Selected pass-rate vs N
The critic is a flag on pillbox dispatch.
Pillbox's Ghost loop forks k workers, grades each with a verifier, and pulls the winner. The verifier is the reward — exact, exit-derived, unforgeable — and also the slow last step. --critic typesafe puts a calibrated probability in front of it, and because the verifier still runs on every graded worker, every dispatch becomes a labeled (p, passed) pair: the harness measures its own critic for free.
# the verifier still decides; the critic reorders what it's spent on export TYPESAFE_API_KEY=… pillbox dispatch --from-bookmark seg-3 -k 4 --rubric grade.txt \ --agent opencode --temperature 0.7 \ --critic typesafe --critic-policy order \ -- "Implement the parser for segment 3." # verdict --json (additive; byte-identical without --critic) "critic": { "kind": "typesafe", "policy": "order", "model": "jev-latest", "verifier_runs": 1, "verifier_runs_saved": 3 } "workers": [ { "session": "…", "status": "scored", "critic": { "p": 0.91, "latency_ms": 180, … } }, { "session": "…", "status": "unverified", "critic": { "p": 0.12, … } }, … ] # and on each worker's own §0 log, class=signal (poolable) pillbox session log <id> --type artifact | grep critic_verdict
Three policies. record scores every worker before each grade and changes nothing — measurement. order settles all forks to idle, grades in descending p, and stops at the first passer; the rest are unverified. select grades only the argmax — if it fails there is no winner, and the exit code says so. The honest cost of trusting the critic is visible, not hidden.
Same state, offline and live. The critic sees the worker's edits from its own §0 log, rendered exactly the way this page's harness renders them (a golden test pins the format). A Brier number measured here is the Brier number the loop gets.
Rust, unit-tested over the mock driver like the rest of the loop: record leaves sends/grades identical; order stops at the first passer and falls through when the critic is wrong; select can end with no winner; a critic HTTP failure never turns a gradeable worker into an errored one. Docs updated first, per the repo's own rule.
1,554 graded agent runs, published.
Every row is a real Pillbox session: an opencode worker's final edits on an aider-polyglot / sensitivity / toolz task, and the rubric verifier's verdict from the session's own scored event. Paths and injected memory preambles stripped; grader output dropped (it leaks the label). Split is by task, so no decider can pad its score on near-duplicate attempts.
- heldout.jsonl — the rows (prompt, edits, label, split)
- results.json — every number on this page, regenerable with
scripts/analyze.mjs - rows.index.json — the picker (no labels, no edits)
- Verbalized vs native confidence. LLM tiers are asked for a number; Jev returns one. That asymmetry is the point of the comparison, and it's stated.
- Shell writes aren't recovered.
cat > f/sed -iedits don't appear in the state. Same blind spot for every decider. - Truncation. States over 14k chars are cut from the edit tail; the count is in
results.json. - One generator. Almost every run is
opencodeon a small set of tasks. Ranking within a task is what best-of-N measures; generalization across generators is not claimed. - Prices move. $/1k uses each vendor's billed or listed price on the run date, recorded per row.
Building the measurement is the job. This page is the measurement.
Data platforms at MetaMask and Meta, an eval-and-observability harness for a production data agent at Phantom, and a shelf of agent runtimes that treat calibration as the ship gate.