# Reproduction bundle — typesafe.vu1n.dev

Everything on the page regenerates from what is in this directory, which mirrors the
source repository's layout (data/ · deploy/ · scripts/). No API keys are needed to
reproduce the numbers; keys are only needed to re-score.

    npm test          # metrics + analysis unit tests
    npm run regen     # scripts/analyze.mjs + scripts/evolution.mjs
                      # → deploy/public/data/results.json and evolution.json

    data/scores/       one JSONL per decider or variant: id, task, split, y (verifier label),
                       p (decider's P(pass)), latency, per-call cost from the vendor, usage.
                       jev-v*_*.jsonl are the evolution ladder; jev-v6/v7 also carry the
                       LLM-written checks and Jev's per-check answers.
    data/heldout.jsonl the 1,554 rows (prompt, agent edits, verifier label, task-level split)
    data/bestofn_index.json task → candidate ids for best-of-N draws
    data/summary.json  dataset build stats
    data/extract_v3.py how heldout.jsonl was built from Pillbox §0 session logs
    scripts/reconstruct_files.py how final files / visible tests were rebuilt for v3+ variants
    deploy/            metrics.js (Brier, ECE, AUC — ported from granola/eval/metrics.py),
                       analysis.js (cascade, best-of-N with seeded draws), deciders.js
                       (the exact prompts / questions / API calls), variants.js (the ladder),
                       and their tests
    scripts/           analyze.mjs, evolution.mjs, earlykill.mjs (regeneration); precompute.mjs,
                       advisor.mjs, compare.mjs, score_prefixes.mjs (how scores were produced —
                       re-scoring needs keys in deploy/.dev.vars); build_prefixes.py rebuilds the
                       early-kill prefix set (every run cut at each edit, with elapsed-time
                       fractions) from Pillbox §0 logs — the prefix states themselves are not
                       shipped (they are reconstructed workspaces); scores/jev-prefix.jsonl is.

Protocol notes that matter for reading the numbers:
- Split is by TASK. Tasks with ≥40 rows are the test split (6 tasks, 1,195 rows); the rest
  are dev (359 rows). Learned combiners ("weights fit on dev") never see test rows.
- Best-of-N compares selectors on ONE shared pool: test rows scored by every decider, same
  seeded draws for each selector (seed 11, 300 draws per task per N).
- Constant baseline: always answering the test base rate gives Brier p(1−p) = 0.250 on
  these rows. Baseline Jev (0.279) is below that floor; the dev-fitted composite (0.246)
  clears it. Cost ratios compare per-call cost of the specific variant: the composite is
  one Jev call with six questions ($0.102 per 1k) vs Claude Sonnet 5 ($4.72 per 1k) → ~46×.
- Early kill (Experiment 4): the combiner is fit on dev-split prefixes only; the sweep is over
  test runs, killing at the first edit whose score falls below θ; "saved" is the run's own
  recorded remaining wall-clock at that edit. Winners killed = passing runs killed / all passing.
- Baseline Jev's p≥0.8 tail: 107 test rows, all passed (mean p 0.82 → observed precision
  1.00, i.e. under-confident there, not "perfectly calibrated"); 70 are ap_dot_dsl and 37
  ap_grade_school.
- Costs are the vendor's per-call billed amount (OpenRouter `usage.cost`), Workers AI
  neurons × $0.011/1k, or Jev input tokens × $0.042/M. Latency is wall-clock per call.
- Known blind spots: shell-written files (`cat > f`, `sed -i`) are not in the edit list;
  states over 14k chars are truncated from the edit tail (count in results.json);
  visible tests exist for ~15% of rows; nearly all rows come from one generator
  (opencode) on a small task set.