06 · Budget, selection and judging

06 · Budget, selection and judging

In short. Harness² compares two ways of improving a task-executor's harness (a parallel mode and a sequential mode) and the comparison is only fair if both spend exactly the same. A small ledger records every executor rollout and every improver session as it happens and refuses to finish unless the totals are exactly R = 4k rollouts and P = 3k improvement sessions for the scale k. After that, a blind selector picks one of the mode's k composed final outputs: it sees anonymised, shuffled artifacts (never labels, harnesses or scores), and its pick is then reflected on to write a lesson into a per-domain experience library. Only after all of that is finished, and from a separate command, does a benchmark judge score the chosen output. Nothing in the method's run path imports the judge. A judge failure is recorded as None and left out of the denominator; it is never a zero.

All code references are pinned to commit 1cfe254. Files covered in full: harness2/core/ledger.py, harness2/core/selection.py, harness2/core/judge.py, harness2/core/report.py, harness2/cli/judge.py and harness2/cli/report.py. Also covered: the parts of recursion.py, schedule.py, library.py, context.py and prompts.py that call them.

Where this chapter sits in a task's life

Each task goes through the same stages. The table shows which module owns each one and whether it has any view of a score.

stateDiagram-v2
    [*] --> Base: run_task rolls base1..basek
    Base --> Candidates: 2k stages INV/DOM, each rolled once
    Candidates --> Compositions: k stages agg_i or c_t
    Compositions --> Finals: each composition rolled once
    Finals --> BudgetCheck: assert_identity
    BudgetCheck --> ModeFailed: BudgetError
    BudgetCheck --> Selected: run_selection or trivial_selection
    Selected --> Reflected: reflect_adoption, if a library is attached and sel.ok
    Reflected --> Library: Library.add, if has_experience
    Selected --> Summary: summary.json
    Library --> Summary
    Summary --> Judged: cli.judge, a separate process, later
    Judged --> Reported: cli.report
    Reported --> [*]
    ModeFailed --> Summary: mode recorded ok false

In the sequential mode, "Candidates → Compositions → Finals" repeats k times as nested rounds (inv_t, dom_t, then c_t). The totals come out the same.

Part 1: The budget identity

The two constants

The whole budget is two one-line functions in ledger.py:

def expected_rollouts(k: int) -> int:
    """R = 4k: k shared base + 2k candidates + k finals."""
    return 4 * int(k)


def expected_sessions(k: int) -> int:
    """P = 3k: k INV + k DOM + k compositions (parallel: agg1..aggk; sequential: c1..ck)."""
    return 3 * int(k)

(expected_rollouts, expected_sessions)

The module docstring (L15-L19) states what is counted and what is not: "Base executions count toward each mode. Retries are separate invocation counts; selection and reflection are recorded outside this ledger."

Deriving 4k and 3k from the drivers

The constants are not just declared. They are what the two recursion drivers actually do, as you can check by tallying every ledger.rollout / ledger.session call.

Parallel mode (run_parallel, on-disk token agg):

Sequential mode (run_sequential, token seq). There are k rounds t = 1..k, and each one builds on the harness from the round before (base, then c1, c2, …):

So both modes get the same number of executor rollouts and the same number of improver sessions, and each produces exactly k finals (agg1..aggk vs c1..ck, see L434 and L564). The difference between the modes is only the shape of the spend: width (independent compositions over a shared pool) against depth (nested rounds). This is why the comparison is fair. If one mode were allowed one extra rollout or one extra improver session, any gain could simply be the extra compute.

Bases are shared, but counted in both ledgers. run_task rolls base1..basek once, in the parallel namespace, and passes the same RunViews to every mode. Each mode's ledger records them (L360-L361, L469-L470). A run of both modes therefore makes k + 3k + 3k = 7k distinct executor runs, but each mode is charged 4k. That is the right number for comparing the modes: each mode alone would have needed its own k bases.

What counts and what does not

Selection and reflection are left out because both modes do them in the same way, once per mode. They are bookkeeping around the comparison, not part of the improvement process being compared. Their dollar cost is still recorded (on the selection record) and shows up in the report's spend line (see Part 4).

Dollar accounting. Every Entry carries usd, but in practice only sessions get a value: _record_sessions passes stage.usd, while _roll/_roll_many and the base loop call ledger.rollout(label) with no cost. So Ledger.rollout_usd is always 0.0 in this release, and Ledger.usd equals the improver session spend. Executor cost is not tracked by the ledger. (RunView has a cost_usd field, but _roll does not fill it.)

Entry and Ledger, field by field

Entry is one line in the ledger:

Ledger is "Everything one mode spent on one task, recorded as it happens."

to_dict keys: mode, task_id, k, R, expected_R, P, session_attempts, expected_P, usd, rollout_usd, session_usd, rollouts, sessions. session_attempts is the honest call count (retries included), so you can see the retry overhead without it changing P.

Recording order under concurrency

Stages and rollouts run in thread pools. The ledger is written only after each pool has drained, in the caller's list order. _record_sessions says "Record stages in stable order after their concurrent phase finishes", and _roll_many says "The ledger is written after the pool drains, so a concurrent run records the same sequence of labels as a serial one." Inside the pool, _roll is called with record=False (L332) and the labels are appended afterwards (L337-L339). As a result the ledger is deterministic and the same for every worker count, and the label lists in summary.json can be compared directly across runs.

When it doesn't balance: BudgetError

def assert_identity(self) -> None:
    want_r, want_p = expected_rollouts(self.k), expected_sessions(self.k)
    rl, sl = self.labels("rollout"), self.labels("session")
    dupes = sorted(
        {label for label in rl if rl.count(label) > 1}
        | {label for label in sl if sl.count(label) > 1}
    )
    if not dupes and self.R == want_r and self.P == want_p:
        return
    ...
    raise BudgetError("\n".join(detail))

(L89-L113)

• BudgetError subclasses AssertionError. It means the code broke its own invariant, not that a model misbehaved.
• Duplicates are checked before the counts, "because a duplicate can satisfy them". Suppose two stages both wrote inv1 and nothing wrote dom1. The count would still be 4k, but one run directory would have silently overwritten the other. The error message spells this out (L107-L112).
• The message lists mode-k task: R=… (want …), P=… (want …) plus every rollout and session label, so you can see what is missing or extra.

What happens next: assert_identity is the last line of each driver, so the BudgetError escapes recursion.run_mode into _run_one_mode. There it is caught by run_task's per-mode except Exception (L654-L660). The mode is written as {"ok": False, "error": "BudgetError: …", "traceback": …}, an error is appended, no selection happens for that mode, and the task's status becomes partial. Nothing is judged for that mode, because collect_selections finds no selection record. Every failure mode of the improver (a fallback, an empty reply, a dead rollout) still produces exactly one entry, so a BudgetError in practice points to a programming bug, not to bad luck.

The test suite checks the identity end to end: test_recursion_budget_and_resume asserts (ledger["R"], ledger["P"]) == (4 * k, 3 * k) for both modes at k = 1 and 2 (L73).

Part 2: Blind selection

Who calls it

_run_one_mode runs right after the recursion driver returns:

candidates = selection.selection_pool(mode, k, r.rollouts, base)
if pool_policy == "finals" and k == 1:
    sel = selection.trivial_selection(...)
else:
    sel = selection.run_selection(..., prop=selector, ..., pool_kind="finals")
if mode_lib and sel.ok:
    selection.reflect_adoption(...)

(L513-L538)

The selector proposer comes from _selector_for. It uses the adapter's "selector" channel spec and reuses the improver proposer only if the specs match. run_task accepts only pool_policy="finals" (L595-L599).

Step 1: the pool is exactly the k finals

selection_pool:

del base
labels = (
    [f"agg{i}" for i in range(1, int(k) + 1)]
    if mode == "agg"
    else [f"c{t}" for t in range(1, int(k) + 1)]
)
return {lbl: rollouts[lbl] for lbl in labels if lbl in rollouts}

The pool contains no bases and no intermediate inv/dom candidates, only the k composed finals. The docstring says this means "there is no abstention by construction": the selector cannot fall back to the original harness's output. base is accepted and then deleted on purpose, "so that exclusion is visible where the pool is built."

Step 2: drop non-answers (_render_pool)

_render_pool turns RunViews into rendered text:

1. A candidate is dropped (its label goes into sel.dropped) if not (rv.ok and rv.delivered), if adapter.deliverable_text raises (recorded as "<label> (render failed: …)", truncated to 200 chars), or if the text is blank.
2. Fallback for state deliverables. If nothing survived and the benchmark declares spec.state_deliverable (tasks whose "answer" is a changed environment rather than a file), each ok candidate's trajectory is shown instead, prefixed with a note that the task produced no file deliverable. sel.evidence becomes "trajectory" (default "deliverable"), and the labels that made it back in are removed from dropped.

This happens before anonymisation, so the anonymous id letters are only ever given to real answers. sel.n_candidates is the surviving pool size. If it is empty, run_selection returns at once with error="no candidate produced an artifact" and ok=False (L288-L292).

Step 3: anonymise with a seeded shuffle

assign_ids and _anon_id:

seed = int(hashlib.sha1(f"{task_id}|{k}|{mode}".encode()).hexdigest()[:12], 16)
ordered = sorted(labels)
random.Random(seed).shuffle(ordered)
return {_anon_id(i): lbl for i, lbl in enumerate(ordered)}

• Canonical sort first, then shuffle. The result does not depend on dict order or PYTHONHASHSEED.
• Seed = first 48 bits of SHA-1 of task|k|mode. Reruns of the same task give the same mapping, so resumes and caches line up ("stable across reruns"). The mapping differs between tasks and between modes, so A does not systematically mean agg1. Note that the sweep's --seed (stream order) is not part of this seed.
• Ids are spreadsheet-style: A..Z, AA..AZ, BA… (index 0 → A, 25 → Z, 26 → AA, 52 → BA, 702 → AAA). Every id matches the parser's [A-Za-z0-9_-]+ (L80).

Computed with the same code (outside the repo), here is what the shuffle gives on the demo tasks:

Step 4: build the selector's evidence directory

The rest of run_selection sets up a fresh directory work_dir/selection/<mode><k>/. It is wiped with rmtree first so stale files never leak in.

selection/agg2/
  task.md                 adapter.task_prompt(task_id) -- the request verbatim
  source/                 context.copy_source(...) -- task materials, when supplied
  candidates/A.md         header comment + COMPLETE rendered deliverable
  candidates/A/<asset>    optional rendered evidence (adapter-named files only)
  candidates/B.md
  selection.json          written afterwards by _persist

_write_candidates writes one file per anonymous id. Each starts with a comment like <!-- candidate A: 12,345 chars, 210 lines. COMPLETE -- nothing trimmed. … -->. Assets are copied only if the adapter names them through an optional deliverable_assets(run_dir) hook. The docstring explains why: "a path into the run dir would also carry the grader's own output." The wording for assets comes from optional adapter hooks deliverable_assets_note / deliverable_assets_line, with defaults _DEFAULT_ASSETS_NOTE / _DEFAULT_ASSETS_LINE, through _assets_note, _assets_line and _evidence_block. An adapter flag deliverable_assets_reference_only changes the wording from "rendered evidence" to "raw originals".

What the selector never sees: labels (agg1, c2), harness trees, trajectories (except in the state-deliverable fallback), costs, the other mode's outputs, the ledger, and above all any score. Two guards enforce the last point:

• Prompt side: ANSWER_FREE_SELECT tells the model that there is no rubric, answer key or score, and that it must never reason about "what the grader wants". POOL_FINALS_SELECT tells it the ids are arbitrary and shuffled.
• File side: context.assert_no_grader_verdicts(root) is called on the whole directory just before the session (L330). It scans file names and the contents of text-like files for GRADER_VERDICT_MARKERS (deterministic_eval_summary, "pass_rate", …) and raises ValueError on any hit. This call sits outside the try, so a leak is a crash, not a fallback.

Step 5: the prompt and the reply format

The mission is prompts.select(bp, n, pool_kind="finals", assets_note=…), plus a # YOUR EVIDENCE block listing the files and ending with "Open every candidate before choosing… Choose exactly one of: A, B." (L310-L328). The prompt orders the criteria: delivery → conformance (unless selection_conformance == "value_only") → fidelity against context/source/ → internal consistency → operational fitness → finish, and "Length and polish are not merits in themselves." The required output (L1095-L1099):

<why>
At most 6 lines. Name the criterion that decided it, and the closest runner-up ...
</why>

```choice
<the id of the single candidate to submit>
```

It is parsed by the module-level regexes (L33-L36):

The session runs with label select.<mode><k> (L333) and reply.cost_usd is added to sel.usd.

Step 6: failure handling

try:
    reply = prop.call(mission, root, f"select.{mode}{k}")
    sel.usd += reply.cost_usd
    m = _CHOICE.search(reply.text)
    if not m:
        raise ValueError("no ```choice block in the reply")
    anon = m.group(1).strip()
    if anon not in sel.id_map:
        raise ValueError(f"chose {anon!r}, which is not one of {ids}")
    ...
    sel.ok = True
except (ProposerError, ValueError) as exc:
    if isinstance(exc, ProposerError):
        sel.usd += exc.cost_usd
    if not _fallback(sel, pool, pool_kind, exc):
        return sel

(L332-L350)

There are three ways to fail: the proposer gives up (ProposerError, whose spend is still charged), there is no choice block, or the chosen id is not in the pool. _fallback picks a neutral stand-in: the alphabetically first base label in the pool if there is one, otherwise, for pool_kind == "finals", sorted(pool)[0] (so agg1 or c1). It sets fallback=True and records the error. With a finals pool the base branch can never fire, because the pool has no base* labels.

Important subtlety: _fallback never sets sel.ok = True. A fallback selection is persisted with its chosen_label, but it stays ok=False. Downstream that means:

• _run_one_mode appends "<mode>: selection failed: …" to the task errors (L550-L551), and the task status is partial.
• No reflection runs, because reflect_adoption requires sel.ok, and nothing enters the library.
• The judge does not score it: judge_selections turns any ok=False record into an ungraded row (judge.py L101-L113). The test test_failed_selection_is_never_judged checks exactly this: the selector raises, adapter.score is never called, and the row has graded=False, n_passed=None. The selector's $3 is still charged to selections[mode].usd.

The docstring sums up the policy: "A selector failure falls back… and never raises; an adapter bug does raise, because it is a bug and not a selection outcome." Exceptions from adapter.task_prompt, copy_source, _write_candidates or the verdict tripwire are not caught in selection.py. They propagate to run_task's per-mode handler.

After a choice (or fallback), chosen_run_id and chosen_is_base are filled (L352-L353) and _persist writes selection.json (the full record plus chosen_run_dir). A disk error does not raise: it sets ok=False and appends could not persist selection: … to error.

trivial_selection: k = 1 has nothing to choose

With k = 1 the pool holds one final, so trivial_selection delivers it without a selector session (no spend). It applies the same drop rules through _render_pool, raises ValueError if handed more than one candidate, and sets trivial=True, chosen_id="A", id_map={"A": label} and a fixed why. It writes selection.json at the same path and in the same schema, "so judge watchers find it once." If the single final produced nothing, the record is ok=False with "finals-only pool has no base fallback". The test above also asserts that no select.agg1 / select.seq1 session ever happens (L88-L94).

The Selection record

Selection is what ends up in summary.json under selections.<mode>. _run_one_mode adds three more keys: chosen_run_dir, chosen_tree_is_base and harness_fingerprint (L539-L549).

to_dict is a plain copy of __dict__.

order_graded_first, a helper that lives here but is used elsewhere

order_graded_first(items, graded) stable-sorts items so that those where graded(x) is true come first, keeping relative order and discarding nothing. Nothing in the selection path calls it. Its only caller is the WorkBuddy adapter's _rank ("Newest-first, graded preferred but never filtered out"), which orders an executor's trial directories. The code does not explain why it lives in selection.py.

Part 3: Reflection, experience and the library

Selection compares outputs with the harnesses hidden. Reflection is the step after adoption, and it is deliberately not blind: it looks at what each harness edit actually did. Its product, the "experience", is what later tasks in the same domain read before they make their own edits.

reflect_adoption

reflect_adoption(task_id, sel, candidates, *, adapter, prop, work_dir, decisions) runs only if a library is attached and sel.ok (schedule L529-L538). It is passed all of the mode's rollouts (r.rollouts: bases, candidates and finals), not just the selection pool. It also gets decisions: each stage's <notes> text, or its error.

1. It returns {} unless sel.ok and the chosen label is among the candidates.
2. It builds work_dir/reflection/<mode><k>/ with context.build (Chapter 05 covers the context builder). The adopted harness is harnesses/<chosen_label>, the pool is every non-base label with its harness dir and rollout, and the decision notes are included.
3. It copies the adapter's base harness to base_harness/.
4. It writes adoption.json with chosen_label, candidate_ids (the id_map) and reason (the selector's why). This lets the reflector translate "B was better because…" into real labels.
5. It runs two tripwires: assert_no_leak_paths (benchmark-declared answer-key globs) and assert_no_grader_verdicts.
6. It calls _distil, stores the result on sel.experience, and re-persists selection.json in the selection directory with the experience included.

_distil: two tagged records

_distil builds the mission from prompts.experience(bp) plus a # WHAT HAPPENED block that names the adopted label and quotes the selection reasoning. It runs in a copy of the reflection dir (<root>.experience, ignoring any selection.json), after one more verdict tripwire, under label experience.<mode><k>.

The prompt warns: "Selection compared DELIVERABLES with the harnesses hidden, so the choice says which output read best -- not which edit worked." It asks for (L1150-L1160):

<edits>
component: <name as it appears in the harness tree>
candidates: <the labels that carried this edit>
change: <what the edit did>
observed: <the execution difference, and where you saw it>
verdict: <helped|hurt|no observed effect|confounded>
(repeat these five lines for each changed component)
</edits>

<experience>
cue: <the shape of situation this applies to>
lesson: <the transferable operation, naming the component>
evidence: <the edit rows this rests on, and the execution contrast>
confidence: <high|medium|low, and why>
</experience>

If there is no reusable contrast, the model writes NO_ENTRY instead of the <experience> block.

Parsing:

The reply's cost is always added to sel.usd, which is why Selection.usd = selection + reflection.

Note that the tripwires in reflect_adoption/_distil raise ValueError, and selection.py does not catch it. It reaches run_task's per-mode handler and marks the mode as failed. A leak is treated as a bug, as in selection.

What reaches the library

Back in _run_one_mode:

if mode_lib and sel.ok and library_store.has_experience(sel.experience):
    mode_lib.add(task_id, chosen_tree, why=sel.why, label=sel.chosen_label,
                 run_id=sel.chosen_run_id, experience=sel.experience)

• has_experience requires non-blank string cue, lesson and evidence.
• Library.add checks the tree and the experience text for grader verdicts again, writes the full adopted harness tree to harnesses/<task>/harness/, and writes EXPERIENCE.md with the sections Why this was chosen (the selector's why), What each edit did (the edit rows) and Transferable lesson (cue / lesson / evidence / confidence). It then calls rebuild_index, which lists each entry with its component inventory and cue.
• Libraries are separate per mode (library_root/<mode>/<domain>). _resolve_libraries refuses a single shared library across two modes because it "would leak one mode into another".
• On resume, _restore_libraries replays the saved choice into the library under the same conditions (ok plus has_experience).

If there is no lesson, the task is still adopted and judged. It just leaves nothing for later tasks.

Selection flowchart

flowchart TD
    Pool["selection_pool: the k finals"] --> Render["_render_pool: drop non-answers"]
    Render -->|"empty"| Empty["ok false: no artifact"]
    Render --> Ids["assign_ids: sort, sha1-seeded shuffle"]
    Ids --> Write["_write_candidates: anonymised files"]
    Write --> Trip["assert_no_grader_verdicts"]
    Trip --> Call["selector session select.mode k"]
    Call -->|"choice parsed"| Chosen["ok true, chosen_label"]
    Call -->|"error or bad id"| Fallback["_fallback: neutral stand-in, ok false"]
    Chosen --> Persist["_persist: selection.json"]
    Fallback --> Persist
    Persist -->|"ok and library"| Reflect["reflect_adoption"]
    Reflect --> Distil["_distil: edits + experience"]
    Distil -->|"has_experience"| Lib["Library.add: tree + EXPERIENCE.md"]
    Distil -->|"NO_ENTRY"| NoLib["nothing stored"]
    Persist --> Judge["later: judge_selections"]
    click Pool href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L93-L111"
    click Render href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L114-L153"
    click Ids href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L71-L76"
    click Write href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L183-L219"
    click Trip href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L454-L490"
    click Call href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L332-L345"
    click Fallback href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L237-L252"
    click Persist href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L255-L265"
    click Reflect href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L402-L451"
    click Distil href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L454-L505"
    click Lib href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/library.py#L106-L152"
    click Judge href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/judge.py#L90-L143"

Part 4: Judging and reports

Why nothing in the method's run path imports the judge

The docstring of core/judge.py opens with: "Measurement of the run that selection chose, after it chose it. Nothing in the method imports this module." This holds at the pinned commit. Searching harness2/ for imports of judge finds only:

• core/report.py (from .judge import Scored, the data class only), which is itself only imported by the CLI;
• cli/judge.py and, through _load_rows, cli/report.py;
• outside the package, the test suite (tests/test_workflow.py).

schedule.py, recursion.py, selection.py, library.py, context.py and prompts.py do not import it. The adapter contract says the same thing from the other side: BenchmarkAdapter.score is "Called only by core.judge, only after selection committed." And RunView, the only thing the method knows about a rollout, "Never carries a score."

The reason is leakage. If a score could reach an improver session, the selector or the library, the method would be optimising against the benchmark's rubric. It would no longer be improving the harness from the task and its evidence. That would inflate results and make the parallel-vs-sequential comparison meaningless. Keeping the method and the measurement in separate import graphs, plus running judging as a separate process afterwards, plus the verdict tripwires on every context directory, gives three layers of defence. (The tripwire's own docstring calls itself "not the defence itself". The real defence is that the read paths never touch grader output.)

Scored: one judged row

Scored:

judge_one and judge_selections

judge_one calls adapter.score(run_dir, task_id):

• if it raises → error="<Type>: <msg>", n_* stay None;
• if it returns a falsy value → error="judge returned nothing";
• otherwise it copies raw, n_passed and n_criteria, and computes all_pass.

judge_selections(adapter, selections, *, parallel=8) takes Selection.to_dict() records. A record that is not ok, has no chosen_label or has no chosen_run_dir becomes an ungraded row with "selection did not complete successfully" and is never sent to the judge (this covers fallbacks, see Part 2). The rest are judged in a ThreadPoolExecutor. An exception from a worker also becomes an ungraded row. Row order follows completion order; cli.judge re-sorts. _mode_of reads mode, or the legacy key arm.

write(rows, path) dumps r.__dict__ plus the derived frac and graded as JSON.

Failure is None, never zero, and why denominators matter

A judge crash, a timeout or an API outage says nothing about the run. Counting it as 0 would bias a mode downwards whenever the infrastructure was flaky during its tasks. So:

• frac is None for an ungraded row;
• summarise computes good = [r for r in group if r.graded], sets n = len(good) and n_ungraded = len(group) - len(good), and averages only over good;
• tasks that never produced a selection (base failed, BudgetError, crash) have no row at all, so they are also outside the denominator. The report docstring says "Ungraded rows and tasks that never completed are excluded from the denominator rather than counted as failures."

The consequence is that two modes can have different n. The report shows n and ungraded on each line so the reader can judge coverage. The code does not reconcile denominators across modes, for example by restricting to tasks graded in both. If you need that, it is not covered by the code.

cli/judge.py: the judging command

python3 -m harness2.cli.judge --bench jb --substrate oc --k 2 --modes parallel (docstring L15-L29)

main step by step:

1. Parses the cell (--bench, --subset, --substrate), the run identity (--k, --seed, --modes from add_run_identity_args) and --bases. There is no --tag, so tagged namespaces are not reachable from this CLI.
2. config.validate_models(judge=True) requires HARNESS2_JUDGE_MODEL (and the executor/improver model checks).
3. Resolves the namespace with summary_namespace. This is always the agg namespace, because run_task keeps one summary.json for both modes there.
4. Judge-model namespace check (L169-L171):

identity = ns_dir / "config.json"
if identity.is_file() and json.loads(identity.read_text()).get("judge") != config.JUDGE_MODEL:
    raise SystemExit("Judge model changed for this experiment; choose a new HARNESS2_RUNS_DIR")

config.json is written by the sweep's _check_identity and records method_version, executor, improver, judge, endpoints, split, tasks, seed and so on. Re-judging a namespace with a different judge model would mix two measurement instruments in one judged.json, so the CLI refuses. If config.json is missing (for example the tasks were run from Python directly), the check is skipped.

1. Exits if the namespace dir does not exist.
2. --bases → judge_bases: scores every recorded base<i> run directly through adapter.score, so the initial harness can be reported beside the adapted one. It writes base_judged.json as {task, label, run_id, score} rows, where score is the raw dict or None. Legacy base entries without a run_dir are counted and skipped. It returns 0 only if every row is graded, else 2. cli.report does not read this file. The code includes no consumer for it.
3. Otherwise collect_selections reads selections.<mode> from every */summary.json (skipping unreadable files, mapping legacy arm through _selection_record).
4. J.judge_selections(adapter, sels, parallel=_PARALLEL), where _PARALLEL = 8.
5. Merge with the cache through _merge, keyed by (task, mode, k): a row for a different run_id replaces the old one (the selection changed). For the same run, a graded old row beats an ungraded new one, "so a failed re-judge never erases an existing score." The exception is a new row whose error starts with selection, which always replaces. Rows come out sorted by key.
6. Writes judged.json and prints N row(s), G graded, U ungraded (judge failure or no pick — NOT zeros). Exit code 0 only if this run produced rows and all were graded, else 2.

_load_rows rebuilds Scored objects from judged.json. It drops the derived keys _DERIVED = ("frac", "graded") and maps legacy arm → mode. An unreadable cache raises ValueError ("move it aside to rebuild") rather than being silently overwritten.

core/report.py: from rows to a figure

Summary, one per (mode, k):

render(summaries, *, title, bench) prints the lines sorted by (k, mode), then the notes:

• k = 1 noise floor (L122-L131): at k = 1 the two modes are the same procedure (one INV, one DOM, one composition; compare prompts.aggregate(..., k=1) in run_sequential L536-L540). Any gap between them is pure run-to-run noise, and an effect at k ≥ 2 "has to clear it." This is an honest built-in error bar.
• Own final rarely chosen: chose_final <= 25% of n. This can only happen with records from outside the finals pool (older runs).
• Mostly kept base: kept_base >= 80% of n, likewise.
• Stage fallbacks: any fell_back → inspect the stage errors.
• Mixed scoring units: more than one metric. The message notes that weighted points and unweighted criteria counts are "about 20 pt apart on JobBench".

cli/report.py: the report command

python3 -m harness2.cli.report --bench jb --substrate oc --k 1,2 (docstring). It is read-only and never calls the judge.

main: it takes a comma list of --k, --seed and --modes. For each k it loads judged.json via _load_rows (and warns "run cli.judge for k=… first" if missing), keeps rows of the requested modes, and collects stage fallbacks and spend with _spend_and_fallbacks:

• fell_back[(mode,k)] = the summed length of modes.<mode>.fell_back across task summaries (read through _modes_of, legacy arms included);
• usd[(mode,k)] = the sum of every stage's usd plus selections.<mode>.usd, which is improver + selector + reflector spend. Executor spend is not included (see Part 1).

It then prints R.render(...) with a title like jb-oc seed=… modes=parallel,sequential, followed by improver (editors, composer, selection, reflection) spend across reported modes: $X. It raises SystemExit("nothing to report") if no rows were found.

Part 5: The data model in one picture

classDiagram
    class Entry {
        +str kind
        +str label
        +float usd
        +bool cached
        +int attempts
    }
    class Ledger {
        +str mode
        +str task_id
        +int k
        +list entries
        +rollout(label, usd)
        +session(label, usd, attempts)
        +R() int
        +P() int
        +assert_identity()
        +to_dict() dict
    }
    class BudgetError
    class Stage {
        +str label
        +str dim
        +bool fell_back
        +float usd
        +int attempts
    }
    class ModeResult {
        +list stages
        +dict rollouts
        +list finals
        +Ledger ledger
    }
    class RunView {
        +str label
        +str run_id
        +bool ok
        +bool delivered
    }
    class Selection {
        +str chosen_id
        +str chosen_label
        +dict id_map
        +bool ok
        +bool fallback
        +float usd
        +dict experience
    }
    class Library {
        +add(task_id, harness, experience)
        +rebuild_index()
    }
    class TaskResult {
        +dict modes
        +dict selections
    }
    class Scored {
        +int n_passed
        +int n_criteria
        +bool all_pass
        +graded() bool
        +frac() float
    }
    class Summary {
        +int n
        +int n_ungraded
        +float score
        +float all_pass
        +line(bench) str
    }
    Ledger "1" *-- "many" Entry
    Ledger ..> BudgetError : raises
    ModeResult "1" *-- "1" Ledger
    ModeResult "1" *-- "many" Stage
    ModeResult "1" o-- "many" RunView
    Selection ..> RunView : chosen from the k finals
    Selection ..> Library : experience stored
    TaskResult o-- ModeResult : to_dict
    TaskResult o-- Selection : to_dict
    Scored ..> Selection : judged from record
    Summary ..> Scored : aggregates graded rows
    click Entry href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/ledger.py#L40-L46"
    click Ledger href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/ledger.py#L49-L132"
    click BudgetError href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/ledger.py#L36-L37"
    click Stage href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L41-L55"
    click ModeResult href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L58-L105"
    click RunView href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L175-L187"
    click Selection href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L46-L68"
    click Library href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/library.py#L57-L181"
    click TaskResult href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L239-L262"
    click Scored href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/judge.py#L32-L53"
    click Summary href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/report.py#L31-L65"

Information flows one way: Ledger → ModeResult → summary.json; Selection → summary.json → Scored → Summary. Nothing on the right-hand side (Scored, Summary) is ever read by anything on the left.

Part 6: Worked example, k = 2, parallel mode

The task is ledger-1 from the bundled demo (harness2/demo.py), run at k = 2 with a library attached. Namespaces: bases and summary.json live in demo-oc-agg2, and the parallel work dir is demo-oc-agg2/ledger-1/.

1. Rollouts and sessions into the ledger

summary.json → modes.agg.ledger then looks like this (labels exact, dollar values illustrative):

{"mode": "agg", "task_id": "ledger-1", "k": 2,
 "R": 8, "expected_R": 8, "P": 6, "session_attempts": 6, "expected_P": 6,
 "usd": 0.0, "rollout_usd": 0.0, "session_usd": 0.0,
 "rollouts": ["base1","base2","inv1","dom1","inv2","dom2","agg1","agg2"],
 "sessions": ["inv1","dom1","inv2","dom2","agg1","agg2"]}

If, say, a bug made the dom2 stage also be labelled inv2, the counts would still be 8/6, but the duplicate check would raise BudgetError: agg-2 ledger-1: R=8 (want 8), P=6 (want 6) … DUPLICATE labels: ['inv2']. The agg mode would be recorded as failed, with no selection.

(The seq mode on the same task would record base1, base2, inv1, dom1, c1, inv2, dom2, c2 as rollouts and inv1, dom1, c1, inv2, dom2, c2 as sessions, which is also 8/6.)

2. Select

1. selection_pool("agg", 2, …) → {agg1, agg2}. Bases and inv*/dom* are excluded.
2. _render_pool: both delivered answer.txt, so both stay and n_candidates = 2.
3. assign_ids(["agg1","agg2"], task_id="ledger-1", k=2, mode="agg") → seed int(sha1("ledger-1|2|agg")[:12], 16), which gives {A: agg1, B: agg2}. (For ledger-2 the same call gives {A: agg2, B: agg1}.)
4. Directory demo-oc-agg2/ledger-1/selection/agg2/ gets task.md, source/ledger.csv, candidates/A.md, candidates/B.md. The tripwire passes.
5. The selector session select.agg2 reads them. The demo's scripted selector (DemoProposer.call) recomputes the signed total from the source and replies:

<why>The signed total matches the source ledger.</why>
```choice
A
```

Both finals print Total: 85.15, and DemoProposer answers with the first candidate file (in sorted order) that contains the expected total, so it picks A.

1. Parsed: chosen_id="A", chosen_label="agg1", ok=True, chosen_run_id="demo-oc-agg2/ledger-1/agg1", chosen_is_base=False. selection.json is written. Selection spend goes to sel.usd, not to the ledger.

3. Reflect

1. reflect_adoption builds reflection/agg2/ from all 8 rollouts (the pool shows the 6 non-base harnesses with their diffs), plus base_harness/ and adoption.json = {"chosen_label": "agg1", "candidate_ids": {"A": "agg1", "B": "agg2"}, "reason": "The signed total matches…"}.
2. _distil runs experience.agg2 in reflection/agg2.experience/. The demo reply has an <edits> row (scripts/summarize.py, verdict helped) and an <experience> block with cue / lesson / evidence / confidence. edits.md is written and sel.experience is set. selection.json is re-persisted, and sel.usd now includes the reflection cost.
3. has_experience is true, so Library.add writes library/…/agg/accounting/harnesses/ledger-1/harness/… and EXPERIENCE.md, and rebuilds INDEX.md. The next accounting task's improvers will see this entry. The selector never will.
4. _run_one_mode stores selections.agg in summary.json with chosen_run_dir, chosen_tree_is_base=False and harness_fingerprint.

4. Judge (later, separate process)

python3 -m harness2.cli.judge --bench … --k 2 --modes parallel (the demo adapter is not a CLI bench choice; with JobBench, for example, the flow is identical):

1. Checks HARNESS2_JUDGE_MODEL, and that config.json's judge equals it.
2. collect_selections → [selections.agg] from ledger-1 and from every other task.
3. judge_one(adapter, "ledger-1", "agg", 2, "agg1", run_id, run_dir) → adapter.score returns e.g. {"n_passed": 3, "n_criteria": 4}, giving Scored(n_passed=3, n_criteria=4, all_pass=False), graded=True, frac=0.75.
4. Suppose a second task's judge call times out. Its row is n_passed=None, n_criteria=None, error="TimeoutError: …", so graded=False, frac=None.
5. _merge with any earlier judged.json. If the timed-out task had a graded row for the same run before, that row is kept.
6. Prints judged.json: 2 row(s), 1 graded, 1 ungraded (… NOT zeros) and exits with code 2 (not all graded in this run).

5. Report

python3 -m harness2.cli.report --bench … --k 2 --modes parallel:

• summarise: group (agg, 2) has 2 rows, good = 1, so n=1, n_ungraded=1, score = 100 × mean([0.75]) = 75.00, all_pass = 0.0, chose_final=1 (label agg1), kept_base=0.
• fell_back = the number of stages across tasks that fell back; usd = stage spend + selection/reflection spend.
• Output: parallel k=2: n= 1 | score 75.00% | picked output: final 1 / candidate 0 / base 0 | ungraded 1, then the spend line. The timed-out task lowers coverage (n) but not the score. Counting it as zero would have reported 37.50%.

Questions this chapter answers

• Why R = 4k and P = 3k, and what if spend doesn't match? Because both drivers do exactly k shared bases + 2k candidate rollouts + k final rollouts, and 2k proposal + k composition stages. Equal spend makes the width-vs-depth comparison fair. A mismatch, or any duplicate label, raises BudgetError. The mode is then recorded as failed and never selected or judged.
• How is selection kept blind? The pool is only the k finals. Non-answers are dropped first. Labels are replaced by ids from a sorted, SHA-1-seeded shuffle. The selector gets only the task, the source and the complete deliverables, with no harnesses, labels, costs or scores. The prompt says there is no rubric, and a verdict tripwire scans the directory before the call.
• Why does nothing in the method's run path import the judge, and how are failures counted? So that no score can leak into improvement, selection or the library. Judging is a separate later command over finished selection.json records. A judge failure, or a selection that didn't complete, is None/ungraded and excluded from the denominator. A failed re-judge never overwrites an existing score.

Key takeaways

• The budget is enforced, not just reported: assert_identity is the last line of each mode, and duplicates are checked before counts.
• Retries, cached replies and stage fallbacks never change R or P. Retries show up only in session_attempts. Selection and reflection are outside the ledger, and their spend sits on Selection.usd.
• Dollar accounting covers improver/selector/reflector sessions. Executor rollout cost is not tracked by the ledger or the report.
• The selector chooses only among the mode's own k finals, so there is no abstention to the base. Anonymous ids are stable per (task, k, mode) and independent of the sweep seed.
• A selector failure leads to a persisted neutral fallback with ok=False. It is never reflected, never stored in the library and never judged. Adapter bugs and leaks raise.
• Reflection is the one deliberately non-blind step. It writes edit verdicts and at most one lesson, and only a complete cue/lesson/evidence lesson enters the per-mode, per-domain library.
• Measurement lives outside the method: core/judge.py is imported only by the report, the CLIs and the tests. None is not zero, and denominators are graded rows only, so always read n and ungraded.
• cli.judge refuses to re-judge a namespace with a different judge model. cli.report prints the k = 1 noise floor and sanity warnings.

Where to go next

• 00 · Start here: the map of the guide.
• 01 · Architecture overview: where budget, selection and judging sit in the whole system.
• 02 · Code structure: the package layout around core/ and cli/.
• 03 · One task, end to end: the same flow as this chapter's worked example, from the scheduler's side.
• 04 · The harness tree: what the inv/dom/agg/c stages actually edit, and the diffs reflection reads.
• 05 · Evidence and safety: context.build, the leak globs and the grader-verdict tripwires used here.
• 07 · Executors and benchmarks: adapter.score, deliverable_text, state_deliverable and the judges behind Scored.
• 08 · Setup and running: HARNESS2_JUDGE_MODEL, run_sweep, then cli.judge and cli.report in practice.