03 · One task, end to end

03 · One task, end to end

This chapter follows one benchmark task through the whole Harness² run lifecycle, at commit 1cfe254. A sweep turns a benchmark split into a seeded, ordered stream of (domain, task) pairs. It runs domains concurrently, and the tasks inside each domain one after another, so a per-domain experience library carries lessons forward. For each task, the scheduler rolls k shared base executions on the unmodified harness. It then runs one or both recursion modes: parallel (agg, breadth) and sequential (seq, depth). Each mode spends exactly 4k rollouts and 3k improver sessions, builds k final harnesses, has a blind selector pick one final, reflects on why it won, and adds the adopted harness plus a lesson to the library for the next task. Everything sits under deterministic namespaces, so an interrupted sweep resumes cheaply. The central files are harness2/core/schedule.py (namespaces, stream order, the drivers) and harness2/core/recursion.py (the two modes). harness2/cli/run_sweep.py is the command-line front end.

All links point at commit 1cfe254e8d25b15f37048c3730134686abe60335. Line numbers come from that commit.

The call stack at a glance

flowchart TD
  CLI["run_sweep.py main()"] --> SW["schedule.run_sweep()"]
  SW --> TS["task_stream()"]
  TS --> SO["stream_order()"]
  SW --> RD["run_domain() x domains"]
  RD --> RT["run_task() x tasks, in order"]
  RT --> RB["_roll_bases(): k base rollouts"]
  RT --> OM["_run_one_mode() per mode"]
  OM --> RM["recursion.run_mode()"]
  RM --> RP["run_parallel()"]
  RM --> RS["run_sequential()"]
  OM --> SEL["selection.run_selection()"]
  OM --> REF["selection.reflect_adoption()"]
  OM --> LIB["Library.add()"]
  click CLI href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/run_sweep.py#L121-L206"
  click SW href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L780-L850"
  click TS href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L736-L777"
  click SO href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L130-L163"
  click RD href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L669-L733"
  click RT href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L566-L666"
  click RB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L430-L477"
  click OM href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L480-L563"
  click RM href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L569-L578"
  click RP href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L343-L449"
  click RS href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L452-L566"
  click SEL href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L268-L357"
  click REF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L402-L451"
  click LIB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/library.py#L106-L152"

1. The CLI: harness2.cli.run_sweep

run_sweep.py is a thin argparse layer over schedule.run_sweep. Its docstring sets the cost expectation: each task costs R = 4k task-executor rollouts and P = 3k improver sessions per mode. It also explains why a real run checks the judge before the first rollout: the namespace records the judge model, and harness2-judge refuses to grade under a different one. Finding a missing judge only after the sweep would mean paying for the sweep twice.

Arguments

_build_parser combines three argument groups. Two of them come from cli/_common.py.

Notes:

• --modes defaults to parallel only (MODES_DEFAULT), even though schedule.run_sweep defaults to both modes. parse_modes maps the names to on-disk tokens and returns them in MODES order, so sequential,parallel becomes ("agg", "seq").
• The CLI always passes tag="" and never passes only_tasks. Tags and task filters are only reachable from Python.

_main: the order of operations

_main does these steps in order:

1. Validates --k ≥ 1 and --workers ≥ 1, then parses the modes.
2. Dry run: _print_stream prints the namespace for each mode, the summary and library roots, every (domain, task) in stream order, and the per-domain task counts (the legal --only-domains names). It returns exit code 1 when the stream is empty. It creates no Proposer, touches no substrate and validates no model setting.
3. Builds the adapter with make_adapter, then resolves the stream once with _stream_for. That call raises early on a bad --only-domains name.
4. Runs preflight.require_ready(adapter, judge=True), which raises ConfigError("Setup incomplete; nothing has run yet…") before any money is spent.
5. Computes runs_root (runs_root_for = <spec.runs_root>/adapt), library_root (library_root_for) and the summary namespace (summary_namespace, always the agg token).
6. Runs _check_identity. It writes <runs_root>/<summary_ns>/config.json with the method version, executor/improver/judge models, Vertex project and location, judge endpoint, Codex endpoint, split, the full task stream, max_tasks, domains and seed. If the file already exists and differs, it raises: "This run directory belongs to different models or tasks. Choose a new HARNESS2_RUNS_DIR." This check is what makes resume safe: the caches can never mix models or task partitions.
7. Creates the improver Proposer from the benchmark's proposer channel. Its audit and cache directory is <runs_root>/_audit/<summary_ns>.
8. Calls schedule.run_sweep(..., pool_policy="finals") and writes the returned record to <runs_root>/_sweeps/<summary_ns>-<UTC stamp>.json.
9. Prints the done and failed totals and a ready-made harness2.cli.judge command line.

Exit codes: 1 on a config/runtime error, an empty sweep ("refusing to report success") or any domain that died with an exception. 2 when some tasks failed. 0 when everything succeeded. main turns ConfigError, RuntimeError, ValueError and FileNotFoundError into a one-line error: message.

2. Modes, namespaces and tags

Constants

schedule.py#L41-L45

MODES = ("agg", "seq")
MODE_NAMES = {"agg": "parallel", "seq": "sequential"}
_MODE_ALIASES = {"agg": "agg", "seq": "seq", "parallel": "agg", "sequential": "seq"}

ROLLOUT_PARALLEL = recursion.PHASE_WIDTH

mode_token(name) lower-cases and strips the name, looks it up in _MODE_ALIASES, and raises ValueError for anything else. The name "agg" is historical: the parallel mode aggregates 2k independent candidates. The code calls it "parallel" everywhere a human reads it.

ns() and Namespace: the only ways to name a run

The module docstring states the rule (L15-L19): ns() is the only function that builds a namespace, and Namespace is the only thing that globs over one. The scheme is:

<bench>-<substrate>-<mode><k>[-s<seed>][-<tag>]      e.g.  jb-oc-agg2-s7

ns(benchmark, substrate, mode, k, *, seed=None, tag="") rejects an unknown mode ("the method has exactly two"). It rejects an empty substrate, because otherwise "the two substrates' caches [would] collide on every label". It also validates the tag. The seed segment comes before the tag segment.

validate_tag(tag) enforces two rules. An empty tag is fine.

• The tag must match [A-Za-z0-9_.][A-Za-z0-9_.\-]*: no / and no leading -, because it becomes both a path segment and a run-id segment.
• It must not match s\d+(-|$). A tag like s7 would spell a seed segment, so jb-oc-agg2-s7 could mean "seed 7" or "tag s7". The check makes seeded and tagged namespaces impossible to confuse. (s7x is allowed, because the regex needs - or the end of the string after the digits.)

Namespace is a frozen dataclass holding the full identity of one setting:

Why the namespace matters. The run id is the cache key for a rollout. The JobBench adapter turns run_id into a directory label (jobbench.py#L717-L753) and reuses a completed rollout only when its run mark and harness fingerprint still match. Putting benchmark, substrate, mode, k, seed and tag into the id keeps two experiments from ever reading each other's rollouts. Within one task, the stage label (inv1, agg2, c1, …) is the last segment. Labels can repeat across modes because the mode token differs.

_modes_of(record) reads the per-mode records of a summary. It falls back to a legacy key "arms" so older cached runs still resume.

The on-disk layout of one task

Derived from the code paths above (with ns_agg = jb-oc-agg2-s7, ns_seq = jb-oc-seq2-s7):

<runs_root>/                                  # <spec.runs_root>/adapt
  _audit/<ns_agg>/                            # Proposer cache + audit (both modes share it)
  _sweeps/<ns_agg>-<stamp>.json               # run_sweep record
  <ns_agg>/config.json                        # _check_identity
  <ns_agg>/<task>/summary.json  (+ .lock)     # ONE summary for all modes
  <ns_agg>/<task>/missing/                    # dead_dir placeholder for failed base rollouts
  <ns_agg>/<task>/harnesses/{base,inv1,dom1,inv2,dom2,agg1,agg2}/
  <ns_agg>/<task>/context/<label>/            # evidence built for each stage
  <ns_agg>/<task>/selection/agg2/selection.json
  <ns_agg>/<task>/reflection/agg2/ (+ agg2.experience/)
  <ns_seq>/<task>/harnesses/{base,inv1,dom1,c1,inv2,dom2,c2}/
  <ns_seq>/<task>/selection/seq2/ ... reflection/seq2/ ...
<library_root>/agg/<domain>/INDEX.md, harnesses/<task>/{harness/,EXPERIENCE.md}
<library_root>/seq/<domain>/...

The base rollouts and the summary always live in the agg** namespace**, even when only seq runs (run_task L606-L609; summary_namespace documents this). That is how both modes share the same k base executions: the base run ids are "<ns_agg>/<task>/base1"…, whichever mode asks for them. The library root is <spec.runs_root>/library/<bench>/k<k>-<substrate>[-s<seed>][-<tag>] (root_for), with one subdirectory per mode (created in run_domain). Separate seeds get separate libraries because they are separate streams.

3. Sweep, domain, task: the stream

Why order matters: continual learning

Harness² treats each domain as a continual-learning stream. Before a task runs, the improver can read that domain's library: harnesses adopted on earlier tasks, each with an EXPERIENCE.md lesson. Results therefore depend on order. Task 5 benefits from tasks 1–4 and must never see tasks 6+. Three requirements follow:

1. The order must be reproducible, so the same seed always gives the same stream, and a resumed run or an independent reproduction sees the same history.
2. The order must be local to a domain, so filtering out one domain cannot change another domain's stream.
3. A task's library must contain exactly the tasks before it, never stale entries from later tasks or another order.

stream_order handles (1) and (2). run_domain handles (3) with Library.retain.

stream_order

schedule.py#L130-L163

domains = sorted(tasks_by_domain)
if seed is None:
    missing = [d for d in domains if d not in set(reference_domain_order)]
    if missing:
        raise ValueError(...)   # frozen order list is stale
    ordered_domains = [d for d in reference_domain_order if d in tasks_by_domain]
else:
    ordered_domains = list(domains)
    random.Random(f"{seed}:__domains__").shuffle(ordered_domains)
out = []
for d in ordered_domains:
    tasks = sorted(tasks_by_domain[d])
    if seed is not None:
        random.Random(f"{seed}:{d}").shuffle(tasks)
    out.extend((d, t) for t in tasks)
return out

It returns a flat list of (domain, task_id) pairs. Design points:

• Every shuffle starts from the sorted order, and each domain gets its own RNG seeded with the string f"{seed}:{domain}". The result depends only on (seed, domain, task set): not on dict iteration order, not on PYTHONHASHSEED, and not on which other domains are present (docstring L135-L143).
• Unseeded runs use a frozen domain order and sort tasks inside each domain. A domain missing from the frozen list raises an error instead of landing "in an arbitrary position".
• Domain order only affects reporting and submission order. Domains do not share a library (libraries are per domain), and run_sweep runs them concurrently anyway.

_reference_domain_order(adapter, by_domain) uses the adapter's optional reference_domain_order() when it has one. Otherwise it falls back to _default_reference_domain_order: descending task count, then name.

task_stream

task_stream(adapter, *, split="all", seed=None, only_domains=(), only_tasks=(), max_tasks=None) resolves the ordered stream "for execution, previews, and resume identity". The CLI's dry run, _check_identity and run_sweep all call it, so all three see the same list.

1. Groups adapter.tasks(split) by adapter.domain_of(t).
2. only_domains: unknown names raise ValueError (listing the domains the split has). Then it filters.
3. only_tasks: unknown ids raise. Filtering happens after ordering, so the surviving tasks keep their stream positions relative to each other.
4. max_tasks < 1 raises. Otherwise it keeps the first max_tasks tasks per domain in stream order.
5. Returns {domain: [ordered tasks]}. The dict's insertion order is the domain stream order.

run_sweep

schedule.py#L780-L850. Parameters:

It builds the stream, resolves the selector once with _selector_for and passes it to every domain as selector_prop. It submits one run_domain per domain to a ThreadPoolExecutor(max_workers=domain_parallel) and collects results with as_completed. A domain that raises becomes a row {"domain", "error"}. The return value is a record with benchmark, substrate, k, modes, seed, tag, domain_order, max_tasks, only_tasks, start and finish timestamps, n_domains and the per-domain rows. Rows arrive in completion order, not stream order. domain_order keeps the stream order.

_selector_for(adapter, prop, selector_prop): an explicit selector_prop wins. If the improver's channel spec equals the benchmark's selector channel spec, it reuses the improver (which "preserves fake/custom proposers" in tests). Otherwise it builds a new Proposer(selector spec) with the same attempts, project and region, and an audit dir <prop.audit_dir>/selector.

Concurrency is across domains only. Inside a domain, tasks are strictly sequential, which the library requires. Inside a task, modes run one after another (for mode in modes), and parallelism happens inside each phase (proposals and rollouts, width 2).

run_domain: threading the library forward

schedule.py#L669-L733

libs = {
    mode: library_store.Library(root=Path(library_root) / mode, domain=domain, surface=surface)
    for mode in modes
}
...
for i, tid in enumerate(task_ids, 1):
    try:
        for lib in libs.values():
            lib.retain(task_ids[: i - 1])
        before = {mode: len(lib.entries()) for mode, lib in libs.items()}
        r = run_task(tid, k, ..., libraries=libs, ...)

• One library per mode (<library_root>/agg/<domain>, <library_root>/seq/<domain>). Parallel and sequential are compared head to head, so neither may learn from the other's adoptions. _resolve_libraries enforces the same rule for direct callers. Passing a single library with more than one mode raises "a shared mutable library would leak one mode into another". Passing both library and libraries also raises.
• Library.retain(task_ids[:i-1]) runs before every task and deletes every entry that is not one of the preceding tasks (then rebuilds INDEX.md). After a crash, a reordering or a re-run with a different max_tasks, the library can hold entries from tasks that are later in the current stream. retain removes them, so the improver never sees the future.
• library_entries_before records how many entries each mode's library had when the task started. This is a useful audit of the continual-learning setup.
• Per-task outcome: ok/skipped → done += 1. Any other status (partial, base-failed) → failed += 1, with errors joined by ; . An exception escaping run_task → status exception. A failed task does not stop the domain. The next task still runs, with whatever the library holds.
• After each task it prints [<domain>] i/N <status> <task> :: <error>.

Returns {"domain", "n_tasks", "seed", "done", "failed", "tasks": [rows]}.

The library only feeds improver sessions through Library.context_dir(). It returns the domain directory only when INDEX.md exists and at least one EXPERIENCE.md is present, and None otherwise. The first task of every domain therefore runs without any library. (How the library is laid out and shown to the improver is covered in 04 The harness tree and 05 Evidence and safety.)

4. run_task: one task, start to finish

schedule.py#L566-L666. The docstring summarises it as "k shared reference executions -> every mode -> blind selection per mode -> summary.json."

Sequence

sequenceDiagram
  participant D as run_domain
  participant T as run_task
  participant S as summary.json
  participant A as adapter (executor)
  participant M as recursion.run_mode
  participant P as improver (Proposer)
  participant Sel as selector
  participant L as Library (per mode)
  D->>L: retain(previous tasks)
  D->>T: run_task(task, k, libraries)
  T->>S: summary exists? _resume()
  alt cached ok for every mode and artifacts intact
    S-->>T: status=skipped
    T->>L: _restore_libraries (re-add chosen harness)
    T-->>D: TaskResult(skipped)
  else run
    T->>A: _roll_bases: base1..basek (agg namespace)
    A-->>T: k RunViews
    alt no base ok
      T->>S: status=base-failed
    else for each mode (agg, then seq)
      T->>M: run_mode(mode, base, library_dir)
      M->>P: 2k INV/DOM proposals + k compositions (3k sessions)
      M->>A: 3k rollouts of the new harnesses
      M-->>T: ModeResult (finals, trees, rollouts, ledger)
      alt k == 1
        T->>T: trivial_selection (no session)
      else k > 1
        T->>Sel: run_selection over k finals (1 session)
      end
      T->>Sel: reflect_adoption -> experience (1 session)
      T->>L: add(chosen tree, why, experience)
      T->>S: _write_summary(status=partial)
    end
    T->>S: _write_summary(final merged status)
    T-->>D: TaskResult
  end

Step by step

1. Validate k, modes and pool_policy (L588-L599). Resolve the per-mode libraries and the selector.
2. Locate the work directory: base_ns = Namespace(bench, sub, "agg", k, seed, tag), work = base_ns.task_dir(...), summary_path = work/summary.json (L604-L609).
3. Resume if summary.json exists and _resume accepts it (section 7). The task returns skipped without spending anything.
4. Read the base harness tree: harness_tree.read_tree(surface, adapter.base_harness()). Every mode starts from this tree, and selection uses it to decide whether a chosen final is really just the base.
5. Roll k bases with labels base1..basek and run ids <ns_agg>/<task>/base<i> through _roll_bases. res.base stores label, run_id and run_dir for each.
6. Base gate (L629-L633): if no base rollout is ok, the status becomes base-failed, the error "no base rollout produced anything" is recorded, the summary is written and the task returns. No improver session is spent, because the improvers would have no evidence. One surviving base is enough to continue. A base that ran but delivered nothing still counts as ok.
7. For each mode in modes order: build Namespace(bench, sub, mode, …) and call _run_one_mode. Any exception is caught and stored as res.modes[mode] = {"ok": False, "error", "traceback"(last 1500 chars)}, plus an error line. After every mode, the status is set to partial and the summary is written, so a crash in the next mode loses nothing.
8. Finish: res.status = _merged_status(res.to_dict()), then a final _write_summary.

One mode failing does not abort the other. If agg raises (for example a BudgetError), seq still runs on the same bases.

_roll_bases: the shared reference executions

schedule.py#L430-L477. Input: pending = [(label, run_id), …]. Output: {label: RunView}. Base rollouts call adapter.execute(task_id, None, rid), where None** for the harness dir means "the adapter's own base harness"**.

• Batched path. If there is more than one pending rollout and the adapter has execute_many (WorkBuddy does, at workbuddy.py#L826), it makes one call. If the batch returns too few directories, the missing ones become dead RunViews ("batch adapter returned too few run directories"). A directory that does not exist gives ok=False. If the batch raises, every pending base becomes dead with that exception.
• Threaded path. Otherwise each base goes through _one, which runs execute, or a dead RunView on exception, in a pool of width min(len(pending), ROLLOUT_PARALLEL=2). A single pending base runs inline. Unlike the batched path and _roll, _one sets ok=True without checking that the returned directory exists (schedule.py#L462-L470).
• _dead(label, rid, dead_dir, error) builds RunView(ok=False, delivered=False, run_dir=work/"missing", error=…). Failures are values, never exceptions, so one bad rollout cannot crash a task.

RunView is "what the method knows about one finished rollout. Never carries a score."

_run_one_mode: recursion → selection → reflection → library

schedule.py#L480-L563

work_dir = n.task_dir(runs_root, task_id)
lib_dir = mode_lib.context_dir() if mode_lib else None
r = recursion.run_mode(mode, task_id, k, base, adapter=adapter, prop=prop,
                       work_dir=work_dir,
                       run_id_of=lambda lbl, _n=n: _n.run_id(task_id, lbl),
                       base_tree=base_tree, library_dir=lib_dir)
res.modes[mode] = r.to_dict()
candidates = selection.selection_pool(mode, k, r.rollouts, base)
if pool_policy == "finals" and k == 1:
    sel = selection.trivial_selection(...)
else:
    sel = selection.run_selection(..., prop=selector, ..., pool_kind="finals")

What happens, in order:

1. Recursion in the mode's own namespace directory. run_id_of binds the namespace, so every new rollout gets the id <ns_mode>/<task>/<label>. The base RunViews passed in keep their agg-namespace ids.
2. Selection pool. selection_pool returns exactly the k finals: agg1..aggk or c1..ck. The bases are excluded on purpose ("there is no abstention by construction"). base is passed in only so the exclusion is visible at the call site (del base).
3. Pick a final. At k = 1, trivial_selection delivers the single final without a selector session, still applying the drop rules for finals with no answer, and writes selection.json in the standard schema. At k > 1, run_selection anonymises the finals (stable seeded ids A, B, …), shows only their deliverables and the task to the selector, and parses a ```choice block. If the selector fails, _fallback picks a base if one is in the pool. A finals pool never contains a base, so in practice it takes the alphabetically first final, with fallback=True but ok left False: the failure is recorded as an error, the task ends partial, and the pick is not reflected, not added to the library and not judged. (Details are in 06 Budget selection and judging.)
4. Reflection. Only when the mode has a library and selection succeeded, reflect_adoption builds evidence: the chosen harness, the non-base candidates, the base harness, adoption.json and the per-stage decision notes. It then runs one experience session (_distil) through the selector Proposer, which must return cue, lesson and evidence (and optionally confidence and edits). The result goes on sel.experience, and selection.json is rewritten.
5. Record the selection on res.selections[mode]: sel.to_dict() plus
   • chosen_run_dir: the run directory of the chosen final, used by resume,
   • chosen_tree_is_base: _tree_is_base runs harness_tree.diff_summary against the base tree and reports whether the "improved" harness is actually unchanged (every stage fell back, or the edits were no-ops),
   • harness_fingerprint: runmark.fingerprint of harnesses/<chosen_label>, a 16-hex-char SHA-1 over relative paths and bytes (ignoring __pycache__, .pyc and .DS_Store).
6. Errors. "<mode>: selection failed: …" if not sel.ok. "<mode>: selected run <id> produced no deliverable" if the chosen final delivered nothing.
7. Library update. If there is a library, sel.ok is true and has_experience(sel.experience) holds (non-empty strings for cue, lesson and evidence), then mode_lib.add(task_id, chosen_tree, why=…, label=…, run_id=…, experience=…) stores the full tree and an EXPERIENCE.md, after checking that neither contains grader verdicts. A task with no distillable lesson leaves no library entry, even though its final was delivered.

TaskResult and status

TaskResult is the per-task record. to_dict is exactly what summary.json contains.

_merged_status(merged) returns "ok" only when there is at least one mode, there are no errors, and for every mode the mode record is ok, its selection is ok, and the selection has a chosen_label. Otherwise it returns "partial". A ModeResult is ok when it has finals and no errors (recursion.py#L71-L73). A selection that used the fallback has a chosen_label but keeps ok=False (_fallback never sets it), so a task whose selector failed ends up partial, and _resume re-runs it later. Its fallback/error fields record what happened.

_write_summary: merging under a lock

schedule.py#L289-L328. The docstring says modes "may be written by separate processes", for example one process running --modes parallel and another --modes sequential on the same task. Both write the same summary.json in the agg namespace, so the write must be a merge:

• An exclusive fcntl.flock on summary.lock serialises writers (POSIX only).
• The modes and selections dicts are unioned with the old file, and this process's modes win.
• Old errors are carried forward, except:
  • "no base rollout produced anything" when this write's status is ok/partial (bases have since succeeded),
  • errors whose "<mode>:" prefix names a mode this process just completed successfully (stale failures are dropped),
  • duplicates (compared as JSON).
• The merged status is recomputed with _merged_status only when the written status is ok or partial. base-failed is written as is.
• The write is atomic: write to summary.json.tmp, then os.replace. A crash can never leave half a JSON file. If it somehow does, _load_old treats an unreadable file as {}.

The TaskResult that run_task returns reflects only this process's modes. The file can hold more.

5. Parallel vs sequential (recursion.py)

The module docstring (recursion.py#L15-L19) states the contract: each mode schedules 4k rollouts and 3k improvement stages, including shared base executions. Invalid proposals receive one retry before inheriting the current harness.

Constants

In the worst case, one stage makes 2 × 2 = 4 model invocations. The ledger still counts it as one session (P), and stores the real number of invocations in attempts.

Stage and ModeResult

Stage is one improvement stage, including its retries:

ModeResult:

_run_stage: one improver session with one retry

recursion.py#L135-L251

Inputs: label, dim, job (the prompt text from prompts.propose/aggregate/compose_round), the fallback tree, the adapter, the Proposer, ctx_dir, task_id, the evidence rollouts (always the base RunViews), harness_dir (the current harness on disk), library_dir, pool (earlier candidates as (label, harness dir, RunView)) and decisions ({label: notes or error} of earlier stages).

1. Build evidence with context.build(ctx_dir/label, …) and the mission text with context.mission(tree, job=job). If this raises, the stage falls back immediately with attempts=0 and the error "could not build evidence: …". (What the evidence contains is covered in 05 Evidence and safety.)
2. Up to STAGE_ATTEMPTS* attempts.** Each one calls prop.call(mission [+ "# YOUR PREVIOUS ATTEMPT WAS REJECTED\n\n<err>…"], tree.root, f"{label}.a{attempt}", harvest="context/harness", max_attempts=TRANSPORT_ATTEMPTS). The edited harness comes back by *harvesting context/harness from the session's working directory.
3. Each attempt can be rejected in these ways (continue → retry with the reason):
   • reply.workspace is None: the harness directory did not come back,
   • the harvested tree is empty ("do not delete them all"),
   • files outside the editable surface (rejected paths), with a hint about naming rules,
   • _accept_tree fails. Before the retry, the rejected tree is written back into the evidence's harness/ directory, so the second attempt edits its own previous attempt instead of starting over.
4. A ProposerError (transport exhausted) breaks out without a second stage attempt, because retrying a dead channel at the stage level would only spend more. Its cost and attempts are still counted.
5. On success it returns a Stage with the new tree, kept_base, the cost, attempts, cached and the notes.
6. Otherwise it returns Stage(tree=dict(fallback), fell_back=True, error=err[:400]).

_accept_tree(surface, adapter, new, fallback, label) first repairs the tree with harness_tree.autotrigger (and prints which skills it auto-triggered). It computes kept (no added, removed or modified files compared with the fallback), then validates: harness_tree.validate (subagent policy, reserved scripts), adapter.validate_tree, and context.assert_tree_has_no_grader_verdicts. It returns (kept, "") or (kept, error).

Session caching. Proposer.call content-addresses each call by label, mission, system prompt, the resolved context path and every evidence file (_call_fingerprint), and replays a stored response if one exists (proposer.py#L143-L147). Its docstring explains why the path is part of the key: "the two modes can have byte-identical k=1 evidence and must still be independent draws." A re-run with unchanged evidence therefore costs nothing (cached=True, attempts=0).

_roll, _roll_many, _record_sessions

_roll(adapter, ledger, task_id, label, tree, run_id_of, work_dir, *, record=True) writes the tree to work_dir/harnesses/<label> (if it is not None), runs adapter.execute(task_id, hdir, run_id_of(label)), and returns a RunView. Exceptions become a dead RunView: "a dead rollout is recorded, never raised". With record=True it appends ledger.rollout(label).

_roll_many(..., items, ..., width) rolls independent harnesses in a pool of min(len(items), width) threads with record=False, then records them after the pool drains, in the caller's order:

with ThreadPoolExecutor(max_workers=n) as pool:
    got = list(pool.map(lambda it: _roll(..., record=False), items))
for (label, _tree), rv in zip(items, got):
    ledger.rollout(label)
    out[label] = rv

This is why "a concurrent run records the same sequence of labels as a serial one". The ledger is deterministic whatever the thread scheduling. _record_sessions does the same for stages: they are recorded in list order after their concurrent phase finishes.

run_parallel: width

recursion.py#L343-L449. Docstring: "WIDTH: 2k iid candidates from shared evidence, then k INDEPENDENT compositions."

1. Creates a Ledger("agg") and records the k base rollouts in it. They count toward this mode's R even though they are shared. Writes the base tree to harnesses/base.
2. Phase 1, propose (2k sessions). specs = [(f"{d.lower()}{i}", d, i) for i in 1..k for d in ("INV","DOM")] → inv1, dom1, inv2, dom2, …. Each stage gets the same evidence (the base rollouts and base harness), the library and prompts.propose(..., dim, k=k, index=i). The fallback is always the base tree. They run in a pool of min(2k, propose_workers=2). Because the inputs are identical except for the prompt's dimension and index, the 2k candidates are iid draws: none of them sees another.
3. Phase 1, roll (2k rollouts) of all 2k candidate trees.
4. Phase 2, compose (k sessions). agg1..aggk, each with dim AGG and prompts.aggregate(..., k=k, index=i). Each sees the base evidence plus the pool of all 2k candidates (harness dir and rollout) plus their decision notes. The fallback is again the base tree. The k compositions are independent: same inputs, different index, and none sees another composition.
5. Phase 2, roll (k rollouts) of the k finals. result.finals = [agg1..aggk].
6. result.similarity = harness_tree.diversity(...) over the finals. Then ledger.assert_identity().

run_sequential: depth

recursion.py#L452-L566. Docstring: "DEPTH: k nested passes over the same shared base evidence."

current_harness = dict(base_tree)
history = []
for t in range(1, k + 1):
    current_label = f"c{t - 1}" if t > 1 else "base"
    current_dir = harness_tree.write_tree(surface, current_harness, work_dir/"harnesses"/current_label)
    specs = [(f"inv{t}", "INV"), (f"dom{t}", "DOM")]
    ...  # 2 proposals on top of current_harness, pool=history; roll both
    round_pool = list(history) + [inv_t, dom_t]
    comp = _run_stage(label=f"c{t}", dim="COMPOSE", fallback=current_harness, pool=round_pool, ...)
    result.rollouts[ck] = _roll(...)            # roll c_t
    history = round_pool + [(ck, ..., result.rollouts[ck])]
    current_harness = comp.tree
result.finals = [f"c{index}" for index in range(1, k + 1)]

Each round t (1..k):

1. Writes the current harness (base in round 1, then c{t-1}) to disk. That is the starting point that round edits.
2. Two proposals inv{t}, dom{t} in a pool of min(2, propose_workers=2), using prompts.propose(..., index=t, round_t=t if k>1 else None). The fallback is the current harness, not the base. Their pool is the full history of every earlier candidate and composition with its rollout, and their decisions include the notes of every earlier stage.
3. Two rollouts of inv{t} and dom{t}.
4. One composition c{t}, with dim COMPOSE. At k=1 it uses prompts.aggregate(k=1, index=1), exactly like the parallel composition. At k>1 it uses prompts.compose_round(t=t, k=k). Its pool is the history plus this round's two candidates. The fallback is the current harness.
5. One rollout of c{t}. c{t} becomes the next round's current harness, and its rollout joins the history.

The finals are every round's composition c1..ck, not just the last one. The selector can therefore pick an earlier, shallower final if deeper editing made things worse.

Side by side

flowchart TB
  subgraph PAR ["Parallel (agg): width"]
    PB["k base rollouts (shared)"] --> PP["2k proposals at once<br/>inv1..invk, dom1..domk<br/>all from base evidence"]
    PP --> PR["roll 2k candidates"]
    PR --> PA["k independent compositions<br/>agg1..aggk, each sees all 2k"]
    PA --> PF["roll k finals"]
  end
  subgraph SEQ ["Sequential (seq): depth"]
    SB["k base rollouts (shared)"] --> S1["round t: inv_t + dom_t<br/>edit current harness"]
    S1 --> S2["roll inv_t, dom_t"]
    S2 --> S3["compose c_t from history + round"]
    S3 --> S4["roll c_t"]
    S4 --> SD{"t < k ?"}
    SD -->|"yes: current = c_t"| S1
    SD -->|no| SF["finals c1..ck"]
  end
  PF --> SEL["selection over k finals"]
  SF --> SEL
  click PP href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L368-L387"
  click PR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L392-L402"
  click PA href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L404-L434"
  click PF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L435-L445"
  click S1 href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L478-L517"
  click S2 href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L518-L528"
  click S3 href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L530-L558"
  click S4 href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L559-L562"
  click SF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L564"
  click SEL href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/selection.py#L268-L357"

The budget is the same, but the information flow differs. Parallel spends its sessions on diversity: many fresh looks at the same evidence, then several ways to combine them. Sequential spends them on refinement: each round builds on the previous composition and sees everything tried so far. Holding R and P equal is what makes the comparison fair.

run_mode(mode, *a, **kw) dispatches "agg" → run_parallel and "seq" → run_sequential, and raises on anything else.

The ledger: R = 4k, P = 3k, enforced

ledger.py defines expected_rollouts(k) = 4k (k base + 2k candidates + k finals) and expected_sessions(k) = 3k (k INV + k DOM + k compositions). Each Ledger holds Entry rows (kind, label, usd, cached, attempts). assert_identity raises BudgetError unless the counts are exactly 4k/3k and no label repeats. Duplicates are checked first, "because a duplicate can satisfy" the counts while hiding a candidate that was never produced. Selection and reflection sessions are outside the ledger (docstring). A stage that fell back still counts as one session, and a dead rollout still counts as one rollout: the ledger measures scheduled work, not successful work. BudgetError subclasses AssertionError, so run_task's except Exception catches it and marks the mode failed.

6. Worked example: one task, k = 2, both modes

Setup: run_sweep --bench jb --substrate oc --k 2 --seed 7 --modes parallel,sequential. Take one task T in domain D that is the 3rd task of its domain's stream. Namespaces: ns_agg = jb-oc-agg2-s7, ns_seq = jb-oc-seq2-s7.

0. Before the task. run_domain calls retain([t1, t2]) on both libraries, so each holds at most the entries for t1 and t2. context_dir() returns the domain directory if any EXPERIENCE.md exists, else None.

1. Resume check. No summary.json yet in <runs_root>/jb-oc-agg2-s7/T/, so the task runs.

2. Bases (2 rollouts). jb-oc-agg2-s7/T/base1, …/base2 run on the unmodified harness (harness_dir=None), two threads.

3. Parallel (agg). Ledger starts with base1, base2.

Selection: pool {agg1, agg2}, k > 1 → run_selection (1 selector session, select.agg2). Reflection → 1 experience session (experience.agg2). If the lesson is complete, agg library ← T.

4. Sequential (seq). Ledger starts with base1, base2 (the same two rollouts, counted again).

Selection over {c1, c2} → 1 selector session (select.seq2). Reflection → 1 experience session. seq library ← T.

5. Totals for this task.

The same task at k = 1 in both modes: 1 base. Parallel runs inv1, dom1, agg1 (3 sessions, 1 + 2 + 1 = 4 rollouts). Sequential runs inv1, dom1, c1 (3 sessions, 4 rollouts). That is 7 distinct rollouts, 6 improver sessions, 0 selector sessions (trivial selection) and 2 reflection sessions. At k = 1 the two modes differ only in their prompts, as run_sequential notes when it uses aggregate(k=1) for c1. The Proposer cache keeps them independent draws because each mode has its own context path.

7. Namespaces and resume

Resume works at three levels, from coarse to fine.

Level 0: experiment identity (CLI)

_check_identity refuses to reuse a runs directory whose config.json names different models, endpoints, split, seed, domains, max_tasks or task stream. Every cache below therefore assumes the same experiment.

Level 1: task skip (_resume)

schedule.py#L350-L421. A task is skipped only if all of these hold:

1. summary.json parses (otherwise it prints "unreadable summary.json … re-running").
2. Its status == "ok". A partial or base-failed task always re-runs.
3. Every requested mode has an ok mode record and an ok selection with a chosen_label. A summary that covers only agg re-runs when seq is requested ("cached summary covers [...], re-running for [...]").
4. For each mode, the adopted output is still on disk and unchanged:
   • chosen_run_dir is recorded and exists,
   • the artifact is non-empty: the trajectory when the selection used evidence == "trajectory" on a state-deliverable benchmark, else deliverable_text (and only if delivered),
   • adapter.artifacts_intact(run_dir) returns true, when the adapter has that method,
   • harnesses/<chosen_label> is still a valid tree,
   • the harness fingerprint still matches harness_fingerprint.

If any check fails, it prints "cannot resume : …; re-running" and the whole task runs again. On success, res is filled from the cache with status = "skipped", and _restore_libraries re-adds each mode's adopted harness (read from <ns_mode>/T/harnesses/<chosen_label>) with its saved why and experience to the library, provided the selection was ok and has a complete experience. This matters because retain may have removed or rebuilt entries. Without the replay, a skipped task would leave a hole in the library for every later task. The result matches what the original run would have added.

Level 2: rollout and session caches (re-running is mostly free)

When a task re-runs, run_task starts from the top, but most of the work replays:

• Rollouts. The run id is deterministic (<ns>/<task>/<label>). Adapters reuse a finished rollout when its run mark is complete for the same harness fingerprint and its artifacts are intact. Otherwise they print "re-rolling : " and run it again (JobBench: jobbench.py#L717-L753). Changing a harness changes its fingerprint, which forces a fresh rollout.
• Sessions. Proposer.call replays a stored response when label, mission and evidence hash the same. The audit and cache dir is per summary namespace (_audit/<ns_agg>), and the context path (which contains the mode's namespace) is part of the key, so the modes never share a reply.

The two layers compose. If a crash interrupts seq round 2, the re-run replays the bases, all of agg, and seq round 1 from cache, then pays only for the remaining work. Whether a given session hits the cache depends on its evidence being byte-identical to last time. The code does not guarantee that in general, so treat replays as likely, not certain.

Why the summary lives in the agg namespace

Both modes need the same k bases, and resume needs one place to look. Putting both in the parallel namespace means one summary covers both modes, and two processes running different --modes on the same task merge under the flock instead of overwriting each other.

8. Failure modes and edge cases

What the code here does not show: whether a shared Proposer is thread-safe across concurrent domains (it is shared by every run_domain thread; its internals belong to the proposer chapter), any wall-clock or cost cap per task, and how adapter.execute enforces timeouts. None of these are covered by the scheduling code.

9. Answers to the chapter's questions

How is the task stream ordered, and why does order matter? stream_order puts domains in a frozen reference order (unseeded) or a Random("<seed>:__domains__") shuffle, and tasks within each domain sorted (unseeded) or shuffled with Random("<seed>:<domain>"), always starting from the sorted order. Order matters because each domain is a continual-learning stream: the library a task sees holds exactly the adopted harnesses and lessons of the tasks before it (retain(task_ids[:i-1])). Domains run concurrently, tasks sequentially. Per-domain RNGs mean filtering domains never changes another domain's stream.

What exactly differs between parallel and sequential? The dependency structure, not the budget. Both spend 4k rollouts and 3k sessions on the same k shared bases. Parallel draws 2k independent INV/DOM edits of the base, then k independent compositions over all of them, with finals agg1..k. Sequential runs k rounds: each round makes two edits of the current harness that can see all history, then composes c_t, which becomes the next round's starting point. Its finals are c1..ck, one per depth. Fallbacks, prompts, labels and concurrency follow from that (see the side-by-side table).

How do resume and namespaces work? ns() builds <bench>-<substrate>-<mode><k>[-s<seed>][-<tag>], and every rollout id is <ns>/<task>/<label>, so experiments never share caches. Bases and the summary live in the agg namespace. A task is skipped when its summary is ok for every requested mode and the adopted outputs, harnesses and fingerprints are intact. The library is then replayed. Otherwise the task re-runs, and deterministic run ids (rollout marks) and content-addressed Proposer replies make the re-run cost only the missing work. config.json blocks mixing models or task sets in one runs directory.

Key takeaways

• One task = k shared bases + (per mode) 3k sessions + 3k new rollouts + one selection + one reflection. The ledger enforces R = 4k and P = 3k per mode and fails loudly otherwise.
• Parallel = width, sequential = depth, at an identical budget. That equality is what makes the comparison meaningful.
• Failures are values. Dead rollouts, rejected proposals and selector failures become records, fallbacks and partial statuses, never crashes that stop a sweep.
• Determinism everywhere: seeded per-domain streams, stable anonymous ids, ledger order fixed after each concurrent phase, content-addressed caches.
• The library is the only thing carried between tasks. It is per mode and per domain, trimmed to the past before each task, written only when a complete lesson exists, and replayed on resume.
• Namespaces are the cache keys. ns() and Namespace are their only constructors, tags can never spell seeds, and the summary is pinned to the agg namespace.

Where to go next

• 00 · Start here: what Harness² is and how to read this guide.
• 01 · Architecture overview: where the scheduler sits among adapters, substrates and the proposer.
• 02 · Code structure: a module-by-module map of harness2/.
• 04 · The harness tree: read_tree, write_tree, validate, autotrigger, diff_summary and diversity, all used throughout this chapter.
• 05 · Evidence and safety: what context.build puts in front of each stage, and the grader-verdict and leak guards.
• 06 · Budget, selection and judging: the ledger in depth, run_selection, reflect_adoption and judging the chosen finals.
• 07 · Executors and benchmarks: adapter.execute/execute_many, run marks and artifact checks.
• 08 · Setup and running: preflight, environment variables and real run_sweep / judge commands.