05 · Evidence and safety

05 · Evidence and safety

Harness² improves a harness by handing an "improver" model (also called the proposer, the editor or the selector, depending on its job) a folder of evidence and letting it read and edit files there. This chapter covers the two modules that control what that model sees and how it runs. core/context.py builds the evidence tree, a folder holding the task, the full trajectories, the deliverables, the harness being edited, the earlier-task library and the candidate pool. It also runs the leak tripwires, which refuse to hand over any folder that contains answer-key paths or the grader's own verdicts. core/proposer.py runs each improver session in a throwaway temporary workspace, through either OpenCode or the Claude CLI, with the prompt sent on stdin. It reads the answer from output/answer.md or the final message, copies edited files back out, and retries, caches and audits every call. Together they enforce the method's main safety rule: the model adapting a harness learns from what the solver did, never from how it was scored.

All code links point to commit 1cfe254e8d25b15f37048c3730134686abe60335. Line numbers come from that commit.

Contents

1. Where these modules sit
2. The threat model: what "a leak" means here
3. context.py reference: every symbol
4. The evidence tree: every path, when, from where, what is cut
5. The mission pointer and MISSION.md
6. Leak tripwires: what, when, what they raise
7. The other evidence folders: selection and reflection
8. proposer.py reference: every symbol
9. Improver sessions: isolation, prompt passing, answer reading, retries
10. Caching and the audit trail
11. Worked example: the evidence tree for ledger-1, stage inv2
12. Diagrams
13. Key takeaways
14. Where to go next

1. Where these modules sit

The module docstring of context.py puts it plainly: "The evidence tree an agentic proposer reads. The only way evidence reaches a model." Every model session in the method gets its evidence as files on disk. The prompt itself is only a short pointer. There are three callers, and all of them go through Proposer.call:

The method code calls context.build and context.mission, then hands the folder to prop.call(mission, folder, label, ...). The benchmark adapter (see chapter 07) supplies all the raw material through the BenchmarkAdapter protocol: task_prompt, task_files, trajectory, deliverable_files, deliverable_text and a few optional hooks. context.py never reads a run directory on its own. That is deliberate, and section 2 explains why.

2. The threat model: what "a leak" means here

A Harness² run touches three kinds of data for each task:

1. What the solver saw and did. The task prompt, the source files, the trajectory (actions and observations) and the files it produced. This is legitimate evidence.
2. The answer key. Rubrics, expected outputs, hidden tests, reference solutions. Benchmarks ship these next to the task (WorkBuddy's tests/**, solution/** and environment/**, JobBench's RUBRICS.json and expected/**, the Lab benchmark's task.json).
3. The grader's verdict. Pass rates, failed-check names, judge findings. These are written into the run's trial directory once the run has been scored.

If the improver ever reads (2) or (3), three things break:

• The current task is contaminated. An improver that reads the rubric can write a harness that satisfies the rubric's wording rather than the work. Its score then measures leakage, not adaptation.
• The contamination spreads. Adopted harnesses go into the per-domain library (see Library.add) and are served as evidence on later tasks. One leaked string in a harness would show up in every later task's context. That is why the verdict tripwire also runs on authored harness trees before they are stored.
• Selection becomes a reward oracle. The selector picks a final output. If it could see verdicts, it would pick by score, and the reported number would mean nothing.

The code defends against this in layers:

The docstring of the verdict tripwire states the design stance outright. It is "A tripwire on the adapters' read boundary, not the defence itself: if it fires, find the read path that produced the match rather than widening the marker list." (context.py L457-L460). The tripwires exist to catch an adapter bug loudly. They are not meant to clean data up quietly.

3. context.py reference: every symbol

3.1 Public and private symbols at a glance

3.2 ContextTree fields

ContextTree is the return value of build. Its docstring reads "A built evidence tree, plus what actually landed in it." The point of the "what actually landed" part is that the mission never lists a folder that does not exist. An improver told about context/library/ when there is none would waste steps, or worse, start guessing.

3.3 ContextTree.layout

layout returns a fixed-width listing. Each line appears only when its condition holds:

One detail: the rollouts line is driven by got_r, so a rollout whose trajectory failed is not named there. Its rollouts/<label>.md file still exists and holds the (trajectory unavailable: …) placeholder.

3.4 build(...)

def build(
    dest: Path | str,
    adapter: BenchmarkAdapter,
    task_id: str,
    rollouts: Sequence[RunView] = (),
    *,
    harness_dir: Path | None = None,
    library_dir: Path | None = None,
    pool: Sequence[tuple[str, Path, RunView | None]] = (),
    decisions: Mapping[str, str] | None = None,
    obs_trunc: int | None = None,
) -> ContextTree:

(context.py L105-L116)

The order of operations matters, because later steps read earlier output (the pool diff compares against harness/) and the final tripwires scan everything:

1. Wipe and recreate dest.
2. Write task.md from adapter.task_prompt(task_id) (L131).
3. copy_source → source/ (L132).
4. Write decisions/<label>.md for each decision. An empty note becomes "No decision notes returned." (L133-L136).
5. If there are rollouts, create rollouts/ and deliverables/, then _write_rollout for each one (L141-L151).
6. If harness_dir is a folder: tripwire assert_no_leak_paths(harness_dir, …), then _copy_tree → harness/ (L153-L156).
7. _write_pool → pool/ (L158).
8. If library_dir is a non-empty folder: tripwire on it, then shutil.copytree → library/ (L160-L165).
9. Tripwire assert_no_leak_paths(dest, …) over the whole tree (L167).
10. Tripwire assert_no_grader_verdicts(dest) (L169).
11. If rollouts were requested and none rendered, raise RuntimeError (L171-L175): "no trajectory could be rendered for ANY rollout -- the proposer would be adapting on nothing".
12. Return the ContextTree.

Error handling. The docstring (L117-L123) states the policy. Only the trajectory and deliverable readers are caught, "because their failure is itself evidence". A failure to render is written into the file so the model can see it. Everything else, including adapter bugs and tripwire ValueErrors, propagates. In _run_stage the whole build + mission call is wrapped (recursion.py L154-L174). An exception there turns into a Stage with fell_back=True and error="could not build evidence: …". That stage keeps its fallback harness and no model session runs. A tripped wire therefore costs nothing and leaks nothing. The stage simply proposes no change.

3.5 copy_source(adapter, task_id, root) -> int

copy_source copies the task's source material to root/source/, skipping anything that matches a leak glob:

rel = f.relative_to(src).as_posix()
if any(
    fnmatch.fnmatch(rel, g.lstrip("./")) or fnmatch.fnmatch(rel, f"*/{g.lstrip('./')}")
    for g in adapter.spec.leak_globs
):
    continue

(context.py L204-L209)

• Input: the adapter's optional task_files(task_id) hook (spec.py L216-L217). If the hook is missing, not callable, returns nothing or returns a non-folder, the function returns 0 and writes nothing.
• Filtering: it skips non-files, anything under __pycache__, and any relative path matching a leak glob, either at the top level or at any depth through the */ prefix. Files are walked in sorted order. shutil.copy2 keeps their timestamps.
• Then a tripwire: if anything was copied, assert_no_leak_paths(dst, …) checks the result (L214-L215). This is belt and braces. The filter and the check use the same matching rule, so the check guards against future edits to the filter.
• Output: the number of files copied.

Here source files are filtered (skipped silently), while harness, library and pool folders are rejected (the tripwire raises). The reason is that a benchmark task folder legitimately holds both the inputs and the answer key. WorkBuddy's task_files returns the whole task folder, and its docstring says "spec.leak_globs strips the answer key before any copy" (workbuddy.py L711-L715). A harness or library containing an answer-key path, on the other hand, would mean the method itself had copied one somewhere, which is a bug.

copy_source is public because run_selection also calls it (selection.py L301).

3.6 _write_rollout(dest, r, adapter, trunc, names_only) -> str

_write_rollout writes one rollout and returns "" on success or a failure string. It has two halves.

Trajectory (L221-L230). It calls adapter.trajectory(r.run_dir, obs_trunc=trunc) and writes the result to rollouts/<label>.md. On any exception it writes (trajectory unavailable: <Type>: <msg>) instead and returns the failure detail. The trajectory is written in full. Only individual observations may be cut, at obs_trunc characters. BenchSpec refuses obs_trunc < 10_000 because that range "fabricates defects" (spec.py L163-L167): an improver that sees clipped observations diagnoses failures the solver never had.

Deliverables. There are three cases:

The per-file folder holds:

• images/<name>: up to 8 images from the optional deliverable_images hook. An OSError on copy is silently skipped (L250-L256).
• <name>.txt for each name -> body from deliverable_files. Slashes in the name become __, an empty name becomes unnamed, and .txt is appended unless the name already ends in .txt (L257-L260).
• raw/<name>: binary originals from the optional deliverable_assets hook (L261-L267). This copy has no try, so an OSError here propagates and becomes a stage error.
• INDEX.md: "# <label> produced N file(s)", one line per file with its extracted character count, and a paragraph explaining that each .txt "is the artifact converted to text by the SAME reader the work is assessed through". When raw/ exists, it adds a note that the .txt view is CANONICAL and the raw file should never be used "to overrule it" (L268-L286).

That "canonical" wording closes a subtle gap. A spreadsheet formula with no cached value looks fine when opened in a spreadsheet program, but the grader's reader sees an empty cell. If the improver trusted the raw file, it would wrongly decide the deliverable was correct.

Note that _write_rollout never lists the run directory. Every byte comes from a named adapter hook. The comment on _listing says the same: names come "from the adapter, never a directory scan" (L338). A directory scan would pick up whatever the grader wrote next to the run.

3.7 _write_pool(dest, pool, adapter, trunc, names_only) -> list[str]

_write_pool writes pool/<label>/ for each (label, cand_dir, roll):

If either read raises, rollout.md is overwritten with (unavailable: <exc>) (L331-L332). The label is always appended to the returned list.

Two consequences follow from the code:

• The diff baseline is dest/"harness". If build got no harness_dir, that folder does not exist, read_tree returns {}, and every candidate file shows up as added.
• read_tree only reads files on the editable surface (tree.py L127-L148). The diff therefore ignores README stubs and anything off the surface, while harness/ itself is a full _copy_tree copy.

3.8 Names-only helpers

• _listing(adapter, run_dir) calls the optional deliverable_listing hook. If the adapter declared names mode but implements no hook, the result is a one-item list explaining that. On an exception it returns (listing unavailable: …). It never scans directories.
• _WITHHELD_NOTE explains the policy to the model. Its key sentence: "on this benchmark the extracted-deliverable tree lives inside the same trial directory as the grader's own output, so serving file bodies here once served the grader's pass rate and failed-check names along with them." It then tells the model to reason from the trajectory and the file set, because "a missing or misnamed file is a finding".
• _listing_md(label, names) builds the heading, a label: [...] repr line, a bullet list and the withheld note.
• _write_listing(d, label, names) writes only INDEX.md.

At this commit names mode is used by exactly one benchmark cell: WorkBuddy's code subset (workbuddy.py L666). JobBench (L521) and Lab (L493) use "contents".

3.9 _images and _copy_tree

• _images(adapter, run_dir) returns the optional deliverable_images hook's paths, or [] if there is no hook. Its docstring: "Images the benchmark's own reader would pick up".
• _copy_tree(src, dst) copies files recursively in sorted order, skipping __pycache__ and .pyc, and returns the count. Used for harness/ and pool/<label>/harness/. The library, by contrast, goes through a plain shutil.copytree (L164).

4. The evidence tree: every path, when, from where, what is cut

The module docstring (L17-L23) gives the intended layout. The full set of paths build can write, plus what the session adds, is:

Every path, any file: the final two tripwires scan the whole tree. A leak-glob match or a grader-verdict marker anywhere aborts the build.

The session adds these, outside context/:

4.1 Which callers pass what

The library folder comes from Library.context_dir, which returns the domain folder only when INDEX.md exists and at least one EXPERIENCE.md does, and None otherwise. The scheduler passes it on at schedule.py L499. The library therefore holds only earlier tasks in the same domain. Entries get in only through Library.add, which itself runs the verdict tripwire on the harness and the experience text before writing (library.py L120-L123).

5. The mission pointer and MISSION.md

mission(tree, *, job) builds the whole prompt for a stage. In order:

1. job.strip(): the role instructions from prompts.propose, prompts.aggregate or prompts.compose_round (chapter 04).
2. # YOUR EVIDENCE and "It is on disk, in context/. Read it -- do not answer from this message alone."
3. tree.layout, the listing from section 3.3.
4. A note that every trajectory EVENT is present, and that any renderer-truncation marker "is ours, not behavior by the solver. Do not diagnose it as a solver failure."
5. The reading method: read task.md and the current harness first; inventory and inspect every rollout or candidate; compare each candidate with its baseline; "Cite the specific trajectory moment behind every change; one you cannot trace to a moment is speculation."

Why a pointer and not the evidence itself? The module docstring answers: "Evidence is served as files and the prompt is a short pointer, so nothing is truncated to fit a prompt." (L25-L26). Trajectories run to megabytes; JobBench reads up to 8,000,000 characters per trajectory (jobbench.py L880). Pasting that into the prompt would force clipping, and clipping fabricates defects.

MISSION.md is a file the OpenCode channel writes at the root of the session workspace:

(cwd / "MISSION.md").write_text(
    (f"{system}\n\n---\n\n" if system else "") + mission, encoding="utf-8"
)

(proposer.py L254-L256)

Its contents are the optional system text, a --- separator, then the mission string. For a recursion stage, that mission is the context.mission(...) output, plus a # YOUR PREVIOUS ATTEMPT WAS REJECTED block on a retry (recursion.py L182-L194). The command-line prompt is a fixed one-liner telling the model to read it (section 9.3). The Claude CLI channel does not write a MISSION.md; the mission goes on stdin. No in-repo caller passes system at this commit (all three prop.call sites, recursion.py L182, selection.py L333 and L482, omit it). In practice MISSION.md is therefore exactly the mission.

6. Leak tripwires: what, when, what they raise

6.1 Tripwire 1: assert_no_leak_paths(root, patterns)

for p in root.rglob("*"):
    if not p.is_file():
        continue
    rel = p.relative_to(root).as_posix()
    for pattern in patterns:
        pat = str(pattern).replace("\\", "/").lstrip("./")
        if fnmatch.fnmatch(rel, pat) or fnmatch.fnmatch(rel, f"*/{pat}"):
            hits.append(rel)
            break
if hits:
    shown = ", ".join(sorted(hits)[:8])
    raise ValueError(f"answer-key path(s) would enter model context: {shown}")

(context.py L405-L416)

Notes on fnmatch semantics. * matches / too, so tests/** behaves like "anything under tests/", and */tests/** catches tests/ at any depth. lstrip("./") strips characters, not a prefix. A glob like .env would become env. None of the shipped globs start with a dot, so this has no effect at this commit.

Shipped leak_globs:

When it runs:

Checking source folders before copying matters. Otherwise the answer key would be written to disk under the work directory, where a later session could find it, even though the final check would still raise.

Without this wire: if a harness edit, a library entry or a candidate ever contained a copy of tests/test_outputs.py or RUBRICS.json (an improver could create one, for example, if a solver's trajectory printed it), that file would be shown to every later session.

6.2 Tripwire 2: assert_no_grader_verdicts(root, markers=GRADER_VERDICT_MARKERS)

GRADER_VERDICT_MARKERS:

The JSON keys are matched with their double quotes, so prose like "the pass rate was low" in a trajectory does not trip it. A JSON verdict file does. The code does not name which benchmark emits which marker.

Behaviour (L462-L490):

1. It walks every file in sorted order.
2. Name check: if any marker is a substring of the relative path, it raises ValueError saying the path "matched marker … in its NAME. The tripwire in core.context fired -- this is a leak, not a false positive to widen around. Find the read path that produced it."
3. Content check: only for suffixes in _MARKER_SCAN_SUFFIXES: .md .txt .json .csv .tsv .yaml .yml .html .htm .log .py .js .sh .xml .ini .toml .patch .diff and no suffix (""). It reads the file as UTF-8 with errors="replace" and skips it on OSError. On the first marker found it raises ValueError with the path, the marker, the offset and about 180 characters of surrounding text (60 before, 120 after).
4. Binary files (e.g. .xlsx, .png under raw/ or images/) are checked by name only.

When it runs:

A hit in run_selection is raised before the try at L332, so it propagates out of selection. That matches the docstring: "an adapter bug does raise, because it is a bug and not a selection outcome" (L279-L280).

6.3 Tripwire 3: assert_tree_has_no_grader_verdicts(tree, markers=...)

assert_tree_has_no_grader_verdicts applies the same checks to an in-memory {rel: body} mapping, with the same name and suffix rules. Its error messages are addressed to the author: "rename it -- files carrying these strings cannot be served as evidence to later tasks", and "remove or rename that string". Its docstring explains the timing: run it "before it can be rolled or stored: such a tree could never afterwards be served as evidence."

When it runs:

Why the authored-tree check matters: suppose an improver writes a helper script that prints "pass_rate", or a memory file quoting a grader. That harness is harmless for the current rollout. But if it were adopted and stored, every later task's build would hit tripwire 2 on library/ and fall back. The library would quietly turn into a denial-of-service on the whole domain. Blocking the string at authoring time keeps the library servable. The test test_complete_evidence_and_library_prefix checks exactly this with {"EXPERIENCE.md": '"pass_rate": 1'} (L128-L129).

6.4 Summary: all tripwires in one table

7. The other evidence folders: selection and reflection

7.1 Selection

run_selection does not call build. It writes its own folder under work_dir/selection/<mode><k>/ (selection.py L296-L305):

selection/agg2/
  task.md                 adapter.task_prompt
  source/                 context.copy_source (leak_globs filtered)
  candidates/<anon>.md    one file per candidate, anonymised, "COMPLETE -- nothing trimmed"
  candidates/<anon>/      deliverable_assets copied by name (when the adapter has the hook)

• The pool is built by _render_pool, which drops candidates that did not deliver or rendered empty. For benchmarks with state_deliverable=True and no file deliverable, it falls back to showing the trajectory.
• _write_candidates copies only the assets the adapter names. Its docstring gives the reason: "a path into the run dir would also carry the grader's own output."
• Candidate labels are hidden behind anonymous ids (assign_ids), so the selector cannot favour, say, c2 over c1 by name (bases are never in the pool).
• Only tripwire 2 runs on the whole folder (L330). Tripwire 1 runs only on source/ inside copy_source. Candidate assets are not checked against leak_globs; they are named one by one by the adapter.

7.2 Reflection and experience

reflect_adoption runs after selection has committed. It calls build into work_dir/reflection/<mode><k>/ with all candidates, adds base_harness/ (a copy of adapter.base_harness()) and adoption.json (chosen label, the id map, the selector's reason), then runs tripwires 1 and 2 again because new files were added after build returned. _distil copies that folder to a sibling <name>.experience folder, dropping any selection.json (shutil.ignore_patterns("selection.json"), L479). It runs tripwire 2 a third time on the copy, calls the proposer, and runs tripwire 3 on the parsed lesson before returning it.

The demo test checks that the reflection folder contains source/ledger.csv, rollouts/base1.md, rollouts/c2.md, pool/inv1/harness.diff, decisions/c1.md, adoption.json, base_harness/scripts/summarize.py and harness/scripts/summarize.py (test_workflow.py L113-L123).

8. proposer.py reference: every symbol

8.1 Symbols at a glance

8.2 ProposerError

ProposerError(RuntimeError): "A call did not produce a usable reply. Never swallowed into an empty string." It carries attempts: int and cost_usd: float, so callers can still account for money spent on failed attempts. _run_stage adds them to the stage totals (recursion.py L195-L200), and selection and _distil add exc.cost_usd to sel.usd.

8.3 Reply fields

8.4 Proposer fields and methods

8.5 ChannelSpec (from spec.py)

ChannelSpec is a frozen dataclass: "How to reach one model, on one proposer/selector channel."

resolve_channel_model is the one place a channel is matched to a model. Kind and model must be overridden together. OpenCode needs a / in the model id and the Claude CLI must not have one. The defaults are DEFAULT_PROPOSER_CHANNEL = "opencode" and PROPOSER_MODEL_BY_CHANNEL → HARNESS2_IMPROVER_MODEL (config.py L174-L175, L101).

9. Improver sessions: isolation, prompt passing, answer reading, retries

9.1 The call loop

for attempt in range(1, tries + 1):
    if attempt > 1:
        time.sleep(10 * (attempt - 1))
    tmp = Path(tempfile.mkdtemp(prefix="harness2-prop-"))
    try:
        if context_dir and Path(context_dir).is_dir():
            shutil.copytree(context_dir, tmp / "context", dirs_exist_ok=True)
        with _SEM if _SEM is not None else contextlib.nullcontext():
            text, cost, err = (
                self._claude_cli if self.spec.kind == "claude_cli" else self._opencode
            )(tmp, mission, system, wall)

(proposer.py L154-L164)

Step by step:

1. Cache lookup (L143-L147). Compute _call_fingerprint. A cache hit is appended to _calls and returned. No subprocess runs, and the cost is 0.
2. Budget: wall = timeout or spec.timeout, tries = max(1, max_attempts or self.attempts) (L149-L150).
3. Each attempt:
   • Back off 10 × (attempt − 1) seconds (none on the first try, 10 s before the second).
   • Make a fresh harness2-prop-* temp folder and copy the evidence into tmp/context/. The original evidence folder is never handed to the model, so the model can neither damage it nor carry edits from one attempt into the next.
   • Take the semaphore (at most 2 sessions at once across all threads), then run the channel.
   • Add the attempt's cost to spent_usd, even if the reply turns out empty.
   • Empty text → last = err or "empty reply" and try again (L166-L168).
   • Otherwise build a Reply(ok=True, attempts=attempt, cost_usd=spent_usd, ...). If harvest is set, copy that subfolder out first. If the subfolder is missing, delete any stale workspace at the cache path. Then write the audit and return.
   • subprocess.TimeoutExpired → last = "timeout after <wall>s". Any other exception → last = "<Type>: <msg>" (L185-L188). Because the harvest and the audit write sit inside the same try, an error there also counts as a failed attempt and triggers a retry.
   • finally: shutil.rmtree(tmp, ignore_errors=True). The workspace always disappears.
4. All attempts failed (L192-L208): append a failed Reply, write its audit (ok: false, error), and raise ProposerError("[<label>] <kind>/<model>: <last>", attempts=tries, cost_usd=spent_usd).

9.2 Two retry layers

A ProposerError ends the stage at once (break, L195-L200), so the worst case is 2 × 2 = 4 subprocess runs per stage. On a validation failure the rejected tree is written back into the evidence harness/ folder (L226-L229). The retry then starts from the model's own draft. Because that changes the evidence contents (and the label becomes .a2), the retry gets a new cache key.

9.3 OpenCode channel (_opencode)

_opencode prepares the workspace, runs OpenCode under bun, and reads the answer.

Workspace files: MISSION.md, an empty output/, and opencode.json containing {"agent": {"build": {"steps": spec.steps}}}, which caps the agent loop at 120 steps by default.

Prompt (argv): a fixed one-liner, never the mission itself:

"Read MISSION.md first. Do what it asks. Reply with the requested output as your final message AND save it verbatim to output/answer.md."

(L263-L266). The size of the mission therefore never reaches the command line.

Isolation (L269-L274):

env["OPENCODE_PERMISSION"] = json.dumps(
    {"external_directory": {"*": "deny", str(cwd): "allow", f"{cwd}/**": "allow"}}
)
env["XDG_DATA_HOME"] = str(cwd / ".ocdata")
env["XDG_STATE_HOME"] = str(cwd / ".ocstate")

• external_directory is denied everywhere except the session folder. The model cannot read the real run directories, the benchmark task folders (with their answer keys), the library on disk or other sessions' folders.
• Per-session XDG_DATA_HOME/XDG_STATE_HOME give each session its own OpenCode database. Without them, OpenCode's shared session history in the user's home folder would let one session read an earlier session's conversation, which might include evidence from a different mode or task. The test isolation case runs two calls and asserts that the second sees no database from the first: previous_sessions == [False, False] (test_workflow.py L130-L146).

Command: bun run --cwd <OPENCODE_DIR>/packages/opencode --conditions=browser src/index.ts run "<prompt>" --dir <tmp> --model <model> --format json [--variant <effort>], run with cwd=tmp and a timeout of wall (L276-L302). config.BUN and config.OPENCODE_DIR point into dependencies/ (config.py L130-L131). _opencode_dir raises FileNotFoundError with a hint to run scripts/setup_benchmark.py if the checkout is missing. That exception is caught by call and counts as a failed attempt.

Answer reading, in order:

1. output/answer.md, if it exists and is non-blank (L304-L306).
2. Otherwise the final message from the JSONL stream via _opencode_stdout. It joins all text events (part.type == "text") with blank lines, so a trailing terminator event cannot swallow the answer. error events are collected into an error string (last 300 chars). Non-JSON output is returned as-is after stripping the banner.
3. Otherwise ("", cost, "rc=<code>: <error or stderr tail or 'empty reply'>"). The stderr tail is the last 300 characters after _strip_oc_banner removes the one-time Performing one time database migration / sqlite-migration: / Database migration complete. lines (_OC_BANNER). The banner would otherwise fill the error message, because every session starts with a fresh database.

Why prefer answer.md? The final message of an agent loop can be a short sign-off ("Done, see the file"). The file holds the verbatim, complete answer.

Cost: _opencode_cost sums numeric part.cost over step-finish/step_finish events, rounded to 6 decimals.

9.4 Claude CLI channel (_claude_cli)

_claude_cli:

cmd = [cli, "--dangerously-skip-permissions", "-p", "--output-format", "json",
       "--effort", self.spec.effort]
stdin = mission
if system:
    if len(system) > SYSTEM_ARG_MAX:
        stdin = f"# SYSTEM INSTRUCTIONS\n\n{system}\n\n# END\n\n{mission}"
        cmd += ["--append-system-prompt",
                "Follow the SYSTEM INSTRUCTIONS block at the top of the message."]
    else:
        cmd += ["--append-system-prompt", system]

(condensed from L213-L232)

• Prompt on stdin: the mission is always sent on stdin (input=stdin), never as an argument, so its size is limited by nothing but the model's context.
• SYSTEM_ARG_MAX = 120_000: a system prompt of up to 120,000 characters goes through --append-system-prompt. Anything longer is moved into stdin under a # SYSTEM INSTRUCTIONS … # END header, and the argument becomes a pointer to it. The code does not say why 120,000. A likely reason (an inference, not stated) is the OS limit on a single argv string, e.g. Linux's 128 KiB MAX_ARG_STRLEN. As noted in section 5, no in-repo caller passes system, so this branch is unused at this commit.
• Working directory: cwd=tmp, so the model starts in the folder that holds context/. --dangerously-skip-permissions lets it read and edit without prompting. Unlike OpenCode, no directory restriction is configured for this channel in this code. Isolation here is the temporary copy and the working directory only. The code adds no sandbox that would stop the CLI from reading paths outside tmp.
• Answer reading: _parse_json parses stdout as one JSON object. Failing that, it takes the last line that looks like {…}, then the span from the first { to the last }, and finally falls back to {"result": <stdout>}. The text is payload["result"] or payload["text"]. This channel reads only the final message; there is no answer.md.
• Errors: if is_error is set or subtype is not None/"success", the text is dropped and err = "CLI error: <subtype> <first 200 chars>". If the return code is non-zero with no text, err = "rc=<code>: <stderr tail 300>". The cost is total_cost_usd.

9.5 Environment (env())

env() copies os.environ minus _DROP_ENV: ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, ANTHROPIC_BEDROCK_BASE_URL, CLAUDE_CODE_USE_BEDROCK. Then:

Dropping the Anthropic and Bedrock variables means every session is routed through Vertex AI with the configured project. A stray key in the shell cannot silently send sessions to a different endpoint or account, where the model and the billing would differ from what the audit records.

9.6 Harvest: how edits come back

_run_stage passes harvest="context/harness" (recursion.py L192). After a successful reply, _harvest copies tmp/context/harness to <audit_dir or a new harness2-harvest-* temp folder>/<safe label>.<cache key>.workspace before the temp folder is deleted. The safe label replaces every run of characters outside [A-Za-z0-9_.-] with _. That durable copy becomes Reply.workspace. _run_stage reads it with harness_tree.read_tree(surface, reply.workspace, rejected=…). Only files on the editable surface are applied, and anything else is reported back as an error. If the folder did not come back, the stage re-prompts: "the harness directory did not come back from the session; edit the files in context/harness/* in place"*.

Selection and experience calls do not harvest; their answer is only the reply text, parsed with regexes (_CHOICE, _WHY, _EXP, _EDITS, selection.py L33-L36).

10. Caching and the audit trail

10.1 _call_fingerprint

_call_fingerprint(label, mission, system, context_dir) hashes with SHA-256, using \0 separators: the label, the mission, the system text, the resolved absolute path of the context folder, and every file under it (relative path plus bytes, read in 1 MiB chunks; __pycache__ skipped; unreadable files hashed as an <unreadable:…> marker). It returns the first 16 hex characters.

Why the path is included, in the docstring's words: "the two modes can have byte-identical k=1 evidence and must still be independent draws." Without it, parallel and sequential mode at k=1 would share one cached reply, and the comparison between modes would be fake.

10.2 Audit files

_paths (only when audit_dir is set):

_cached returns a Reply(cached=True, cost_usd=0.0) when response.md exists and is non-blank. When the call needs a harvest, it also requires the .workspace/ folder. Failed calls write an empty response.md, so they are never served from cache. _drop_workspace removes a stale workspace when a fresh successful reply brought no harness back, so a later cache read cannot pair that new response with an old harness.

Without audit_dir (the demo, the tests) there is no cache and no audit, and harvested workspaces go to fresh temp folders.

11. Worked example: the evidence tree for ledger-1, stage inv2

We use the bundled demo, because it runs with no model and no benchmark (DemoAdapter, started by cli/demo.py with --k 2, both modes). The task ledger-1 asks: "Sum every signed amount in ledger.csv. Return Total: with two decimal places." Its source folder holds ledger.csv. The demo test also plants reference_answer.txt there, containing "pass_rate": 1, which is both a leak-glob path and a verdict marker (test_workflow.py L109). We follow sequential mode, k = 2, round 2, stage inv2. This is the case the test checks with context/inv2/pool/c1/rollout.md (L124).

What happened before this stage:

• The scheduler rolled base1 and base2 with the base harness (labels from schedule.py L619).
• Round 1 ran inv1, dom1 (proposals), then c1 (composition), and rolled each one. history now holds [(inv1, …), (dom1, …), (c1, …)] (recursion.py L561).
• ledger-1 is the first task in the accounting domain, so Library.context_dir() returns None and there is no library.

_run_stage(label="inv2", …) calls context.build(work/context/inv2, adapter, "ledger-1", [base1, base2], harness_dir=work/harnesses/c1, library_dir=None, pool=history, decisions={inv1:…, dom1:…, c1:…}).

Step 1: wipe. runs/demo-oc-seq2/ledger-1/context/inv2/ is deleted if it exists and recreated.

Step 2: task.md. DemoAdapter.task_prompt writes the one-line instruction.

Step 3: source/. copy_source walks sources/ledger-1/:

• ledger.csv: no glob matches → copied.
• reference_answer.txt: matches reference_answer.txt → skipped silently.
• Count = 1 → assert_no_leak_paths(source/) passes. The test asserts that no reference_answer.txt exists anywhere under the work folder (L125).

Step 4: decisions/. inv1.md, dom1.md, c1.md get each stage's <notes> text. In the demo that is "Keep signed amounts: the base total omitted credits in the source ledger." (demo.py L160).

Step 5: rollouts. For base1 and base2, trajectory() returns the run's trajectory.md ([ACTION] Run scripts/summarize.py on ledger.csv / [OBSERVATION] …). The demo implements deliverable_files → {"answer.txt": …}, so this is per-file mode. Each rollout gets deliverables/base1/answer.txt (no extra .txt, because the name already ends in .txt) and INDEX.md. There is no images/ (no hook) and no raw/ (no hook).

Step 6: harness. assert_no_leak_paths(harnesses/c1) passes, then _copy_tree copies it to harness/. A harness written by write_tree contains the four always-on files of the OpenCode surface (systemprompt.md, guardrails.md, LongTermMEMORY.md, tool_descriptions/tool_guidance.md, created empty if absent; opencode.py L44-L49), the README stubs (scripts/README.md, sub_agents/README.md, plugin/README.md, skills/README.md; L64-L87), and the edited scripts/summarize.py.

Step 7: pool. For inv1, dom1, c1: tripwire on the candidate folder, copy to pool/<label>/harness/, write harness.diff against harness/ (which is c1's tree, so pool/c1/harness.diff is empty), then rollout.md and produced.md (contents mode → deliverable_text, i.e. answer.txt's body).

Step 8: library. Skipped (library_dir=None).

Step 9: whole-tree checks. assert_no_leak_paths(dest) finds no reference_answer.txt. assert_no_grader_verdicts(dest) scans every .md, .txt, .py and suffix-less file; no marker is present, because the only file holding "pass_rate" was filtered in step 3.

Step 10: return. ContextTree(rollouts=["base1","base2"], deliverables=["base1","base2"], harness_files=N, library=False, pool=["inv1","dom1","c1"], source_files=1, decisions=3, deliverables_mode="contents").

The resulting folder:

runs/demo-oc-seq2/ledger-1/context/inv2/
├── task.md
├── source/
│   └── ledger.csv                      (reference_answer.txt filtered out)
├── decisions/
│   ├── inv1.md
│   ├── dom1.md
│   └── c1.md
├── rollouts/
│   ├── base1.md
│   └── base2.md
├── deliverables/
│   ├── base1/
│   │   ├── INDEX.md
│   │   └── answer.txt
│   └── base2/
│       ├── INDEX.md
│       └── answer.txt
├── harness/                            (copy of harnesses/c1)
│   ├── systemprompt.md
│   ├── guardrails.md
│   ├── LongTermMEMORY.md
│   ├── tool_descriptions/tool_guidance.md
│   ├── scripts/README.md
│   ├── scripts/summarize.py
│   └── sub_agents/README.md, plugin/README.md, skills/README.md
└── pool/
    ├── inv1/  harness/  harness.diff  rollout.md  produced.md
    ├── dom1/  harness/  harness.diff  rollout.md  produced.md
    └── c1/    harness/  harness.diff (empty)  rollout.md  produced.md

context.mission(tree, job=prompts.propose(…, "INV", …, index=2, round_t=2)) produces the job text, then # YOUR EVIDENCE, then this layout:

context/task.md                  the task as the agent received it
context/source/                 task source materials
context/decisions/              earlier edits and composition decisions
context/rollouts/               FULL trajectories: base1.md, base2.md
context/deliverables/<label>/    the files each rollout produced, one per artifact (browse INDEX.md for the set)
context/harness/                the harness you are editing (N files)
context/pool/                   3 candidates, each with its harness and its rollout

With a real Proposer on OpenCode, call(mission, tree.root, "inv2.a1", harvest="context/harness", max_attempts=2) copies this folder to /tmp/harness2-prop-XXXX/context/, writes MISSION.md, output/ and opencode.json, and runs OpenCode, which can read only inside that folder. When it finishes, output/answer.md (or the final message) becomes Reply.text, and context/harness/ is harvested as the new tree. That tree then has to pass harness_tree.validate, adapter.validate_tree and tripwire 3 before it is rolled. (In the demo, DemoProposer.call skips all of this and edits summarize.py directly in the evidence folder; demo.py L120-L163.)

What if harnesses/c1 had somehow contained reference_answer.txt? Step 6 would raise ValueError("answer-key path(s) would enter model context: reference_answer.txt"). _run_stage would return Stage(fell_back=True, error="could not build evidence: ValueError: …"), and stage inv2 would roll the unchanged c1 harness without any model session.

12. Diagrams

12.1 Building the evidence tree, with tripwires

flowchart TD
  Start["_run_stage / reflect_adoption"] --> Build["context.build"]
  Build --> Wipe["wipe dest"]
  Wipe --> Task["task.md from adapter.task_prompt"]
  Task --> Src["copy_source: skip leak_globs"]
  Src --> SrcChk{"assert_no_leak_paths on source/"}
  SrcChk --> Dec["decisions/label.md"]
  Dec --> Roll["_write_rollout per RunView"]
  Roll --> Mode{"proposer_deliverables"}
  Mode -->|names| List["_write_listing: INDEX.md only"]
  Mode -->|contents| Files["deliverable files, images, raw, INDEX.md"]
  List --> HChk{"assert_no_leak_paths on harness_dir"}
  Files --> HChk
  HChk --> Harn["_copy_tree to harness/"]
  Harn --> Pool["_write_pool: harness, diff, rollout, produced"]
  Pool --> LChk{"assert_no_leak_paths on library_dir"}
  LChk --> Lib["copytree to library/"]
  Lib --> AllPath{"assert_no_leak_paths on whole tree"}
  AllPath --> Verd{"assert_no_grader_verdicts"}
  Verd --> Any{"any trajectory rendered?"}
  Any -->|no| Err["RuntimeError: adapting on nothing"]
  Any -->|yes| Tree["ContextTree"]
  Tree --> Mis["mission: pointer prompt"]
  Mis --> Call["Proposer.call"]
  SrcChk -->|hit| Fail["ValueError: stage falls back, no session"]
  HChk -->|hit| Fail
  LChk -->|hit| Fail
  AllPath -->|hit| Fail
  Verd -->|hit| Fail
  click Start href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L135-L251"
  click Build href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L105-L188"
  click Wipe href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L126-L128"
  click Task href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L131"
  click Src href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L191-L216"
  click SrcChk href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L399-L416"
  click Dec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L133-L136"
  click Roll href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L219-L287"
  click Mode href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L146"
  click List href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L372-L375"
  click Files href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L248-L286"
  click HChk href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L399-L416"
  click Harn href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L386-L396"
  click Pool href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L290-L334"
  click LChk href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L399-L416"
  click Lib href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L160-L165"
  click AllPath href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L399-L416"
  click Verd href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L454-L490"
  click Any href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L171-L175"
  click Err href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L171-L175"
  click Tree href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L42-L102"
  click Mis href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L521-L546"
  click Call href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/proposer.py#L126-L208"
  click Fail href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L154-L174"

12.2 One improver session (OpenCode channel, with harvest)

sequenceDiagram
  participant S as _run_stage
  participant P as Proposer.call
  participant C as Audit cache
  participant W as tmp workspace
  participant O as OpenCode via bun
  S->>P: mission, evidence dir, label inv2.a1, harvest context/harness
  P->>C: _call_fingerprint, then _cached
  C-->>P: miss
  loop up to TRANSPORT_ATTEMPTS
    P->>W: mkdtemp, copy evidence to context/
    P->>W: write MISSION.md, output/, opencode.json
    P->>O: argv prompt "Read MISSION.md first", env with OPENCODE_PERMISSION and per-session XDG dirs
    O->>W: read MISSION.md, context/task.md, rollouts, harness
    O->>W: edit context/harness/, write output/answer.md
    O-->>P: JSONL events on stdout, exit
    P->>W: read output/answer.md, else join text events
    alt empty reply or timeout
      P->>W: rmtree, back off 10s, retry
    else usable reply
      P->>C: _harvest context/harness to label.key.workspace
      P->>C: _write_audit mission, response, meta
      P->>W: rmtree
      P-->>S: Reply with text and workspace
    end
  end
  P-->>S: ProposerError if every attempt failed

For the Claude CLI channel the differences are: no MISSION.md and no output/; the mission goes on stdin; the command is claude --dangerously-skip-permissions -p --output-format json --effort <effort>; the answer is the result field of the single JSON object on stdout.

12.3 Classes

classDiagram
  class ContextTree {
    +Path root
    +str task_id
    +list rollouts
    +list deliverables
    +int harness_files
    +bool library
    +list pool
    +list failed
    +str deliverables_mode
    +int source_files
    +int decisions
    +layout() str
  }
  class Proposer {
    +ChannelSpec spec
    +Path audit_dir
    +int attempts
    +str project
    +str region
    +env() dict
    +call(mission, context_dir, label) Reply
  }
  class Reply {
    +str label
    +str text
    +bool ok
    +float cost_usd
    +int attempts
    +bool cached
    +str channel
    +Path workspace
  }
  class ProposerError {
    +int attempts
    +float cost_usd
  }
  class ChannelSpec {
    +str kind
    +str model
    +str effort
    +int steps
    +int timeout
  }
  class BenchSpec {
    +tuple leak_globs
    +str proposer_deliverables
    +int obs_trunc
    +dict channels
  }
  class RunView {
    +str label
    +str run_id
    +Path run_dir
    +bool ok
    +bool delivered
  }
  class BenchmarkAdapter {
    +task_prompt(task_id) str
    +task_files(task_id) Path
    +trajectory(run_dir, obs_trunc) str
    +deliverable_files(run_dir) dict
    +deliverable_text(run_dir) str
  }
  class Library {
    +Path root
    +str domain
    +context_dir() Path
    +add(task_id, harness) Path
  }
  class Stage {
    +str label
    +bool fell_back
    +str error
  }
  Proposer --> ChannelSpec : uses
  Proposer ..> Reply : returns
  Proposer ..> ProposerError : raises
  BenchSpec --> ChannelSpec : channels
  BenchmarkAdapter --> BenchSpec : spec
  ContextTree ..> RunView : built from
  ContextTree ..> BenchmarkAdapter : read through
  ContextTree ..> Library : copies context_dir
  Stage ..> ContextTree : built per stage
  Stage ..> Proposer : calls
  click ContextTree href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/context.py#L42-L102"
  click Proposer href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/proposer.py#L90-L387"
  click Reply href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/proposer.py#L70-L87"
  click ProposerError href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/proposer.py#L61-L67"
  click ChannelSpec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L36-L48"
  click BenchSpec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L132-L173"
  click RunView href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L176-L187"
  click BenchmarkAdapter href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L200-L250"
  click Library href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/library.py#L57-L181"
  click Stage href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L42-L56"

13. Key takeaways

• Evidence is files, never pasted text. context.build writes a folder, and context.mission writes a short pointer to it. Trajectories go in whole; only individual observations are cut, at obs_trunc (≥ 10,000 characters).
• The adapter is the only reader of run directories, through named hooks. context.py never scans a run folder, because grader output lives there.
• Three tripwires. Leak-glob paths (on source, harness, pool, library and the whole tree). Verdict markers by name and content (on every folder before a session). Verdict markers in authored harness trees (before rolling and before storing in the library). They raise ValueError loudly and are not to be widened. In a stage, a trip means "fall back, no session".
• Names-only mode withholds deliverable contents on benchmarks where extracted deliverables share a folder with grader output (WorkBuddy code).
• Every session is disposable. A fresh harness2-prop-* copy per attempt, deleted in finally. OpenCode is limited to that folder, with its own state database. The Claude CLI channel is isolated only by its working directory and the copy.
• Prompts go through stdin or MISSION.md, never argv (only a system prompt ≤ SYSTEM_ARG_MAX uses argv). Answers come from output/answer.md first, then the final message (OpenCode), or from the JSON result (Claude CLI). An empty answer is an error, never "".
• Two retry layers: 2 transport attempts per call (10 s back-off) × 2 stage attempts with rejection feedback. Every call is content-addressed (including the folder path), cached and audited.

14. Where to go next

• 00 · Start here: what Harness² is and how to read this guide.
• 01 · Architecture overview: where the evidence tree and improver sessions sit in the whole loop.
• 02 · Code structure: the module map, including core/context.py and core/proposer.py.
• 03 · One task, end to end: base rollouts → recursion → selection → library, the run that produces the evidence described here.
• 04 · The harness tree: the c1..c8 editable surface, read_tree/write_tree, and validation, which decide what a harvested context/harness/ may contain.
• 06 · Budget, selection and judging: the selector session in full, anonymisation, and why judging happens only after selection.
• 07 · Executors and benchmarks: how each adapter implements trajectory, deliverable_files and leak_globs, the read boundary the tripwires protect.
• 08 · Setup and running: HARNESS2_IMPROVER_MODEL, Vertex settings, the OpenCode checkout and the audit folder layout.