07 · Executors and benchmarks

07 · Executors and benchmarks

Harness² keeps three things separate: the method (recursion, prompts, selection, library), the substrate (the agent framework a solver rollout runs under: OpenCode or Codex CLI), and the benchmark (a BenchSpec value plus an adapter class). This chapter covers the two plug-in points that connect the method to the outside world: harness2/substrate/ (executors) and harness2/benchmarks/ (JobBench, LAB, WorkBuddy-Bench). It also covers the contract that joins them, harness2/core/spec.py, and the vendored upstream runners in third_party/, which are pinned and patched. By the end you should know what each layer owns, how one JobBench task becomes a RunView under OpenCode, and exactly which methods you must write to add a new benchmark or a new executor.

All links point at commit 1cfe254e8d25b15f37048c3730134686abe60335.

1. Why three axes

The module docstring of core/spec.py states the design rule:

"""The benchmark seam: everything a benchmark must supply, and nothing more.

Three axes stay separate -- method (recursion, prompts, selection, library), substrate (the
agent framework a solver rollout runs under) and benchmark (a `BenchSpec` value plus an
adapter). `ChannelSpec` describes the proposer/selector channel, which is independent of both.
"""

In practice this means:

• Core never imports a benchmark. It talks only to the BenchmarkAdapter protocol and reads BenchSpec fields. The one place a CLI turns --bench/--subset/--substrate into a concrete adapter is cli/_common.make_adapter.
• Core needs almost nothing from a substrate: SubstrateLike is just name plus surface(), the editable c1..c8 tree schema. Core uses surface() to write and validate harness trees (for example recursion._roll calls harness_tree.write_tree(adapter.substrate.surface(), …)), and name for namespacing (schedule.py L225).
• A benchmark × substrate pair is a "cell". The BenchSpec docstring says so: "Static description of one (benchmark, substrate) cell. A value, not a code path." Every adapter in the repo builds a different BenchSpec per substrate, and WorkBuddy builds one per (subset, substrate).
• Launching is the benchmark's job, not the substrate's. Both substrates take a launcher callable in their constructor. If it is missing, execute() raises "the benchmark adapter must supply one (it owns runner choice, env wiring and read-back locations)" (OpencodeSubstrate.execute).

So the same method code runs unchanged across 3 benchmarks × 2 substrates (with WorkBuddy split into 3 subsets), and none of that code knows about Harbor, bun, codex exec or rubric files.

Component diagram

flowchart TB
  subgraph CORE ["core (method)"]
    REC["core.recursion._roll"]
    SCH["core.schedule._roll_bases"]
    JUD["core.judge (score after selection)"]
  end
  SPEC["core.spec: BenchmarkAdapter / BenchSpec / RunView / SubstrateLike"]
  REC --> SPEC
  SCH --> SPEC
  JUD --> SPEC
  subgraph BENCH ["benchmarks (adapters)"]
    JB["JobBenchAdapter"]
    LAB["LabAdapter"]
    WB["WbAdapter"]
  end
  SPEC --> JB
  SPEC --> LAB
  SPEC --> WB
  SUB["substrate.base: Substrate protocol"]
  subgraph EXEC ["substrates (executors)"]
    OC["OpencodeSubstrate"]
    CX["CodexSubstrate"]
  end
  SUB --> OC
  SUB --> CX
  JB --> OC
  JB --> CX
  LAB --> OC
  LAB --> CX
  WB --> OC
  WB --> CX
  subgraph TP ["third_party runners (pinned + patched)"]
    JBR["jb_opencode_runner.py / run_benchmark_codex_cli_adapt.sh"]
    LABR["lab_runner: run_opencode.py / run_codex_lab.py"]
    WBR["wb_runner: Harbor agents + harness configs"]
  end
  JB --> JBR
  LAB --> LABR
  WB --> WBR
  click REC href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/recursion.py#L260-L300"
  click SCH href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L430-L477"
  click JUD href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/judge.py#L70-L80"
  click SPEC href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L176-L250"
  click JB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/jobbench.py#L622-L1162"
  click LAB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/lab.py#L498-L1002"
  click WB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/workbuddy.py#L671-L1004"
  click SUB href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/base.py#L47-L80"
  click OC href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/opencode.py#L173-L320"
  click CX href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/codex.py#L109-L228"
  click JBR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/jb_opencode_runner.py#L211-L277"
  click LABR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/third_party/lab_runner/opencode/run_opencode.py"
  click WBR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/third_party/wb_runner/agents/oc_harness_agent.py"

2. The contract: core/spec.py in full

2.1 Constants and type aliases

2.2 ChannelSpec and resolve_channel_model

ChannelSpec is a frozen dataclass describing how to reach one model on the proposer/selector channel:

resolve_channel_model(kind, model, *, effort, steps, timeout) is "the one channel->model mapping". Its rules:

1. Override both or neither: (kind is None) != (model is None) raises, because "a bare-id/provider-prefixed mismatch fails silently downstream".
2. Defaults come from config.DEFAULT_PROPOSER_CHANNEL and config.PROPOSER_MODEL_BY_CHANNEL[kind].
3. An empty default model is allowed (returns a ChannelSpec with model=""), but an empty override is not.
4. opencode needs a provider-prefixed id (contains /). claude_cli needs a bare Anthropic id (no /).

All three adapters call resolve_channel_model(None, None, effort="high") and use the same channel for proposer and selector (for example jobbench._channels).

2.3 BenchPrompts: the benchmark-shaped text

BenchPrompts is the text that the shared method prompts interpolate. It is the only way a benchmark influences what the proposer and selector read.

__post_init__ enforces a firm rule. In work, inv_levers, dom_levers, yardstick, domain_guidance and selection_checks, the words benchmark, evaluation, grader, rubric, score are forbidden (case-insensitive):

raise ValueError(
    f"BenchPrompts.{field_name} must describe the WORK, not its "
    f"measurement (found {bad!r}). Say what the practitioner needs."
)

(spec.py L119-L126). The reason: the method must improve the work, not game the measurement. Text that mentions grading would teach the proposer to optimise for the grader.

resolve_prompts(adapter, task_id) returns adapter.prompts_for(task_id) when the adapter has that optional method, and otherwise adapter.spec.prompts. It raises TypeError if the result is not a BenchPrompts. Core calls it in recursion.py L365 and in selection.py L286.

2.4 BenchSpec: one cell as a value

BenchSpec (frozen):

Validation in __post_init__ checks: install mode known; proposer_deliverables valid; both channel roles present; obs_trunc >= 10_000; reserved_scripts are basenames ending in .py with no /.

2.5 RunView: what the method knows about a rollout

RunView is a mutable dataclass, documented as "What the method knows about one finished rollout. Never carries a score."

The adapter never builds a RunView. Core builds it around adapter.execute, in recursion._roll and schedule._roll_bases, and turns a raised exception into a dead view instead of propagating it ("A dead rollout is recorded, never raised"). Leaving the score out is deliberate: selection must happen without seeing grades. Scores arrive only through adapter.score, which core.judge calls after selection has committed.

2.6 SubstrateLike and BenchmarkAdapter

SubstrateLike (runtime-checkable Protocol): name: str, surface() -> SurfaceSpec. It is "the subset core needs". The full interface lives in substrate/base.py (section 3).

BenchmarkAdapter is documented as "Implement these and the whole method works. Nothing here is method logic."

Optional hooks that core probes with getattr. These are not in the Protocol, but core uses them when present:

Class diagram

classDiagram
  class BenchmarkAdapter {
    <<Protocol>>
    +BenchSpec spec
    +SubstrateLike substrate
    +tasks(split) list
    +domain_of(task_id) str
    +task_prompt(task_id) str
    +task_files(task_id) Path
    +execute(task_id, harness_dir, run_id, timeout) Path
    +delivered(run_dir) bool
    +trajectory(run_dir, obs_trunc) str
    +deliverable_files(run_dir) dict
    +deliverable_text(run_dir) str
    +base_harness() Path
    +validate_tree(tree) None
    +score(run_dir, task_id) dict
  }
  class SubstrateLike {
    <<Protocol>>
    +str name
    +surface() SurfaceSpec
  }
  class Substrate {
    <<Protocol>>
    +str name
    +surface() SurfaceSpec
    +materialise(tree_dir, spec, ctx) Install
    +execute(task_ref, install, run_ids, timeout) list
    +read_trajectory(stream_path, obs_trunc, max_chars, args_trunc) str
    +preflight() None
  }
  class BenchSpec {
    +str name
    +Path runs_root
    +str install_mode
    +BenchPrompts prompts
    +dict channels
    +tuple leak_globs
    +bool allow_subagents
    +frozenset reserved_scripts
    +str proposer_deliverables
    +bool state_deliverable
    +int obs_trunc
  }
  class BenchPrompts {
    +str work
    +str domain_word
    +str inv_levers
    +str dom_levers
    +str yardstick
    +str domain_guidance
    +str selection_checks
    +str prompt_pack
  }
  class ChannelSpec {
    +str kind
    +str model
    +str effort
    +int steps
    +int timeout
  }
  class RunView {
    +str label
    +str run_id
    +Path run_dir
    +bool ok
    +bool delivered
    +float cost_usd
    +float wall_s
    +str error
  }
  class Install {
    +Path staging_dir
    +str fingerprint
    +Mapping extras
  }
  class RunRequest {
    +str task_ref
    +tuple run_ids
    +int timeout
  }
  class SurfaceSpec {
    +str name
    +tuple always_on
    +int always_on_max
    +str skill_dir
    +str hooks_file
  }
  class OpencodeSubstrate
  class CodexSubstrate
  class JobBenchAdapter
  class LabAdapter
  class WbAdapter
  BenchmarkAdapter --> BenchSpec
  BenchmarkAdapter --> SubstrateLike
  BenchSpec --> BenchPrompts
  BenchSpec --> ChannelSpec
  Substrate --|> SubstrateLike
  Substrate --> Install
  Substrate --> RunRequest
  SubstrateLike --> SurfaceSpec
  OpencodeSubstrate ..|> Substrate
  CodexSubstrate ..|> Substrate
  JobBenchAdapter ..|> BenchmarkAdapter
  LabAdapter ..|> BenchmarkAdapter
  WbAdapter ..|> BenchmarkAdapter
  click BenchmarkAdapter href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L200-L250"
  click SubstrateLike href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L190-L197"
  click Substrate href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/base.py#L47-L80"
  click BenchSpec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L132-L173"
  click BenchPrompts href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L89-L126"
  click ChannelSpec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L36-L48"
  click RunView href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/spec.py#L176-L187"
  click Install href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/base.py#L26-L32"
  click RunRequest href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/base.py#L35-L41"
  click SurfaceSpec href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/tree.py#L41-L60"
  click OpencodeSubstrate href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/opencode.py#L173-L320"
  click CodexSubstrate href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/substrate/codex.py#L109-L228"
  click JobBenchAdapter href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/jobbench.py#L622-L1162"
  click LabAdapter href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/lab.py#L498-L1002"
  click WbAdapter href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/benchmarks/workbuddy.py#L671-L1004"

Substrate --|> SubstrateLike is structural: Substrate is a superset of SubstrateLike. Python does not declare that inheritance.

3. Substrates (executors)

3.1 The interface: substrate/base.py

The file's docstring calls it "The Substrate interface: what an agent framework owns, benchmark-free."

Substrate methods:

• surface(): the c1..c8 tree schema plus validators (SurfaceSpec, defined in core/tree.py; see chapter 04).
• materialise(tree_dir, spec, ctx): stage the editable tree as substrate-native files, according to spec.install_mode. ctx carries benchmark-owned values: {{token}} interpolation slots and file:<rel> entries for integration files that travel with the staged tree.
• execute(task_ref, install, run_ids, timeout): run rollouts through the benchmark-supplied launcher; one run_dir per run_id.
• read_trajectory(stream_path, *, obs_trunc, max_chars, args_trunc=400): render this substrate's capture format into the shared grammar. obs_trunc and max_chars come from the cell. args_trunc is a per-substrate default (400 for OpenCode, 2000 for Codex) that no caller overrides.
• preflight(): health checks "before money is spent". It raises RuntimeError with the remediation command and "never starts any part of the model chain".

The package init re-exports Install, Launcher, RunRequest, Substrate, OpencodeSubstrate, OPENCODE_SURFACE, CodexSubstrate, CODEX_SURFACE.

3.2 What a substrate owns, and what it does not

The base harness is notably not the substrate's. Each adapter supplies base_harness() per cell: JobBench ships trees in third_party/jb_runner/, while LAB and WB write them on first use.

3.3 OpencodeSubstrate

Source: substrate/opencode.py. name = "oc" (L176).

Surface: OPENCODE_SURFACE

Constants and helpers

The module docstring warns (L19-L24): "The install modes are different delivery channels for the always-on text and are not interchangeable: changing the one a benchmark uses moves that benchmark's baseline."

materialise

1. Reject any mode not in _OC_INSTALL_MODES (the error names codex_root as CodexSubstrate's).
2. Reject a directory that is not an OpenCode tree (HT.is_tree).
3. Copy the whole tree to tempfile.mkdtemp(prefix="adapt-oc-<mode>-").
4. Write each ctx entry with key file:<rel> into the staged tree (normalised relative path).
5. For home_dir_jinja only: compose the prompt, move tool_descriptions/tool_guidance.md to the root, run assert_jinja_safe, and write _composed/prompt.j2.
6. On any exception, delete the staging dir and re-raise.
7. Return Install(staging_dir, fingerprint=RM.fingerprint(tree_dir), extras={"install_mode": mode}).

For workspace_agents_md and workspace_config, OpenCode materialise is a plain copy. The rendering (AGENTS.md, .opencode/, or LAB's render_tree) happens in the benchmark's launcher. The fingerprint is a digest of the source tree, not the staged copy: SHA-1 over sorted relative paths plus bytes, ignoring __pycache__, .pyc, .DS_Store and the marker file, truncated to 16 hex chars (runmark.fingerprint). None digests to "base".

Reference CLI contract. run_argv and run_env are documented as "The reference argv; the real launchers build their own." OpenCode runs from source through bun: bun run --cwd <OPENCODE_DIR>/packages/opencode --conditions=browser src/index.ts run <prompt> --dir <ws> --model <m> --title <t> --format json [--agent] [--variant] [--session]. "There is no opencode binary in the contract." With agent="", the rendered AGENTS.md is appended to the provider prompt. run_env sets OPENCODE_PERMISSION={"external_directory": {"*": "deny", <ws>: "allow", <ws>/**: "allow", …}} (without this, "the session auto-rejects reads of its own working tree") and the output-token ceiling.

execute wraps the arguments into a RunRequest and calls the launcher. read_trajectory delegates to render_events, which parses the --format json JSONL defensively ("the event schema varies by OpenCode version"):

• error objects → [ERROR] … (300 chars)
• text → [ASSISTANT], reasoning → [THINKING] (clipped at obs_trunc)
• tool/tool_use/tool_call/tool-invocation → [ACTION] name(args) (args clipped at args_trunc), then [OBSERVATION] or [OBSERVATION ERROR]
• Overall clip at max_chars with the marker …(+N chars — render truncation) (_TRUNC_FMT, _clip).

preflight checks that config.OPENCODE_DIR/packages/opencode/src/index.ts exists (the pinned checkout, "v1.14.18") and that config.BUN is a file. It starts nothing.

3.4 CodexSubstrate

Source: substrate/codex.py. name = "cx". It targets codex-cli CODEX_VERSION = "0.144.6". The docstring says: "Per-deployment differences are values in Install.extras, never subclasses."

Surface: CODEX_SURFACE

CODEX_ROOT_ENTRIES = ("AGENTS.md", ".agents", "scripts", ".codex").

materialise accepts only codex_root. It requires AGENTS.md, then copies exactly those four entries (if present) into mkdtemp(prefix="adapt-cx-root-"). There is no render step: "the CLI injects AGENTS.md itself". Unlike OpenCode, ctx entries without the file: prefix go into Install.extras. file: entries are not written at all in the Codex path.

Reference CLI (exec_argv, exec_env):

str(Path(config.CODEX_DIST) / "codex"), "exec", prompt, "--model", model,
"-c", f'model_reasoning_effort="{reasoning_effort}"',
"--skip-git-repo-check",
"--dangerously-bypass-approvals-and-sandbox",
"--dangerously-bypass-hook-trust",
"--ephemeral", "--json", "-C", str(workspace), "--add-dir", str(workspace),

Codex's own sandbox is bypassed, and hook trust is bypassed because, in the docstring's words, "Hook trust enables the editable hooks" (the c8 .codex/hooks.json); --json with --ephemeral "preserves ephemeral sessions". The env drops OPENAI_BASE_URL, sets CODEX_HOME (the custom provider lives there, "not a global OpenAI URL override") and prepends CODEX_DIST/codex-path to PATH.

preflight(*, require_dist=False, require_bridge=False) requires HARNESS2_CODEX_BASE_URL (an HTTP(S) URL) and HARNESS2_CODEX_API_KEY, and checks that the codex binary exists. With require_bridge it also requires HARNESS2_CODEX_CONTAINER_URL. It then opens a TCP socket to the endpoint (5 s timeout). This is the one preflight that touches the network. require_dist is accepted but not read in the body.

Trajectory. parse_items reads lines that start with {"type": or {"timestamp": and dedupes thread items by item.id ("last state wins, first-appearance position kept"). render_events handles two formats:

• codex exec --json thread items (_render_item): command_execution → [ACTION] shell: … plus [OBSERVATION( ERROR)] … | exit_code=N; agent_message → [ASSISTANT]; reasoning → [REASONING]; todo_list → [ASSISTANT] plan: [x] …; error → [HARNESS]; user messages that start with # AGENTS.md instructions → [HARNESS AGENTS.md], so harness text is never mistaken for the user.
• ~/.codex/sessions rollout payloads (_render_rollout_payload): function_call, function_call_output, message, etc., deduped against consecutive repeats by _emit.
• turn.completed adds a token-usage [HARNESS] line; turn.failed/error → [ERROR].

final_message(path) returns the last agent message (or task_complete.last_agent_message). Nothing in harness2/ or scripts/ calls it at this commit.

3.5 OpenCode vs Codex at a glance

4. Shared benchmark plumbing

4.1 benchmarks/tasks.py

The docstring reads: "Task discovery and optional user-created partitions; no bundled task lists."

discover(bench, subset=None) -> dict[str, str] returns sorted task id → domain:

No tasks found raises FileNotFoundError with the setup hint (harness2-setup <bench> [--subset …]).

select(tasks, split, path, *, benchmark): split must be all, train or test. Without a split file only all is allowed (sorted ids). A split file is JSON {"benchmark": <name>, "train": [...], "test": [...]}. It is validated for the right benchmark, list types, no duplicates, no train/test overlap, and no ids missing from the dataset. The repo bundles no split; you create one with scripts/split_tasks.py, as the error message says.

4.2 Patterns all three adapters share

• A per-substrate spec factory (make_spec, lab_spec, spec_for) that returns a BenchSpec value.
• self.substrate = OpencodeSubstrate(launcher=self._launch…) or CodexSubstrate(launcher=…): the adapter injects its own launcher.
• Prompt overlays: a base BenchPrompts, plus tables of domain lenses joined into domain_guidance by prompts_for, using a forgiving key match (_norm/_lookup: lowercase, non-alphanumerics → _).
• Idempotent execute using the completion marker in core/runmark.py (.adapt_complete, MARK): is_complete(where, harness_dir=…) compares the recorded harness fingerprint, and why_not gives the log sentence. JB and LAB also record an output_fingerprint so a modified output dir is re-rolled.
• Dead chains are never cached. Each adapter raises if the runner produced no output or completion record, which leaves the rollout unmarked.
• Deliverables are read with the judge's own reader (JB, LAB) or a strict allowlisted renderer (WB), so what the selector sees matches what the judge grades.
• score** returns** None** on judge failure**, never 0.
• run_detached (JB, LAB): Popen(start_new_session=True), then killpg(SIGKILL) on the wall timeout, so a hung bun/codex process tree cannot survive (jobbench L55-L69, lab L406-L420).

5. JobBench adapter

Source: benchmarks/jobbench.py. The docstring: "Rollouts are addressed by LABEL, not by run directory: the run store is dataset-addressed and shared by every setting, so the run_id becomes the label."

5.1 Tasks, domains, prompts

• Tasks: <profession>/<taskN> directories under job-bench-eval's dataset/main, each with task_folder/TASK_INSTRUCTIONS.txt and a RUBRICS.json next to it.
• Domain: domain_of returns the profession (the part before /). reference_domain_order is the sorted set.
• JOBBENCH_BASE: "Professional workplace analysis across occupations…"; domain_word="occupational domain", deliverable_word="work product", a "COVERAGE OF WHAT WAS NAMED" selection_lead, selection_conformance="value_only", prompt_pack="tts".
• JOBBENCH_DOMAINS: 7 SOC-style group lenses (Business / Financial Ops, Other (Legal·Sales·Sci·Edu), Office / Admin Support, Computer / Mathematical, Architecture / Engineering, Management, Arts / Media).
• JOBBENCH_PROFESSIONS: 35 occupation lenses keyed by dataset slug (biostatisticians, lawyers, web_administrators, …).
• jobbench_prompts(*, domain, profession) appends both lenses to domain_guidance.

A detail in the code worth knowing: prompts_for passes the profession slug as both domain and profession. The domain table is keyed by group names like "Management", so the domain lens applies only if a profession slug normalises to one of those keys. The code has no profession → group mapping. In practice the profession lens is the one that applies.

• task_prompt: TASK_INSTRUCTIONS.txt verbatim ("as the agent received it"). Rubrics live in a separate file, which is why nothing needs stripping. task_files returns the dataset task_folder. _dataset_task rejects ids without /, ids containing .., or absolute ids.

5.2 Spec and base harnesses

make_spec(substrate): name="jb", runs_root=RUNS/jb, install mode from _INSTALL_MODE (oc → workspace_agents_md, cx → codex_root), domain_word="profession", leak_globs=("RUBRICS.json", "eval_result/**", "expected/**", "reference/**"), allow_subagents only on oc, obs_trunc=config.JB_OBS_TRUNC, and proposer_deliverables="contents". Default timeouts (_DEFAULT_TIMEOUT): oc 2400 s, cx 3600 s.

Paths (L46-L52): OC_RUNNER = the first-party benchmarks/jb_opencode_runner.py; CX_RUNNER = third_party/jb_runner/run_benchmark_codex_cli_adapt.sh; OC_BASE_HARNESS = third_party/jb_runner/harness_baseline_oc; CX_BASE_HARNESS = third_party/jb_runner/codex_base_harness. base_harness raises if the vendored tree is missing, because "an absent base would make every base rollout silently harness-free". validate_tree adds no invariants.

5.3 Rendering the OpenCode tree (workspace_agents_md)

The launcher, not the substrate, turns the staged tree into OpenCode-loadable files:

• compose_agent_body(staged, ctx) concatenates c1 systemprompt.md, # Operating Guardrails + guardrails.md, tool_guidance.md, # Long-Term Memory + LongTermMEMORY.md, and a # Helper scripts (run via Bash) listing that uses each script's first docstring line. Each part goes through _interp (only {{ name }} tokens are replaced; single braces stay) and _strip_comments (HTML comments removed, so a stub README renders to nothing).
• _render_ctx(domain) supplies env_description, domain_focus, date (UTC today) and working_directory.
• render_opencode_tree(staged, render_dir, ctx) writes AGENTS.md, copies whole skill directories to .opencode/skills/<n>/, writes each sub-agent as .opencode/agent/<sub>.md with mode: subagent enforced (_ensure_mode_subagent), copies non-underscore scripts/*.py, and copies non-underscore plugin/*.ts to .opencode/plugin/.

5.4 Launchers

_launch_common enforces one run per invocation ("no batch substrate here") and converts / → __ in the label.

_launch_opencode: substrate preflight → render_opencode_tree into mkdtemp("hrender_<label>_") → env (TASK_IDS, LABEL, HARNESS_RENDER_DIR, AGENT_NAME="", TASKS_BASE_DIR=self.store, OPENCODE_DIR, MODEL_ID=config.SOLVER_MODEL_OC, TIMEOUT_PER_TASK, MAX_RETRIES=2, MAX_CONCURRENT=1, OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX=64000) → _run_runner with wall 2*t + 600 → _require_artifacts.

_launch_codex: substrate preflight, then codex_config_error() (refuses if CODEX_HOME/config.toml no longer matches the live endpoint/model), then exec_env(os.environ, codex_home=CODEX_HOME_DIR) plus HARNESS2_CODEX_BIN, HARNESS_RENDER_DIR=install.staging_dir (the raw codex_root stage), BENCHMARK_MODELS=SOLVER_MODEL_CX, MAX_CONCURRENT_PER_MODEL=1, VERTEX_PROXY_KEY. It runs the vendored bash script with wall t + 600.

_run_runner prepends the judge interpreter's and bun's directories to PATH, runs .sh files with bash and everything else with sys.executable, and on wall timeout raises "artifacts left UNMARKED so the next attempt re-rolls instead of adopting a truncated rollout". _require_artifacts raises if model_output/<label> is empty and distinguishes "trajectory without deliverables" from "dead agent", because "A trajectory is not liveness: an attempt abandoned on a 401 or a rate-limit storm still leaves .jsonl events behind."

5.5 The first-party OpenCode runner: jb_opencode_runner.py

It replaced an upstream shell fork (see jb_runner/PROVENANCE.md). Its contract is entirely environment variables:

5.6 Execute, read-back, judge

JobBenchAdapter.init(substrate="oc", *, timeout, store_root, split_file) sets store = store_root or RUNS/jb/dataset, the dataset-addressed run store.

execute layout: outputs at <store>/<task>/model_output/<label>/, trajectories and the completion marker at <store>/<task>/model_traj/<label>/, judge details at <store>/<task>/eval_result/…. See the worked example in section 9 for the full step order. artifacts_intact requires the marker's output_fingerprint to match the current output, plus either a delivery or at least one event-bearing .jsonl. delivered = any file in the output dir.

trajectory: the newest *.jsonl in model_traj/<label>, rendered by the substrate with max_chars=8_000_000.

Judge extractor. _extract runs _EXTRACT_DRIVER in a subprocess under config.JB_JUDGE_PYTHON. The driver imports upstream eval/judge.py and calls its convert_file_to_text, extract_all_file_contents and collect_image_paths. If the extractor lacks optional readers (an [ERROR: … not available] string, _extractor_lacks_deps) or fails, it falls back to _raw_extract (UTF-8 reads, 200,000 chars per file, up to 8 images). Results are cached per run dir in _EXTRACT_CACHE. deliverable_files, deliverable_images and deliverable_text (the judge's "flat" rendering) read from it.

Scoring (score):

1. Try existing detail files in order: a "reporting judge" pattern from config.JB_REPORTING_JUDGE_DETAILS, then the configured judge's eval_result/eval_<label>/<model>_judge.json (_details_candidates).
2. _read_details moves judge-failed files aside to .judge_failed (_judge_failed detects transport errors and "opus judge failed" style reasons). It refuses partial files (fewer rubrics than total_count).
3. If nothing was delivered: _no_output_score is a real zero built from the rubric weights (no_output: True).
4. Otherwise _run_judge runs upstream judge.py --output-dir … --rubrics-file … --details-file … --judge-model JB_JUDGE_MODEL --api-base JB_JUDGE_API_BASE --max-workers 8 --max-retries 3 --timeout-per-rubric 240 --evaluated-model <label>. The token comes from JB_JUDGE_API_KEY or gcloud auth application-default print-access-token, and JUDGE_REASONING_EFFORT is passed through (the patched judge reads it; see section 8).
5. _parse_details returns {n_passed: total_score, n_criteria: max_score, all_pass, metric: "weighted", criteria_passed, criteria_count, judge_model}, or an "unweighted-fallback" when max_score is 0.

preflight: substrate preflight; upstream judge.py present; for cx, HARNESS2_CODEX_API_KEY set and codex_config_error() clean; judge interpreter present and able to import pandas, openpyxl, docx ("Without them the judge grades '[ERROR: ... not available]' text as a real zero"); a warning if HARNESS2_JUDGE_BASE_URL is empty.

6. LAB (harvey-labs) adapter

Source: benchmarks/lab.py. The docstring: "one class for both substrate cells. Upstream tools run in subprocesses under config.LAB_PYTHON. _task_json strips grading criteria before exposing task data to the improver."

6.1 Tasks, domains, prompts

• Tasks: <practice-area>/<task> with task.json (instructions, deliverables, work_type, tags, criteria) and a documents/ data room. Domain = practice area.
• LAB_BASE: legal work for a supervising lawyer; domain_word="practice area", deliverable_word="legal deliverable".
• LAB_WORK_TYPES: analyze, review, draft, research.
• LAB_PRACTICE_AREAS: 26 practice lenses (antitrust-competition … white-collar-defense-investigations).
• lab_prompts(*, practice_area, work_type); prompts_for reads work_type from the criteria-stripped task.
• _task_json: "task.json WITHOUT criteria — the ONLY place this adapter reads a task." task_prompt(task_id, *, max_chars=12_000) renders title, work type, tags, instructions and deliverables as markdown. task_files = documents/.

6.2 Spec, constructor and modes

lab_spec(substrate_name): name="lab", install mode workspace_config (oc) or codex_root (cx), leak_globs=("task.json",), allow_subagents=False, reserved_scripts=_runtime_scripts() on oc (the basenames of upstream harness/skills/*/scripts/*.py, L455-L461, so a harness cannot shadow runtime scripts), obs_trunc=50_000.

LabAdapter.init(*, substrate="oc", model, timeout, skill_mode="native", solver_effort="high", judge_model=JUDGE_MODEL, split_file). skill_mode native loads the docx/xlsx/pptx manuals on demand; inline puts them in the always-on body ("which every proposal must re-emit. Switching moves the recorded baseline."). Results live under RUNS/lab/results/{oc,cx}/<run_id>. Default timeout 3600 s. JUDGE_MODEL = config.LAB_JUDGE_MODEL.

6.3 Base harnesses and validation

base_harness materialises the tree once per process (and per pid, under _BASE_LOCK). On oc, systemprompt.md = _opencode_base_body(skill_mode). That function runs _BASE_BODY_SRC in the LAB interpreter: upstream compose_prompt(DEFAULT_SKILLS) with the overlay's ENV_DELTA stripped off the end (the runner re-appends it). On cx, AGENTS.md = CODEX_BASE_BODY (workspace layout, scripts/read_doc.py for binary documents, write to output/).

validate_tree enforces two environment invariants that would otherwise fail silently:

• oc (_validate_oc): writes the tree, runs _RENDER_CHECK_SRC (upstream overlay render_tree), and requires the rendered prompt to mention output/ and read_doc.py.
• cx (_validate_cx): the AGENTS.md body must contain both strings.

6.4 Launchers and execute

execute: reuse if the marker matches and artifacts_intact (requires metrics.json plus a matching output_fingerprint over output/). Otherwise it removes the stale dir with _safe_rmtree, which refuses anything outside the results root. Then _install → substrate.execute → require metrics.json ("the runner refuses to write it when the session stream had no parseable events") → _check_tool_gate (every tools entry in the emitted workspace/opencode.json must agree with permission) → mark_complete in the run dir.

_install: with harness_dir=None, the base harness is a value (Install(staging_dir=base_harness(), fingerprint="base", extras={"harness": "base"})); otherwise substrate.materialise plus extras["harness"]="staged".

• _launch_oc: LAB_PYTHON -u <lab_repo>/opencode/run_opencode.py --task … --run-id … --results-root … --model … --tool-mode base --reasoning-effort … --skill-mode … --timeout … --opencode-dir … --bun …, plus --harness-dir <stage> unless base. Wall 2*timeout + 600. A wall timeout is logged, not raised, and the missing metrics.json then fails the rollout.
• _launch_cx: <lab_repo>/external/adapt_runs_cx/run_codex_lab.py with ADAPT_LABCX_* env (codex bin, results, scratch, base URL) and VERTEX_PROXY_KEY. Wall timeout + 900.
• _lab_env: PYTHONPATH=<lab_repo>, and PATH gets the LAB python, bun and codex-path directories.

6.5 Read-back and judge

• delivered: any file under output/.
• trajectory: opencode_session.jsonl or codex_session.jsonl, through the substrate renderer, max_chars=10**9 ("effectively uncapped").
• deliverable_files: runs _EXTRACT_SRC inside the LAB repo. It imports the judge's evaluation.scoring._read_file_as_text at the top of the driver (a missing reader is a hard failure, not a per-file degradation), caps text at config.LAB_DELIV_CAP, and raises if extraction fails. deliverable_text joins the files with ----- FILE: name ----- headers. deliverable_images returns up to 8 images.
• score: if scores.json is absent, run python -m evaluation.run_eval --run-id <ABSOLUTE run dir> --task … --judge-model … (the absolute path is required because "run_eval joins its own RESULTS_DIR with it, and an absolute right-hand side wins that join") and log to judge.log. OSError or timeout → None. Returns {n_passed, n_criteria, all_pass}.
• preflight: substrate preflight; LAB interpreter present; the runner overlay present; on oc, _require_runtime_scripts() refuses an empty exemption set; import evaluation.scoring must work.

7. WorkBuddy-Bench adapter

Source: benchmarks/workbuddy.py and benchmarks/wb_render.py. The docstring: "one (subset, substrate) cell per WbAdapter instance. Execution is a Harbor batch, not a single run: one label is one rollout-store namespace and one batch is one generated job YAML with n_attempts=k. Everything a model sees of a deliverable comes through wb_render."

7.1 Subsets

Subset (frozen): name, dataset_id, domain_field, deliverable, scoring, leak_globs, llm_judge_model="", plus the property tasks_dir. SUBSETS:

Constants: SCOREABLE = code/web/office (exported, not referenced elsewhere at this commit); SUBSTRATES = ("oc", "cx"); TRAJ_MAX_CHARS = 2_000_000; EPHEMERAL_PREFIX = "_ab-"; MAX_JOBS/MAX_TRIALS from config (WB_MAX_JOBS = {"oc": 1, "cx": 1}, WB_MAX_TRIALS = {"oc": 2, "cx": 2}, config L170-L171), with one semaphore per substrate.

Prompts: WORKBUDDY_BASES (one BenchPrompts per subset; web has a "DOES IT WORK" selection_lead), WORKBUDDY_DOMAINS and workbuddy_prompts(subset, *, domain).

spec_for(subset, substrate): name="wb-<subset>", runs_root=RUNS/wb/<subset>, install mode home_dir_jinja (oc) or codex_root (cx), allow_subagents on oc, proposer_deliverables="names"** for code** (the proposer sees only changed file names), and state_deliverable=True.

7.2 Harbor batches

The harness slugs are _harness_oc(fork) (opencode/<ver> for the stock harness, oc-harness/<ver> when a harness workspace is supplied) and _harness_cx() (codex-harness/<CODEX_VERSION>). WB_OC_HARNESS_VER = "1.14.18".

_job_yaml(...) generates the job file: model: WB_SOLVER_MODEL, harness, dataset, harness_backend: local, model_connection: WB_MODEL_CONNECTION (local_proxy), n_attempts: k, jobs_dir, a name-based task selection, and n_concurrent_trials. For oc it adds harness_params_override with OPENCODE_VARIANT: high, WB_HARNESS_WS and prompt_template_path (the _composed/prompt.j2 from home_dir_jinja). For cx it adds extra_allowed_hosts: host.docker.internal, a read-only bind mount of CODEX_DIST at /opt/codex, env VERTEX_PROXY_KEY, VERTEXAI_LOCATION, CODEX_BRIDGE_URL, and CODEX_MODEL/WB_HARNESS_WS. Web adds llm_judge_override.

run_batch(...): under a per-label lock it skips done tasks and resumes partially (asking for k - have attempts). It writes configs/jobs/_ab-<label>-<sha1>.yaml and runs uv run ./scripts/run.sh --job <slug> in its own session, inside the substrate semaphore. On timeout it calls _kill_group (SIGTERM, 180 s grace, then SIGKILL). The YAML is always deleted afterwards, and the function returns only the run dirs this invocation created. A non-zero exit is logged ("partial results are still usable"). sweep_orphans deletes stale _ab-*.yaml files; nothing in harness2/ calls it at this commit.

Trial names are mapped back to task ids with resolve_task, because Harbor truncates names to 32 chars (_HARBOR_NAME_MAX). Ambiguous prefixes raise.

7.3 WbAdapter

• init(subset="office", *, substrate="oc", split_file): both substrates get the same self._launch.
• _label(run_id, n) requires the run id to start with <spec.name>-<substrate.name>- ("labels are constructed by core.schedule.ns only") and appends __x<n> for batches.
• _cell_identity records solver model, OC harness version and connection in the marker. _reusable requires a finished trial (verifier/score.json exists; only score() may read its value), a harness-fingerprint match and an identity match.
• _install: base = Install(staging_dir=runs_root, fingerprint="base", extras={"install_mode": "none"}), which means stock OpenCode or Codex with no harness workspace.
• _launch: k = len(run_ids); on cx it calls preflight(require_dist=True, require_bridge=True); runs one batch; returns the fresh finished trials ranked newest-first, graded first (order_graded_first).
• execute: reuse the newest reusable trial, or run a one-task batch, mark the chosen trial with the cell identity and return it.
• execute_many: k replicates in one batch. "Only the trials this invocation produced are marked, so a killed batch's leftovers cannot be laundered into the new identity."
• delivered: at least 200 rendered body characters.
• trajectory: agent/oc-output.txt or agent/codex-output.jsonl, max_chars=TRAJ_MAX_CHARS.
• deliverable_files/deliverable_listing/deliverable_text/deliverable_assets all go through wb_render. Assets (raw office binaries) exist only for office.
• base_harness: an empty OpenCode tree (four empty always-on files plus stubs), or a one-sentence Codex AGENTS.md.
• validate_tree: on oc, it runs assert_jinja_safe(compose_prompt(tree)), so "a brace in a skill's description: is a rejected proposal rather than a dead trial". On cx it checks nothing.
• preflight: uv on PATH, scripts/run.sh in the checkout, and the harness version YAMLs present (they come from third_party/wb_runner/configs/harnesses/), plus substrate preflight.
• score: if nothing was delivered and the stream ends on a provider error (_killed), it returns None, because the verifier's zero would be a judge-side artefact. Otherwise it reads verifier/score.json with score_from_payload (precedence reward > overall > test_pass_rate > tests_passed/tests_total; values outside [0,1] are rejected, not clamped) and returns {n_passed: round(sc*100), n_criteria: 100, all_pass: sc >= 1.0, raw_score}.

7.4 wb_render: the leak-proof deliverable view

A WB trial directory contains both the agent's work and the verifier's output from the same trial. wb_render reads only agent-authored bytes:

• ALLOWLIST: office → verifier/raw_artifacts/**, verifier/agent.patch; web → verifier/agent.workspace.tar.gz; code → verifier/agent.patch; sec → artifacts/logs/artifacts/** (a subset the renderer knows but SUBSETS does not expose).
• _admit(trial, path, subset): every read goes through it. Anything else raises RenderInputError: "Everything else in a WB trial directory is grader output … and reading it is the leak this module replaces."
• _is_verifier_authored: for office/sec, it quarantines files named judge_evidence*, files whose head declares verifier provenance, or files containing verdict markers (pass_rate, failed_checks…).
• render / render_report (the selector's per-file view plus the quarantine log), flat_text (one document with a "Files produced" index), body_chars, raw_files, listing (the proposer's names-only view, taken from patch headers or tar members). Binary .xlsx/.pptx are rendered with openpyxl/python-pptx, or through a bridge interpreter; failure raises RenderBackendError rather than degrading to a stub.

7.5 Special requirements, by benchmark

The WB setup also requires Gemini-on-Vertex model ids for solver and judge (install.workbuddy_models). Chapter 08 covers setup.

8. Vendored runners: provenance, pins, patches, licenses

8.1 What is in third_party/

From third_party/README.md: "derived benchmark runners, compatibility patches, and upstream license notices… included in wheels so setup works outside a source checkout." Research logic belongs in harness2/core/, benchmark interfaces in harness2/benchmarks/. Bundled asset hashes are recorded in dependencies/ASSETS.json and checked with python scripts/check_provenance.py.

All three PROVENANCE files say the integration "adds editable harness installation and configurable models, paths, and endpoints", and that local provider names, env vars and log paths use the harness2 namespace.

How the harness tree reaches the agent inside a WB container: OcHarnessAgent installs skills, agents, plugins and scripts from WB_HARNESS_WS into the agent's home (.config/opencode/{skill,agent,plugin}, ~/harness_scripts), and an empty workspace reproduces stock OpenCode (oc_harness_agent.py). CodexHarnessAgent installs the four codex_root entries at /workspace and commits them before the agent starts (codex_harness_agent.py). The version YAMLs disable auto-update and telemetry and set OPENCODE_DISABLE_PROJECT_CONFIG=1, so that "a repo under test could [not] redefine the provider block or re-enable a denied tool".

8.2 How upstreams are pinned

config.DEPENDENCY_PINS is "the authoritative pin table" (dependencies/PINNED.md), and DEPENDENCY_ORIGINS gives the remotes:

Codex is pinned by version (CODEX_VERSION = "0.144.6"), and OpenCode uses Bun 1.3.11 with the patched lock (PINNED.md).

The mechanics live in harness2/install.py:

• clone(name): git init + fetch --depth 1 origin <pin> + checkout --detach FETCH_HEAD. It refuses a checkout at a different HEAD ("move this checkout aside"), a non-empty directory, or a wrong origin.
• PATCHES maps each repo to its diff. patch(name) is idempotent: if git apply --reverse --check succeeds the patch is already applied and is skipped; otherwise git apply --check and then git apply. preflight.py L71-L74 uses the same reverse check to verify patches are in place.
• overlays(name): copies lab_runner/{opencode,tts_lab,adapt_runs_cx} into the LAB checkout (opencode/, external/tts_lab, external/adapt_runs_cx), and wb_runner/{agents,configs/harnesses} into WB (src/workbuddy_bench/agents, configs/harnesses), ignoring __pycache__ and PROVENANCE.md.
• RUNNERS and missing_runners(): a copy of the code without third_party/ still runs the demo and tests, but setup reports which files are missing.

8.3 What each patch changes

8.4 Licenses

Each runner directory also keeps its own LICENSE copy. Harness² code is Apache 2.0 (file headers).

9. Worked example: one JobBench task under OpenCode

Take a JobBench task id of the form <profession>/task<N>, say lawyers/task1. The id is illustrative: the lawyers profession lens exists in the table, but whether that exact task number exists depends on the downloaded dataset. Core asks for one rollout of a candidate harness tree.

sequenceDiagram
  participant Core as core.recursion._roll
  participant JB as JobBenchAdapter
  participant OC as OpencodeSubstrate
  participant L as JB._launch_opencode
  participant R as jb_opencode_runner.py
  participant Bun as bun + OpenCode
  Core->>Core: write_tree(surface, tree, work_dir/harnesses/label)
  Core->>JB: execute(task_id, hdir, run_id)
  JB->>JB: label from run_id (slash to double underscore)
  JB->>JB: reuse check (runmark + artifacts_intact)
  JB->>JB: wipe stale output, traj, judge details
  JB->>JB: _ensure_store_task (copy task_folder once)
  JB->>OC: materialise(hdir, spec, {})
  OC-->>JB: Install(staging_dir, fingerprint)
  JB->>OC: execute(task_id, install, [run_id], timeout)
  OC->>L: launcher(RunRequest, Install)
  L->>OC: preflight()
  L->>L: render_opencode_tree into hrender dir
  L->>R: python subprocess, env contract, wall 2t+600
  R->>R: stage /tmp/hevo_ws workspace
  loop up to MAX_RETRIES=2
    R->>Bun: timeout T bun run src/index.ts run PROMPT --format json
    Bun-->>R: JSONL events, files in output/
  end
  R-->>L: exit 0 if model_output/label non-empty
  L->>L: _require_artifacts
  L-->>OC: [model_output/label]
  OC-->>JB: [run_dir]
  JB->>JB: rm staging, mark_complete(model_traj/label, output_fingerprint)
  JB-->>Core: run_dir
  Core->>JB: delivered(run_dir)
  Core->>Core: RunView(label, run_id, run_dir, ok, delivered)

Step by step, with the code:

1. Adapter construction. make_adapter("jb", None, "oc") → JobBenchAdapter(substrate="oc"): spec = make_spec("oc") (install mode workspace_agents_md), substrate = OpencodeSubstrate(launcher=self._launch_opencode), timeout = 2400, store = RUNS/jb/dataset.
2. Core materialises the tree. _roll writes the candidate tree with write_tree(adapter.substrate.surface(), tree, work_dir/"harnesses"/label) and calls adapter.execute(task_id, hdir, run_id). For a base rollout, hdir is None.
3. Idempotency. execute computes label = run_id.replace("/", "__"), out = <store>/lawyers/task1/model_output/<label>, mark = …/model_traj/<label>. If RM.is_complete(mark, harness_dir=hdir) (marker version 1 and the same tree fingerprint) and artifacts_intact(out), it returns out immediately. Otherwise it prints why it is re-rolling, deletes out and mark, evicts the extract cache, and deletes the old judge details so a stale score cannot attach to a new rollout.
4. Seed the store. _ensure_store_task copies the dataset's clean task_folder into the store once, atomically (temp dir + os.rename under a lock).
5. Stage. OpencodeSubstrate.materialise(src, spec, {}) copies the tree to /tmp/adapt-oc-workspace_agents_md-XXXX and returns Install(fingerprint=RM.fingerprint(src), extras={"install_mode": "workspace_agents_md"}).
6. Execute via launcher. OpencodeSubstrate.execute builds RunRequest(task_ref="lawyers/task1", run_ids=(run_id,), timeout=2400) and calls _launch_opencode.
7. Render. The launcher runs self.substrate.preflight(), then render_opencode_tree into hrender_<label>_XXXX/: AGENTS.md (system prompt + guardrails + tool guidance + memory + scripts list, with {{domain_focus}} = lawyers), .opencode/skills/*, .opencode/agent/*.md, scripts/, .opencode/plugin/*.ts.
8. Runner subprocess. _run_runner(OC_RUNNER, env, wall=2*2400+600) runs python jb_opencode_runner.py in its own process group. run_one stages /tmp/hevo_ws/lawyers_task1_<label>_<pid>/ with task_folder/, output/, .opencode/, AGENTS.md and scripts/.
9. OpenCode runs. _attempt runs timeout --kill-after=30s 2400 bun run --cwd <opencode>/packages/opencode --conditions=browser src/index.ts run "<PROMPT>" --dir <ws> --model <SOLVER_MODEL_OC> --title "lawyers_task1 | <label> | a1" --format json, with OPENCODE_PERMISSION fencing reads to the workspace and /tmp. AGENT_NAME is empty, so AGENTS.md is appended to the provider prompt. Event lines go to model_traj/<label>/lawyers_task1_<label>_attempt1_<stamp>.jsonl. If output/ is still empty, the task folder is reset and attempt 2 runs.
10. Collect. Deliverables move to model_output/<label>/ and the workspace is deleted. The runner exits 0 only if deliverables landed.
11. Verify and mark. _require_artifacts confirms output exists. Back in execute, the staging dir is removed and RM.mark_complete(mark, run_id, harness_dir, extra={"output_fingerprint": …}) writes .adapt_complete into model_traj/<label>/ atomically. Note that the marker lives in the trajectory dir, not the output dir, so the output stays exactly what the agent produced. execute returns out.
12. RunView. Core builds RunView(label, run_id, run_dir=out, ok=True, delivered=adapter.delivered(out)). Had any step raised, for example the runner exiting 1 because of a dead proxy, core would record RunView(ok=False, delivered=False, error="RuntimeError: JobBench produced no output … left UNMARKED …") and the next attempt would re-roll.
13. Afterwards. When core builds proposer/selector contexts it calls adapter.trajectory(out, obs_trunc=spec.obs_trunc) (newest .jsonl → render_events → [ACTION] bash({...}) / [OBSERVATION] …) and adapter.deliverable_text(out) (upstream judge extractor in the judge interpreter). Only after selection commits does core.judge call adapter.score(out, "lawyers/task1"), which runs upstream judge.py against RUBRICS.json and returns the weighted pass totals, or None if the judge failed.

10. Adding your own benchmark

Everything below is grounded in the BenchmarkAdapter protocol and in how the three shipped adapters satisfy it. The smallest complete example in the repo is demo.DemoAdapter, which runs a local script instead of an agent.

Step 1: discover tasks. Implement tasks(split="all") -> list[str] in a stable, sorted order. If your data has a train/test partition, reuse tasks.select (you would add a branch to discover for your dataset layout). Implement domain_of(task_id) -> str. Optionally implement reference_domain_order() -> list[str] so the schedule's frozen order covers all domains.

Step 2: write the prompt profile. Build a BenchPrompts with work, inv_levers, dom_levers, yardstick and selection_checks that describe the work, not its measurement. The constructor rejects the words benchmark/evaluation/grader/rubric/score. Optionally implement prompts_for(task_id) to add per-domain lenses into domain_guidance (see jobbench_prompts, lab_prompts, workbuddy_prompts).

Step 3: build the BenchSpec** per substrate.**

from harness2.core.spec import BenchSpec, resolve_channel_model

def my_spec(substrate: str) -> BenchSpec:
    ch = resolve_channel_model(None, None, effort="high")
    return BenchSpec(
        name="mybench",
        runs_root=config.RUNS / "mybench",
        install_mode="workspace_agents_md" if substrate == "oc" else "codex_root",
        prompts=MY_BASE,
        channels={"proposer": ch, "selector": ch},
        leak_globs=("answers/**", "rubric.json"),   # every answer-key path
        allow_subagents=(substrate == "oc"),
        proposer_deliverables="contents",            # or "names" + deliverable_listing
    )

Pick the install mode that matches how your runner delivers always-on text. The modes are not interchangeable baselines.

Step 4: give the agent the task, minus the answers. task_prompt must strip criteria and expected output (LAB pops criteria in exactly one place, _task_json). task_files returns the source material or None. Put every answer-key path in leak_globs: core asserts none of them appears in a context directory.

Step 5: supply a base harness. base_harness() -> Path must be a valid tree for self.substrate.surface(). Either ship one (JB) or write it with harness_tree.write_tree(SURFACE, {...}, d) once per process (LAB, WB). Raise if it is missing, so base runs can never be silently harness-free.

Step 6: write the launcher and execute. Construct the substrate with your launcher, OpencodeSubstrate(launcher=self._launch). The launcher has the signature Launcher = Callable[[RunRequest, Install], list[Path]] and must return one run dir per run id. In execute(task_id, harness_dir, run_id, *, timeout=None) -> Path:

1. Reuse when RM.is_complete(where, harness_dir=harness_dir) (and optionally your own artifacts_intact).
2. install = self.substrate.materialise(harness_dir or self.base_harness(), self.spec, ctx).
3. dirs = self.substrate.execute(task_id, install, [run_id], timeout), then remove install.staging_dir in a finally.
4. Raise if the runner produced nothing. Never mark an empty rollout.
5. RM.mark_complete(where, run_id=run_id, harness_dir=harness_dir, extra={...}), then return the run dir.

If your runner batches natively, also implement execute_many (as WB does).

Step 7: read back. delivered(run_dir); trajectory(run_dir, *, obs_trunc), usually self.substrate.read_trajectory(stream, obs_trunc=obs_trunc, max_chars=…) so the grammar is shared; deliverable_text/deliverable_files from agent-authored bytes only, ideally through the judge's own reader so the selector sees what the judge grades. Optional: deliverable_images, deliverable_listing, deliverable_assets.

Step 8: validate trees. validate_tree(tree) raises one ValueError that lists every environment invariant the proposal broke, in prose the proposer can act on (see LAB's output/ and read_doc.py rules, and WB's Jinja check).

Step 9: score. score(run_dir, task_id) -> dict | None with at least n_passed, n_criteria and all_pass (the shape all three adapters return). Return None when the judge failed, and a real zero only when the agent genuinely delivered nothing. Do not call it anywhere else; core.judge owns the timing.

Step 10: preflight and wiring. Add preflight() that checks every interpreter, checkout, binary and credential before any spend. Then register the bench in cli/_common.make_adapter and BENCHES (L30), add discover support, and, if you vendor an upstream, add a pin to DEPENDENCY_PINS/DEPENDENCY_ORIGINS, a patch in PATCHES, overlays in install.overlays, a PROVENANCE.md and a license copy.

11. Adding your own executor (substrate)

Step 1: define the surface. Write a SurfaceSpec (core/tree.py): name, always_on files and always_on_max with a reason, skill_dir and skill_name_rule, trigger_hint/autotrigger_file/trigger_line, subagent_requires_mode, plugin_dir and/or hooks_file (+ hook_events), optional compile_skill_scripts/reject_empty_always_on, and stubs READMEs for each component directory. Chapter 04 explains how core validates trees against it.

Step 2: add an install mode. BenchSpec.__post_init__ accepts only values in INSTALL_MODES, so a new delivery channel means a new entry there.

Step 3: implement the Substrate** protocol** (base.py):

class MySubstrate:
    name = "my"                                   # short id, used in namespaces

    def __init__(self, launcher: Launcher | None = None):
        self._launcher = launcher                 # benchmark-supplied

    def surface(self) -> SurfaceSpec: ...
    def materialise(self, tree_dir: Path, spec, ctx: Mapping[str, str]) -> Install:
        # check spec.install_mode, check HT.is_tree(SURFACE, tree_dir),
        # stage into a mkdtemp, clean up on error,
        # return Install(staging_dir, fingerprint=RM.fingerprint(tree_dir), extras={...})
    def execute(self, task_ref: str, install: Install, run_ids: list[str], timeout: int) -> list[Path]:
        if self._launcher is None: raise RuntimeError("...adapter must supply one...")
        return self._launcher(RunRequest(task_ref, tuple(run_ids), int(timeout)), install)
    def read_trajectory(self, stream_path: Path, *, obs_trunc: int, max_chars: int,
                        args_trunc: int = 400) -> str:
        # emit [ACTION] / [OBSERVATION] / [OBSERVATION ERROR] / [ASSISTANT] / [ERROR] lines,
        # clip with the shared "…(+N chars — render truncation)" marker
    def preflight(self) -> None:
        # check binaries/checkouts/env; raise RuntimeError with the remedy; start nothing

Keep per-deployment differences in Install.extras, not subclasses (Codex's rule). Fingerprint the source tree with RM.fingerprint, so reuse is keyed by what the proposer wrote.

Step 4: optionally publish a reference CLI contract (like run_argv/run_env or exec_argv/exec_env) for launchers to copy, and make the sandbox decisions explicit there.

Step 5: teach each adapter to use it. Every adapter switches on substrate in its spec factory (install mode, allow_subagents, reserved_scripts), its constructor (which class, which launcher), base_harness and validate_tree, and the CLI's SUBSTRATES tuple (cli/_common.py L32). WB additionally needs a Harbor harness entry and agent class under third_party/wb_runner/, plus a registration in the WB runtime patch.

Key takeaways

• Three axes, one seam. Method code sees only BenchmarkAdapter, BenchSpec, RunView and SubstrateLike (name + surface()). Benchmarks and executors plug in without core changes.
• Substrates own the surface, staging, fingerprint and trajectory grammar. Benchmarks own launching. Both substrates refuse to run without a benchmark-supplied launcher.
• OpenCode has 4 always-on files, 3 install modes, runs from a pinned source checkout via bun, and is fenced with OPENCODE_PERMISSION. Codex has one AGENTS.md (8k cap), copies four root entries, bypasses its own sandbox and hook trust, and is configured through CODEX_HOME.
• RunView** never carries a score**, and score() returning None means "the judge failed", not zero. Dead rollouts are never marked, so they are re-rolled rather than graded.
• Leak control is layered: leak_globs, criteria stripping in task_prompt, the judge's own extractors, and WB's allowlist-plus-quarantine renderer.
• Upstreams are pinned by commit, patched idempotently and overlaid by setup, with provenance, asset hashes and licenses kept next to the vendored code.
• Not determined by the code: values in RunView.cost_usd/wall_s (never set), the concrete JobBench task numbers in your download, and how selection_conformance="value_only" is consumed (chapter 06). SCOREABLE, sweep_orphans and codex.final_message are exported but unused at this commit.

Where to go next

• 00 · Start here: the map of the whole guide.
• 01 · Architecture overview: where the three axes sit in the full system.
• 02 · Code structure: package layout around substrate/, benchmarks/ and third_party/.
• 03 · One task, end to end: the method's view of the rollout traced in section 9.
• 04 · The harness tree: SurfaceSpec, c1..c8, write_tree/is_tree and validation.
• 05 · Evidence and safety: leak_globs, context building, verdict quarantine.
• 06 · Budget, selection and judging: how RunViews become selections, and when score() is called.
• 08 · Setup and running: setup_benchmark.py, pins, patches, overlays and preflight in practice.