08 · Setup and running

08 · Setup and running

This chapter takes you from an empty Linux machine to a judged, reported Harness² benchmark run. Harness² ships seven console commands (harness2-demo, harness2-setup, harness2-check, harness2, harness2-judge, harness2-report, harness2-bridge). You normally run them in a fixed order: setup → check → (dry-run) → run → judge → report. Each step writes to a predictable place under dependencies/ or runs/, and each refuses to go on when something is missing. Real runs cost money: every task costs R = 4k executor rollouts and P = 3k improver sessions per recursion mode. So the chapter shows how to keep a first run down to a single task, how to try the whole workflow for free with the local demo, and why you must block the executor's web access before you evaluate. Every claim below links to the code at commit 1cfe254.

All links point at commit 1cfe254e8d25b15f37048c3730134686abe60335. Line numbers were computed from that commit.

1. The map: seven commands and where they live

The entry points are declared in pyproject.toml [project.scripts]:

There are also three convenience wrappers in scripts/. scripts/setup_benchmark.py and scripts/check_setup.py are compatibility aliases for setup and check; many error messages still say "run scripts/setup_benchmark.py <bench>", which is the same as harness2-setup <bench>. scripts/run_lab.sh, run_jobbench.sh and run_workbuddy.sh chain check → run → judge → report for you.

flowchart TD
  I["Install: uv venv + uv pip install -e .[render]"] --> D{"Free demo first?"}
  D -->|yes| DEMO["harness2-demo<br/>no credentials"]
  D -->|no| ENV["Write .env + gcloud ADC login"]
  DEMO --> ENV
  ENV --> S["harness2-setup bench"]
  S -->|"missing git/uv/npm, pin or patch conflict"| SF["setup failed: exit 1"]
  S --> C["harness2-check --bench"]
  C -->|"FAIL lines"| CF["Setup incomplete: exit 1<br/>fix, rerun setup"]
  C --> DRY["harness2 ... --dry-run<br/>prints task stream, no cost"]
  DRY --> LOCK["Block executor web access<br/>huggingface.co, github.com"]
  LOCK --> R["harness2 --bench --k --mode<br/>cost: 4k rollouts + 3k sessions per task"]
  R -->|"ConfigError: Setup incomplete"| RF["error: exit 1, nothing spent"]
  R -->|"config.json mismatch"| RI["new HARNESS2_RUNS_DIR"]
  R -->|"some tasks failed: exit 2"| RR["rerun same command = resume"]
  RR --> R
  R --> J["harness2-judge same flags"]
  J -->|"ungraded rows: exit 2"| JR["rerun later e.g. 429 quota"]
  JR --> J
  J --> P["harness2-report same flags"]
  P --> OUT["Partial / All-pass or score<br/>+ improver spend"]
  click DEMO href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/demo.py#L27-L54"
  click S href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/setup.py#L26-L116"
  click SF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/setup.py#L109-L110"
  click C href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/check.py#L25-L45"
  click CF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/check.py#L36-L40"
  click DRY href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/run_sweep.py#L91-L118"
  click LOCK href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/docs/EXPERIMENTS.md#L93-L109"
  click R href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/run_sweep.py#L131-L206"
  click RF href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/preflight.py#L147-L160"
  click RI href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/run_sweep.py#L209-L235"
  click RR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/core/schedule.py#L350-L421"
  click J href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/judge.py#L150-L200"
  click JR href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/judge.py#L65-L81"
  click P href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/cli/report.py#L58-L110"
  click ENV href "https://github.com/google-research/harness2/blob/1cfe254e8d25b15f37048c3730134686abe60335/harness2/config.py#L35-L63"

2. Requirements

2.1 Platform and Python

• OS: Linux is the supported platform for benchmark runs (README Install, SETUP Requirements). pyproject.toml declares only Operating System :: POSIX (L12-L16). The code also needs POSIX fcntl for summary locking (writesummary).
• Python: requires-python = ">=3.12,<3.14" (pyproject L11). The docs ask for 3.12 for benchmark runs. 3.13 is supported for the package itself (SETUP L9-L10). The benchmark environments are always built with Python 3.12 (uv sync --locked --python 3.12, setup.py L68).
• Core package dependencies: none (dependencies = [], L20). There are two optional extras (L22-L24):
  • render = openpyxl, python-pptx. WorkBuddy office checks these in the main environment (preflight L118-L123). The README installs it by default.
  • bridge = litellm[proxy]==1.95.0, fastapi==0.139.2, google-auth==2.56.3. Only needed for harness2-bridge (the Codex executor on Vertex).

2.2 System tools: who needs what

The Debian/Ubuntu one-liner from the docs (SETUP L12-L15):

sudo apt-get update
sudo apt-get install -y git curl unzip ca-certificates pandoc poppler-utils

2.3 Which benchmark needs what

The OpenCode checkout (anomalyco/opencode at 23fb5e0…) is installed for every benchmark, even with --substrate cx. harness2-check always runs OpencodeSubstrate().preflight and requires dependencies/opencode/node_modules (L76-L77).

2.4 Install the package

From the repository root (README L86-L91):

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'          # add ,bridge for the Codex executor

Where things resolve. config.ROOT is the repository root when pyproject.toml sits beside the package (an editable/source install). Otherwise it is the current working directory. ROOT/.env, ROOT/dependencies/, ROOT/third_party/ and the default ROOT/runs/ all hang off it. For a wheel install, SETUP "Wheel installations" says to work from a dedicated directory. Note that config.THIRD_PARTY = ROOT / "third_party" (L25), and setup refuses to run when the runners/patches are absent there (missing_runners). With a wheel you therefore need a third_party/ folder in that working directory. The wheel's packages.find only includes harness2* (pyproject L35-L36).

3. Configuration: .env and environment variables

3.1 How .env is loaded

load_dotenv() runs at import time of harness2.config (L63). It reads ROOT/.env and puts each KEY=VALUE into os.environ with setdefault, so a shell variable always wins over .env. It accepts blank lines, # comments, an optional export prefix, single- or double-quoted values, and trailing # comments on unquoted values. A missing file is not an error. Keys must be alphanumeric/underscore.

env(name, default) treats a set-but-empty variable as unset. require(name, needed_by=…) raises ConfigError with the fixed remediation line:

<NAME> is required for <needed_by>; set it in .env or the shell (see .env.example)

.env.example** is not in the tree at this commit.** The README (L100-L104) and SETUP (L30) say cp .env.example .env, and MANIFEST.in lists it, but git ls-tree at 1cfe254 has no such file. Write .env by hand from the table below.

A minimal .env for a Vertex Gemini LAB run (the model IDs are placeholders, so use IDs your project can serve):

GOOGLE_CLOUD_PROJECT=my-gcp-project
# GOOGLE_CLOUD_LOCATION=global           # default
HARNESS2_SOLVER_MODEL=google-vertex/<gemini-model-id>
# HARNESS2_IMPROVER_MODEL=google-vertex/<model-id>   # defaults to the solver
HARNESS2_JUDGE_MODEL=<gemini-model-id>   # bare id, no provider prefix
# HARNESS2_RUNS_DIR=runs/lab-first-try

3.2 Every variable

All values are read once, at import (config.py L91-L178). Changing .env mid-run has no effect on a running process.

These variables are set by Harness² for subprocesses, so you do not set them yourself:

• The improver session environment strips ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, ANTHROPIC_BEDROCK_BASE_URL and CLAUDE_CODE_USE_BEDROCK (proposer DROPENV). It adds GOOGLE_VERTEX_PROJECT/GOOGLE_VERTEX_LOCATION/VERTEXAI_LOCATION, or the Claude-CLI Vertex trio for the claude_cli channel (Proposer.env).
• HARNESS2_BRIDGE_KEY is generated by the bridge and handed to LiteLLM (bridge L97-L104).
• HARNESS2_WB_VERTEX_URL points WorkBuddy's job-local proxy at the Vertex chat endpoint (workbuddy L623).
• HARNESS2_CODEX_BIN and VERTEX_PROXY_KEY are set for the JobBench Codex runner (jobbench L815-L826).

There are also fixed constants you cannot change from the environment (edit config.py if you must): PHASE_WIDTH = 2 (rollout width), TRANSPORT_ATTEMPTS = 2, PROPOSER_CONCURRENCY = 2 (L176-L178), LAB_DELIV_CAP = 60000, JB_JUDGE_REASONING_EFFORT = "low", CODEX_VERSION = "0.144.6", WB_MAX_JOBS/WB_MAX_TRIALS (L136-L172), and METHOD_VERSION = 2, which enters the run identity (L26).

3.3 Authentication

Every provided configuration goes through Google Application Default Credentials:

gcloud auth application-default login

The alternatives listed in SETUP L54-L66 are an attached service account, --no-browser, GOOGLE_APPLICATION_CREDENTIALS=<key.json>, and --impersonate-service-account=<account>. The check only proves that a token can be minted. It never proves your account can serve a model. That is first learned at the first real call (SETUP L101-L103, require_ready docstring).

3.4 Model validation

validate_models(judge=…) is the first gate in every model-touching command. It checks that:

1. HARNESS2_SOLVER_MODEL is set.
2. The solver and the improver both contain / (OpenCode provider/model syntax).
3. GOOGLE_CLOUD_PROJECT is set if either is a google-vertex… model.
4. With judge=True, HARNESS2_JUDGE_MODEL is set.

The improver channel is built by resolve_channel_model with the default channel opencode. It again rejects an improver ID without / (L76-L80), and Proposer.post_init refuses an empty model.

4. Commands in order

4.0 harness2-demo: the free local demo

harness2-demo --output runs/demo --k 2

The demo builds a DemoAdapter with two tasks (ledger-1, ledger-2) in one domain (accounting). It uses a deterministic DemoProposer and real Python executions of a summarize.py script, so no credentials and no model calls are involved. It calls the same run_domain as a real sweep, so you see the real on-disk layout: runs/demo/runs/demo-oc-agg2/<task>/summary.json, runs/demo/library/k2/{agg,seq}/…, and a runs/demo/demo.json report with execution_count and editor_calls. Exit code is 1 if any task failed (L49-L54).

4.1 harness2-setup <bench>: install one benchmark

harness2-setup lab                         # or jb, or wb [--subset office|web|code]
harness2-setup lab --dry-run               # print the plan, change nothing

What it does, step by step (setup.py L41-L115):

1. Runner check. missing_runners() confirms the four patches (PATCHES) and three runner files (RUNNERS) exist under third_party/. Otherwise it fails with setup failed: benchmark runners and patches are not included in this copy … (only a note under --dry-run).
2. Binaries. It checks git, uv, npm (+ docker, curl for wb), or fails with Missing <binary>; see README.md (L48-L53).
3. OpenCode. clone("opencode") does git init + fetch --depth 1 origin <pin> + checkout --detach. It is idempotent at the right pin and refuses a checkout at another revision or a nonempty non-git directory. Then patch skips the patch if git apply --reverse --check says it is already applied. Then npm install --prefix dependencies/tools bun@1.3.11 and bun install --frozen-lockfile --ignore-scripts (L54-L63).
4. The benchmark. Clone + patch + overlays. The overlays copy third_party/lab_runner/* into harvey-labs/opencode, external/tts_lab and external/adapt_runs_cx, or third_party/wb_runner/* into WorkBuddy's src/workbuddy_bench/agents and configs/harnesses. Then uv sync --locked --python 3.12 in the checkout (L64-L68).
5. Per benchmark (L69-L106):
   • jb: uv pip install python-docx into its venv, then download_jobbench. That resolves the dataset's current HF revision, downloads dataset/** into a temp dir, verifies TASK_INSTRUCTIONS.txt exists, moves it to dependencies/job-bench-eval/dataset/main/, and records .harness2-download.json.
   • lab: uv pip install google-auth[requests] into its venv. No download, because tasks ship in the checkout.
   • wb: workbuddy_models writes configs/models/harness2/{solver,judge}.yaml (JSON content, ADC via gcloud … print-access-token, 64k/16k output tokens, high/low reasoning effort). Then bash scripts/dataset/fetch-dataset.sh <subsets>, and for oc, build-harness-mounts.sh --harness opencode/1.14.18.
6. Codex (--substrate cx): install.codex installs @openai/codex@0.144.6 and stages the native binary at dependencies/codex/dist/codex, readable by the container user (sharewith_containers). configure_codex then writes dependencies/codex/home/config.toml and requires HARNESS2_CODEX_BASE_URL.
7. It prints Installation complete. Run harness2-check before an experiment. (or Setup plan complete. for a dry run).

Writes: dependencies/{opencode,tools,harvey-labs|job-bench-eval|workbuddy-bench,codex}/ only. Nothing goes under runs/.

Network during setup (SETUP L143-L158): github.com, registry.npmjs.org, huggingface.co + CDN, a Docker registry (WB image), and astral.sh (CPython 3.12 if the host lacks it).

Re-running setup. Every step is idempotent at the pinned revision, and patches are skipped once applied. Rerun setup after you change the WB solver/judge model (SETUP L50-L52) or the Codex endpoint/model (L135-L138). Otherwise harness2-check reports the config as stale.

4.2 harness2-check --bench <bench>: offline readiness

harness2-check --bench lab
harness2-check --bench wb --subset office --substrate cx

It calls preflight.check(adapter, judge=True) and prints one FAIL: … per problem, then Setup incomplete: N issue(s). See docs/BENCHMARKS.md. (exit 1). Or it prints Ready: <bench>/<substrate>; <n> tasks. Model access is not exercised; … (exit 0) (check.py L36-L45). The checks, in order:

1. Runners and patches are present (missing_runners; if this fails, check stops here) (L29-L31).
2. validate_models(judge=True), task discovery (adapter.tasks), the OpenCode preflight, and the adapter preflight. Each exception becomes one error line (L46-L55).
3. The benchmark and OpenCode checkouts are at their pins, and their patches are applied (git apply --reverse --check) (L57-L75).
4. dependencies/opencode/node_modules exists (L76-L77).
5. ADC token, when any Vertex model or judge is involved (L79-L85).
6. Per benchmark:
   • LAB: pandoc and pdftoppm, plus import evaluation.scoring; import google.auth, anthropic in the LAB venv (L86-L94).
   • JobBench (with judge): import pandas, openpyxl, docx, mammoth, pdfplumber, pptx, openai, a judge endpoint, and a key for non-Gemini judges (L95-L115).
   • WorkBuddy: the Docker daemon, render libs (office), harbor + the three harness adapters registered, and the solver/judge YAML matching the current models (L116-L143).

Every subprocess probe has a 45 s timeout (L34-L44).

4.3 harness2 … --dry-run: preview the task stream

harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1 --dry-run

With --dry-run, _main builds the adapter and calls printstream. It does not create a Proposer, touch a substrate, validate any model setting, or write anything. It prints:

• The cell header: # lab-oc split=all seed=None k=1 modes=parallel.
• The namespace per mode (e.g. lab-oc-agg1), the summaries directory, the library root, and the worker count.
• The numbered (domain, task) stream.
• Every domain of the split with its task count. These are the legal --only-domains names.

It exits 0 if at least one task was selected, else 1. It still needs the benchmark installed, because task discovery raises No lab tasks found at … Run harness2-setup lab. otherwise (tasks.py L54-L58).

4.4 harness2: the refinement sweep (this costs money)

harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1

All flags (buildparser plus the shared helpers in _common.py):

What happens in a real run (_main):

1. Adapter and stream. make_adapter(...), then _stream_for validates the domain/task selection before any spending (L141-L142).
2. Preflight with the judge. require_ready(adapter, judge=True) runs the same checks as harness2-check and raises ConfigError("Setup incomplete; nothing has run yet: …"). The module docstring explains why the judge is checked now: the namespace records the judge model and harness2-judge refuses to grade under a different one, so a missing judge discovered after the sweep "means paying for the sweep twice" (L22-L26).
3. Identity lock. checkidentity writes <runs_root>/adapt/<ns>/config.json on the first run. It holds the method version, executor, improver, judge, project, location, judge endpoint, Codex endpoint, split, exact task list, max_tasks, domains and seed. On a later run any difference raises This run directory belongs to different models or tasks. Choose a new HARNESS2_RUNS_DIR for a new experiment.
4. Proposer on the benchmark's proposer channel, with the audit trail in <runs_root>/adapt/_audit/<ns>/ (L151).
5. Sweep. schedule.run_sweep runs run_domain per domain in a thread pool of --workers. Each task goes through run_task: k shared base rollouts, then each mode (recursion, then selection, then reflection), with summary.json checkpointed after each mode. Chapters 03 and 04 cover the recursion itself.
6. Record. <runs_root>/adapt/_sweeps/<ns>-<UTC stamp>.json holds the full sweep report (L175-L178).
7. Summary line sweep <ns>: N tasks ok, M failed, D domain(s) died, then a next: line printing the matching judge command (L191-L202).

Exit codes (L183-L206 and main):

4.5 harness2-judge: grade the selected run

harness2-judge --bench lab --k 1 --mode parallel
harness2-judge --bench lab --k 1 --mode parallel --bases     # score the reference runs too

There is no --split-file here. The judge builds the adapter without one (L166) and works purely from the summaries on disk. Flow (main):

1. validate_models(judge=True). This is not wrapped in a try, so a missing model surfaces as a Python traceback ending in the ConfigError message. The full require_ready preflight is not run here.
2. Locate <runs_root>/adapt/<ns>, where <ns> is always the parallel-token namespace, e.g. lab-oc-agg1. Summaries for both modes live there (summary_namespace).
3. If config.json records a different judge, exit with Judge model changed for this experiment; choose a new HARNESS2_RUNS_DIR (L169-L171).
4. No namespace dir → nothing to judge: … has the sweep … run on this machine?. No selections → … tasks may still be running, or the wrong --k/--seed was given (L172-L185).
5. collect_selections reads every */summary.json. Then J.judge_selections(adapter, sels, parallel=8) (L45, core judge). Only the run the blind selector chose is scored.
6. _merge with the existing judged.json. A new pick replaces the old row. For the same run_id, a graded row is never overwritten by a failed re-judge. The result is written with J.write.
7. Exit 0 only if every row judged in this invocation is graded, else 2. Ungraded rows are explicitly not zeros (L194-L200).

For LAB, each grading run's output goes to judge.log in the run directory (LabAdapter.score). A Vertex 429 RESOURCE_EXHAUSTED there means quota: rerun harness2-judge later (BENCHMARKS L34-L39).

--bases (judge_bases) scores summary["base"] entries with adapter.score(run_dir, task_id) and writes base_judged.json. It exits 2 if nothing was recorded or anything is ungraded. harness2-report does not read this file (EXPERIMENTS L87-L91), so compare base scores yourself.

4.6 harness2-report: the score table

harness2-report --bench lab --k 1 --mode parallel
harness2-report --bench jb --k 1,2,3 --mode parallel,sequential

The report is read-only. For each k it loads judged.json (warning no judged rows at … run cli.judge for k=… first if it is missing). It reads fell-back counts and improver USD from the summaries (spendand_fallbacks) and renders via core.report. LAB reports Partial (fraction of criteria passed) and All-pass; the other benchmarks report one score. The table also shows pick provenance and sanity warnings, for example when selection kept a reference output on ≥ 80 % of tasks, when stages fell back, or when scoring units are mixed. It ends with improver (editors, composer, selection, reflection) spend across reported modes: $X. Exit 0, or nothing to report (exit 1).

Executor rollout cost is not included in that dollar figure. The report sums only improver stage and selection usd fields. Executor and judge spend has to be read from your cloud billing. This is not covered by the code.

4.7 harness2-bridge: Codex on Vertex (optional)

uv pip install -e '.[render,bridge]'
harness2-bridge                 # WorkBuddy: harness2-bridge --host 0.0.0.0

The bridge refuses to start without LiteLLM next to the interpreter or without GOOGLE_CLOUD_PROJECT (L75-L79). It persists {port, key} in dependencies/codex/bridge.json (mode 0600, reused on restart so setup's Codex config stays valid, _endpoint) and writes dependencies/codex/bridge.yaml (LITELLM_CONFIG). It prints the lines to paste into .env: HARNESS2_CODEX_BASE_URL, HARNESS2_CODEX_API_KEY, HARNESS2_CODEX_MODEL, plus HARNESS2_CODEX_CONTAINER_URL=http://host.docker.internal:<port>/v1 when not bound to localhost (L87-L95). Then it execves LiteLLM. Then run harness2-setup <bench> --substrate cx and pass --substrate cx to every later command.

4.8 The wrapper scripts and the partitioner

bash scripts/run_lab.sh [extra harness2 args] runs harness2-check (aborting on failure), then harness2, harness2-judge and harness2-report with the same --bench --substrate --k --mode. It reads the environment variables SUBSTRATE (default oc), K (1), MODE (parallel), and for WorkBuddy SUBSET (office) (run_lab.sh L24-L36, run_workbuddy.sh L24-L37). Extra arguments go only to harness2. If they include --dry-run, the script stops after it. The exit status is the sweep's if nonzero, else the judge's. The scripts prepend ./.venv/bin to PATH, so they assume the venv lives at .venv.

python scripts/split_tasks.py --bench lab --seed 42 --output splits/lab.json makes a domain-stratified train/test partition (split_tasks.py L29-L65):

python scripts/check_provenance.py verifies the SHA-256 of every bundled runner, patch and license against dependencies/ASSETS.json (check_provenance.py L23-L39).

5. What lands on disk: the runs/ layout

Each benchmark's root is RUNS/<bench> (runs/lab, runs/jb) or RUNS/wb/<subset> (lab_spec L485, jobbench L513, workbuddy L658). Names come from ns(): <bench>-<substrate>-<agg|seq><k>[-s<seed>], e.g. lab-oc-agg1, jb-cx-seq3-s7. Task directories replace / in the task ID with __ (Namespace.task_dir).

$HARNESS2_RUNS_DIR (default ./runs)
└── lab/
    ├── adapt/                                   # runs_root_for(): scheduler state
    │   ├── lab-oc-agg1/                         # summary namespace (both modes' summaries)
    │   │   ├── config.json                      # identity lock (_check_identity)
    │   │   ├── judged.json                      # harness2-judge
    │   │   ├── base_judged.json                 # harness2-judge --bases
    │   │   └── <area>__<task>/
    │   │       ├── summary.json (+ .lock)       # status, base runs, modes, selections, errors
    │   │       ├── harnesses/{base,inv1,dom1,agg1}/
    │   │       ├── context/                     # evidence handed to improver sessions
    │   │       ├── selection/agg1/selection.json
    │   │       └── reflection/agg1/
    │   ├── lab-oc-seq1/<area>__<task>/          # sequential work dirs (c1…ck)
    │   ├── _audit/lab-oc-agg1/                  # every improver session: mission, response, meta
    │   └── _sweeps/lab-oc-agg1-<UTC>.json       # one record per harness2 invocation
    ├── library/lab/k1-oc/{agg,seq}/<domain>/    # harnessing-level library (library_root_for)
    └── results/oc/lab-oc-agg1/<area>/<task>/<label>/   # executor run dirs; judge.log here

Sources for the layout: runs_root_for, library_root_for + library.root_for, run_domain libraries, the LAB results root (lab.py L524-L529) and run IDs <ns>/<task>/<label> (Namespace.run_id), the selection dir (selection.py L296), and reflection (L416). The exact contents of context/ and the audit files are covered in chapters 03–05. The executor run-directory internals differ per benchmark and are covered in chapter 07.

5.1 Resume behaviour

Repeat the exact same command to resume (README L173-L181). Three mechanisms make this safe:

1. Per-task skip. run_task looks for summary.json. _resume only reuses it if status == "ok", every requested mode has an ok record with an ok selection and a chosen label, the adopted run directory and output still exist, the artifacts are intact, and the adopted harness's fingerprint is unchanged. Otherwise it prints why and re-runs the task. A resumed task counts as skipped (= done) and its library entries are restored.
2. Identity lock. config.json (above) stops a resume with different models, endpoints, tasks, --max-tasks, domains or seed. Use a new HARNESS2_RUNS_DIR for a new experiment (EXPERIMENTS L64-L68).
3. Judge merge. Re-running harness2-judge re-judges every selection but never replaces a graded row with a failed one (_merge).

Adding a mode later works: _write_summary merges per-mode records under an flock, and _resume re-runs only when a requested mode is missing (L289-L328). Note that the identity lock includes the task list. Widening --max-tasks or --only-domains in the same runs dir is refused, so plan your selection before the first paid run.

6. Costs and keeping a first run small

6.1 The budget identity

ledger.expected_rollouts and expected_sessions define, per task and per mode:

• R = 4k executor rollouts = k shared base + 2k candidates + k finals.
• P = 3k improver stages = k INV + k DOM + k compositions (agg1…aggk in parallel, c1…ck in sequential).

On top of that, outside the ledger (ledger docstring):

• Selection. For k = 1 with the release's finals pool, selection is trivial and calls no model (runone_mode L514-L527). For k > 1 it is one selector session.
• Reflection. One session after a successful adoption, when a library is in use (L529-L538).
• Retries. Invalid proposals get one retry (recursion docstring), and transport errors get TRANSPORT_ATTEMPTS = 2.
• Judging. One judge call per selected run (plus k per task with --bases).

With both modes in one invocation (--mode parallel,sequential), the k base rollouts are shared (run_task L620-L626), so one task costs k + 2·3k = 7k rollouts and 6k stages instead of 8k and 6k. The two ledgers still record 4k rollouts each (8k in all), because each mode is charged for the shared bases (chapter 06).

Dollar amounts depend on your models' pricing and are not computed by the code. Only improver spend is reported (4.6).

6.2 How to keep it small

1. Run the demo first. It is free and exercises the same scheduler, recursion and library code.
2. --dry-run to see the domains and pick one.
3. --k 1, one mode.
4. --only-domains <one>** and** --max-tasks 1. Remember --max-tasks is per domain: without --only-domains, --max-tasks 1 still runs one task in every domain.
5. --workers 1 (the default) keeps concurrency, and so spend rate, low.
6. A dedicated HARNESS2_RUNS_DIR per experiment so a later full run does not collide with the trial's identity lock.

7. Web access: lock it down before evaluation

Why. All three benchmarks are public. JobBench publishes each task's RUBRICS.json on Hugging Face. Executor network access is unrestricted, and Codex runs with its sandbox disabled (README L166-L171). The docs report that in a validation run "a Codex rollout on JobBench downloaded its task's rubric without being instructed to" (EXPERIMENTS L93-L98). The code itself never restricts network access. The adapters only keep criteria out of the improver's evidence (for example, LAB pops criteria from task.json, taskjson). Nothing stops the executor from fetching answers over the network.

What to block, after setup and before harness2 (EXPERIMENTS L100-L109):

• huggingface.co, hf.co, *.hf.co
• github.com, raw.githubusercontent.com

How: at the network level (an egress firewall or DNS policy on the executor host), not via a tool setting, because Codex's sandbox is off. Keep the model endpoints reachable (aiplatform.googleapis.com or <region>-aiplatform.googleapis.com, plus your bridge or judge endpoint). Setup needs the blocked hosts, so lock down only after harness2-setup, and lift the block when you install another benchmark. For WorkBuddy, rollouts run inside Docker containers, so apply the policy to container egress too. Report the restrictions you used alongside results.

8. Troubleshooting: every check and its fix

Two failure classes are not caught by any check: a model your account cannot serve, and quota (429). Both show up at the first call, as failed stages/rollouts in summary.json or ungraded judge rows.

9. Worked example: a first LAB run on one task

Goal: one LAB task in the antitrust-competition practice area, k = 1, parallel recursion, OpenCode executor, Vertex Gemini models.

Step 0: install and try the demo.

sudo apt-get install -y git curl unzip ca-certificates pandoc poppler-utils   # + Node 20, uv, gcloud
git clone https://github.com/google-research/harness2 && cd harness2
uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2       # free; writes runs/demo/demo.json

Step 1: configure.

cat > .env <<'EOF'
GOOGLE_CLOUD_PROJECT=my-gcp-project
HARNESS2_SOLVER_MODEL=google-vertex/<gemini-model-id>
HARNESS2_JUDGE_MODEL=<gemini-model-id>
HARNESS2_RUNS_DIR=runs/lab-first
EOF
gcloud auth application-default login

Step 2: install LAB. This creates dependencies/opencode/ (patched, node_modules), dependencies/tools/ (Bun 1.3.11), and dependencies/harvey-labs/ (pinned, patched, overlays copied, .venv built, google-auth[requests] added).

harness2-setup lab --dry-run    # optional: review every command
harness2-setup lab              # "Installation complete. Run harness2-check before an experiment."

Step 3: check. You want Ready: lab/oc; <n> tasks. Model access is not exercised; ….

harness2-check --bench lab

Step 4: preview. Confirm the namespace (lab-oc-agg1) and that exactly one task is listed. The trailing domain list gives every legal --only-domains name.

harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1 --dry-run

Step 5: lock down executor egress (huggingface.co, hf.co, *.hf.co, github.com, raw.githubusercontent.com). Keep Vertex reachable.

Step 6: run. Cost: 4 rollouts (base1, inv1, dom1, agg1), 3 improver stages (inv1, dom1, agg1), no selector session (k = 1), and at most one reflection session.

harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1
# sweep lab-oc-agg1: 1 domain(s) in parallel, tasks within a domain in stream order
#   [antitrust-competition] 1/1 ok           antitrust-competition/<task>
# sweep lab-oc-agg1: 1 tasks ok, 0 failed, 0 domain(s) died
# record: …/runs/lab-first/lab/adapt/_sweeps/lab-oc-agg1-<UTC>.json
# next:   python3 -m harness2.cli.judge --bench lab --substrate oc --k 1 --modes parallel

Files produced under runs/lab-first/lab/:

• adapt/lab-oc-agg1/config.json is the identity lock.
• adapt/lab-oc-agg1/antitrust-competition__<task>/summary.json plus harnesses/{base,inv1,dom1,agg1}/, context/, selection/agg1/selection.json and reflection/agg1/.
• adapt/_audit/lab-oc-agg1/… holds the improver session transcripts.
• adapt/_sweeps/lab-oc-agg1-<UTC>.json is the sweep record.
• results/oc/lab-oc-agg1/antitrust-competition/<task>/{base1,inv1,dom1,agg1}/ are the executor runs and deliverables.
• library/lab/k1-oc/agg/antitrust-competition/ holds the adopted harness and lessons, if reflection produced experience.

Step 7: judge and report. judged.json appears beside config.json. A LAB judge.log appears in the selected run's directory. The report prints Partial and All-pass for lab-oc modes=parallel at k = 1, the pick provenance and the improver spend. Rerun the judge if it exits 2 (for example on a 429 in judge.log).

harness2-judge  --bench lab --k 1 --mode parallel
harness2-judge  --bench lab --k 1 --mode parallel --bases   # optional base comparison
harness2-report --bench lab --k 1 --mode parallel

The whole thing as one command:

HARNESS2_RUNS_DIR=runs/lab-first bash scripts/run_lab.sh --only-domains antitrust-competition --max-tasks 1

10. Deployment picture

The project's canvas holds the deployment diagram: your machine (CLI, .env, dependencies/, runs/), the model channels on Vertex AI (executor, improver/selector, judge), the executors (OpenCode via Bun, Codex via the bridge), Docker for WorkBuddy, the upstream data sources, and the network boundary that should be closed to the executor during evaluation. The sequence below shows which component talks to which model during one task:

sequenceDiagram
  participant CLI as harness2 CLI
  participant EX as Executor (OpenCode or Codex)
  participant V as Vertex AI
  participant IM as Improver sessions (OpenCode channel)
  participant J as harness2-judge
  CLI->>EX: k base rollouts (base1..k)
  EX->>V: HARNESS2_SOLVER_MODEL
  CLI->>IM: k INV + k DOM proposals
  IM->>V: HARNESS2_IMPROVER_MODEL
  CLI->>EX: 2k candidate rollouts
  CLI->>IM: k compositions
  CLI->>EX: k final rollouts
  CLI->>IM: selection (k>1) + reflection
  Note over CLI: summary.json, library entry
  J->>V: HARNESS2_JUDGE_MODEL on selected run only
  J-->>CLI: judged.json

Key takeaways

• The order is setup → check → dry-run → run → judge → report. Judge and report must use the same --bench/--subset/--substrate/--k/--mode/--seed as the run.
• Configuration is environment variables read once at import. .env loads with "shell wins" semantics, and .env.example is referenced but absent at this commit. You need HARNESS2_SOLVER_MODEL, HARNESS2_JUDGE_MODEL, and GOOGLE_CLOUD_PROJECT for Vertex. Codex adds HARNESS2_CODEX_BASE_URL/_API_KEY (and _CONTAINER_URL for WorkBuddy).
• Checks are offline. They prove installs, pins, patches, ADC tokens and judge config, never that a model is reachable. A real run repeats them, judge included, so a missing judge can't waste a sweep.
• Cost per task per mode is 4k rollouts + 3k improver stages (+ reflection, + a selector at k > 1, + one judge call). Start with the free demo, then --k 1 --only-domains X --max-tasks 1. --max-tasks is per domain.
• Resume by rerunning. config.json blocks resumes with changed models or tasks, and judged.json never loses a graded row. Use a fresh HARNESS2_RUNS_DIR per experiment.
• Block the executor's access to Hugging Face and GitHub after setup and before evaluating. Rubrics are public, Codex's sandbox is off, and the code itself restricts nothing.

Where to go next

• 00 · Start here: the guide's map and reading order.
• 01 · Architecture overview: how the two recursion levels fit together.
• 02 · Code structure: where cli/, core/, benchmarks/, substrate/ sit.
• 03 · One task, end to end: what run_task does between the base rollouts and summary.json.
• 04 · The harness tree: what lives in harnesses/<label>/ and the eight components.
• 05 · Evidence and safety: what the improver may see (leak globs, criteria stripping) and why.
• 06 · Budget, selection and judging: the R = 4k / P = 3k ledger, blind selection and judged.json in depth.
• 07 · Executors and benchmarks: OpenCode vs Codex, and each adapter's run directories and scoring.