feat: multimodal browser-agent debug environment

[mikasenghaas]

Summary

• New v1 environment: a sandboxed local-app browser-agent environment (flight-search web-app tasks). Each task boots a local SPA + headless-Chromium CDP service inside a per-task Docker image; a browser agent drives it by screenshots → vision model → click/type actions, then submits a structured JSON result, which a structured-output LLM judge (default openai/gpt-4.1-mini on Prime inference) scores against a deterministic answer key (answer_key reward, weight 1.0). Requires a multimodal model.
• The browser agent is private and is not vendored in this repo. The harness fetches it at run time from a private, auth-gated GitHub repo (pinned to a commit), caches it locally, and stages it into the sandbox. Configure via --harness.agent-repo / --harness.agent-ref with a token in --harness.agent-token-env; a --harness.agent-path local override is provided for development. A .gitignore keeps the agent out of version control.
• Tasks are pulled dynamically from the private Prime hub, not bundled in the repo: load_tasks fetches the dataset via prime env pull and caches it under ~/.cache/verifiers/. Override with --taskset.dataset_path, or repoint --taskset.hub_env_id / --taskset.hub_version.
• The judge config is a JudgeConfig subconfig (--taskset.judge.model + a verifiers BaseClientConfig), so the endpoint, team header, and API key auto-resolve to Prime inference. The judge uses strict structured output (json_schema) so its verdict is always valid JSON (with a tolerant parser as a backstop).
• Co-packages the taskset and a thin (public) harness shim in one module (both resolved via __all__, so --taskset.id and --harness.id share the name). Registered as an editable workspace member (pyproject.toml + uv.lock).

Adapted to the current feat/nano-as-v1 API: boundary error types (HarnessError / TasksetError); module discovery via __all__ (no load_environment wiring). Config is kept minimal — timeouts, task shuffling, and selection are left to the framework; the sandbox image is set per-task.

Run

The task image is a Prime-registry image, so use the prime runtime; supply the agent token + pinned ref and a vision model:

export <AGENT_TOKEN_ENV>=<token>
uv run eval <env-id> --harness.id <env-id> \
  --harness.runtime.type prime --harness.agent-ref <agent-commit-sha> \
  -m <multimodal-model> -n 1 -r 1 -c 1

Status

Verified end-to-end on the prime runtime: dynamic hub task pull, plugin discovery, private-source agent staging, sandbox provisioning, the agent completing a task and submitting a result, and the structured-output judge scoring it. A sample run scored 0.83 (5/6 fields) — the agent solved the search but omitted baggage fees from the total.

🤖 Generated with Claude Code

[macroscopeapp]

🟢 Low harness/__init__.py:239

The _download_agent staging path is hard-coded (dest / "<package>.tmp"), so concurrent processes race on it: one may rmtree the directory while another is copytree-ing into it, causing a failed or corrupted copy. The final os.replace atomically commits the finished directory but does not protect the staging work from interleaving. Use a unique staging directory per process (e.g. tempfile.mkdtemp inside dest) to prevent collisions.

-            staging = dest / (self.config.agent_package + ".tmp")
-            if staging.exists():
-                shutil.rmtree(staging)
-            shutil.copytree(matches[0], staging)
+            staging = Path(tempfile.mkdtemp(dir=dest, prefix=self.config.agent_package + ".tmp."))
+            shutil.copytree(matches[0], staging)
             os.replace(staging, dest / self.config.agent_package)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/mini_browse_apps_platform_v1/mini_browse_apps_platform_v1/harness/__init__.py around lines 239-243:

The `_download_agent` staging path is hard-coded (`dest / "<package>.tmp"`), so concurrent processes race on it: one may `rmtree` the directory while another is `copytree`-ing into it, causing a failed or corrupted copy. The final `os.replace` atomically commits the finished directory but does not protect the staging work from interleaving. Use a unique staging directory per process (e.g. `tempfile.mkdtemp` inside `dest`) to prevent collisions.

Evidence trail:
environments/mini_browse_apps_platform_v1/mini_browse_apps_platform_v1/harness/__init__.py lines 196-197 (TOCTOU guard), lines 238-243 (deterministic staging path and non-atomic staging operations) at REVIEWED_COMMIT.

[macroscopeapp]

🟢 Low harness/__init__.py:230

tar.extractall(extract, filter="data") raises TypeError: extractall() got an unexpected keyword argument 'filter' on Python 3.10.0–3.10.11 and 3.11.0–3.11.3, because the filter parameter was only backported in 3.10.12 / 3.11.4. Since _download_agent runs on the host (not inside the 3.12 Docker container), hosts on those earlier patch versions will crash with an unhelpful error. Consider guarding with hasattr(tarfile, 'data_filter') to fall back gracefully.

 with tarfile.open(archive) as tar:
-                tar.extractall(extract, filter="data")
+                if hasattr(tarfile, 'data_filter'):
+                    tar.extractall(extract, filter="data")
+                else:
+                    tar.extractall(extract)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/mini_browse_apps_platform_v1/mini_browse_apps_platform_v1/harness/__init__.py around lines 230-231:

`tar.extractall(extract, filter="data")` raises `TypeError: extractall() got an unexpected keyword argument 'filter'` on Python 3.10.0–3.10.11 and 3.11.0–3.11.3, because the `filter` parameter was only backported in 3.10.12 / 3.11.4. Since `_download_agent` runs on the host (not inside the 3.12 Docker container), hosts on those earlier patch versions will crash with an unhelpful error. Consider guarding with `hasattr(tarfile, 'data_filter')` to fall back gracefully.

Evidence trail:
environments/mini_browse_apps_platform_v1/mini_browse_apps_platform_v1/harness/__init__.py lines 230-231 (REVIEWED_COMMIT): `tar.extractall(extract, filter="data")`. environments/mini_browse_apps_platform_v1/pyproject.toml line 4: `requires-python = ">=3.10"`. Root pyproject.toml line 14: `requires-python = ">=3.11,<3.14"`. Python 3.11 docs (https://docs.python.org/uk/3.11/library/tarfile.html): 'New in version 3.11.4' for extraction filters. Python 3.14 docs (https://docs.python.org/3/library/tarfile.html): 'Changed in version 3.12: Added the filter parameter.' and recommendation to use `hasattr(tarfile, 'data_filter')`. scikit-learn issue #31521 for identical bug.