Benchmark-agnostic evaluation protocol (BenchmarkProtocol + registry) #107

Motivation

After PR #85 lands, the simulation backend supports policy rollout, evaluation, dataset recording, and physics introspection — but the benchmarking surface is a single eval_policy(success_fn=...) where success_fn is typed str | Callable and the only registered string value is "contact". That works for ad-hoc sparse-success experiments but does not generalize to established benchmarks (LIBERO, Meta-World, RoboSuite, ManiSkill, user-authored tasks), each of which has a richer notion of "what constitutes a task."

A LIBERO-shaped register_task(scene, goal_predicate, init_sampler) API would fix LIBERO specifically but wouldn't fit Meta-World's dense reward or ManiSkill's procedural scenes. The correct abstraction is the protocol the eval loop calls into, not a benchmark-specific schema.

Proposal

Introduce a BenchmarkProtocol ABC and a string-keyed registry, mirroring the existing register_urdf shape. Each adapter is kinematically tied to one or a small set of robots (LIBERO's BDDL and scene files reference Panda body names, Meta-World hardcodes Sawyer, RoboSuite parameterizes per-task over a fixed robot list), so robot compatibility is first-class metadata on the protocol — without it, agents will silently evaluate with the wrong robot.

# strands_robots/simulation/benchmark.py

class BenchmarkProtocol(ABC):
    """Protocol every benchmark (LIBERO, Meta-World, custom) implements."""

    max_steps: int

    # --- Robot compatibility (first-class metadata) ---

    @property
    @abstractmethod
    def supported_robots(self) -> list[str]:
        """Registry config names this benchmark accepts.
        Empty list means any robot (unusual; dense-reward benchmarks rarely generalize)."""

    @property
    @abstractmethod
    def default_robot(self) -> str:
        """Robot on_episode_start loads when the sim is empty."""

    # --- Lifecycle hooks ---

    def on_episode_start(self, sim: SimEngine, rng: random.Random) -> None:
        """Per-episode init. Called after sim.reset() and before the first obs.
        Default impl ensures a supported robot exists: if the sim has no robots,
        add default_robot; otherwise validate the loaded robot's data_config is
        in supported_robots. Override to layer on per-episode randomization."""

    @abstractmethod
    def on_step(self, sim: SimEngine, obs: dict, action: dict) -> StepInfo:
        """Return dense reward + done flag + info dict for this step."""

    @abstractmethod
    def is_success(self, sim: SimEngine) -> bool:
        """Terminal success predicate."""

    def is_failure(self, sim: SimEngine) -> bool:
        """Optional early-termination failure condition. Default: False."""

PolicyRunner.evaluate grows a spec: BenchmarkProtocol | None parameter. Before episode 1 it must validate the sim's current robot against spec.supported_robots and return a structured error dict (not an exception) on mismatch, surfacing the allowed list. The loop itself becomes:

for ep in range(n_episodes):
    sim.reset()
    spec.on_episode_start(sim, rng)
    cumulative_reward = 0.0
    for step in range(spec.max_steps):
        obs = sim.get_observation(robot_name=robot_name)
        action = policy.get_actions(obs, instruction)[0]
        sim.send_action(action, robot_name=robot_name)
        info = spec.on_step(sim, obs, action)
        cumulative_reward += info.reward
        if info.done or spec.is_success(sim) or spec.is_failure(sim):
            break

The existing success_fn: str | Callable path subsumes cleanly: "contact" and user callables both become trivial BenchmarkProtocol adapters under the hood, kept for backwards compatibility.

Registry + tool-spec surface

Three new agent-tool actions (same pattern as register_urdf / list_urdfs):

• list_benchmarks — enumerate registered benchmarks.
• evaluate_benchmark(name, policy_provider, policy_config, n_episodes, ...) — run any registered benchmark.
• register_benchmark_from_file(name, spec_path) — load a user-authored YAML/JSON spec (declarative named-predicate DSL, no Python eval) so agents can author benchmarks without writing code.

Declarative spec format sketch:

name: drawer-open
scene: /path/to/drawer_scene.xml
max_steps: 300
init_state:
  - randomize: { type: uniform, joint: drawer_slide, low: -0.05, high: 0.0 }
success:
  all:
    - joint_above: [drawer_slide, 0.15]
failure:
  any:
    - body_below_z: [gripper, -0.1]
dense_reward:
  - term: { type: distance_neg, body_a: gripper, body_b: drawer_handle, weight: 1.0 }
  - term: { type: joint_progress, joint: drawer_slide, weight: 5.0 }

This is LLM-authorable and safely sandboxed.

Adapters (separate follow-up PRs)

With the protocol in place, each real benchmark becomes a thin adapter, not a core change. Adapters live in
optional extras (strands-robots[benchmark-libero] etc.); the core package stays dependency-free.

Tracked as sub-tasks:

• Benchmark adapter: Meta-World (MetaWorldAdapter) #108 — Meta-World adapter (suggested reference adapter — pure Python, dense-reward)
• Benchmark adapter: RoboSuite (RoboSuiteAdapter) #109 — RoboSuite adapter (MuJoCo-native, dense-reward; pins end-to-end reward threading)
• Benchmark adapter: LIBERO (LiberoAdapter + BDDL parser) #110 — LIBERO adapter (BDDL parser + ~130 tasks across LIBERO-{spatial,object,goal,10,90})

Scope for this issue

In scope:

1. BenchmarkProtocol ABC + StepInfo dataclass in strands_robots/simulation/benchmark.py.
2. Widen PolicyRunner.evaluate to accept spec: BenchmarkProtocol alongside the existing success_fn (kept for backcompat).
3. register_benchmark / list_benchmarks / evaluate_benchmark tool actions + tool_spec.json entries.
4. Named-predicate helper library (on_table, grasped, joint_above, inside_region, body_above_z, …) as a foundation for declarative specs.
5. Declarative YAML/JSON spec loader (register_benchmark_from_file), restricted to the named-predicate DSL.
6. One reference adapter (suggest: MetaWorldAdapter, since it's Python-native and doesn't need BDDL parsing).
7. Tests: protocol contract, cumulative-reward accounting, per-episode randomization reproducibility by seed, declarative-spec loader.

Out of scope (follow-up issues):

• BDDL parser / LIBERO adapter.
• RoboSuite / ManiSkill adapters.
• Dense-reward curriculum tooling, RL-style training harness.
• Replacing the current success_fn path (kept working for the whole backcompat window).

Why this belongs on the roadmap

The PR #85 architecture (SimEngine ABC + backend-agnostic PolicyRunner + string-keyed register_urdf) already does 80% of the work. Adding BenchmarkProtocol uses the same patterns and unlocks every standard benchmark through adapters — without committing the core to any single benchmark's conventions.

Relevant PR: #85. Related labels: simulation, enhancement, roadmap.