feat: scripted demonstration generator for sim-to-real data collection

[mikegc-aws]

Summary

The MuJoCo backend (#85) can create scenes, step physics, and record datasets via DatasetRecorder. But there's no way to generate demonstrations programmatically. Without a demonstration source, the sim-to-real pipeline dead-ends: you have a simulated arm and a recording tool, but nothing to record.

For sim-to-real training, users need hundreds or thousands of diverse demonstrations. A scripted IK-based policy that generates pick-and-place (and other common manipulation primitives) would bridge this gap.

Motivation

The current paths to generate demonstrations are:

1. Real teleoperation — works but limited to ~50-100 episodes per session (human fatigue)
2. Agent in sim — the LLM agent + MockPolicy just waves the arm; no task completion
3. External RL policy — requires a separate training loop, reward engineering, etc.

What's missing is the middle ground: a scripted policy that uses MuJoCo's built-in inverse kinematics (or a planner like mink) to generate goal-directed trajectories for common manipulation primitives.

Proposal

A ScriptedPolicy (or DemoGenerator) that produces demonstrations for common tasks:

from strands_robots.simulation import Simulation
from strands_robots.simulation.demo_generator import DemoGenerator

sim = Simulation()
sim.create_world()
sim.add_robot("so100")
sim.add_object("cube", position=[0.2, 0.0, 0.02], color=[1, 0, 0, 1])

gen = DemoGenerator(sim, robot_name="so100")

# Generate 500 pick demonstrations with randomized cube positions
dataset = gen.generate(
    task="pick",
    target_object="cube",
    n_episodes=500,
    randomize={
        "object_position": {"x": [0.1, 0.3], "y": [-0.15, 0.15]},
        "lighting": {"intensity": [0.5, 1.5]},
        "camera_noise": {"sigma": 0.01},
    },
    dataset_path="/tmp/sim_pick_demos",
)

Manipulation primitives to support

Primitive	Description
reach	Move end-effector to target position
pick	Reach → close gripper → lift
place	Move to target → open gripper → retract
pick_and_place	Pick from A, place at B
push	Contact object and slide to target
stack	Pick object, place on top of another

Implementation approach

1. Use MuJoCo's mj_jac for Jacobian-based IK (already available in the physics engine)
2. Waypoint-based trajectory: approach → align → grasp → lift (for pick)
3. Smooth interpolation between waypoints (cubic or quintic spline)
4. Domain randomization applied per-episode: object pose, lighting, texture, camera noise
5. Record via existing DatasetRecorder with dual cameras (front + wrist per sim: support body-mounted cameras (wrist/eye-in-hand) via SimCamera.parent_body #137)
6. Output in LeRobot format, directly usable for training

Acceptance criteria

• Generate 100+ pick demonstrations for SO-100 in under 5 minutes
• Demonstrations are diverse (randomized object positions, lighting)
• Output is valid LeRobot v3 dataset with dual-camera observations
• Success rate of scripted policy is >95% in sim (IK solver converges)
• Works with at least SO-100 and one other arm (e.g., Koch, ViperX)
• Optional: task description annotation per episode (for multi-task training)

Related

• feat: MuJoCo simulation backend - AgentTool with 50+ actions #85 (MuJoCo backend) — provides the simulation infrastructure
• sim: support body-mounted cameras (wrist/eye-in-hand) via SimCamera.parent_body #137 (wrist camera) — needed for dual-camera recording that matches real setups
• Benchmark-agnostic evaluation protocol (BenchmarkProtocol + registry) #107-110 (benchmarks) — evaluation after training, not data generation

[cagataycali]

Tracked in #202 (Epic: Training & dataset pipeline).