Skip to content

README.md

Latest commit

 

History

History
276 lines (198 loc) · 12.6 KB

README.md

File metadata and controls

276 lines (198 loc) · 12.6 KB
Raw

GR00T N1.7 on RoboLab

This guide shows the validated path for running the finetuned GR00T N1.7 DROID checkpoint on RoboLab tasks through the GR00T policy server.

Why the DROID Checkpoint

RoboLab directly uses the DROID checkpoint published with the GR00T N1.7 General Availability (GA) release, without any RoboLab-specific finetuning.

Validation Snapshot

The current release baseline uses the public GR00T N1.7 DROID checkpoint and RoboLab's built-in GR00T client:

  • GR00T model: nvidia/GR00T-N1.7-DROID
  • Embodiment tag: OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
  • RoboLab execution horizon: --open-loop-horizon 8
  • Diffusion inference timesteps: model/release default, 4 denoising steps
  • Episodes: 40 per task, 120 tasks
Setup Tasks Episodes Successes Success rate
GR00T N1.7 DROID + RoboLab (client) 120 4,800 412 8.58%
N1.6 reference (branch) 120 1,200 87 7.25%

--open-loop-horizon 8 is part of the reproduced result. It controls how many rows from each predicted action chunk RoboLab executes before querying the GR00T server again. It is separate from the model checkpoint action horizon.

Successful Tasks

The N1.7 full-suite wins are concentrated in visually clear pick/place and short-horizon manipulation tasks:

Task Successes
BananaOnPlateTask 40/40
BananasInBinThreeTotalTask 38/40
UnstackRubiksCubeTask 38/40
SauceBottlesCrateTask 33/40
RubiksCubeOrBananaTask 32/40
BananaInBowlTask 31/40
BananasInBinOneMoreTask 31/40
RubiksCubeTask 25/40
RubiksCubeThenBananaTask 19/40
BananasInCrateTask 18/40
FoodPacking1CansTask 16/40
RedDishesInBinTask 15/40
TakeMeasuringSpoonOutTask 12/40

Failures are more common on cluttered scenes, long-horizon packing, precise shelf placement, and sequential tasks that need recovery after a poor grasp.

Video Showcases

The final full-suite run used --video-mode none, so the success-rate table above is the authoritative quantitative result but not the source of MP4s. The videos below are all unique task families with at least one confirmed successful trial in the recovered diagnostic video runs. Some high-success tasks from the final table are not shown because their video shards were not recoverable.

Click any preview image to open the corresponding MP4.

Task Demo
Rubik's Cube in Bowl Rubik's cube is picked and placed into the bowl
Fast single-object grasp and bowl placement.
Unstack Rubik's Cubes Rubik's cube tower is unstacked
Short-horizon unstacking behavior.
Sauce Bottle to Crate Red sauce bottle is moved into the crate
Bottle grasp and placement into a constrained crate.
Rubik's Cube and Banana Rubik's cube and banana are placed into the bowl
Two-object bowl placement.
Rubik's Cube then Banana Rubik's cube and banana are placed into the bowl in sequence
Sequential cube-and-banana bowl placement.
Red Dishware to Bin Red dishware is moved into the grey bin
Color-conditioned dishware sorting.
Measuring Spoon Out Measuring spoon is taken out of the bowl and placed on the table
Extracts the measuring spoon from the bowl and places it on the table.
Butter on Raisin Box Butter box is placed on top of the raisin box
Places one boxed object on top of another.
Mustard on Raisin Box Mustard bottle is placed on top of the raisin box
Longer placement sequence with a tall object.
Plastic Bottles to Pail Small plastic bottles are placed into the square pail
Multi-object pail placement.
Bowl to Shelf Serving bowl is placed on the shelf
Places a bowl onto the shelf.

N1.6 Reference

The N1.6 comparison used the RoboLab author's N1.6 DROID branch:

The N1.6 runs used 10 trials per task. The N1.7 release run above used 40 trials per task, so compare success rates rather than raw success counts.

Reproduction Steps

Parameters

Setting Value
Model nvidia/GR00T-N1.7-DROID
Embodiment tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
GR00T server wrapper --use-sim-policy-wrapper
Inference timesteps Release default, 4, tried 8, giving similar result
RoboLab execution horizon --open-loop-horizon 8
Observation cameras left exterior + left wrist
Image transport HWC uint8, no letterbox or black padding, 180x320

Install

Set up Isaac-GR00T and RoboLab in separate environments. First, install Isaac-GR00T for the policy server. See the main environment setup guide for platform-specific system requirements:

git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --python 3.12

GR00T N1.7 requires access to the gated nvidia/Cosmos-Reason2-2B backbone. Request access on the model page, then authenticate:

uv run huggingface-cli login

In a separate terminal, install RoboLab for the simulation client. RoboLab's uv sync installs Isaac Sim, Isaac Lab, and the GR00T client dependencies:

git clone https://github.com/NVlabs/RoboLab.git
cd RoboLab
sudo apt install ffmpeg
uv venv --python 3.11
source .venv/bin/activate
uv sync

RoboLab should provide the GR00T runner:

uv run python policies/gr00t/run.py --help

Start GR00T

Start one GR00T server:

cd Isaac-GR00T

CUDA_VISIBLE_DEVICES=0 uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-DROID \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --device cuda \
    --host 127.0.0.1 \
    --port 5555 \
    --use-sim-policy-wrapper

The server is ready when it prints:

Server is ready and listening on tcp://127.0.0.1:5555

The command above assumes the server and client run on the same host. For a separate server host, bind the server to an appropriate private interface, pass its reachable hostname to RoboLab's --remote-host, and restrict port 5555 to the intended network boundary.

Run RoboLab

Isaac Sim checks EULA acceptance when the process starts. Export the variable in every new terminal before launching RoboLab:

export OMNI_KIT_ACCEPT_EULA=Y

To avoid repeating this command, persist it in your shell startup file (for example, ~/.bashrc) or environment launcher.

Run a smoke test first:

cd RoboLab

CUDA_VISIBLE_DEVICES=0 uv run python policies/gr00t/run.py \
    --headless \
    --remote-host 127.0.0.1 \
    --remote-port 5555 \
    --task BananaOnPlateTask \
    --num-envs 10 \
    --num-runs 1 \
    --open-loop-horizon 8 \
    --instruction-type default \
    --video-mode none

For a more stable estimate, use the largest --num-envs that fits GPU memory:

CUDA_VISIBLE_DEVICES=0 uv run python policies/gr00t/run.py \
    --headless \
    --remote-host 127.0.0.1 \
    --remote-port 5555 \
    --task BananaOnPlateTask \
    --num-envs 40 \
    --num-runs 1 \
    --open-loop-horizon 8 \
    --instruction-type default \
    --video-mode none

Useful setup-smoke tasks:

  • BananaOnPlateTask
  • BananaInBowlTask
  • BananasInBinThreeTotalTask
  • UnstackRubiksCubeTask
  • SauceBottlesCrateTask

Example 4-GPU Layout

For faster evaluation on a 4-GPU node, run two GR00T servers and two RoboLab workers:

GPU Process
0 GR00T server A
1 GR00T server B
2 RoboLab eval worker A
3 RoboLab eval worker B

Each RoboLab worker should point to a different server port and receive a disjoint task shard. Keep the same model, embodiment tag, horizon, camera, image, and video settings across shards.

The GR00T server and Robolab client can also share a GPU, with smaller --num-envs parallel environments.

RoboLab Observation Contract

The N1.7 DROID client sends this request shape to the policy server:

Group Key Shape Dtype
Video video.exterior_image_1_left [B, T, H, W, 3] uint8
Video video.wrist_image_left [B, T, H, W, 3] uint8
State state.eef_9d [B, T, 9] float32
State state.joint_position [B, T, 7] float32
State state.gripper_position [B, T, 1] float32
Language annotation.language.language_instruction [B] string

Use T=1 for the current baseline. Map RoboLab over_shoulder_left_camera to GR00T exterior_image_1_left, and RoboLab wrist_cam to GR00T wrist_image_left.

The GR00T response contains chunked action.joint_position and action.gripper_position. Concatenate those actions and execute only the first 8 rows before querying the server again.

Image Handling

Do not letterbox. Do not add black bars.

The validated RoboLab client sends 16:9 HWC uint8 images at 180x320. This is only the transport size; the N1.7 processor still applies its model-side image transform after receiving the image.

Avoid square client images unless deliberately running an ablation. Square stretching changes scene geometry, and square padding reintroduces letterbox bars.

Ablation Notes

Useful checks from the validation sweep:

Factor Outcome
8 denoising steps instead of 4 No meaningful full-suite improvement; roughly doubles DiT inference work.
Additional camera inputs No stable improvement over left exterior + left wrist in the tested configs.
History frames No stable improvement in the tested configs.

Outputs And Dashboard

For throughput sweeps, use --video-mode none. RoboLab still writes run summaries, task logs, HDF5 trajectories, timing, and per-task result rows.

For inspectable diagnostic runs, use:

--video-mode all --enable-subtask

--video-mode all writes both policy/sensor and viewport mp4s beside the task outputs. --enable-subtask populates score and failure-reason fields when task subtask tracking is available.

The RoboLab dashboard reads an output directory containing run folders:

robolab-dashboard --output-dir RoboLab/output --port 8080

The dashboard uses episode_results.jsonl as the canonical per-episode summary and discovers mp4s, per-env logs, and HDF5 files from each task directory.

Troubleshooting

If the server returns SVD did not converge, save the task name and server log. This is a numerical robustness issue in the action decode path, not a RoboLab installation problem.

If RoboLab crashes before launching IsaacSim, check for duplicate argparse flags between RoboLab and IsaacLab's AppLauncher.