GAM-RoboTwin
GAM (Grounded Action Model) grounds the objects named in the instruction with an open-vocabulary 3D detector
(WildDet3D), turns each object into a 3D box, a point cloud and a handful of image tokens, and predicts action
chunks with a flow-matching transformer. This repository holds everything the released code needs to run GAM on
RoboTwin 2.0: the RoboTwin-tuned detector, one action expert for each of the 50 tasks. All files are referenced from configs/robotwin/<task>.yaml and download on first use.
The videos are evaluation rollouts of the released experts; the wireframes are the 3D boxes the policy was given at that step (predicted by the detector from the RGB image alone, labelled with the text prompt of the slot).
| click_bell | shake_bottle | beat_block_hammer | handover_block |
![]() | ![]() | ![]() | ![]() |
| grab_roller (randomized) | stack_bowls_two (randomized) | place_can_basket | |
![]() | ![]() | ![]() |
Quick Links
- π Paper: Grounded Action Model: 3D Grounding as a Foundation for Robotics
- π» Code: GehaoZhang6/Grounded-Action-Model
- π Models: GAM-LIBERO-PRO, GAM-WildDet3D (general detector + universal span head)
- π Task files: gam-libero-pro-assets
Intended Use
Evaluate the released experts on RoboTwin 2.0 (clean and randomized scenes), or train your own expert from your own
demonstrations. The configuration is monocular: the detector sees RGB only
(zero depth map), object geometry is its predicted 3D box and the point cloud is back-projected from its own
depth; no simulator depth or object state is used at training or evaluation. Every object is grounded from its
name on every frame (grounding: text).
Results
Success rate (%) of the 50 experts, clean (Easy) and randomized (Hard) scenes, 20 episodes each, monocular
(no simulator depth or object state). For eight tasks (blocks_ranking_rgb, blocks_ranking_size,
place_cans_plasticbox, place_container_plate, place_object_basket, press_stapler, rotate_qrcode,
stack_blocks_two) the released checkpoint is an intermediate training step, picked by this evaluation.
| Easy | Hard | Avg | |
|---|---|---|---|
| mean over 50 tasks | 67.6 | 44.0 | 55.8 |
Per task
| task | Easy | Hard | Avg |
|---|---|---|---|
adjust_bottle |
100 | 100 | 100.0 |
beat_block_hammer |
85 | 45 | 65.0 |
blocks_ranking_rgb |
15 | 15 | 15.0 |
blocks_ranking_size |
35 | 30 | 32.5 |
click_alarmclock |
95 | 80 | 87.5 |
click_bell |
100 | 60 | 80.0 |
dump_bin_bigbin |
90 | 50 | 70.0 |
grab_roller |
100 | 100 | 100.0 |
handover_block |
90 | 50 | 70.0 |
handover_mic |
100 | 75 | 87.5 |
hanging_mug |
30 | 15 | 22.5 |
lift_pot |
85 | 10 | 47.5 |
move_can_pot |
60 | 25 | 42.5 |
move_pillbottle_pad |
35 | 15 | 25.0 |
move_playingcard_away |
85 | 45 | 65.0 |
move_stapler_pad |
25 | 0 | 12.5 |
open_laptop |
100 | 75 | 87.5 |
open_microwave |
50 | 15 | 32.5 |
pick_diverse_bottles |
55 | 60 | 57.5 |
pick_dual_bottles |
65 | 65 | 65.0 |
place_a2b_left |
50 | 30 | 40.0 |
place_a2b_right |
40 | 35 | 37.5 |
place_bread_basket |
35 | 25 | 30.0 |
place_bread_skillet |
35 | 0 | 17.5 |
place_burger_fries |
90 | 95 | 92.5 |
place_can_basket |
80 | 80 | 80.0 |
place_cans_plasticbox |
80 | 15 | 47.5 |
place_container_plate |
90 | 20 | 55.0 |
place_dual_shoes |
50 | 25 | 37.5 |
place_empty_cup |
90 | 35 | 62.5 |
place_fan |
60 | 5 | 32.5 |
place_mouse_pad |
15 | 5 | 10.0 |
place_object_basket |
70 | 50 | 60.0 |
place_object_scale |
35 | 10 | 22.5 |
place_object_stand |
80 | 80 | 80.0 |
place_phone_stand |
55 | 35 | 45.0 |
place_shoe |
90 | 35 | 62.5 |
press_stapler |
90 | 75 | 82.5 |
put_bottles_dustbin |
60 | 70 | 65.0 |
put_object_cabinet |
60 | 55 | 57.5 |
rotate_qrcode |
65 | 10 | 37.5 |
scan_object |
70 | 20 | 45.0 |
shake_bottle |
100 | 90 | 95.0 |
shake_bottle_horizontally |
95 | 95 | 95.0 |
stack_blocks_three |
30 | 15 | 22.5 |
stack_blocks_two |
75 | 30 | 52.5 |
stack_bowls_three |
80 | 60 | 70.0 |
stack_bowls_two |
85 | 95 | 90.0 |
stamp_seal |
50 | 25 | 37.5 |
turn_switch |
75 | 50 | 62.5 |
Install
git clone https://github.com/GehaoZhang6/Grounded-Action-Model && cd Grounding-Action-Model
conda create -n gam python=3.11 && conda activate gam
bash scripts/install.sh --robotwin /path/to/RoboTwin # GAM + RoboTwin 2.0 in one environment
export ROBOTWIN_ROOT=/path/to/RoboTwin
Evaluate
# clean + randomized, 20 episodes each; --overlay also writes the videos above (head camera + 3D boxes)
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --overlay
# your own checkpoint, or RoboTwin in a separate environment
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --ckpt runs/robotwin/click_bell/latest.pt
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --sim-python ~/envs/robotwin/bin/python
eval.chunking selects how action chunks are executed: replan (the released setting: 15 of 50 steps per
query, seam-blended), tempens (temporal ensembling) or rtc (real-time chunking). The detector runs in bf16
at evaluation (eval.precision).
Train
# from your own demonstrations: precompute, then train β or end-to-end (frozen detector, no cache)
python -m GAM.precompute --config configs/robotwin/click_bell.yaml && python -m GAM.train --config configs/robotwin/click_bell.yaml
python -m GAM.train --config configs/robotwin/click_bell.yaml --e2e
Measured on an L40S: batch 48 -> 34 GB, 32 -> 24 GB, 16 -> 14 GB (--batch-size 16 --grad-accum 3 on a 24 GB card).
Files
| path | what |
|---|---|
checkpoints/wilddet3d_robotwin.pt |
WildDet3D fine-tuned on RoboTwin 2.0 scenes (50 tasks x clean / randomized), monocular refresh β 4.7 GB |
checkpoints/gam_policy_<task>.pt |
action expert of <task> (EMA weights) β 1.2 GB each |
All 50 RoboTwin 2.0 tasks have a checkpoint; configs/robotwin/<task>.yaml names the task's slots (the objects the
detector is prompted with). Training caches are not released: GAM.precompute builds one from your demonstrations.
Cache format (written by GAM.precompute)
<cache_dir>/episode<i>.npz: state (T,14), action (T,14), patches (T,1296,1024) float16, box2d (T,K,4),
xyz (T,K,3), boxgeo (T,K,9) (extents + rot6d), pcd_pts (T,K,512,3) float16, pcd_mask (T,K,512),
clicks (T,K,2), image_wh, instruction. K is the number of slots in meta.json (1 to 4; seven tasks
(beat_block_hammer, click_bell, grab_roller, handover_block, place_can_basket, shake_bottle, stack_bowls_two)
were trained with three, padded with __none__).
License
The GAM code is MIT (GehaoZhang6/Grounded-Action-Model). The WildDet3D weights here are fine-tuned from WildDet3D (Allen Institute for AI), which builds on SAM 3 and LingBot-Depth, and are distributed under the SAM License; the policy checkpoints and span heads are MIT.
Citation
@misc{zhang2026groundedactionmodel3d,
title = {Grounded Action Model: 3D Grounding as a Foundation for Robotics},
author = {Gehao Zhang and Weikai Huang and Shailesh Shailesh and Yiyan Peng and Jiafei Duan and Ranjay Krishna},
year = {2026},
eprint = {2609.23863},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2609.23863}
}
- Downloads last month
- 49






