GAM β€” Grounded Action Model

GAM-RoboTwin

GAM (Grounded Action Model) grounds the objects named in the instruction with an open-vocabulary 3D detector (WildDet3D), turns each object into a 3D box, a point cloud and a handful of image tokens, and predicts action chunks with a flow-matching transformer. This repository holds everything the released code needs to run GAM on RoboTwin 2.0: the RoboTwin-tuned detector, one action expert for each of the 50 tasks. All files are referenced from configs/robotwin/<task>.yaml and download on first use.

The videos are evaluation rollouts of the released experts; the wireframes are the 3D boxes the policy was given at that step (predicted by the detector from the RGB image alone, labelled with the text prompt of the slot).

click_bellshake_bottlebeat_block_hammerhandover_block
grab_roller (randomized)stack_bowls_two (randomized)place_can_basket

Quick Links

Intended Use

Evaluate the released experts on RoboTwin 2.0 (clean and randomized scenes), or train your own expert from your own demonstrations. The configuration is monocular: the detector sees RGB only (zero depth map), object geometry is its predicted 3D box and the point cloud is back-projected from its own depth; no simulator depth or object state is used at training or evaluation. Every object is grounded from its name on every frame (grounding: text).

Results

Success rate (%) of the 50 experts, clean (Easy) and randomized (Hard) scenes, 20 episodes each, monocular (no simulator depth or object state). For eight tasks (blocks_ranking_rgb, blocks_ranking_size, place_cans_plasticbox, place_container_plate, place_object_basket, press_stapler, rotate_qrcode, stack_blocks_two) the released checkpoint is an intermediate training step, picked by this evaluation.

Easy Hard Avg
mean over 50 tasks 67.6 44.0 55.8
Per task
task Easy Hard Avg
adjust_bottle 100 100 100.0
beat_block_hammer 85 45 65.0
blocks_ranking_rgb 15 15 15.0
blocks_ranking_size 35 30 32.5
click_alarmclock 95 80 87.5
click_bell 100 60 80.0
dump_bin_bigbin 90 50 70.0
grab_roller 100 100 100.0
handover_block 90 50 70.0
handover_mic 100 75 87.5
hanging_mug 30 15 22.5
lift_pot 85 10 47.5
move_can_pot 60 25 42.5
move_pillbottle_pad 35 15 25.0
move_playingcard_away 85 45 65.0
move_stapler_pad 25 0 12.5
open_laptop 100 75 87.5
open_microwave 50 15 32.5
pick_diverse_bottles 55 60 57.5
pick_dual_bottles 65 65 65.0
place_a2b_left 50 30 40.0
place_a2b_right 40 35 37.5
place_bread_basket 35 25 30.0
place_bread_skillet 35 0 17.5
place_burger_fries 90 95 92.5
place_can_basket 80 80 80.0
place_cans_plasticbox 80 15 47.5
place_container_plate 90 20 55.0
place_dual_shoes 50 25 37.5
place_empty_cup 90 35 62.5
place_fan 60 5 32.5
place_mouse_pad 15 5 10.0
place_object_basket 70 50 60.0
place_object_scale 35 10 22.5
place_object_stand 80 80 80.0
place_phone_stand 55 35 45.0
place_shoe 90 35 62.5
press_stapler 90 75 82.5
put_bottles_dustbin 60 70 65.0
put_object_cabinet 60 55 57.5
rotate_qrcode 65 10 37.5
scan_object 70 20 45.0
shake_bottle 100 90 95.0
shake_bottle_horizontally 95 95 95.0
stack_blocks_three 30 15 22.5
stack_blocks_two 75 30 52.5
stack_bowls_three 80 60 70.0
stack_bowls_two 85 95 90.0
stamp_seal 50 25 37.5
turn_switch 75 50 62.5

Install

git clone https://github.com/GehaoZhang6/Grounded-Action-Model && cd Grounding-Action-Model
conda create -n gam python=3.11 && conda activate gam
bash scripts/install.sh --robotwin /path/to/RoboTwin      # GAM + RoboTwin 2.0 in one environment
export ROBOTWIN_ROOT=/path/to/RoboTwin

Evaluate

# clean + randomized, 20 episodes each; --overlay also writes the videos above (head camera + 3D boxes)
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --overlay
# your own checkpoint, or RoboTwin in a separate environment
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --ckpt runs/robotwin/click_bell/latest.pt
python benchmarks/robotwin/eval.py --config configs/robotwin/click_bell.yaml --sim-python ~/envs/robotwin/bin/python

eval.chunking selects how action chunks are executed: replan (the released setting: 15 of 50 steps per query, seam-blended), tempens (temporal ensembling) or rtc (real-time chunking). The detector runs in bf16 at evaluation (eval.precision).

Train

# from your own demonstrations: precompute, then train β€” or end-to-end (frozen detector, no cache)
python -m GAM.precompute --config configs/robotwin/click_bell.yaml && python -m GAM.train --config configs/robotwin/click_bell.yaml
python -m GAM.train --config configs/robotwin/click_bell.yaml --e2e

Measured on an L40S: batch 48 -> 34 GB, 32 -> 24 GB, 16 -> 14 GB (--batch-size 16 --grad-accum 3 on a 24 GB card).

Files

path what
checkpoints/wilddet3d_robotwin.pt WildDet3D fine-tuned on RoboTwin 2.0 scenes (50 tasks x clean / randomized), monocular refresh β€” 4.7 GB
checkpoints/gam_policy_<task>.pt action expert of <task> (EMA weights) β€” 1.2 GB each

All 50 RoboTwin 2.0 tasks have a checkpoint; configs/robotwin/<task>.yaml names the task's slots (the objects the detector is prompted with). Training caches are not released: GAM.precompute builds one from your demonstrations.

Cache format (written by GAM.precompute)

<cache_dir>/episode<i>.npz: state (T,14), action (T,14), patches (T,1296,1024) float16, box2d (T,K,4), xyz (T,K,3), boxgeo (T,K,9) (extents + rot6d), pcd_pts (T,K,512,3) float16, pcd_mask (T,K,512), clicks (T,K,2), image_wh, instruction. K is the number of slots in meta.json (1 to 4; seven tasks (beat_block_hammer, click_bell, grab_roller, handover_block, place_can_basket, shake_bottle, stack_bowls_two) were trained with three, padded with __none__).

License

The GAM code is MIT (GehaoZhang6/Grounded-Action-Model). The WildDet3D weights here are fine-tuned from WildDet3D (Allen Institute for AI), which builds on SAM 3 and LingBot-Depth, and are distributed under the SAM License; the policy checkpoints and span heads are MIT.

Citation

@misc{zhang2026groundedactionmodel3d,
  title         = {Grounded Action Model: 3D Grounding as a Foundation for Robotics},
  author        = {Gehao Zhang and Weikai Huang and Shailesh Shailesh and Yiyan Peng and Jiafei Duan and Ranjay Krishna},
  year          = {2026},
  eprint        = {2609.23863},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2609.23863}
}
Downloads last month
49
Video Preview
loading

Paper for gehaozhang/gam-robotwin