Open
New views, object layouts, and interaction scenarios keep the task space from collapsing into memorization.
Research release 2025
Spatial intelligence
that survives
the viewpoint shift.
ROCKET-3 teaches a visuomotor agent to connect a goal it can see with a world it is standing in—then refine that skill through reinforcement learning.
100K+synthesized tasks
4×interaction success
0-shotunseen worlds
Goal viewlandmark found
RL post-trainingreasoning unlocked
One policy. Many views. Zero-shot transfer.
01 / AT A GLANCE
100,000+automatically synthesized
cross-view tasks
cross-view tasks
72Minecraft instances
in parallel
in parallel
~1,000frames per second
collection throughput
collection throughput
41%real-world ball-finding
success lift
success lift
02 / THE CORE IDEA
The goal is a view,
not a sentence.
Language names a task. A visual goal specifies where the task lives. ROCKET-3 turns that cross-view relationship into a reusable interface for multi-task reinforcement learning.
Task representationCVGS
T = ⟨O₁,Og,Mg,E⟩
O₁ agent view
Og goal view
Mg target mask
E interaction
“Can the agent act on a target it cannot currently see?”
Verifiable
Every task has a measurable target and outcome, making it useful for both expert trajectories and RL rewards.
Scalable
Procedural synthesis turns a Minecraft world into a stream of training tasks with almost no manual labeling.
Curricular
Distance and visibility provide a natural path from easy alignment to occluded, long-range interaction.
03 / TRAINING LOOP
Foundation → finesse
→ transfer.
Imitation learning gives the policy a broad visual prior. PPO post-training then rewards the spatial reasoning that matters when the target is distant, hidden, or described from another camera.
04 / EVIDENCE
RL makes the
hard cases move.
Across Approach, Break, Interact, and Hunt tasks, post-training lifts the average success rate from 7% to 28% while preserving the spatial signals learned during pre-training.
Four times the average success
RL turns a 7% baseline into 28% across interaction types. Archery rises from below 1% to 28%.
KL keeps learning stable
A constraint against the pre-trained policy prevents collapse while the agent learns long-horizon interaction.
Representations stay useful
Visibility and centroid heads retain their performance after RL, a signal that the policy is learning more than a shortcut.
05 / ZERO-SHOT TRANSFER
One spatial skill.
New worlds.
With a light action mapping, the same policy moves from Minecraft into DMLab, Unreal, and a real robot car—without retraining on the target world.
06 / IN MOTION
Watch the policy
close the loop.
Short rollouts from the release. Switch environments to see the same interaction vocabulary travel from a Minecraft task to unseen worlds.
07 / OPEN RESEARCH
Take the recipe
with you.
Read the full method, inspect the training code, or start from the released checkpoint. ROCKET-3 is built to be a useful research artifact, not just a result.
quickstart.sh
# install the environment
python -m pip install -r requirements.txt
# run a local checkpoint smoke test
python smoke_test.py \
--checkpoint /path/to/rocket3.ckptCITATION
If this work helps,
please cite it.
@misc{cai2025scalable,
title={Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents},
author={Cai, Shaofei and Mu, Zhancun and Xia, Haiwen and Zhang, Bowei and Liu, Anji and Liang, Yitao},
year={2025},
eprint={2507.23698},
archivePrefix={arXiv},
primaryClass={cs.RO}
}