01 · Input
T + 1 frames
Configuration Sequence
tt+1t+2t+T
.52.52.57.81
−.10−.08.02.27
.31.28.19.12
.05.03.00−.08
Abstract
In this technical report, we propose Pelican-Sim 1.0, a general world model simulator for embodied intelligence that predicts future observations from visual context and robot actions to support downstream learning and decision making. The model incorporates four key design features: (1) Unified action representation: a 28-dimensional action value space covering most mainstream embodiments, keeping one model valid across heterogeneous devices. (2) Action-visual injection: URDF- and camera-rendered action videos bridge actions and pixels, giving markedly better controllability across embodiments, scenes, and tasks (PSNR +0.904 over alternative fusion baselines). (3) Sparse mixture-of-experts (MoE): sparse MoE layers add capacity for heterogeneous dynamics and absorb the action modality while reducing inter-modality conflict (FVD −6.530 vs. the dense backbone). (4) Efficient rollout generation: causal adaptation and few-step distillation yield a four-step autoregressive simulator, achieving a 5.67× speedup over the 35-step model.
Benefiting from these designs, we train on approximately one million real-world and simulated trajectories and obtain large gains in action controllability and video quality: PSNR improves over the strongest evaluated baselines by 4.636 on AgiBotWorld Beta, 2.080 on RoboMIND, and 10.343 on RoboTwin, with the adapted EWMBench DYN score up 0.426 on RoboTwin. Relying on this, four downstream applications on RoboTwin succeed: 500 generated trajectories added to 50 demonstrations per task raise policy success from 70% to 93%; policy evaluation reaches a Pearson correlation of 0.994 across five checkpoints; and relative success gains reach 47.7% for action selection and 20.3% for policy improvement. Qualitative generalization across trajectory, scene, object, embodiment, and viewpoint shifts highlights its potential as a general-purpose world model simulator.
Method
Pelican-Sim 1.0 predicts future observations from an initial RGB image, a frame-aligned robot-configuration trajectory, and an optional task instruction. The trajectory supplies numerical configurations and, through URDF rendering with camera calibration, synchronized visual action conditions.
01
Complementary Action Representations
Forward Kinematics
Camera Projection
Action Video
gripper state
Open
Transitional
Closed
02
Dual-Branch Injection with Sparse Experts
Interleaved Action Conditioning
In the 28-layer Cosmos-Predict 2.5 DiT, numerical action embeddings apply scale and shift within odd-indexed blocks; action-video Context Blocks add aligned residuals after even-indexed blocks. Indices are zero-based.
Unified Action Representation
Each side occupies 14 dimensions: seven arm joints, one parallel-gripper opening, and six hand joints. Unused joints and absent sides are zero-filled. Both action conditions span the conditioning frame and all target frames.
Sparse Mixture of Experts
Every main DiT block uses one always-active shared expert and eight routed experts, with two routed experts selected per token. This expands capacity through input-dependent activation.
Experiments & Results
01
Experimental Setup
- Training trajectories
- ~1M
- Robot interaction video
- 8,000 hours
- Real & simulated sources
- 7 datasets
The corpus combines AgiBotWorld Beta, RealSource World, and RoboMIND with LIBERO, ManiSkill2, RoboTwin, and RoboCasa. A 90% / 5% / 5% training, validation, and test partition is established before clip extraction. Video prediction is evaluated on held-out splits of AgiBotWorld Beta, RoboMIND, and RoboTwin; baselines are retrained on the same mixed-domain corpus.
02
Video Prediction Benchmark
Pelican-Sim 1.0 achieves the best scores on all five video-quality metrics and the adapted EWMBench overall score among the methods evaluated on each dataset.
AgiBotWorld Beta
+4.636 dBPSNR over the best baseline
EnerVerse-AC → Pelican-Sim
17.640 → 22.276 dB
RoboMIND
+2.080 dBPSNR over the best baseline
Ctrl-World → Pelican-Sim
21.770 → 23.850 dB
RoboTwin
+10.343 dBPSNR over the best baseline
Ctrl-World → Pelican-Sim
20.040 → 30.383 dB
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓ | FVD ↓ | Overall ↑ |
|---|---|---|---|---|---|---|
| AgiBotWorld Beta | ||||||
| IRASim | 15.796 | 0.544 | 0.028 | 64.418 | 62.548 | 3.571 |
| Ctrl-World | 17.550 | 0.682 | 0.022 | 46.600 | 45.100 | 4.401 |
| OSCAR | 16.644 | 0.608 | 0.023 | 27.805 | 32.241 | 4.303 |
| Wan-Move | 12.610 | 0.429 | 0.447 | 88.496 | 95.260 | 3.244 |
| EnerVerse-AC | 17.640 | 0.769 | 0.023 | 19.510 | 21.390 | 4.431 |
| Pelican-Sim 1.0 | 22.276 | 0.796 | 0.013 | 14.640 | 10.850 | 4.997 |
| RoboMIND | ||||||
| IRASim | 20.102 | 0.709 | 0.019 | 44.597 | 43.414 | 3.079 |
| Ctrl-World | 21.770 | 0.716 | 0.015 | 21.800 | 16.700 | 4.276 |
| EnerVerse-AC | 17.930 | 0.680 | 0.293 | 67.350 | 63.690 | 2.982 |
| Pelican-Sim 1.0 | 23.850 | 0.835 | 0.012 | 17.610 | 15.930 | 4.666 |
| RoboTwin | ||||||
| IRASim | 17.121 | 0.666 | 0.025 | 69.777 | 57.350 | 3.383 |
| Ctrl-World | 20.040 | 0.804 | 0.016 | 31.400 | 26.600 | 3.971 |
| EnerVerse-AC | 19.680 | 0.779 | 0.221 | 32.450 | 29.700 | 3.610 |
| Pelican-Sim 1.0 | 30.383 | 0.942 | 0.006 | 8.910 | 4.980 | 5.044 |
03
Qualitative Generation Results
Action-conditioned rollouts across seven real-world and simulated data sources, including parallel-gripper and dexterous-hand embodiments.
AgiBotWorld Beta · Gripper
Real
1 / 8
Ground-truth video
Rendered Action
Prediction
Ground-truth video
Rendered Action
Prediction
Applications
The four-step causal simulator supports data generation and policy applications. A fine-tuned Qwen3-VL-2B-Instruct evaluator supplies task-conditioned success, progress, and visual-validity scores for policy evaluation, action selection, and policy improvement, and remains frozen during downstream use.
DATA GENERATION
Scalable Data Generation
Pelican-Sim generates ten rollouts per demonstration from edited initial frames and the original actions. With 10, 30, and 50 randomized demonstrations per task, π0.5 success improves from 28.5%, 57.0%, and 70.0% to 64.5%, 87.0%, and 93.0%, respectively (Table 9). A separate cross-domain study reaches 72.0% using 50 clean and 500 generated trajectories, compared with 70.0% using 50 directly collected randomized trajectories (Table 8). Both studies use 200 unseen evaluation seeds per task.
Generalization under Distribution Shifts
Scene & Appearance
5 generated scenesContributors
Core contributors · Shilong Zou, Shilin Zhang
Shilong Zou: data preparation, model architecture design, world-model training, and downstream policy training. Shilin Zhang: data preparation, downstream policy training, and visualization of experimental results.
Contributors
Yingji Zhang, Yuhang Huang, Yi Zhang, Zeyuan Ding, Han Dong, Junwei Liao
Tech leads
Yong Dai, Shilong Zou
Corresponding authors
Jian Tang, Xiaozhu Ju
Contact: jack.zou@x-humanoid.com · vito.dai@x-humanoid.com · jian.tang@x-humanoid.com · jason.ju@x-humanoid.com.
Citation
BibTeX
@misc{zou2026pelicansim,
title={{Pelican-Sim 1.0}: A General World Model Simulator for Embodied Intelligence},
author={Shilong Zou and Shilin Zhang and Yingji Zhang and Yuhang Huang and Yi Zhang and Zeyuan Ding and Han Dong and Junwei Liao and Yong Dai and Jian Tang and Xiaozhu Ju},
year={2026},
eprint={2609.12036},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2609.12036}
}