Build World Models for Interactive Simulation with Nvidia Cosmos

NVIDIA Cosmos is presented as a platform for adapting physical-AI world models to specific sensors, tasks, and data, then deploying them reliably. Its modular Cosmos Framework supports training and fine-tuning across text, images, video, audio, and actions; reinforcement-learning methods including DPO, GRPO, PPO, OPD, and DiffusionNFT; and optimization through distillation and quantization. It can work with Cosmos 3, Alpamayo, GR00T, Qwen, pi0.5, and compatible Hugging Face models, using tools such as PyTorch, vLLM, and NVIDIA NIM. Agent skills and recipes are intended to automate bounded development tasks while preserving controls such as pinned revisions, held-out evaluation, and recorded quality, memory, and latency.

The practical examples center on Cosmos 3 Nano simulation. For autonomous driving, Transfer-Auto generates synchronized multi-camera video and LiDAR from prompts and map controls, producing up to 400 frames at 30 fps; Sim-Auto instead generates observations incrementally, closing the loop between a policy, the AlpaSim runtime, and the simulated world. Robotics workflows translate simulator depth or other visual inputs into realistic RGB while retaining motion, or generate bimanual robot observations from actions. A surgical model is described as learning visual dynamics from synchronized video and robot kinematics to generate or predict actions and video. Cosmos NIM provides GPU-accelerated model serving; the supplied summary reports roughly two- to threefold speedups on H200 GPUs through techniques including quantization, speculative decoding, and cache-aware routing.