Introduction

Recent advancements in neural rendering techniques, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have shown great promise in creating photorealistic 3D scene reconstructions from real-world data like monocular images and videos. These techniques open up exciting new possibilities for robot learning by allowing robots to be trained in highly realistic simulated environments derived from real-world scenes. This emerging “real-to-sim” approach aims to close the gap between simulation and reality by transitioning from real-world data to reconstructed neural representations, training robots in these realistic environments, and then deploying them back into the real world with a minimized sim-to-real gap.