Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning

ArXi:2507.09180v4 Announce Type: replace Depth information is robust to scene appearance variations and inherently carries 3D spatial details. Thus, a visual backbone based on the vision transformer is proposed to fuse RGB and depth modalities for enhancing generalization in this paper. Different modalities are first processed by separate CNN stems, and the combined convolutional features are delivered to the scalable vision transformer to obtain visual representations.