VR Teleop and Diffusion Policy
Meta Quest 3 Pro teleoperation for a Franka Panda, feeding a demonstration pipeline that trains and deploys a real diffusion policy
The Franka Panda cell: wrist + external RealSense D435 cameras, a fixed workspace, and the objects used for demonstrations.
What it is
This is the data-collection and deployment backbone behind my Diffusion Policy and Latent Safety Filters work: a VR teleoperation system for a real Franka Panda arm, built to make collecting good demonstrations fast enough that training a diffusion policy on real (not simulated) data is actually practical.
How it works
- Teleoperation: a Meta Quest 3 Pro controller drives the arm in real time — controller pose maps to end-effector Cartesian commands, sent to the robot via Polymetis for realtime control. A browser page (
quest_controller.html) hosted from the robot machine streams controller pose and button state from the headset itself, so no extra tracking hardware is needed. - Recording: each demonstration (“episode”) logs end-effector pose and gripper state at 10 Hz, alongside synchronized RGB frames from both cameras at 30 Hz, direct to an SSD mounted on the robot machine.
- Dataset conversion: recorded episodes are converted into the Zarr format the Diffusion Policy training pipeline consumes.
- Deployment: trained policies are hosted on a separate GPU server; the robot computer acts as a client, sending observations and receiving actions over a persistent connection at 10 Hz — the same client-server pattern used for the safety-monitor work.
Collecting a demonstration via Meta Quest 3 Pro teleoperation.
Hardware setup
- Arm: Franka Emika Panda, controlled via Polymetis for low-latency Cartesian control.
- Cameras: two Intel RealSense D435s — one external, one wrist-mounted — for the dual-view observations the policy is trained on.
- Compute split: a dedicated robot-control machine runs the teleop/recording/client code, while training and inference run on a separate GPU machine, kept in sync over the network.
The trained diffusion policy executing a pick-and-place rollout, closing the loop from demonstration to deployment.
Notable engineering details
A few things that weren’t obvious going in:
- The headset has to be worn, not just powered on — the controller’s pose is tracked relative to the headset, so without it on your head the poses come out meaningless. Cost real debugging time before the fix turned out to be that simple.
- Camera calibration matters more than expected — since it’s easy to bump a camera between recording sessions, an uncalibrated dual-camera setup makes the trained policy noticeably more brittle than a single fixed view.
- Gripper control needed a real fix upstream — the stock Polymetis gripper client didn’t expose a blocking stop command, so I forked and patched it (tracked against facebookresearch/fairo#1417) rather than working around it in the teleop script.
- Deliberately diverse demonstrations — varying lighting, object placement, and even recording recovery actions after a failed grasp, rather than only clean successful episodes, since that diversity is what the eventual policy generalizes from.