Streaming RealSense RGB and depth from the robot's mini PC to the browser-based operator panel.
Our teleoperation system (FR20 controlled via FR3, with controller, digital twin and operator UI) gains a second compute node: a mini PC mounted on the robot, intended to offload workloads from the operator PC. camera-node is the first service running on it, and the first of several planned Rust applications. It lives in a Cargo workspace (edge/) so future services can share crates.
Its job is narrow on purpose: capture RGB and depth from Intel RealSense cameras connected to the mini PC and deliver low-latency live video to the operator's browser over the robot's wired LAN. Vision processing, recording and point clouds are explicitly out of scope for v1 and slot in later without restructuring.
Everything ships as one process: capture, encoding and signalling. There is no separate media server, no external broker, and only one port to know (ws://minipc:8443). The operator UI connects to that address, discovers the available streams, and attaches them to video elements.
The deciding constraint is that our operator panel is browser-based. A browser can only receive live video through a handful of mechanisms, and for teleoperation the field narrows fast:
HLS / DASH: segment-based, 2 to 10 s latency even in low-latency modes. Built for video-on-demand, unusable for teleop.
MJPEG over HTTP: trivially simple, but roughly 10× the bandwidth of H.264 and no hardware encoding. Kept only as a debug fallback.
Plain RTP / RTSP: excellent on a LAN, but browsers cannot consume it. Only viable for native UIs, which ours is not.
WebSocket + WebCodecs: workable and simple conceptually, but we would hand-build jitter management, fan-out to multiple viewers, and per-client bitrate adaptation ourselves.
WebRTC: purpose-built for exactly this. Sub-100 ms glass-to-glass on a LAN, hardware decode in the browser, built-in congestion control and multi-consumer support, lands directly in a <video> tag. Chosen.
Because both machines sit on the same network, no STUN or TURN infrastructure is needed: WebRTC's ICE negotiation finds the direct route. The stack works identically over wired ethernet or shared Wi-Fi (same subnet). Wired is preferred for teleoperation, and WebRTC's congestion control absorbs Wi-Fi jitter by adapting bitrate if we ever need the wireless path.
RealSense depth frames are 16-bit distance maps and not directly viewable. Conveniently, librealsense ships a built-in colorizer: it converts each depth frame into a ready-made color image (near to far mapped onto a colormap). camera-node therefore builds no depth visualization of its own. It simply requests the colorized depth image from the SDK and pushes it through the exact same encode-and-stream path as the RGB feed. Each camera produces two named streams, cam0-rgb and cam0-depth, and the operator UI shows two videos.
What we deliberately do not stream is the raw 16-bit data: lossy video codecs would corrupt the actual millimeter values. If the UI later needs true depth values for measurements or point clouds, those will travel as losslessly compressed data on demand over a separate channel.
| Component | What it does | Why chosen |
|---|---|---|
| librealsense2 | Intel's official camera SDK: device access, RGB and depth streams, built-in depth colorizer | The canonical, maintained way to talk to RealSense hardware |
| realsense-rust | Rust bindings over librealsense2 (Tangram Vision) | Safe, maintained wrapper; keeps the whole service in Rust |
| GStreamer + gstreamer-rs | Media pipeline framework and its official Rust bindings | Industry-standard pipelines with first-class Rust support; enables hardware encoding (VA-API on the Intel mini PC), which drops per-camera CPU from ~90% to a few percent |
| webrtcsink (gst-plugins-rs) | Batteries-included WebRTC producer element, itself written in Rust | Handles encoder selection, connection negotiation, encryption, congestion control and multi-consumer fan-out, which is the hard 80% of WebRTC |
| Embedded signalling | WebSocket handshake server, run inside webrtcsink | Zero extra processes or custom protocol code in v1 |
| gstwebrtc-api | JavaScript client library (npm) for the browser side | Matches the signalling protocol exactly; the operator UI consumes streams in roughly 40 lines |