The playout buffer,
inner workings
One bounded, seq-ordered queue at the robot-node's action ingress. Setpoints join the line as datagrams arrive; a drain thread releases exactly one per ServoJ period. Network jitter becomes queue-depth fluctuation instead of arm-motion fluctuation, and delay is simply time standing in line. This is the receiver-side playout buffer that video streaming uses (the top recommendation of the buffer streaming report), grown a safety layer video never needed: a robot must not "catch up" after a stall.
How to read this document. Every claim carries a file:line reference into rust/robot-node/src/buffer.rs (branch feat/buffer-tab-netem, tip 44bd7b8). That commit restructured the file to read in runtime order: tunables → the wire contract (config, verdicts, reports) → the queue state → the arrival path → the drain ladder → the flush paths → the telemetry surface → the drain thread. This document's section order and the file's order now roughly match, so you can read them side by side; line numbers are approximate anchors next to the function/type names, which are the stable reference. The integration points live in node.rs and main.rs. The figures use synthetic traces generated by simulating the drain rules in this page, so picture and prose cannot drift apart. Two design rules to keep in mind throughout: no cross-host clock math anywhere, and setpoint values are never smoothed or filtered — though since the loss-concealment change a hole of at most 3 missing setpoints is bridged at release by joint-space interpolation toward the next real setpoint (tagged, guarded, one synthetic per tick; §5 and figure H). Bigger holes keep the jump and count as true loss.
1 · Why this exists: the delayed-replay runaway
The incident that shaped every policy in this file: a network stall held a burst of setpoints in flight; when the link recovered, the stale tail was delivered and applied, and the arm replayed an abandoned trajectory at full speed (cap_delta warning, then error=14). A naive queue makes this worse, not better: it faithfully stores the stale tail and plays it back. So the queue here is built inside-out around one question: when is a queued setpoint no longer safe to send?
The three report recommendations it implements (report §6):
Rec 1 · Playout buffering
A receiver-side jitter buffer with prefill, reorder repair and loss/stall attribution from sender timestamps, exactly the RTP receiver shape.
Rec 2 · Fail closed
Where video conceals a gap and plays on, the robot flushes and e-stops: post-live underrun windows and hole-ahead detection turn "catch-up jump" into "controlled stop".
Rec 3 · Receiver reports
An RFC 3550-style health snapshot (jitter EWMA, loss %, depth, spans) shipped operator-ward so the sender side can see what the network is doing to the stream.
2 · Where it sits
The buffer lives in front of the driver, at the node's action ingress. Both transports feed the same queue: UDP datagrams carry their wire seq (node.rs:582 ingest_wire_action), the ordered gRPC stream gets a synthetic monotonically increasing seq at ingress (node.rs:565 ingest_action, buffer.rs:769 next_synthetic_seq). Since commit 97eebdb the playout queue is the sole ServoJ queue: the drain releases straight into the apply path, no mailbox hop. The driver underneath (guard suite, hold/starvation, e-stop latch) is untouched.
flowchart LR
subgraph OP["Operator PC (Python, unchanged)"]
BR["remote_follower_bridge.py"]
end
subgraph NODE["robot-node (Rust)"]
UDP["UDP action listener<br>wire seq"]
GRPC["gRPC StreamActions<br>synthetic seq at ingress"]
OFF["PlayoutBuffer::offer<br>seq-ordered VecDeque"]
DR["playout_drain thread<br>one pop per cmdT"]
SJ["apply path → ServoJ<br>(direct, mailbox bypassed)"]
end
BR -->|"S7A1 datagrams :50063"| UDP
BR -->|"ordered stream :50061"| GRPC
UDP --> OFF
GRPC --> OFF
OFF --> DR
DR -->|"DrainTick::Release"| SJ
UDP -.->|"e-stop: flush queue,<br>apply on arrival thread"| SJ
The passthrough contract
buffer.rs:249 PlayoutConfig::from_comm returns None when
playout_buffer_ms rounds to a depth of 0 (that includes the default 0, negatives and NaN).
No queue, no thread, and the ingress code path is exactly the unbuffered one
(main.rs:164 only attaches and spawns when Some). The buffer is a
feature you turn on per robot YAML, not a layer everything always pays for.
E-stops never queue
An e-stop is applied on the arrival thread immediately, and the queue is flushed and sealed so nothing pre-e-stop can replay (node.rs:674 enqueue_or_bypass, buffer.rs:1213 flush_for_estop). E-stops are dual-pathed by design (datagram + one-shot gRPC), so a synthetic-seq e-stop can land in a wire-seq deployment; only an in-space seq may seal the floor (buffer.rs:207 ActionSeq::estop_floor), because comparing a synthetic seq against wire seqs is wrap-math nonsense that could blackhole the live stream.
Upstream contract: producers never go silent
The underrun policy (§5) makes silence mean something: an engaged stream that stops is a stall,
and the node e-stops. So the producers must never go silent on purpose. The orchestrator honors that with
station-keeping (servo7/nodes/orchestrator_node.py:498 _drain_controller_action,
:529 _hold_follower_pose): in ROBOT_CONTROLLER and TRAJECTORIES modes, any
dispatch tick where the controller yields no action — or yields one that doesn't target the follower —
streams the follower's measured pose instead, at the dispatch rate (the follower's own
publish_rate_hz). The second case is the align-leader break fixed 2026-07-26: a leader-only alignment
trajectory walked the leader for seconds while the follower's playout stream starved into the underrun
e-stop 342 ms later. Holds re-stamp timestamp with the local clock (the bridge's stale-tick
gate judges build time), are skipped entirely while the follower is latched (a re-published e-stop state
would ride the bridge's e-stop path), and never happen in teleop, where the leader is the sole producer.
3 · The edge cases, one picture each
Every rule in buffer.rs exists because one of these pictures used to end with a broken robot. Each figure below is the same kind of timeline: arrivals as the network delivers them, releases as the drain hands them to the arm, and queue depth underneath, all generated by simulating the actual drain rules in this page (N = 5 for legibility). Hover any figure for the tick-by-tick account.
How to read all of them: blue is what the network did, orange is what the arm saw. The whole point of the buffer is that the orange lane stays boring no matter what happens in the blue one, and when it can't stay boring, it stops the arm instead of improvising.
A · The normal case: jitter becomes depth wobble
Packets arrive early, late, and in little clumps. Without a buffer every wobble lands in the arm's motion. With one, the queue breathes and the release lane is a metronome.
Table view
B · Two packets swap places in the network
UDP reorders: seq 5 arrives before seq 4. A latest-wins mailbox would execute 5 then 4, commanding the arm backwards for one tick. The queue just slots 4 into its place.
Repaired
(buffer.rs:841 insert). Reordering is fixed for free by the queue itself.Table view
C · A straggler arrives after its moment has passed
Seq 4 got stuck in the network and shows up after seqs 5 and 6 were already released to the arm. Executing it now would step the trajectory backwards. It dies at the floor.
TooLate, dropped, never sent
(buffer.rs:573 newer_than). The arm is never commanded backwards.Table view
D · The network stalls, then vomits a burst
Five ticks of silence while packets pile up somewhere in a Tailscale relay, then they all land at once. Watch two things: the release lane never misses a beat during the outage (the queue pays it out), and the burst does not become a permanently deeper queue.
Table view
E · Setpoints from a dead stream go stale in the queue
A stream starts, two setpoints arrive, the link dies mid-prefill. Those two entries are now a fragment of an abandoned trajectory. When the link comes back, should the arm execute them? This is the delayed-replay runaway in miniature.
N·cmdT +
max(N·cmdT, 200 ms), so ≈ 2 × the target with a 200 ms floor for shallow buffers — is purged,
never sent (buffer.rs:988 purge_aged, buffer.rs:126
AGE_CAP_MARGIN_FLOOR_S). The model here uses the same 2×N shape. The new stream prefills clean.Table view
F · The link dies for good: fail closed, don't improvise
Arrivals stop and never come back. The queue pays out its depth (that's the jitter budget doing its job), then the drain holds the last setpoint through the grace window, and then, instead of holding forever and maybe replaying whatever shows up next, it e-stops. (Producers never go silent on purpose — §2's station-keeping contract — so a silent link is genuinely dead.)
playout_underrun_estop_after_ms). "Hold forever, then drain the tail" is precisely how the
runaway happened; this replaces it with a controlled stop.Table view
G · A chunk of the trajectory is simply gone
The network eats seqs 7 through 21 whole, then delivery resumes with 22. The queue now holds old setpoints and far-future ones with a 15-tick hole between them. Releasing the head means the arm will eventually leap across that hole in one step.
Table view
H · A small hole is concealed, one synthetic per tick
The network eats seqs 6 and 7 — a hole small enough (≤ 3 missing) that stopping the arm for it would be absurd, but jumping across it is a velocity spike. Instead the drain releases synthetic setpoints, lerped in joint space toward the next real one, at constant speed. When the real 6 finally straggles in, the floor has already sealed its slot.
extras.playout_interpolated, released through the normal path so every guard sees it.
Decided per tick: a straggler that arrives before its slot still plays for real; one that arrives
after is sealed off by the floor. Bigger holes keep the jump and count as lost_at_release
(true loss) — and the pathological spans are figure G's e-stop.Table view
I · Everything at once: one full session
Prefill, steady jitter, a stall-and-burst absorbed by catch-up, and finally a real link death that fails closed. This is A + D + F in one take.
Table view
4 · Arrival: offer() decides in six verdicts
Every non-e-stop setpoint lands in buffer.rs:788 offer(seq, state, now). One lock, an O(depth) decision, six possible verdicts (buffer.rs:299 enum Offer):
Malformed · buffer.rs:567 sender_ts — the sender-local
clock of an arrival is leader_timestamp when stamped, else the state's own build timestamp. A
non-finite value is rejected outright: every downstream health metric and the hole-ahead safety check take
differences of these timestamps, and one NaN would poison them all.
EpochReset · buffer.rs:819 epoch_expired — a restarted
sender resets its wire seq to zero, which would read as "ancient" forever. Mirroring the listener's
ACTION_EPOCH_RESET_S rule: a stale-looking seq arriving after ≥ 2 s of arrival silence flushes
the dead epoch, forgets the seq floor and the timing reference (a restarted sender is a new clock), and
starts prefill over with this entry first.
TooLate · buffer.rs:573 newer_than — the floor is the
newest seq already handed to the driver (or sealed past by a flush). Anything at or below it is dropped:
the arm is never commanded backwards along the trajectory. The comparison is the wrap-aware
SeqGate rule (0 < seq − base < 2³¹ in u32 arithmetic).
Queued / Repaired · buffer.rs:841 insert — entries
insert in wire-seq order. A packet that arrives behind an already-queued newer seq lands in its correct
slot (Repaired, counted): UDP reordering is repaired for free by the queue itself.
Catch-up overflow · buffer.rs:861 drop_overflow — above the high-water mark (target + HIGH_WATER_SLACK = 2, buffer.rs:120) the oldest entries are dropped back down to the current runtime target (§6). A burst after a stall therefore shortens the queue to the freshest N setpoints instead of building a standing lag: the buffer holds delay at the target, it never accumulates it.
Timing health on the way in · buffer.rs:874 note_timing — judged only for arrivals strictly newer than the newest accepted seq (a repair is judged by the hole it fills, not against the tail). Details in §9.
For reference, the six verdicts as one decision tree (you have already seen every branch in action in §3):
flowchart TD
A["offer(seq, state, now)"] --> M{"sender timestamp finite?"}
M -->|"no"| MAL["Malformed<br>rejected, never queued"]
M -->|"yes"| EP{"epoch aged out?<br>idle ≥ 2 s + stale-looking seq"}
EP -->|"yes"| ER["EpochReset<br>flush dead epoch, forget floor,<br>queue as first of new epoch"]
EP -->|"no"| FL{"seq newer than floor?<br>(wrap-aware)"}
FL -->|"no"| TL["TooLate<br>dropped — arm never<br>commanded backwards"]
FL -->|"yes"| DU{"seq already queued?"}
DU -->|"yes"| DUP["Duplicate<br>dropped"]
DU -->|"no"| NT["note timing health<br>loss · spacing · jitter"]
NT --> INS["insert in seq order<br>(reorder repaired)"]
INS --> OV{"depth > target + 2?"}
OV -->|"yes"| DROP["catch-up: drop oldest<br>back down to target"]
OV -->|"no"| Q["Queued / Repaired"]
DROP --> Q
5 · Drain: one pop per tick, and when not to pop
A dedicated thread (buffer.rs:1313 spawn_playout_drain, named
playout_drain_{id}) ticks once per ServoJ period with the same
Instant-paced shape as the driver's servoj_loop: next_tick += period,
sleep the remainder, snap to now if behind. Each tick calls
buffer.rs:953 drain_tick(now) and acts on one of three verdicts
(buffer.rs:320 enum DrainTick):
| Verdict | The node does | Where |
|---|---|---|
Release(state) | Stamp the playout_release trace hop, apply straight to ServoJ (this hop is visible in the dashcam profiler waterfalls). | node.rs:614 release_buffered |
Hold | One stream_hold_tick(): the driver re-sends the last released setpoint so the controller's 100 Hz stream never starves. | node.rs:627 playout_underrun_tick |
Stall(report) | The buffer has already flushed and sealed itself; the node latches the e-stop. Recovery is the operator's reset_emergency + prepare_stream, exactly like a guard trip. | node.rs:639 playout_stall_estop |
Inside one tick the ladder runs in a fixed order — buffer.rs:953 drain_tick reads top to bottom exactly like this list:
1 · Hard age cap · buffer.rs:988 purge_aged — an entry
that has waited longer than N·cmdT + max(N·cmdT, 200 ms) — the buffer's own delay plus an equal
margin, floored at AGE_CAP_MARGIN_FLOOR_S = 0.2 s for shallow buffers
(buffer.rs:126) — is discarded, never sent. With adaptation on, the cap
stretches by twice the extra adapted depth (buffer.rs:292 max_age_for),
so cap ≈ 2 × the runtime target throughout the range. Anything older is stall debris; after a long stall
the queue still effectively drains to newest. This is the direct anti-replay rule.
Why the margin must be a whole buffer: the cap used to be
N·cmdT + 2 stale windows (~40 ms of grace), and that purged healthy heads — an entry legitimately
waits out deepening-slew holds, high-water wobble and interpolation transit (the head ages while synthetics
play). Each wrong purge manufactured a hole at the release cursor; interpolation concealed it by releasing
synthetics without popping; arrivals kept landing, so drop_overflow ate the head
again — self-sustaining churn, visible live as an all-yellow released-mix strip with
aged_dropped climbing on a perfectly clean link. One buffer's worth of grace (floored at
200 ms) absorbs every healthy source of waiting; the fix is pinned by
healthy_head_survives_held_ticks_and_wobble_without_aging_out.
2 · Adapt sample · buffer.rs:1000 adapt — with the adaptive ceiling configured, each live tick feeds the current depth into a 1 s window minimum; on window rollover the runtime target grows or shrinks. The full policy is §6.
3 · Prefill / underrun gate — draining starts only once depth ≥ target, at stream start and again whenever the queue drains to empty. While prefilling nothing is popped and the driver's hold-last machinery covers the gap. This is what buys the jitter budget: a queue that starts draining at depth 1 has no slack to absorb the next late packet. A tick with nothing to release runs the post-live underrun policy below.
4 · Hole-ahead · buffer.rs:1061 hole_ahead — before releasing the head, the drain checks the sender-timestamp span across the queue. A healthy queue spans about one buffer (target × cmdT); a span beyond HOLE_AHEAD_FACTOR = 3 buffers means a loss/stall hole sits between the head and the newest entries, and releasing the head would walk the arm into a catch-up jump across it. With the stall policy armed this fails closed (flush + e-stop); without it, the release proceeds as before, warned and counted.
5 · Deepening hold · buffer.rs:1090 deepening_hold — when the adaptive target has grown and the queue sits meaningfully below it (deficit beyond the high-water slack), hold one tick in every DEEPEN_SLEW_EVERY = 4 instead of releasing. Depth climbs ~25 % per second of wall time without the stream ever pausing outright, and the holds stop by themselves the moment depth reaches target.
6 · Loss concealment · buffer.rs:1104 interpolate_hole —
when the head sits a hole of at most INTERP_MAX_GAP = 3 missing seqs away from
the last released one, release one synthetic setpoint this tick: joint-space lerp toward the head,
seq advancing by one, joint_velocities cleared, tagged extras.playout_interpolated,
through the normal release path so the full guard suite sees it. Decided per tick, so a straggler arriving
before its slot still plays for real; one arriving after is sealed off by the floor (figure
H). Bigger holes fall through to the pop and keep the jump.
7 · Pop · buffer.rs:695 count_release_gap — pop the
front entry, advance the floor, release. If the release cursor jumped seqs that never arrived and can no
longer play, they are counted as lost_at_release — the monotone true-loss counter
(open holes behind the cursor can still heal and never count; concealed seqs count as
interpolated, not lost).
Post-live underrun stall · buffer.rs:1139 underrun_tick —
with playout_underrun_estop_after_ms configured, a queue that stays empty past the window
after the stream has been live means the whole jitter budget is burned: a real stall. The buffer
flushes itself and reports the stall so the node e-stops, instead of holding forever and later draining an
abandoned tail. That "hold forever, then drain the tail" behavior is precisely the delayed-replay runaway.
Initial prefill never escalates, and every flush re-arms the liveness gate: the next window must earn its
first release before an empty queue can count as a stall again.
For reference, the full drain-tick decision tree. You have already watched every branch of it happen: figures E, F, G and H in §3 are the purge, underrun, hole-ahead and interpolation legs; figure I plays a whole session through it:
flowchart TD
T["tick (every cmdT)"] --> PA["purge over-age entries<br>(never sent)"]
PA --> AD["adapt: sample depth into the 1 s window;<br>on rollover grow / shrink the target (§6)"]
AD --> PF{"prefilling?"}
PF -->|"yes, depth < target"| UT["underrun_tick"]
PF -->|"yes, depth ≥ target"| GO["prefill complete"]
PF -->|"no"| EM{"queue empty?"}
EM -->|"yes"| RE["re-arm prefill"] --> UT
EM -->|"no"| GO
GO --> HA{"hole ahead?<br>sender-time span > 3 buffers"}
HA -->|"yes + policy armed"| ST1["flush + seal floor<br>Stall(HoleAhead)"]
HA -->|"yes, no policy"| WR["warn + count,<br>release anyway"] --> IH
HA -->|"no"| DH{"deepening hold?<br>below target − 2, every 4th tick"}
DH -->|"yes"| H3["Hold<br>(slew: depth climbs, stream alive)"]
DH -->|"no"| IH{"small hole at the cursor?<br>gap ≤ 3 missing seqs"}
IH -->|"yes"| SYN["release ONE lerped synthetic<br>seq + 1, tagged playout_interpolated"]
IH -->|"no"| POP["pop front, floor = seq,<br>count lost_at_release on a jump<br>Release(state)"]
UT --> LV{"ever released<br>this window?"}
LV -->|"no"| H1["Hold<br>(prefill is by design)"]
LV -->|"yes"| WIN{"underrun window<br>configured & expired?"}
WIN -->|"no"| H2["Hold<br>(driver holds last setpoint)"]
WIN -->|"yes"| ST2["flush + seal floor<br>Stall(Underrun)"]
6 · Adaptive depth: pay latency only when the network demands it
A fixed depth is a wager placed once, in a YAML file, on how bad the network will be. With
playout_adaptive_max_ms above the base, the wager becomes a runtime servo: the target floats in
[base, max] (buffer.rs:218 PlayoutConfig — base from
playout_buffer_ms, ceiling from playout_adaptive_max_ms, both clamped at 200
entries). The driving signal is not a jitter proxy but the actual failure margin: how close the live
queue came to empty (buffer.rs:1000 adapt).
Measure — every live tick (released at least once, not prefilling) samples the queue depth into a running window minimum; windows roll over every ADAPT_WINDOW_S = 1 s. Idle and prefill ticks feed no samples, so an idle robot never adapts on noise.
Grow fast · buffer.rs:1038 grow_target — a window
minimum at or below ADAPT_GROW_MARGIN = 3 entries means the queue nearly
underran: deepen before the stall, by max(target / 4, 2) entries, up to the ceiling.
The node logs "playout adaptive: deepened target to N entries (Xms)".
Shrink slow — a window whose minimum stayed above 3/5 of the target counts as calm; after ADAPT_SHRINK_CALM_WINDOWS = 10 consecutive calm windows the target shrinks by one entry per further calm window, never below the base. Grow fast, shrink slow — the asymmetry every jitter buffer needs.
Realize without stuttering · buffer.rs:1090 deepening_hold — a deeper target on a live stream is realized by the slew of §5 step 5: hold one tick in every DEEPEN_SLEW_EVERY = 4 while the deficit exceeds the high-water slack. Depth climbs ~25 % per second without the stream ever going silent, and the age cap stretches with the target (buffer.rs:292 max_age_for) so the deeper standing wait stays legal.
What survives a flush — and what must not ·
buffer.rs:662 discard_all — the learned target survives every flush: it is
knowledge about the link, not about the stream. But the in-flight window measurement and the
arrival timing reference are discarded with the queue, for two reasons found live on 2026-07-26: a stall's
drain-down left depth-0 samples in the adapt window, so the first rollover after the e-stop "deepened" the
target and toasted connection unstable — evidence from a stream that was deliberately cut, not a
bad link (pinned by stall_flush_discards_the_adapt_windows_graze_evidence); and the first
packet of the next engagement was spacing-judged against the dead epoch's last packet, producing a
"sender stall ~21622ms" warn on every re-engage (pinned by
first_arrival_after_a_flush_is_not_spacing_judged).
The operator sees it — the bridge watches target_ms vs
min_ms in the health report and toasts
"connection unstable: follower playout buffer deepened to Xms (base Yms) — teleop is
paying extra latency to ride out network jitter/loss", re-warning only when the target climbs
further, and an INFO marks the return to base
(remote_follower_bridge.py:1441 _note_adaptive_depth).
adaptive_target_grows_on_graze_deepens_by_slew_and_shrinks_when_calm.
7 · Your four questions, answered by the code
"When the buffer is empty, do we still send the current state?"
The buffer itself sends nothing on an empty tick
(DrainTick::Hold), but the controller never starves: the node answers every hold with
stream_hold_tick(), so the driver keeps re-sending the last released setpoint at
100 Hz (each such tick is counted in hold_ticks). The empty tick also re-arms prefill. And if
the stall policy is armed and the emptiness outlives the window after a live stream, the answer changes
from "hold" to "e-stop", by design — which is also why the producers upstream never go silent
while engaged (the orchestrator's station-keeping, §2).
"The buffer should check that the stream is healthy via timestamps"
Implemented at buffer.rs:874 note_timing, with differences of sender-local timestamps only. Seq gaps attribute loss (EWMA of the gap fraction); consecutive seqs spaced > 3 periods apart attribute a sender stall (seq continuous, so not the network); transit-time variation feeds an RFC 3550-style jitter EWMA; non-monotonic timestamps are counted (playout stays seq-ordered); non-finite ones are rejected. At the drain, the hole-ahead span check is the same idea pointed at safety.
"Maybe an EMA filter, but I want to test that myself"
Smoothing is deliberately absent: received setpoints are
released verbatim, in seq order — never averaged, filtered or extrapolated, so whatever you measure when
you experiment with smoothing is your filter's effect alone. The one carve-out is loss concealment
(buffer.rs:1104 interpolate_hole): a hole of ≤ 3 missing setpoints is
bridged by joint-space lerp toward the next real one — synthetic frames, clearly tagged
playout_interpolated, counted in interpolated, and only ever standing in for
setpoints that never arrived. Values that did arrive are never touched. Background reading in this folder:
the EMA signal filter explainer.
"Stale trajectories in flight should be disregarded"
They die in four independent ways: (1) the age cap purges anything that waited past the buffer's depth plus an equal margin (≥ 200 ms), (2) catch-up overflow drops the oldest back to the target after a burst, (3) the seq floor drops anything at or below the newest released seq (TooLate), and (4) every flush (e-stop, stall, lifecycle, epoch reset) discards the queue and seals the floor past it so pre-flush stragglers read as too late.
The clock discipline behind all of it
No cross-host clock math anywhere. The realized delay is queue depth × cmdT, a property of the
line, not of any clock pair. Sender timestamps are only ever compared against other sender timestamps
(offsets cancel in every delta); arrival and drain times come from the node's own monotonic clock,
injected as now into every decision method, which is also why the entire test suite runs
without a single sleep.
8 · Fail-closed catalog
Every path that discards queued motion, what seals afterwards, and how the stream comes back. All of them share buffer.rs:662 discard_all: prefill and the liveness gate restart, and the in-flight adapt window and arrival timing reference are discarded with the queue (§6 — a cut stream's drain-down is not jitter evidence, and a fresh epoch gets a fresh spacing baseline):
| Trigger | What happens | Floor seal | Recovery |
|---|---|---|---|
| E-stop arrival flush_for_estop :1213 |
Applied on the arrival thread (never queued); queue discarded. | In-space seq: past both queue and the e-stop itself. Cross-space (gRPC one-shot into a wire-seq deployment): newest queued seq only. | Operator reset; prefill restarts. |
| Post-live underrun underrun_tick :1139 |
Window expired with the queue empty after a live stream: flush + Stall(Underrun), node latches e-stop. |
Newest known seq. | reset_emergency + prepare_stream. |
| Hole ahead hole_ahead :1061 |
Queue span > 3 buffers of sender time: flush + Stall(HoleAhead) (policy armed) or warn + release (policy off). |
Newest known seq (when it stalls). | Same as above. |
| Lifecycle flush flush_lifecycle :1228 |
Stream re-engagement / enter idle: queue discarded, wire-seq epoch survives. | Newest known seq (pre-flush stragglers read TooLate). | Automatic; prefill restarts. |
| Epoch age-out epoch_expired :819 |
≥ 2 s arrival silence + stale-looking seq: dead epoch flushed, floor and timing reference forgotten. | None (new epoch starts fresh). | Automatic; the triggering entry is queued first. |
| Panic / apply error node.rs run_estop_guarded |
Any error or panic out of arrival or release work e-stops this arm (the action-path exception isolation). | Via the e-stop flush. | Operator reset. |
9 · Health & stats surface
Two read-outs, both computed from data the buffer already holds.
buffer.rs:417 PlayoutStats is the cumulative account, logged as one line when the
drain stops and again in the node shutdown totals (main.rs:341 "playout buffer
totals"). buffer.rs:486 PlayoutHealth is the live snapshot the telemetry
publisher ships operator-ward on every frame as extras["playout"]
(telemetry.rs:206 playout_report — report §6 rec 3).
Stats: where did every packet go?
Every arrival ends in exactly one counter:
released, late_dropped, duplicate_dropped,
overflow_dropped, aged_dropped, flushed, or malformed_ts.
Alongside: repaired (holes filled late), seq_holes (gaps at append — loss AND
reorder), lost_at_release (the monotone true-loss counter: seqs the release cursor
crossed unfilled), interpolated (synthetics released to conceal small holes),
hold_ticks (post-live ticks with nothing to send — each one is 10 ms of the arm holding),
sender_stalls, nonmonotonic_ts, epoch_resets,
underrun_episodes, stall_estops, hole_aheads. If a runaway
investigation ever asks "what did the buffer do that session", this line is the answer.
Health: what is the network doing right now?
depth vs target_depth and the adaptive
range as target_ms / min_ms / max_ms (equal when adaptation is off —
the bridge's "deepened" toast is exactly target_ms > min_ms), prefilling,
holding (an underrun hold is in progress), jitter_ms (RFC 3550 EWMA),
loss_pct (EWMA of the seq-gap fraction; true loss is seq_holes − repaired),
span_ms (the hole-ahead quantity: sender-time width of the queue),
release_age_ms (time since the last release), hold_ticks,
lost_at_release, interpolated, released (the degraded-share
denominator), plus the counters above.
last_stall: the diagnosis, not just the event
When a stall e-stop fires, the buffer samples the arrival-side
context at that exact tick and ships buffer.rs:369 StallReport::why()
as the sticky last_stall string, so the reason is still readable after the fact. The
discriminator is the time since the last accepted arrival: a large gap reads
"nothing has arrived for Nms — upstream stopped (network stall, leader freeze, or the bridge deadman
cut the stream)"; a small one reads "datagrams still arriving … but unusable — stale/aged network
debris". Loss %, unrepaired holes, jitter and the drop counters ride along in brackets.
Wire debug: the queue itself, on the wire
With playout_wire_debug: true the report also carries
entries — the queue ladder as (seq, ms behind the newest entry, arrived-as-repair)
triples, oldest first, capped at WIRE_DEBUG_MAX_ENTRIES = 64 so a mis-sized
buffer can never balloon the telemetry frame — and releases, the recent-releases strip as
(seq, interpolated) pairs (same cap). A hole is a missing seq between ladder neighbours;
the strip shows what the arm was actually fed: real, interpolated, or nothing.
Watching it live
tools/netem/playout_watch.py renders all of this as a live page on
:8091: it is a telemetry tee that binds the node's
--state-sink port and forwards every frame untouched to the bridge, parsing
extras["playout"] on the way past — queue ladder, released-mix strip (real / interpolated /
lost / held per sample), depth-vs-target, span, jitter and loss charts, and the last_stall
banner. Impairment sliders (tc netem) live on the companion panel at :8090.
The ↺ counters button zeroes the cumulative counters client-side (they are node-lifetime otherwise);
the baseline clears itself automatically when released regresses, i.e. when the node
restarted.
10 · Threads & locking
One mutex around the queue state, held only for the O(depth) decision and never across the apply path:
the drain pops under the lock, releases the lock, then applies. The arrival threads (UDP listener, gRPC
stream task) and the drain thread are the only writers. A poisoned lock is recovered with
into_inner() (the state is a queue of setpoints, always safe to read). The drain's pacing is the
servoj_loop shape:
DrainTick::Release carries the state by value so the 100 Hz hot path never allocates per
release. Every decision method takes the caller's monotonic now, so the whole state machine is
testable without threads or sleeps; the only place real time exists is this loop.
11 · Config reference
The buffer's knobs live in the robot YAML's communication block, the same block the driver's
sender tick reads (struct rust/fairino-robot/src/config.rs CommunicationConfig;
the playout keys are Rust-node-only — the Python driver never reads them). The node reads its copy
of the robot yaml, so on a remote deployment keep both copies in agreement.
| Key | Default | Meaning | Edge behavior |
|---|---|---|---|
publish_rate_hz |
50.0 | The action rate; cmdT = 1 / publish_rate_hz is the drain period, the depth divisor, and the spacing yardstick. 100 on every deployed remote-node yaml. | Non-positive falls back to the built-in ServoJ period. |
playout_buffer_ms |
0.0 | Base target delay; depth N = round(ms / cmdT). 200 ms at 100 Hz → N = 20. | 0 / negative / NaN → buffer fully off (passthrough, no thread). Absurd values clamp to MAX_TARGET_DEPTH = 200 entries (2 s) with a warning; the age cap is derived from the clamped depth. |
playout_adaptive_max_ms |
0.0 | Adaptive depth ceiling (§6): the runtime target floats in [base, this]. | NaN or ≤ base (incl. the default 0) → fixed depth. Clamped to 200 entries like the base. |
playout_underrun_estop_after_ms |
−1.0 | Post-live empty-queue window before flush + e-stop (§5). | Negative / NaN → off, hold forever. 0 → the first post-live empty tick escalates. |
playout_wire_debug |
false | Ship the queue ladder and releases strip in extras["playout"] (§9, diagnosis panels
only — it grows every frame by up to a few hundred bytes). |
Both surfaces capped at 64 entries. |
servoj_stale_factor |
2.0 | The driver's stale window only (stream.rs:97 stale_after_s: frames older than cmdT × factor are held, not sent). No longer any part of the buffer's age cap — that is derived purely from depth (§5 step 1). | — |
The link hardening around the buffer
The buffer assumes a lossy, reordering, stalling link; the bridge
(servo7/nodes/remote_follower_bridge.py) hardens that link from the sending
side. These live in the yaml's remote_node block or are derived:
| Mechanism | Value | What it does |
|---|---|---|
| v2 action bundles bundle_entries_within_budget |
1200-byte budget, ring depth 10 |
Every setpoint datagram carries the current action plus as many previous ones as fit under the
budget (ACTION_BUNDLE_BUDGET_BYTES = 1200) — the in-band-FEC pattern: a seq
is only truly lost when history+1 consecutive datagrams die, zero extra RTT. The budget keeps the frame
under the 1280-byte IPv6 MTU floor (header + robot-id overhead is why it is 1200, not 1280); real
~400-byte states degrade to fewer copies rather than oversize. The buffer's offer() absorbs the copies as Repaired/
Duplicate. Rollout is node-before-bridge: e-stops stay on v1 single frames so an
un-upgraded node still takes them, and they clear the history ring so nothing pre-stop rides along. |
| Leader deadman _deadman_blocks |
deadman_s, default 1.0(0 disables) |
While engaged, if the follower's telemetry has been silent past the window, stop firing setpoints into the void and alarm — filling a delayed queue makes the eventual replay worse. E-stops bypass the gate; releases with a log as soon as telemetry resumes. |
| Stale-tick gate _drop_stale_tick |
max(3 periods, 30 ms) (30 ms at 100 Hz) |
After a local scheduler stall the bus can hand the bridge a backlog burst; forwarding it replays
stale motion. A setpoint built more than the window ago is backlog, not traffic: dropped (never an
e-stop). The invariant documented at the function: every action producer re-stamps
timestamp with the orchestrator host's clock at build time (the foreign-clock stamp rides
in leader_timestamp, which this gate deliberately ignores) — a new producer that forwards
someone else's timestamp will be judged against the wrong clock here. The gate arms on
the first wall-clock-fresh setpoint, so synthetic-timestamp producers (tests, replays) are never
judged. |
| E-stop dual-pathing _send_estop_dual_path |
always on (UDP mode) | Every e-stop goes out as a UDP datagram AND a one-shot reliable gRPC stream
(wait_for_ready), so a 100 % UDP blackhole still delivers at gRPC speed. Episodes dedupe
re-deliveries; the node applies e-stops on the arrival thread and flushes the queue (§2). |
Deployed values
| Robot yaml (config/robots/) | rate | buffer | adaptive max | underrun e-stop | wire debug |
|---|---|---|---|---|---|
| fairino_fr20_remote_node.yaml | 100 Hz | 200 ms (N = 20) | 400 ms | 20 ms | — |
| fairino_fr3_remote_node.yaml | 100 Hz | 200 ms (N = 20) | 500 ms | 20 ms | — |
| fairino_fr20_remote_node_local.yaml | 100 Hz | 200 ms (N = 20) | 400 ms | 20 ms | — |
| fairino_fr20_simmachine_remote_node.yaml | 100 Hz | 200 ms (N = 20) | 400 ms | 20 ms | — |
| fairino_fr20_simmachine_remote_node_watch.yaml | 100 Hz | 200 ms (N = 20) | 400 ms | 20 ms | true |
| fairino_fr20_simmachine_remote_node_local.yaml | 100 Hz | 0 (off) | — | — | — |
| fairino_fr3_remote_node_local.yaml | 100 Hz | unset (off) | — | — | — |
On boot with the buffer enabled the drain logs one line to grep for: "Playout buffer on: depth 20 (200ms at cmdT 10.0ms), age cap 400ms" — the cap is depth + max(depth, 200 ms), so 200 ms of buffer carries 200 ms of margin.
12 · Test map
The suite at the bottom of buffer.rs (42 tests) pins every rule above, all clock-injected, no sleeps. If you change a policy, the failing test names the rule you changed:
| Rule | Tests |
|---|---|
| Passthrough & clamping | zero_and_garbage_config_mean_no_buffer_at_all · absurd_playout_ms_clamps_to_the_depth_cap · underrun_estop_window_parses_with_negative_meaning_off · adaptive_config_parses_with_zero_meaning_fixed |
| Prefill & pacing | prefill_holds_pops_until_target_depth_then_releases_one_per_tick · jittered_arrivals_equalize_to_one_release_per_tick_in_seq_order · underrun_to_empty_pauses_pops_until_prefill_rebuilds |
| Ordering & dedup | reorder_within_window_is_repaired_into_seq_order · too_late_seq_is_dropped_and_counted · duplicate_queued_seq_is_dropped_and_counted · seq_wrap_is_newer_and_synthetic_seqs_increase |
| Staleness & age cap | over_age_entry_is_discarded_never_released · overflow_above_high_water_drops_oldest_down_to_target · healthy_head_survives_held_ticks_and_wobble_without_aging_out (the all-yellow churn regression, §5 step 1) |
| Loss concealment | small_holes_are_concealed_by_lerp_one_tick_at_a_time · big_holes_keep_the_jump_and_count_as_true_loss |
| Adaptive depth | adaptive_target_grows_on_graze_deepens_by_slew_and_shrinks_when_calm · stall_flush_discards_the_adapt_windows_graze_evidence (the spurious "connection unstable" regression, §6) |
| Timing health | seq_gaps_count_holes_and_feed_the_loss_ewma · consecutive_seq_spacing_blowup_attributes_a_sender_stall · nonmonotonic_sender_ts_is_counted_but_still_queued · clean_pacing_keeps_the_jitter_ewma_near_zero · non_finite_sender_ts_is_rejected_never_queued · first_arrival_after_a_flush_is_not_spacing_judged (the "sender stall ~21622ms" regression, §6) |
| Stall policy | prefill_empty_never_stalls_even_with_the_policy_armed · post_live_underrun_stalls_after_the_window_and_flushes · zero_window_stalls_on_the_first_post_live_underrun_tick · recovery_within_the_window_resumes_without_a_stall · stall_during_re_prefill_flushes_the_partial_queue_and_seals · policy_off_holds_forever_but_still_counts_underrun_episodes |
| Stall diagnosis | stall_diagnosis_attributes_a_dead_upstream · stall_diagnosis_attributes_arriving_but_unusable_debris · hole_ahead_diagnosis_names_the_jump |
| Hole-ahead | hole_ahead_blocks_release_and_stalls_with_the_policy_armed · hole_ahead_without_the_policy_releases_as_today_and_counts |
| Flush & seal | estop_flush_with_in_space_seq_seals_past_the_estop · estop_flush_without_in_space_seq_seals_at_newest_queued_only · cross_space_estop_never_corrupts_a_wrapped_wire_floor · lifecycle_flush_clears_and_restarts_prefill_keeping_the_epoch · flushes_reset_the_liveness_gate · epoch_age_out_accepts_a_restarted_senders_low_seq_and_flushes |
| Telemetry surface | wire_debug_ships_the_queue_ladder_with_span_and_release_age |
Node-level integration (buffer attach, stall → driver e-stop latch, release path) is covered in node.rs's test module. On the Python side, tests/test_orchestrator_node.py TestControllerStationKeeping pins the station-keeping contract of §2 (a leader-only action still station-keeps the follower, a follower-targeted action never double-publishes, an idle controller still holds). And end-to-end, scripts/harness_runaway_replay_check.py replays a sine stream through real netem impairment profiles against the FR20 SimMachine (stale_replay / wifi_burst by default, clean as control) and gates CI on "no silent post-stop travel": the delayed-replay runaway staying dead is an asserted property, not a hope.