Designing the palletizing sim

1. What we want from a sim

Before picking tools we need to be clear on why we'd run a simulator at all. Three reasons matter for atlas — and they're not the same shape of work.

2. The two "sims" — separation of concerns

The word "sim" has been doing two unrelated jobs. We give them different names and never let them touch.

The gate inside RobotNode

class RobotNode:
    def __init__(self, robot: Robot, sim_mode: bool = False):
        self.robot = robot                  # real Fairino driver — instantiated either way
        self.sim_mode = sim_mode            # the flag
        self.sim_engine = KinematicEngine(robot.urdf)  # always present, used when flag is on

    def connect(self) -> None:
        if not self.sim_mode:
            self.robot.connect()            # only touch hardware when in hardware mode

    def handle_action(self, action: RobotState) -> None:
        if self.sim_mode:
            new_state = self.sim_engine.step(action, dt)
            system_state.update_state(self.robot_id, new_state)
        else:
            self.robot.set_state(action)
            system_state.update_state(self.robot_id, self.robot.get_state())

    def set_sim_mode(self, on: bool) -> None:
        """Live toggle. WS message from frontend can call this."""
        if on and not self.sim_mode:
            self.robot.disconnect()         # release hardware
        elif not on and self.sim_mode:
            self.robot.connect()            # reclaim hardware
        self.sim_mode = on

The driver class is always instantiated, but connect() is skipped while the flag is on. This is the live-flip-friendly design: poke a trajectory in sim_mode, flip via a WS message, re-run the same trajectory on hardware without restarting RobotNode.

3. Viewer / engine options

This is where the most confusion happens because the candidates live at different layers. Some are physics engines, some are renderers, some are libraries you build a renderer with. The table is the quickest way to see it:

viser recommended

URDFButtonsGizmosLabelsMIT

three.js + Python + a GUI panel framework. @button.on_click decorators, transform gizmos for waypoint authoring, scene labels and mesh recolor — everything we listed as mandatory, out of the box.

Maintenance: active (v1.0.26, Mar 2026, Nerfstudio team). Forkable if it goes stale (~3k LOC Python).

Risk: small team. We pin the version.

MeshCat stale

URDF (via wrappers)No buttonsMIT

Same three.js renderer as viser, with a Python-WS bridge. No GUI controls layer — for buttons you'd add Streamlit/Gradio alongside, which means two web UIs to iframe.

Maintenance: no commits in 12+ months. Was the standard for Drake/Pinocchio users in 2020–2023.

Why we'd skip: worse than viser on every axis we care about today.

Rerun parallel use

URDF (built-in)Weak callbacksApache-2.0

Excellent telemetry/time-series viewer. URDF loading is first-class (improved again in 0.28/0.29). But custom buttons that call back to Python are not a first-class primitive — fine for "view a rollout afterwards", weak for "approve this now".

Role: not the interactive viewer. Use it alongside viser later for trajectory replay and joint-trace scrubbing.

three.js + urdf-loaders DIY

URDFDIY everything elseApache-2.0

Battle-tested URDF parser (gkjohnson, used at JPL). You'd build the websocket protocol, button bridge, waypoint gizmos, and label layer yourself. Roughly 2 weeks of work that viser hands you for free.

When this wins: we hit a wall with viser and need full control. Not v1.

MuJoCo engine, not viewer

PhysicsDesktop viewer onlyApache-2.0

Best-in-class rigid-body physics with contact. Native OpenGL viewer — not iframeable. MuJoCo-WASM compiles physics to the browser but ships no GUI.

Role: a physics layer for the Robot Controller's internal engine when we want contact-rich behaviour (grasp stability, slip, force control). v1 doesn't need it; v3 does.

viser GUI — what we get for free

Buttons, sliders, dropdowns, file uploads, foldable panels, transform gizmos, 3D click events, scene labels with color overlays. All controlled from Python with decorators. This is the specific reason viser wins for the approve-gate flow.

No JavaScript written on our side.

4. The four leaders, one follower

The follower is always one robot. What changes is who is driving it. Atlas already has Teleop, AI and Emergency Stop. We're adding a fourth leader: Robot Controller.

The Sim Viewer is always rendering the follower's live state. The mode only changes whether the viewer's buttons are active as a command source.

5. Architecture

Everything plugs through SystemState. The Sim Viewer is a sibling Python package living inside atlas's process — it registers a callback on SystemState directly. No new transport layer, no inter-process serialization.

┌─ atlas process ──────────────────────────────────────────────────┐
│                                                                  │
│   ┌────────────────────────────────────────────────┐             │
│   │ RobotNode (one per robot)                      │             │
│   │                                                │             │
│   │   action ──►  sim_mode flag ─┬─► [True]  ──►   │             │
│   │                              │   sim_engine.step  ──┐        │
│   │                              │                      │        │
│   │                              └─► [False] ──►        │        │
│   │                                  robot.set_state ──►│        │
│   │                                  robot.get_state ──►│        │
│   │                                                     │        │
│   │   Sim engine ticks at 100 Hz even when no command   │        │
│   │   (so SystemState stays live).                      │        │
│   └─────────────────────────────────────────────────────┼────────┘
│                                                         │
│                          ──── state ────────────────────┴────────►
│                                                    SystemState
│                                                  ["fairino"]
│                                                        │
│                                                        │ callbacks
│                                ┌───────────────────────┤
│                                ▼                       ▼
│                       ┌─────────────────┐    ┌──────────────────┐
│                       │ Orchestrator    │    │ sim_viewer/      │
│                       │ + WS :8765      │ ◄─►│ (in-process      │
│                       │ + CommandRouter │    │  Python package) │
│                       │ + Mode gate     │    │                  │
│           commands ──►└─────────────────┘    │  viser server    │
│           mode gate           ▲              │  on :8080  ──────┼────►
│           ▲                   │ SET_SIM_MODE │  URDF render     │
│           │                   │              │  callbacks ←─────┤
│  ┌────────┼────────┬──────────┐              │   SystemState    │
│  │        │        │          │              │  buttons → call  │
│ Teleop  AI   Robot         Emergency         │   atlas Python   │
│              Controller    Stop              │   directly       │
│              Node (v2)                       └──────────────────┘
│                                                       │
│  RobotControllerNode (v2) owns:                       │
│    • SimCore + same EngineBackend used by RobotNode   │
│    • Validators (torque, collision, …)                │
│    • Proposal/Approval state machine                  │
│    • Calls sim_viewer.show_proposal(…) directly       │
│                                                       │
└───────────────────────────────────────────────────────┼─────────
                                                        │
        ┌───────────── atlas-frontend :3000 ────────────┴─────────┐
        │ AtlasSimPanel.jsx  →  <iframe src="localhost:8080"/>    │
        │ Plus sim-mode toggle (sends SET_SIM_MODE over :8765)    │
        └─────────────────────────────────────────────────────────┘

The Sim Viewer is a sibling of servo7/ inside the same atlas Python process. Two consequences: state moves as a Python callback (no serialization, no WS hop), and the Approve/Reject buttons inside viser can invoke atlas functions directly (no new WS message types like APPROVE/REJECT ever ship).

The two flows, side by side

Why all heavy math lives in atlas, not the Sim Viewer

• The Sim Engine already lives inside RobotNode for the sim_mode path; SimCore in RobotControllerNode uses the same EngineBackend Protocol. One engine, two consumers.
• Validators need access to the URDF, mass model, and SystemState — all already in atlas.
• Keeps the Sim Viewer container thin: just viser + URDF + a WS client. Trivially restartable; never carries state of its own.
• Swap the viewer (viser → something else, one day) without touching control logic.

The channels at a glance

SimCore layering — so we can swap engines later

class EngineBackend(Protocol):
    def step(self, joint_cmd: list[float], dt: float) -> EngineState: ...
    def reset(self, q: list[float]) -> None: ...
    def add_body(self, urdf_or_mesh: str, pose: Pose) -> BodyHandle: ...

# v1 / v2 — kinematics + collision, no physics
class KinematicEngine(EngineBackend):
    # pinocchio FK + hpp-fcl collision

# v3 — full physics, no atlas changes needed
class MujocoEngine(EngineBackend):
    # contact, gripper success, slip

Rule: no engine types cross the SimCore boundary. No mjData in the rest of atlas, no pinocchio.Model in the Sim Viewer. EngineState is pure data: joint positions, EE pose, optional contact list. Swap engines = one config line; nothing else moves.

6. Phasing

7. The "no jiggle" problem

In MuJoCo, when something was wrong with the physics — load too heavy, integrator unstable, contact unsolvable — the arm would jiggle and explode. That was an in-band signal: the engine was telling us "this won't work".

Kinematics-only doesn't jiggle. Set joint angles → they sit there. A 50 kg payload looks identical to a 5 kg one. Silence is the new failure mode.

The fix isn't to add physics — it's to add explicit analytical checks inside RobotControllerNode that turn implicit jiggle into named failures. Most of what MuJoCo was telling us can be computed directly from the URDF + a candidate trajectory:

Each validator returns CheckResult(pass, severity, message, waypoint_idx). RobotControllerNode attaches them to the TRAJECTORY_PROPOSAL message; the Sim Viewer renders them as a red row in the sidebar with the offending waypoint highlighted in 3D. Arguably better than MuJoCo's wobble: wobble told us something was wrong; this tells us what.

Dependency: the torque validator only works if the URDF has real mass/inertia data. Fairino's URDF quality is a v2 acceptance criterion — if their values are placeholders we backfill from CAD or fall back to the datasheet payload-vs-reach envelope.

8. Open questions for the conversation

9. Deployment — in-process, no Docker

The Sim Viewer is a sibling Python package inside atlas, next to servo7/. It lives in the same process as the orchestrator and RobotNode. The existing devcontainer doesn't change — no compose migration, no second container, no inter-process WS.

The viewer still binds viser to localhost:8080, and the atlas frontend iframes that URL. The browser↔Python WS that viser uses internally is unchanged — that's still how button clicks reach the Python event handlers. The only thing that's gone is a second WS hop between atlas and "the viewer container".

Directory layout

atlas/
├── .devcontainer/            # unchanged
│   ├── devcontainer.json
│   ├── Dockerfile.dev
│   └── Dockerfile.base
├── pyproject.toml            # MODIFIED — adds viser, yourdfpy
├── sim_viewer/               # NEW — sibling Python package
│   ├── __init__.py
│   ├── server.py             # starts viser; called from atlas main
│   ├── scene.py              # URDF load, joint update, button bindings
│   └── urdf/
│       └── fairino30_v6.urdf # vendored from FAIR-INNOVATION/frcobot_ros2
└── servo7/
    ├── nodes/
    │   └── robot_node.py     # MODIFIED — sim_mode flag + SimEngine + 100Hz timer
    └── sim/
        └── kinematic_engine.py  # NEW — EngineBackend (pinocchio FK + hpp-fcl)

The viewer is a normal Python package, importable from atlas's entrypoint:

# atlas main.py (sketch)
from sim_viewer import start_viewer
from servo7.system import system_state

viewer = start_viewer(system_state, robot_id="fairino", port=8080)
# atlas continues — orchestrator, robot_node, etc.

10. Smallest concrete next step — ship v0