IO diagnostics v1

The dashboard, driven

A still from the middle of the video. The video is not in the public copy: its frames show the machine's name and device serials.

Five tree drawings, side by side, same frame

Same host, same records, and the same row selected in all five (at 18:11 the XXXXXXXX CAN bus, at 18:25 the CAN 15 driver). Only the tree renderer changes; the bottom-right switch in the app flips between them live and remembers the choice. All five follow your rules: chevrons only on hover, the status shape on the right, nothing green, greyed when not connected, and a row draws its own level. Pick one below; v1 is the default if you say nothing.

v1 · Quiet IDE tree default

on hover

on hover

VS Code explorer: hover chevrons in a left gutter, type icon + name, muted ids, a status shape at the far right only on problem rows; rollups are a quiet dot.

v2 · Notion toggle

on hover

on hover

Hover a row to swap its icon for the chevron; thin indent guides; a folded parent with problems shows a count pill ("2 ▲").

v3 · Problems first

on hover

on hover

Only the paths that hold a problem open by themselves; each level's healthy rows fold into one muted "N OK" line; a black box shows its own level with a quiet "child in error" hint.

v4 · Status rail

on hover

on hover

Type icon + name + id on the left; a right-edge rail that is invisible when OK and takes the problem shape and colour otherwise.

v5 · Columns tree

on hover

on hover

The tree is also the table: one nested name column, then quiet aligned PORT · ID · RATE · LAST SEEN · FIRMWARE columns, the status shape + message last, only when the row is not OK.

The whole dashboard

Live on this PC, with the real drivers

This PC, right now, through the real host with the real driver plugins in --passive mode (sysfs, by-id links, /proc holders, the rack inventory and firmware cache files; no adapter or camera opened):

• intel_rgbd diagnose_args --passive: usb:6-1 ok (ran in 0.16 s · 8 records); usb:6-2 ok (ran in 0.16 s · 8 records)
• moteus_servo diagnose_args --passive: usb:1-3 ok (ran in 0.28 s · 10 records)

? rack-pc  25 devices  — RealSense D405 (USB XXXXXXXXXXXX): passive run: the SDK was not asked (sysfs only)
  ? USB bus 1 · 480 Mb/s  2 devices  — CAN bus: passive run: the bus was not queried
      Port 1-3  12 Mb/s  — 1 device · 12 Mb/s  [below: unknown]
        fdcanusb XXXXXXXX  XXXXXXXX  — on port 1-3 · USB full-speed · 12 Mb/s  [below: unknown]
          /dev/ttyACM0    — present
        ? CAN bus    — passive run: the bus was not queried
          ? CAN 1    — passive run: the board was not queried  (knock-on)
          ? CAN 14    — passive run: the board was not queried  (knock-on)
          ? CAN 15    — passive run: the board was not queried  (knock-on)
      Port 1-6  12 Mb/s  — 1 device · 12 Mb/s
        ITE Device    — on port 1-6 · USB full-speed · 12 Mb/s · no driver diagnostics
  ? USB bus 6 · 10 Gb/s  2 devices  — RealSense D405 (USB XXXXXXXXXXXX): passive run: the SDK was not asked (sysfs only)
      Port 6-1  5 Gb/s  — 1 device · 5 Gb/s  [below: unknown]
      ? RealSense D405 (USB XXXXXXXXXXXX)  XXXXXXXXXXXX  — passive run: the SDK was not asked (sysfs only)
          /dev/video0 · capture    — present
          /dev/video1    — present
          /dev/video2 · capture    — present
          /dev/video3    — present
          /dev/video4 · capture    — present
          /dev/video5    — present
          colour stream    — configured 640x480 rgb8 at 30 Hz (intel_rgbd RgbdConfig default); diagnostics never opens a stream
          depth stream    — configured 848x480 z16 at 30 Hz (intel_rgbd RgbdConfig default); diagnostics never opens a stream
      Port 6-2  5 Gb/s  — 1 device · 5 Gb/s  [below: unknown]
      ? RealSense D405 (USB XXXXXXXXXXXX)  XXXXXXXXXXXX  — passive run: the SDK was not asked (sysfs only)
          /dev/video6 · capture    — present
          /dev/video7    — present
          /dev/video8 · capture    — present
          /dev/video9    — present
          /dev/video10 · capture    — present
          /dev/video11    — present
          colour stream    — configured 640x480 rgb8 at 30 Hz (intel_rgbd RgbdConfig default); diagnostics never opens a stream
          depth stream    — configured 848x480 z16 at 30 Hz (intel_rgbd RgbdConfig default); diagnostics never opens a stream
  … (USB bus 7, PCI, network, tty: see the next section)

Why "?": in passive mode the drivers say unknown with code PassiveOnly ("the bus was not queried"), which the contract ranks above warn, so the chip reads IO 3 ⚠ with nothing wrong. That is decision 2 below.

Discovery beyond USB

The OS skeleton is passive and always there: 25 devices on this PC at capture time, chip IO ✓, with no driver plugin running. Every source is a file read (sysfs, by-id links, the udev database, the kernel log); nothing is opened.

PCIe link health, this PC

Width below the card's maximum → WARN PcieLinkNarrow (speed is ignored: a GPU drops speed at idle). AER correctable → WARN, nonfatal/fatal → ERROR. A likely frame grabber with no driver bound → WARN NoKernelDriver. On this PC every link is clean, so the chip stays quiet: no false positives.

Hardware this PC does not have: the fake rack machine

Lane B's sysfs spec (tests/data/sysfs-rack-fixture.json) served through the same io_facts sources and the real dashboard: a PCIe frame grabber with no kernel driver, a capture card that trained x1 of x4, two I210 NICs carrying EtherCAT segments (detected, not scanned), and a USB-serial adapter.

▲ PCI root bus 0000:00  8 devices  — Likely frame grabber · Coreco Inc: likely frame grabber / capture card with no kernel driver bound: it cannot capture until its vendor driver loads
  ▲ Likely frame grabber · Coreco Inc    — likely frame grabber / capture card with no kernel driver bound: it cannot capture until its vendor driver loads
    I210 Gigabit Network Connection    — Network controller · kernel driver igb · no driver diagnostics
      enp6s0 · EtherCAT segment (not scanned)    — 100 Mb/s · link up
    I210 Gigabit Network Connection    — Network controller · kernel driver igb · no driver diagnostics
      enp7s0 · EtherCAT segment (not scanned)    — 100 Mb/s · link up
  ▲ Pro Capture Endpoint    — trained narrower than the card can: x1 of x4 · 2.5 GT/s PCIe (max 2.5 GT/s PCIe) (reseat or another slot)
      /dev/video20 · capture    — present
    USB controller    — Serial bus controller · kernel driver xhci_hcd · no driver diagnostics
    USB controller    — Serial bus controller · kernel driver xhci_hcd · no driver diagnostics
    ACP/ACP3X/ACP6x Audio Coprocessor    — Multimedia controller · kernel driver snd_pci_acp6x · no driver diagnostics
    USB controller    — Serial bus controller · kernel driver xhci_hcd · no driver diagnostics

The black-box rule, on a real tree

Zach's rule (2026-10-22): a black box reports for itself. A row draws its own entity's self-report and the host never overrides it. What is worse further down is an advisory hint. A row with nothing to report itself (the host, a USB bus) shows the worst thing below it. And when a parent has failed, the failures under it collapse into that one cause. Below is the real host's output for recorded frame F1, row by row along the path to CAN 15's fault, from io-diagnostics once --json --fixture F1:

  Port 1-3  12 Mb/s  — 1 device · 12 Mb/s  [below: error]
    fdcanusb XXXXXXXX  XXXXXXXX  — on port 1-3 · USB full-speed · 12 Mb/s  [below: error]
      /dev/ttyACM0    — present
      XXXXXXXX CAN bus    — error-active · err 0 · warn 0 · 3 of 3 answer  [below: error]
        CAN 1  XXXXXXXXXXXXXXXX  — answers · 36.5 V · stopped
          CAN 1 driver    — 36.5 V · fault 0
          CAN 1 motor    — 10:1 (GIM4305-10; camera-measured 0.104 output/rotor) · output sign 1 · tag 0
        CAN 14  XXXXXXXXXXXXXXXX  — answers · 36.5 V · stopped
          CAN 14 driver    — 36.5 V · fault 0
          CAN 14 motor    — 10:1 (GIM4305-10) · output sign -1 · tag 1, 21
        CAN 15  XXXXXXXXXXXXXXXX  — answers · 36.5 V · stopped  [below: error]
        ✖ CAN 15 driver    — fault 33 (motor driver) on bridge enable · last trip 13:34:07
          CAN 15 motor    — 10:1 (GIM4305-10) · output sign 1 · tag 2

The plugin contract: what a manufacturer writes

A manufacturer writes one executable and one TOML file. The executable answers three verbs with JSON lines on stdout; the host never imports it, so its dependencies, language and crashes stay its own (it runs in the driver's own uv environment as a subprocess). The smallest real example, examples/acme_gripper/, run through the real host by tests/test_integration.py (which also fails if the README drifts from these files):

"""The smallest IO diagnostics plugin: a made-up USB gripper (1234:5678) with one part, as JSON lines.

    <python> acme_gripper.py describe
    <python> acme_gripper.py diagnose --device usb:1-4

A real plugin reads its device here (inside its own driver's environment); this one only shows the shapes.
"""

import json
import os
import sys
import time

SCHEMA = "diagnostics-record/0"


def emit(**record):
    print(json.dumps({"schema": SCHEMA, **record}))


if sys.argv[1] == "describe":  # what it claims, and the buttons it registers (none here)
    print(json.dumps({"name": "acme_gripper", "version": "1.0", "schema": SCHEMA,
                      "matches": ["usb:1234:5678"], "actions": []}))
elif sys.argv[1] == "diagnose":  # one run for one device; paths are relative to the device's own row ("")
    device = json.loads(os.environ.get("IO_DIAGNOSTICS_DEVICE", "{}"))  # the host's skeleton dict for it
    now = time.time()
    emit(record="entity", path="", name=f"Acme gripper {device.get('serial') or ''}".strip(), kind="device",
         identity={"manufacturer": "Acme", "model": "Gripper G1", "sw_version": "2.3.0"})
    emit(record="entity", path="jaw", name="jaw", kind="part", part_of="")
    emit(record="status", path="", name="gripper: comms", level="ok", message="answers in 2 ms",
         stamp=now, expected_period_s=30)
    emit(record="status", path="jaw", name="jaw: temperature", level="warn", code="JawHot",
         message="jaw motor 71 °C (warn above 70 °C)", stamp=now, expected_period_s=30,
         measurements=[{"key": "temperature", "value": 71.0, "unit": "Cel", "warn_high": 70.0}])

name = "acme_gripper"
command = ["{python}", "{here}/acme_gripper.py"]
timeout_s = 10
period_s = 30
matches = ["usb:1234:5678"]

{"record": "entity", "path": "", "name": "Acme gripper G1-0042", "kind": "device", "identity": {"manufacturer": "Acme", "model": "Gripper G1", "sw_version": "2.3.0"}}
{"record": "entity", "path": "jaw", "name": "jaw", "kind": "part", "part_of": ""}
{"record": "status", "path": "", "name": "gripper: comms", "level": "ok", "message": "answers in 2 ms", "stamp": 1792685757.684107, "expected_period_s": 30}
{"record": "status", "path": "jaw", "name": "jaw: temperature", "level": "warn", "code": "JawHot", "message": "jaw motor 71 °C (warn above 70 °C)", "stamp": 1792685757.684107, "expected_period_s": 30, "measurements": [{"key": "temperature", "value": 71.0, "unit": "Cel", "warn_high": 70.0}]}

Matches: usb:VID:PID · pci:VENDOR:DEVICE · pci-class:… · by-id:<glob> · tty:<glob> · v4l:<glob> · netdev:<glob>, in the TOML or returned by describe. TOML keys: name, command, cwd, timeout_s, period_s, matches, enabled, env, diagnose_args, description. What the host does for you: the device row (port, VID:PID, serial, link speed, plug history) already exists; your rows hang under it; a check not re-sent within max(3 × expected_period_s, 5 s) turns STALE on the host's clock; a crash, a hang or a malformed line becomes a check on the device row, never a dead dashboard.

The two real driver plugins, as they describe themselves

Captured for this report through passive_guard.py (which lets only describe and diagnose --passive through). These are the bam_ws files on branch claude/traj-tracking-sim-to-real-a3da60.

{
 "name": "moteus_servo",
 "version": "0.1.0",
 "schema": "diagnostics-record/0",
 "description": "fdcanusb CAN-FD adapters and the moteus boards on their bus: ids, identity, firmware vs the latest release, read-only config, can status, reply loss",
 "matches": [
  "usb:0483:5740",
  "by-id:*fdcanusb*"
 ],
 "commands": [
  "describe",
  "diagnose",
  "act"
 ],
 "path_root": "--path, else $IO_DIAGNOSTICS_DEVICE_PATH, else fdcanusb-<serial>; the bus is <root>/can (part_of the adapter), each board <root>/can/<id> (via the bus)",
 "global_options": [
  "--inventory DIR",
  "--firmware-cache DIR",
  "--bus-lock FILE",
  "--expect-ids LIST",
  "--expected-period S",
  "--timeout S"
 ],
 "actions": [
  {
   "schema": "diagnostics-record/0",
   "record": "action",
   "name": "run_comms_check",
   "title": "Run comms check",
   "description": "Addressed scan of every id, reply loss over a few queries, read-only config and identity, and the adapter's can status. Query frames and console reads only: no board changes mode.",
   "danger": "query",
   "scope": "device",
   "confirm": "Query every id on {name} and read bus health? No board changes mode.",
   "preconditions": [
    [
     "port_free",
     "No other process has the adapter open (tview, moteus_tool, a peer)"
    ],
    [
     "bus_lock_free",
     "The rack bus lock is free (no experiment running)"
    ]
   ],
   "kind": "touches"
  },
  {
   "schema": "diagnostics-record/0",
   "record": "action",
   "name": "clear_faults",
   "title": "Clear faults",
   "description": "Send stop (mode 0) to one board; moteus clears a latched fault on stop. The motor stays unpowered.",
   "danger": "changes",
   "scope": "node",
   "confirm": "Send stop to CAN {id} on {name}? A latched fault clears; the motor stays unpowered.",
   "preconditions": [
    [
     "port_free",
     "No other process has the adapter open (tview, moteus_tool, a peer)"
    ],
    [
     "bus_lock_free",
     "The rack bus lock is free (no experiment running)"
    ],
    [
     "board_answers",
     "CAN {id} answers a query on this bus"
    ]
   ]
  },
  {
   "schema": "diagnostics-record/0",
   "record": "action",
   "name": "reboot_board",
   "title": "Reboot board",
   "description": "Send `d reset`: the board restarts and is back on the bus in about a second. Any session using it fails.",
   "danger": "interrupts",
   "scope": "node",
   "confirm": "Reboot CAN {id} on {name}? It drops off the bus for about a second; any session using it fails.",
   "preconditions": [
    [
     "port_free",
     "No other process has the adapter open (tview, moteus_tool, a peer)"
    ],
    [
     "bus_lock_free",
     "The rack bus lock is free (no experiment running)"
    ],
    [
     "board_answers",
     "CAN {id} answers a query on this bus"
    ]
   ]
  }
 ]
}

{
 "name": "intel_rgbd",
 "title": "RealSense camera",
 "version": "0.1.0",
 "schema": "diagnostics-record/0",
 "matches": [
  {
   "bus": "usb",
   "vid": "8086",
   "pid": "0b*"
  }
 ],
 "actions": [
  {
   "schema": "diagnostics-record/0",
   "record": "action",
   "name": "check_firmware",
   "title": "Check firmware",
   "description": "Read the installed and the SDK's recommended firmware through camera_info (no stream, no option read), while no other process holds a RealSense and the bus lock is free.",
   "danger": "query",
   "scope": "device",
   "confirm": "",
   "preconditions": [
    [
     "camera_free",
     "No other process holds a RealSense and realsense-viewer is closed"
    ],
    [
     "bus_lock_free",
     "No rack experiment holds the bus lock"
    ]
   ],
   "kind": "touches"
  },
  {
   "schema": "diagnostics-record/0",
   "record": "action",
   "name": "hardware_reset",
   "title": "Hardware reset…",
   "description": "RealSenseCamera.hardware_reset(): the camera drops off USB for about 3 s, re-enumerates, and every option returns to its firmware default (measured on the D435: exposure 250 -> 166, auto-exposure back on). Anything streaming from it must start again.",
   "danger": "interrupts",
   "scope": "device",
   "confirm": "Hardware-reset {name} ({serial})? It drops off USB for about 3 s and every option returns to its firmware default.",
   "preconditions": [
    [
     "camera_free",
     "No other process holds a RealSense and realsense-viewer is closed"
    ],
    [
     "bus_lock_free",
     "No rack experiment holds the bus lock"
    ]
   ]
  }
 ],
 "icon": "camera",
 "sdk": "pyrealsense2 2.58.3.10794",
 "period_s": 60.0,
 "device_ids": "RealSense serial, USB serial, USB port (6-2), usb:8086:<pid>:<USB serial>, /dev/videoN, a /dev/v4l/by-id link, a sysfs path or a host tree path (host/usb-6-2/...)",
 "confirm_token": "hardware_reset takes --confirm <the camera's RealSense serial>: identity.serial_number, the {serial} in its confirm text"
}

Both are declared in plugins.d/ with enabled = false (their default diagnose talks to the hardware, behind a port-holder check and the rack bus lock); --enable NAME turns one on for a single run.

Interop: our records in stock viewers

Decision 1 of the build plan: own the dashboard, and stay backwards compatible through one pure adapter (contract.record.to_diagnostic_status) that strips our added fields. A stock viewer then shows our records with zero code of ours inside it.

A still from the middle of the video. The video is not in the public copy: its frames show the machine's name and device serials.

ROS 2 Humble (docker, no network): 5/5 checks

header:
  stamp:
    sec: 1792677600
    nanosec: 0
  frame_id: ''
status:
- level: "\0"
  name: 'fdcanusb XXXXXXXX: USB'
  message: on port 1-3 as /dev/ttyACM0
  hardware_id: usb:0483:5740:XXXXXXXX
  values:
  - key: port
    value: 1-3
  - key: tty
    value: /dev/ttyACM0
  - key: by-id
    value: /dev/serial/by-id/usb-mjbots_fdcanusb_272813EF-if00
  - key: link_speed
    value: 12 Mbit/s
- level: "\0"
  name: 'CAN bus: state'
  message: error-active, no bus errors
  hardware_id: usb:0483:5740:XXXXXXXX
  values:
  - key: state
    value: error-active
  - key: tx_error_count
    value: '0'
  - key: rx_error_count
    value: '0'
- level: "\0"
  name: 'CAN 1: comms'
  message: answers · 36.5 V · stopped
  hardware_id: moteus:32bb145a
  values:
  - key: serial
    value: XXXXXXXXXXXXXXXX
  - key: firmware
    value: 1.1.2 (657b5c5)
  - key: station

Plus tests/test_integration.py: the real host serves F1 → lane C's --from-host source → stock array with our fields stripped → CDR and back. Not proven: diagnostic_aggregator (not in the image).

Tests and evidence

The pytest counts above were re-run while this report was built, rather than copied from a lane's message. The node counts come from their own result files (tree-variants-report.json, shots.json, hero.json, verdict.json).

Decisions: what needs you

Each one has a recommendation and the reversible default that happens if you say nothing. Your choices are kept in this browser only. Copy puts them on the clipboard as text you can paste into the chat.

Left, and deliberately not done

Done and proved: everything in the green column above, plus the integration fixes. intel_rgbd now accepts the host's own skeleton id usb:6-1; before the fix, every host run ended in DeviceNotFound. once now judges live records at the end of the runs, not at the start. And lane B's io_facts/usb.py fixes are in: a 1.5 Mb/s device no longer breaks the USB source, and hubs no longer list the camera's video nodes. The gaps closed during the completeness check: PCIe link health (tree.pci_statuses), --machine SPEC.json for hardware this PC lacks, --enable NAME and diagnose_args, one command through a hatchling build backend, an end-to-end host → bridge → CDR test, and a README whose manufacturer example is tested against the files.

Not done, all non-blocking:

• Cosmetic: --machine mode's header says 'Live · sysfs-rack-fixture (fake machine)' while the clock mode is still 'live'; the table's Status column truncates 'UNKNOWN'; moteus passive names its bus just 'CAN bus' (no adapter serial).
• Lane B: move the sysfs helpers from io_facts/skeleton.py to io_facts/sysfs.py.
• Lane C: stock.py imports the contract's private _stale_message (needs an additive contract change to make it public; the contract is frozen at c9784d0); foxglove-sdk accepts only the foxglove.sdk.v1 subprotocol; diagnostic_aggregator is not proven (not in the image).
• Lane D: status unresolved until the live run: bounded replies on a real fdcanusb (problem 3). Sibling moteus tools still carry moteus 1.0.0 traps (can_bus_triage.scan_bus batches 8 ids; triage() and list_fdcanusb.scan() open several Fdcanusb in one event loop); lane D's diff is in its handoff.
• Lane F: the per-minute timeline strips were dropped (no API endpoint for them).
• bam_ws: two bam_rack_twin lint tests fail on already-committed code (configcheck.py:148/151, inventory.py:36); the intel_rgbd README/DATASHEET do not mention the plugin yet.
• After the bam_ws merge: switch the worktree paths to ~/bam_ws/src/core/mdp/agent/… in plugins.d/*.toml, tests/fixtures/plugins-live-passive/*.toml, the README commands and tools/safe_pytest_intel_rgbd.py's docstring.

Deliberately not done: no git writes (the orchestrator commits after review). No active hardware run (hard rule: yours). Actions are never executed in v1, by design. The tray was never launched on your desktop. EtherCAT segments are detected but not scanned (as the build plan says). The record contract was not edited, because it is frozen at c9784d0.

Run it

uv run --project ~/io-diagnostics io-diagnostics serve --fixture F2 --offset 330

uv run --project ~/io-diagnostics io-diagnostics serve

uv run --project ~/io-diagnostics io-diagnostics serve --plugins ~/io-diagnostics/tests/fixtures/plugins-live-passive

uv run --project ~/io-diagnostics io-diagnostics serve --machine ~/io-diagnostics/tests/data/sysfs-rack-fixture.json

uv run --project ~/io-diagnostics io-diagnostics tray

uv run --project ~/io-diagnostics io-diagnostics once --json --enable moteus_servo --enable intel_rgbd

uv run --project ~/io-diagnostics --with pillow python ~/io-diagnostics/docs/build_v1_report.py

Built by docs/build_v1_report.py on 2026-10-22T09:21:22-07:00. Host captures were taken 2026-10-22T09:15:56-07:00 on rack-pc through io-diagnostics once --json, and F1 was judged at its own frame clock (2026-10-21T18:11:50-07:00). The images are WebP crops, cut from the proof shots named in each caption.