RobotDx › Reports › Diagnostics rollups: five proposals. Published copy, redacted. Taken out: 10 home paths, 1 hostnames, 1 private sections cut, 21 serials, 3 vault paths. Evidence for D05, D15, D16, D17. Written for Zach at the time, so it says “you”.

Diagnostics rollups: five proposals

How the black-box self-reports and the higher-level diagnostics you specify fit together, and where the Computer's own checks go · 2026-10-23 · robotdiagnostics / io-diagnostics · research and design only, no code changed

Build P3: the expected setup becomes the rig's acceptance spec. First land one contract fix, so a composite's problem is explained by the members it names.

What you already decided

This section quoted Zach's private notes from 2025 and 2026 and is left out of the public copy. Its point survives in the next sections: a diagnostic is the interface's contract test, run once at the bench to accept a part and kept running in deployment to prove the part still meets it.

Three traps every proposal has to close

The tree has one parent edge per entity (part_of or via). summarize() collapses a problem into the nearest failed ancestor on that edge.16 Three probes ran against the real contract code, with no host process and no hardware:

Explanation direction Along via, a failed adapter explains the boards below it. From a composite to its members, a member's failure explains the composite. via: connectivity, explains DOWN dead adapter = one cause; its silent boards are knock-ons host · rack-pc fdcanusb XXXXXXXX CAN bus CAN 1 CAN 14 CAN 15 (Nagios UNREACHABLE) boards read "knock-on of fdcanusb XXXXXXXX" composite → members: explains UP failed member = the cause; the composite's verdict follows 1-DOF test rackWARN · missing: CAN 15 CAN 1 CAN 14 CAN 15 D405 ·4411 caused_by the chip counts and names CAN 15 once; the rack row still shows its own WARN (black-box rule kept)
The contract has one edge kind and one direction of explanation. Composites need the other direction.

Trap 1: the Computer's checks written on host swallow the rack

$ python3 -B knockon_probe.py
--- Computer checks ON host
  chip counts (independent causes): {'error': 1}
  knock_on: [('host/usb1/1-3/can/15', 'host')]
  CAN 15 row: level=error · counted as its own cause: False
--- Computer checks on sibling host/computer
  chip counts (independent causes): {'error': 2}
  knock_on: []
  CAN 15 row: level=error · counted as its own cause: True

A full disk becomes "the cause", and CAN 15's fault 33 becomes its symptom. The computer lane in flight already guards against this: it evaluates the host's own records as a leaf host/:self beside its children. The comment there says "a hot CPU or a crashed computer plugin explains nothing about a dead CAN bus".19

Trap 2: a rig row beside the tree counts one missing board twice

$ python3 -B rig_doublecount_probe.py
independent causes counted by the chip: {'warn': 1, 'error': 1}
knock_on: []

Two records describe one fact:

The chip reads "IO 2 ⚠".

Trap 3: a composite that reports ERROR hides its failed member

$ python3 -B spider_probe.py
spider self-reports WARN  -> chip names: cell/spider/leg3 · counts {'warn': 1, 'error': 1} · knock_on []
spider self-reports ERROR -> chip names: cell/spider · counts {'error': 1} · knock_on [('cell/spider/leg3', 'cell/spider')]

When the spider reports WARN, its record and the leg's are counted as two causes. When it reports ERROR, leg 3 (fault 33), which the spider's own message names, is filed as the spider's symptom.

The shared fix (record v0.2, additive), which every proposal below assumes:

Criteria

Weights come from your notes above. Your 2025 scope rule decides which checks get a second life. A check on code that, once fixed, stays fixed is acceptance only. A check on anything that touches the world (wear, wiring, config, firmware, temperature, drift) stays on as a monitor.1

#CriterionWeightSource
1Black boxes keep their own word. A composite's verdict is the composite's, and the host never overrides a leaf.20Oct 22 rule; Blackbox Design
2One check, two lives. The same check is the bench acceptance test and the deployment monitor (twin, bench, site), with per-context tolerances.20Changes; June 28; bam_docs; Next 2; Oct 16
3Honest status. "Not run yet" and "can't tell" never read green, and one fact is one cause.15record v0 lessons 3 and 5; Vibe Tracks
4Declared, minimal, stable. Data, not scattered code, that survives swapping a board, planner or arm.15DP Diagnostics commitments 1 and 3
5Off the shelf. Stock schemas and tools; the stock ROS view keeps working.10availability before originality
6Calm first screen. One answer up top, then computer → entity → diagnostic.10Oct 23
7A tracer bullet this week, on top of the expected setup in flight.10speed; the 29 unrun tests

One scenario, six answers

The scenario comes from today's file17: CAN 15's moteus-c1 is out for replacement, the last comms sweep ran 26 h ago, and / has 8.5 % free. Below is the top of the page under each option.

Today
1-DOF test rack · 2/2 adapters · 6/7 boards · 2/2 cameraschip: IO 2 ⚠
The rig's WARN and CAN 15's ERROR are counted as two causes for one fact (trap 2). Nothing says whether the rack was ever accepted.
P1 wrappers
1-DOF test rack: actuators · 6/7 boards: CAN 15 out; single-joint tests on the other six still runIO 1 ⚠ · CAN 15
A good sentence, but its rule lives in someone's Python.
P2 groups
1-DOF test rack › Boards · Expected 7, found 6IO 1 ⚠ · CAN 15
ROS's own wording and level: a count mismatch raises the group to at least ERROR.27 It says nothing about firmware, config or readiness.
P3 manifest pick
1-DOF test rack · 6/7 boards · accepted 2/4 · comms sweep not run (26 h ago)IO 1 ⚠ · CAN 15
Missing, accepted and not-run, read in one line. Clicking it opens the check table (level · name · message), as in your Oct 23 drill-down.
P4 tests
test_every_expected_board_answers · failed 09:12 on a3da60 (CAN 15)as of last run
True at 09:12. Between runs it says nothing.
P5 capabilities
3/4 ready · Track on CAN 15: not available (CAN 15 not answering)IO 1 ⚠ · CAN 15
The calmest top line. It answers "what can I do now?" rather than "how is the rack?".

The five proposals

P1 · Wrapper plugins: every composite is a driver

A rig (or arm, or spider) is a plugin like moteus_servo, with the same describe | diagnose | act CLI. It reads its members' latest records and writes its own. The verdict is code, owned by whoever owns the composite. This is closure under composition taken literally: a composite is a black box like any leaf.

rig plugin · one_dof_test_rack/diagnostics.py describe | diagnose --records - | act rig row: WARN BoardOutcaused_by: …/can/15 reads members' latest records (stdin) Computerhost leaf fdcanusb XXXXXXXXCAN 1 · 14 · ● 15 fdcanusb XXXXXXXXCAN 11 · 12 · 13 · 21 D405 ·4411intel_rgbd D405 ·6036intel_rgbd leaves unchanged: every driver keeps self-reporting
The rule lives in the composite's own code. The host only adds the records input.
{"record": "status", "path": "host/rig-1-dof-test-rack", "name": "1-DOF test rack: actuators",
 "level": "warn", "code": "BoardOut", "caused_by": ["host/usb1/1-5/can/15"],
 "message": "6/7 boards: CAN 15 out; single-joint tests on the other six still run"}
# rule in code: one board out -> WARN · CAN 1 out (hard stop + gravity load) or any adapter out -> ERROR
Record v0.1 / black box
The purest fit. The schema gains only caused_by. The plugin protocol gains a records input; today a plugin gets --device only.
Computer
The computer plugin is one more leaf. A wrapper may read it, for example "recording unavailable: disk".
Dev vs deployment
At the bench, pytest calls the wrapper's diagnose() on the F1–F3 fixtures. On site, the host runs it every period. There is no readiness unless every wrapper writes its own.
Stock parts
ROS diagnostic_updater composite tasks26, Apex.Grace fault monitors7, ROS self_test for the on-demand part60, and the existing plugin protocol.
Failure modes
  • Every composite re-implements counting, staleness and "not run". bam_diagnostics fell into exactly this: a check that never ran was reported as OK, "SKIPPING".16
  • Arrival skew. Members report at different times, so the wrapper flaps. Your Oct 23 health-bar note makes the same point.14
  • A crashed wrapper reads STALE while every leaf is fine.
  • The reasoning is invisible on the page.
  • You need one wrapper per rig.

P3 · The expected setup becomes the acceptance spec: one file, three run modes recommended

Keep everything expected.toml has. Add [[rig.check]] entries that each name the status they read, a pass condition and when they run. The three modes are aerospace's built-in-test modes35:

Readiness is a separate state from pass/fail, as in OBD-II: ran, not run since boot, or not run since a change.3738

expected.toml [[rig.device]][[rig.check]] [computer][profile.*] startup · boot, plug, replug continuous · every report initiated · a person's click 1-DOF test rack · 6/7 boards · accepted 2/4 firmware current startup ran 08:51 · ok reply loss < 1 % continuous 0.0 % · ok comms sweep initiated not run (26 h) CAN 12 tracking initiated not run since fw each check: a status on the rig · caused_by = members it read reads members' own statuses (never written onto them) Computer[computer] limits fdcanusb XXXXXXXXCAN 1 · 14 · ● 15 fdcanusb XXXXXXXXCAN 11 · 12 · 13 · 21 D405 ·4411usb 5000 Mb/s D405 ·6036usb 5000 Mb/s bench, site and twin run the same file; only the [profile.*] tolerances differ (Next 2, 2025-12-04)
The manifest is the rig's wrapper, written as data. Expectations belong to the consumer (the rig), so they sit on the rig's row.
[[rig]]
name = "1-DOF test rack"
# ... the [[rig.device]] entries exactly as today ...

[[rig.check]]
name = "firmware current"
when = "startup"                 # every boot, plug and replug; reads only
reads = "can/*: firmware"         # each expected board's own firmware status
pass = "ok"                       # i.e. no UpdateAvailable

[[rig.check]]
name = "reply loss"
when = "continuous"              # judged on the reports the drivers already send
reads = "can/*: servo"
pass = "reply_loss < 1 %"

[[rig.check]]
name = "comms sweep"
when = "initiated"               # a person's click: check_comms on both adapters (query)
action = "check_comms"
valid_for = "24 h"
invalidated_by = ["Restarted", "Plugged", "firmware"]

[[rig.check]]
name = "CAN 12 pendulum tracking"
when = "initiated"
action = "tape"                  # bam_rack_twin hardware tape: MOVES, never scheduled
pass = "tracking_rms < 0.5 deg"
valid_for = "7 d"

[computer]                       # thresholds as data; defaults = ROS diagnostic_common_diagnostics
cpu_warn = "90 %"
memory_warn = "90 %"
disk_free_warn = "5 %"
disk_free_error = "1 %"
usb3_links = 2                   # both D405s at 5000 Mb/s

[profile.twin]                   # the same checks on the MuJoCo twin, wider where physics differs
tracking_rms = "< 1.0 deg"
Record v0.1 / black box
  • Each check is a status on the rig entity, with source: io_diagnostics.expected and caused_by naming the members it read.
  • Not run is unknown with code CheckNotRun, the twin of today's ExpectedNotChecked.
  • An initiated result never goes stale, but it expires:
    • after valid_for, IEC 61508's proof-test interval39;
    • on an invalidated_by event. In OBD, clearing codes resets readiness37; in DEM, re-enabling an event sets "test not completed" (0x50)29.
  • Expectations are the consumer's, as in consumer-driven contracts4243. So they live on the rig's row, and a board's row stays its own self-report. Today's RatioMismatch on the board moves to the rig (decision 2 below).
  • Later, a node's pass condition can default from its part's robotparts diagnostics slot (ok_range).24
Computer
  • [computer] sets the host leaf's thresholds, and rig checks can read it (usb3_links).
  • On this PC, k10temp Tctl has no chip limits, and Zen 4 is designed to sit at its 95 °C limit under load. So CPU temperature warns on sustained throttling, not on a fixed number.5558
  • The NVMe drive reports max 82 / crit 85 °C, used as given.
Dev vs deployment
io-diagnostics accept --rig "1-DOF test rack" --profile bench|site|twin:
  • runs the startup and continuous checks now;
  • lists the initiated checks, with their dry-run tokens, for a person to run;
  • prints JUnit and exits with a Nagios code.
That is Goss's split: validate at dev, serve → /healthz at deploy, one file.44 It is also the factory/site acceptance pair (IEC 62381: FAT, FIT, SAT, SIT).40
Stock parts
  • Goss; InSpec controls (impact, only_if → N/A).46
  • OpenHTF measurement validators (in_range; outcomes PASS / FAIL / UNSET / SKIPPED; marginal).47
  • OBD readiness; DEM; the IEC 61508 proof test.
  • Kubernetes startup vs liveness probes.41
Failure modes
  • pass grows into a mini-language. Cap it at "measurement op value", with no expressions.
  • The file lives in ~/.config, so the copy a site runs needs a committed twin.
  • There are two manifests for one rack: deployment.json already holds UUIDs and firmware history.23
  • Validity windows can nag.
  • A twin pass is not a site pass, so the profile must show on the row.

Scored comparison

CriterionWP1P2P3P4P5
Black boxes keep their word2053434
One check, two lives2032554
Honest status1524544
Declared, minimal, stable1524434
Off the shelf1035444
Calm first screen1043435
Tracer bullet this week1034533
Weighted / 5 3.20
 3.40
 4.45
 3.65
 4.00

The Computer entity, as read on this PC

A passive /proc and /sys snapshot taken at 09:12. No device nodes were opened and io-diagnostics was not run. The stock references are node_exporter's collectors52, the kernel's AER and USB ABI5354, and hwmon55.

$ python3 -B -I computer_snapshot.py
cpu: load 1.54/2.16/2.79 on 24 logical CPUs
memory: 61.9 GiB total, 40.2% available
disk /: 8.5% free
temp nvme/Composite: 39.9 C (max 82, crit 85)
temp nvme/Sensor 1: 39.9 C (max 65262)            # a sentinel, not a limit: must be filtered
temp k10temp/Tctl: 68.4 C (no chip limits)
temp amdgpu/edge: 47.0 C (no chip limits)
usb 1-3 0483:5740 serial XXXXXXXX: speed 12 Mb/s, devnum 10
usb 1-5 0483:5740 serial XXXXXXXX: speed 12 Mb/s, devnum 9
usb 6-1 8086:0b5b serial XXXXXXXXXXXX: speed 5000 Mb/s, devnum 2
usb 6-2 8086:0b5b serial XXXXXXXXXXXX: speed 5000 Mb/s, devnum 3
pcie 0000:01:00.0 (display): x16 of x16, 16.0 GT/s PCIe (max 16.0 GT/s PCIe), TOTAL_ERR_COR 0
Check on the host leafTodayDefault rule (stock source)Verdict
cpuload 1.5 / 24WARN when any core > 90 % (ROS cpu_monitor; it has no ERROR)28ok
memory40 % availableWARN > 90 % used (ROS ram_monitor)28ok
disk /8.5 % freeROS hd_monitor: WARN < 5 %, ERROR < 1 % free.28 Netdata: WARN > 90 % used.56ok by ROS · warn by Netdata
temperature: CPUTctl 68 °CNo chip limits. Zen 4 is designed to sit at its 95 °C limit under load58, so warn on sustained throttling, never on a fixed numberok
temperature: NVMe40 °CThe chip's own max 82 / crit 85 (hwmon temp*_max, _crit)ok
usb links (both D405s)5000 Mb/s ×2WARN at 480 (USB 2 fallback), already judged on the port rowsok
pcie link + AER (GPU)x16/x16 · 0 CORWidth below max, or uncorrectable AER > 0, is a warning. Already judged on the PCI rows (tree.py)ok
re-enumerationdevnum 10 / 9A rising devnum means the device re-enumerated. Today the kernel logged repeated disconnects on 1-3 between 08:07 and 08:10 (device number 6 → 10). Count them per port per hour, and make that a level only past a declared rateevents today; a level only past the rate

GPU health (Xid errors, PCIe replays via nvidia-smi or DCGM57) belongs to an NVIDIA driver plugin of its own, under the rule "drivers bring their own diagnostics".

Recommendation and tracer bullet

Build P3, on the caused_by seam.

  1. Barrier, alone (contract/record.py + tests/test_record.py):
    • Add caused_by and the counting rule. The three probes above become regression tests.
    • Done when: one missing CAN 15 is one cause and the chip names CAN 15; a spider at ERROR names leg 3; stock → v0 → stock is still the identity.
    • Land it after the computer lane commits. Its io_diagnostics/ write set is disjoint from this one, but it imports the contract.
  2. io_diagnostics/expected.py + cli.py:
    • Add [[rig.check]] with when, and two checks: firmware current (startup), and comms sweep (initiated, check_comms, valid 24 h, reset by Restarted / Plugged / firmware).
    • Keep readiness in ~/.local/state/io-diagnostics/acceptance.json.
    • Add the accept verb.
  3. tests/test_accept.py, with the acceptance written first:
    • Fixture F1 (CAN 15 missing) exits 1, naming CAN 15.
    • The full fixture exits 0.
    • A sweep that never ran exits 3, UNKNOWN "comms sweep not run".
  4. A 10-line pytest fixture around accept, so assert rig.ok runs on fixtures in CI and on the rack in your terminal.

What needs you

Each item has my recommendation and the default I'll take if you say nothing.

1. Composites explain upward (caused_by).

Recommend: yes. Default: build it as the barrier.

2. Rig expectations live on the rig's row, and a board shows only a muted "rig expects 0.1".

Recommend: yes. Default: leave RatioMismatch on the board until P3's checks land.

3. Validity windows for click-run checks: comms sweep 24 h, motion checks 7 d, all reset on restart, replug or firmware change.

Default: those.

4. One manifest per rack.

Recommend: deployment.json stays the versioned as-built record, and expected.toml later refers to it by rig id. Default: keep both for now.

5. Spider tolerance for the rack ("6 of 7 is fine").

Recommend: not yet. An acceptance rig with a missing board is WARN ExpectedMissing, as today. Default: as today.

Deliberately left out

The three probe scripts (pure contract code) 
# shared setup
import sys; sys.path.insert(0, "~/io-diagnostics")
from contract.record import DiagnosticStatus, Entity, summarize
NOW = 1761200000.0

# trap 1: host/usb1/1-3 (fdcanusb) -> can (part_of) -> can/15 (via); "disk /" ERROR written on "host"
#         vs on a sibling Entity("host/computer", part_of="host"); CAN 15 ERROR MoteusFault.33
# trap 2: Entity("host/rig-1-dof-test-rack", kind="rig", via="host") WARN ExpectedMissing
#         + host/usb1/1-5/can/15 ERROR BoardNotAnswering (the driver's level)
# trap 3: cell (group) -> spider (part_of) -> leg1..8 (part_of); leg 3 ERROR MoteusFault.33;
#         spider self-reports WARN, then ERROR, "leg 3 failed: walking on 7"
summary = summarize(entities, statuses, NOW)
root = summary["host"].rollup        # or summary["cell"].rollup
print(dict(root.counts), list(root.knock_on), root.origin)

Sources

  1. Changes (Apple Note), 2025-06-27: vault a private note
  2. June 28 - UR pick up in Sim (Notion, 2025): vault a private folder
  3. June 29 - Top Level Design / Python calls for To do list (Notion, 2025)
  4. First deployments (Apple Note), 2025-07-13
  5. How long before I can actually get a robot moving (Apple Note), 2025-07-22
  6. Acceptance Tests (bam_core), 2025-07-23: your items 1–29 only
  7. bam_ros_common, 2025-07-23: the Apex.Grace reaction
  8. ~/bam_ws/src/dev/bam_docs/diagnostics.md
  9. Next 2 (Apple Note), 2025-12-04
  10. Q - What do good sub components look like?
  11. Daily Note - Oct 14 2026: "Computed, measured, calculated vs entered"
  12. Daily Note - Oct 16 2026: Vibe Tracks feature tree
  13. Daily Note - Oct 22 2026: "Feedback on Io Diagonistics"
  14. Daily Note - Oct 23 2026: "feedback on Digoinastics"
  15. MOC - Component Testing: your Oct 3 notes and the AI timeline
  16. ~/io-diagnostics/docs/diagnostics-record-v0.md: record v0 / v0.1; the black-box rule §3; not a safety path §9
  17. ~/io-diagnostics/docs/expected-setup.md and io_diagnostics/expected.py (_rig_row)
  18. ~/io-diagnostics/docs/round3-brief-2026-10-23.md: CLI as the seam, the Monitoring Plugins API
  19. ~/io-diagnostics/io_diagnostics/tree.py: HOST_SELF, in the computer lane, uncommitted at 09:06 on 2026-10-23
  20. This report.
  21. MOC - Diagnostics §7: like_keras bam_diagnostics
  22. ~/bam_archive/v1_bam_ws/src/bam_brain/dev/v3/mdp_machine/docs/diagnostics-architecture.md
  23. bam_ws branch claude/traj-tracking-sim-to-real-a3da60: src/dev/bam_rack_twin/records/one-dof-test-rack/deployment.json; moteus_servo/src/moteus_servo/diagnostics.py (BoardNotAnswering is an error)
  24. ~/robotparts/docs/seams/robotdiagnostics.md
  25. ~/vibetracks/docs/peps/0007-feature-tree-computed-spec.md, 0005-runs-in-kpis-out.md
  26. ROS diagnostic_aggregator README (ros2)
  27. ROS generic_analyzer_base.hpp (ros2): report()
  28. ROS diagnostic_common_diagnostics README (ros2)
  29. AUTOSAR CP R22-11 SWS Diagnostic Event Manager, §7.3.6 and §7.5.1
  30. Nagios Core 4: network reachability
  31. Icinga Business Process: operators
  32. Icinga Business Process: monitoring, --blame --root-cause
  33. Prometheus Alertmanager configuration: inhibit_rules
  34. Datadog composite monitors
  35. MIL-STD-2165A glossary: BIT, fault detection and isolation, false alarm (the PBIT/CBIT/IBIT mode names are programme usage, not this text)
  36. Boeing central maintenance computer, US 4943919A
  37. EPA final rule 66 FR 18156 (2001): OBD readiness
  38. Texas DPS: "Ready or not"
  39. IEC 61508-4 proof-test definition (secondary glossary)
  40. IEC 62381:2024: FAT, FIT, SAT, SIT
  41. Kubernetes: liveness, readiness and startup probes
  42. Pact: can-i-deploy
  43. Martin Fowler: ContractTest (2011)
  44. Goss: validate, serve /healthz, junit/tap/nagios formats
  45. Testinfra
  46. Chef InSpec controls
  47. OpenHTF: measurements (PASS / FAIL / UNSET / PARTIALLY_SET / SKIPPED, marginal), phases, plugs
  48. Labgrid: usage, pytest plugin
  49. Checkly CLI
  50. Autoware diagnostic graph aggregator
  51. ros2_medkit
  52. Prometheus node_exporter
  53. Linux ABI: PCIe AER statistics
  54. Linux ABI: sysfs USB
  55. Linux hwmon sysfs interface
  56. Netdata health.d/disks.conf
  57. NVIDIA DCGM feature overview
  58. AMD community: Ryzen 9 7900X maximum 95 °C (secondary)
  59. Google SRE book: Service Level Objectives
  60. ROS self_test README (ros2)
  61. C - Diagnostics Record (What We Learned Since ROS DiagnosticStatus): Redfish HealthRollup, OpenBMC's 2024 removal
  62. ~/io-diagnostics/docs/research/diagnostics-schema-ros.md §3–4: ros2_medkit entities, Autoware's graph

Vault note: a private note · built by Claude Code (Opus 5.5), scribe lane