Hardware-in-the-Loop

Relevant source files

There is no hardware-in-the-loop CI. Raspberry Pi 5 testing is manual (README, issue 76). Nothing on this page runs in GitHub Actions or in cargo xtask check all. The hardware runbook says it does not authorize a safety-relevant robot deployment and that QEMU and RTL simulation numbers must not be converted into physical timing or safety claims.

Rules from the runbook

docs/operations/hardware-validation-and-benchmarking.md is the canonical checklist. Key rules:

Freeze a clean commit and do not mix artifacts from another revision.
Keep the production Ed25519 public key offline except for exporting the 32-byte public-key file for the Pi build.
Start with motors disconnected or mechanically unloaded, with an independently accessible physical e-stop.
Treat an unexpected reset, kernel panic, missing serial marker, failed provenance hash, watchdog reset, PWM output while e-stop is active, or uncontrolled movement as a stop condition: remove motor power, preserve the capture, file a regression.
Every publishable result needs a commit, command, pinned toolchain, hardware boundary, raw output, and hashes of the raw output and measured artifact.

Flow

Preflight and image


cargo xtask check all --profile full
scripts/verify/fpga-safety-gate.sh
export AXIOM_BPF_TRUSTED_KEY_PATH=/secure/path/axiomos-ed25519.pub
cargo xtask build rpi5 -- release embedded-rpi5
sha256sum -c target/aarch64-unknown-none/release/rpi5-artifacts.sha256
cargo xtask deploy rpi5 -- /path/to/mounted/pi-boot

See Raspberry Pi 5 for build and deploy detail. The runbook requires recording the boot-firmware revision on the card, since the repository does not pin it.

Scripts in scripts/hil

These require physical equipment: a Pi 5 with debug-probe UART, a sigrok-compatible logic analyzer (fx2lafw by default), and for most of them a Shrike RP2040 board driven with mpremote. Serial ports default to specific /dev/serial/by-id/ paths from the author’s lab, so override PI_UART and SHRIKE_UART. Output goes to RUN_DIR, which defaults under artifacts/runs/.

Script	Purpose	Image feature needed
`gate0-cold-boots.sh`	Gate 0: three cold boots (`BOOTS`, default 3, `BOOT_SECONDS` default 30) graded against expected and forbidden markers from the image’s `build.toml`. Expected: `PI5_BENCH_READY`, `INIT_PROCESS_STARTED`, `PI5_BOOT_OK`, `SIGNED_BPF_LOAD_OK`. Forbidden: `PI5_BENCH_FAIL`, `panic`, `fatal`, `watchdog`. `PI5_BENCH_READY` proves route initialization only	`bench` image
`gpio23-probe.sh`	Single-edge GPIO23 interrupt probe: capture UART plus logic analyzer and grade against `PI5_GPIO_IRQ_PROVEN`. The kernel latches the marker once per boot, so power-cycle between attempts	`bench`
`shrike-gpio23-static-high.sh`	Static-high electrical isolation check for Shrike GPIO22 to Pi GPIO23 (2.2 kOhm series; analyzer D0 disconnected)	`bench`
`shrike-gpio23-pulse.sh`	Unloaded repeated GPIO reflex capture. Shrike GP22 to Pi GPIO23, GP21 to GPIO24 (e-stop, active low), analyzer D1 on GPIO12. Runs `PULSE_COUNT` pulses (default 5) and requires `ACTUATORS_MOTORS_DISCONNECTED=YES`	`bench-reflex-rearm`; PWM mode needs `bench-pwm` or `bench-pwm-containment`
`v03d-estop-cycle.sh`	Unloaded e-stop capture at 24 MHz; refuses a normal bench image	`bench-estop-rearm`
`v03d-reduce.py`	Reduces a sigrok `.sr` capture into GPIO24-falling to GPIO12-falling latencies; requires 100 or more ordered pairs at 24 MHz, a safe final state, and a maximum under 1 ms; rejects missing or extra edges and missing re-arms	none (offline)
`v03b-reduce.py`	Reduces `PI5_MC` and `PI5_MA` serial lines to count, min, median, p95, p99, p99.9, max	none (offline)
`pwm-containment-reduce.py`	Offline unloaded PWM containment check (D0 is GPIO23, D1 is GPIO12): 50 percent baseline, clamp to 90 percent, invalid channel rejected, final LOW. Not an e-stop latency or WCET measurement	`bench-pwm-containment`
`shrike-bench.py`	Bounded, unloaded Shrike/FPGA waveform recorder and offline checker (new on this branch; see below)	none for `replay`/`self-test`
`flash-rpi5-gpio-diag.sh`	Flashes a retained GPIO diagnostic image to the Pi boot device and checks its hash against the image’s `build.toml`	n/a

Example, the V03-D e-stop cycle:


cargo xtask build rpi5 -- release embedded-rpi5,bench-estop-rearm
ACTUATORS_MOTORS_DISCONNECTED=YES scripts/hil/v03d-estop-cycle.sh

Wiring sheets live in docs/operations/wiring/ (Pi 5 UART, unloaded e-stop, unloaded PWM).

Bench images use a bounded deferred console. Any PI5_BENCH_LOG_LOSS dropped_records=N marker invalidates serial-derived evidence. PI5_BENCH_TIMING endpoints are software endpoints (M-C is GPIO handler stamp to local output write; M-B ends after local safe writes). They are not physical-edge-to-motor measurements; capture electrically for that claim.

Shrike bench observer

scripts/hil/shrike-bench.py is added on feat/fpga-bringup for the Pi to RP2040 to FPGA bench path. It has three modes and none of them flashes firmware or drives pins:

Mode	Behavior
`capture CONFIG CAMPAIGN RUN --uart DEVICE`	Records D0 to D7 as one byte per sample from a single sigrok process (`--driver` defaults to `fx2lafw`), streams 4 MiB chunks through `zstd -1`, and records a manifest of chunk IDs, sample counts and offsets, SHA-256, timestamps and process exit statuses. The CONFIG is frozen before acquisition
`replay RUN`	Checks every compressed chunk and the independently authored stimulus schedule: count, duty, direction, latency and stop violations, plus UART loss
`self-test`	Synthetic checks only

Channel roles are left_pwm, right_pwm, left_dir, right_dir, estop and stimulus. Each rising stimulus edge advances a list of steps carrying signed per-mille duties (maximum 800), the active-low e-stop level, settling allowance and dwell bounds. The run must begin safe and end in a zero-duty step. The sigrok, libsigrok and FX2 patch binaries are pinned by absolute path and SHA-256.

Storage is bounded: a 50,000,000,000-byte cumulative campaign cap, a controlled stop at 45,000,000,000 bytes, and a 5 GB reserve; observer buffering is kept to 128 MiB or less. The bench plan requires a pilot to fit 1.5 GB per hour. tests/scripts/test_shrike_bench.py and the self-test mode run in the audit gate. The script docstring states that an offline PASS never closes physical acceptance and that it checks a frozen strobe workload, not arbitrary commands, SPI or all V04 populations.

Bench status on this branch

The September 12 baseline recorded in v04-hardware-first.md supersedes the partial September 9 progress report: 10,000 paired monitor samples, 10,000 correlated physical GPIO reflex responses, 200 GPIO stop diagnostic responses and the complete 1,000-request unloaded PWM corpus. The GPIO reflex functional check passed; its 500 ns target and 1 microsecond fallback failed (maximum 9.625 microseconds). A retained 64-bit FX2 sample-counter correction enabled a continuous 48,120,000,000-sample capture; this does not qualify a 24-hour acquisition.

Recorded physical steps on the Shrike board (2026-09-13): USB identity (2e8a:0005, MicroPython v1.26.0-preview), a read-only match of installed flash against the vendor v1.0.0 UF2 payload, ROM restoration with a full 4 MiB flash readback whose hash matched before and after, and firmware-independent BOOT + RST recovery entry. Cold BOOT-held reconnection failed. No custom runtime or FPGA image was flashed.

On 2026-09-15 the remaining physical checks (continuity, configuration timing, the concrete adapter, unloaded chain faults, at least 100 physical FPGA e-stop assertions, the one-hour pilot and the 24-hour soak) were moved to the final v0.5 phase. They remain mandatory before hardware actuation and release. See Shrike FPGA Safety Gate.

Staged checklist

The runbook’s section 4 proceeds in order, with a retained capture and explicit pass decision before the next stage:

Kernel and rootfs: boot markers (INIT_PROCESS_STARTED, SIGNED_BPF_LOAD_OK), no panic, page fault, reboot loop or watchdog reset.
GPIO, timer and BPF dispatch with actuators disconnected: at least 100 injections per edge type, recording successes, drops and unexpected routes; e-stop assertion leaves PWM disabled.
Shrike control link and RP2040: control-link framing, watchdog and fail-safe, invalid frames, link loss. V04_CHUNK tracing needs the trace-control-link feature.
Final PWM safety gate: FPGA board, bitstream hash, logic-analyzer probing of both PWM outputs at and beyond the plus or minus 800 per-mille limit. The powered actuator test comes only after the unloaded tests and is deferred in the electronics campaign.
One-hour pilot, then 24-hour unloaded electronics soak with complete-coverage raw capture. The runbook says the current short-capture scripts and V04 reducer do not alone qualify this full-duration oracle.

A passing run is reported as “24-hour unloaded electronics acceptance”, which unlocks runtime development under the hardware-first plan but does not check off the powered pilot or robot soak.

Benchmark campaigns

Campaign	Command	Notes
Host verifier baseline	`cargo bench --locked -p kernel_bpf --bench verifier --features embedded-profile`	Record executable and raw-log hashes; only Criterion intervals present in raw output are publishable
Pi 5 verifier cost	`cargo xtask bench verifier -- LOG -o CSV --plot PNG --cntfrq HZ`	Needs a separate `verifier-cost` image and `/bin/verifier_bench`; see Benchmarks
GPIO-to-actuation and e-stop latency	`bench` image plus logic analyzer	Report sample count, min, median, p95, p99, max, trigger config, analyzer sample rate and excluded samples; repeat after cold boot, warm boot and load

Kernel benches exist in kernel/crates/kernel_bpf/benches/ (verifier.rs, interpreter.rs, maps.rs). Reducers scripts/benchmark/analyze-v03.py, analyze-v04.py and v04-behavior-runner.py process retained captures; the gate runs self-tests for the reducers but not the physical campaigns.

Evidence review

Raw UART logs, analyzer exports, CSV and plots, command transcripts, and hardware, firmware and bitstream hashes go into a retained campaign directory with a provenance manifest under docs/performance/evidence/COMMIT/. Failures are linked to issues, not deleted. python3 -B scripts/verify/benchmark-provenance.py and the full local gate run after adding evidence, and a human reviews wiring, stop-condition results and the manifest before any new tag or hardware claim. Completing the checklist does not by itself close the outstanding HIL, soak, secure-boot, SMP or runtime-evolution issues.