firmware/test/state-manifest.tsv at develop

mirror of https://github.com/alexhopeoconnor/firmware.git synced 2026-10-13 15:11:20 +10:00

Files

T

Ben MeadorsandClaude Opus 5 83fd62b756 test(native): add 14 suites for routing, persistence, parsing and identity gaps (#11515 )

* test(native): add 14 suites for routing, persistence, parsing and identity gaps

Coverage audit of the native test tree; adds the highest-value untested
logic as 11 new suites and extends 3 existing ones (200 test functions).

New: test_stream_framing, test_nodedb_boot_recovery,
test_nodedb_legacy_migration, test_nodedb_v25_roundtrip,
test_nodedb_identity_hygiene, test_channel_keys, test_reliable_ack_matrix,
test_hop_start_policy, test_routing_response_hops,
test_phone_api_config_dump, test_observer.
Extended: test_rtc, test_mqtt, test_xmodem.

Two source changes the audit produced:

- StreamAPI::handleRecStream copied stream->read()'s `cInt < 0` EOF check
  into the buffer-fed path, where there is no EOF sentinel; with signed
  char any byte >= 0x80 (START1 is 0x94) aborted the parse. Read the byte
  as uint8_t directly. Latent on develop (no callers), pinned by
  test_stream_framing.
- Extract the post-decode pre-hop predicate from Router::handleReceived
  into shouldSkipHandleForPostDecodeHop() (NodeDB.h) so
  test_hop_start_policy drives the exact expression the router calls.
  No behavior change.

test/state-manifest.tsv declares the suites that construct a NodeDB.
Full 68-suite Docker coverage run matches the pre-change baseline.

* test(native): address review - harden observer dispatch, trim comments

Review follow-ups on the coverage-audit suites:

- Observable::notifyObservers() erased list nodes while holding an iterator
  into them, so an observer that unobserves itself from onNotify corrupted the
  dispatch. Today the only self-detacher (PhoneAPI::onNotify ->
  checkConnectionTimeout -> close -> unobserve) survives solely because it
  returns -1 and aborts the chain before the increment; that unwritten contract
  is now gone. Removal during a dispatch nulls the entry and the outermost
  notify sweeps afterwards, which keeps self-detach, next-detach and
  destruction-during-notify all safe without an allocation. Hoisting the next
  iterator instead would have inverted the hazard and broken the existing
  next-detach case. Two regression tests added.

- Correct the documented caller of shouldSkipHandleForPostDecodeHop: the call
  is in Router::dispatchReceived, not handleReceived.

- Cast hop fields to unsigned at the %u call site in test_hop_start_policy.

- Trim the new suites' file headers to the one-or-two-line rule in AGENTS.md.

- Rename eight test functions whose names were exactly `test_` + 35 chars:
  that is the shape of a Lob API key, so trufflehog flagged them as secrets
  and failed the Trunk CI check.

Full 68-suite Docker coverage run matches the pre-change baseline.

* test(native): revert the observer dispatch change, keep the contract test

Backs out the notifyObservers() deferred-removal hardening from the previous
commit. It was reviewer-driven scope creep: nothing in the coverage audit
needed it, no test required it, and it changes dispatch semantics in a header
with ~76 observe() call sites on native verification alone.

The hazard it addressed is not reachable today. The only observer that
unobserves itself from onNotify is PhoneAPI (onNotify ->
checkConnectionTimeout -> close -> unobserve), and it returns -1, which aborts
the chain before the iterator is advanced past the erased node.

test_self_detach_with_abort_during_notify stays: it passes against the
unmodified dispatch and pins that the -1 is load-bearing, so a later cleanup
that "simplifies" it away goes red. The unsafe variant (self-detach returning
0) is documented in a comment rather than tested, since asserting it would be
asserting UB.

* fix(serial): recover the frame behind a stray framing marker

A byte that failed the START2 check was discarded rather than re-tested as
a possible START1, so 0x94 0x94 0xc3 ... lost the real frame: one corrupted
byte on a noisy UART silently dropped the frame behind it. Re-test the byte
in place instead.

Applied to both copies of the receive state machine. readStream() is the one
that matters in the field - it is the serial path every phone client uses -
while handleRecStream() still has no callers on develop.

Strictly widens what the parser accepts; no frame that parsed before parses
differently. test_stream_framing covers it on both receive paths, plus a run
of stray markers and a START1-then-unrelated-byte resync.

This was originally documented as a known gap in the framing suite. Fixing it
instead was NomDeTom's call on review: a passing test asserting the bad
behavior is what makes it hard to change later, and it is the same defect
shape as the signedness fix three functions away.

Also: use Throttle::deadlinePassed() in test_reliable_ack_matrix rather than
a bare millis() compare, matching the house deadline rule.

* test(native): cover the stray-marker resync on the buffer path too

The stray-marker fix went into both copies of the receive state machine, but
only test_stray_start1_before_frame_still_delivers drove both. The repeated-
marker and unrelated-byte cases drove readStream() alone, so a regression in
handleRecStream() would have gone unnoticed by two of the three.

Verified load-bearing: reverting only the handleRecStream() half of the fix
turns test_repeated_stray_start1_before_frame_still_delivers red on the new
assertion. test_start1_then_unrelated_byte_resyncs stays green under that
mutation by design - its failing byte is 0x00, where both branches reset to 0 -
and covers the other half of the ternary.

Also drops the stale header on test_stray_start1_before_frame_still_delivers,
which still described the gap as pinned-as-is after the fix landed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(native): make the hop-start truth table assert the rows it prints

test_truth_table_summary was six TEST_MESSAGE lines and no assertion, so it
reported as a case that could not fail - the anti-pattern #11517 names in its
unfinished assertion-presence lint, and the one exception to NomDeTom's "no
RUN_TEST without an assertion" pass over this PR.

The printed row and the checked expectation now come from one struct, so the
summary cannot narrate a table the predicates no longer implement. It also
covers the consequence columns the per-row tests do not assert together:
classifyHopStart, shouldDropPacketForPreHop and shouldSkipHandleForPostDecodeHop
for the same packet, with the expectations gated on MESHTASTIC_PREHOP_DROP.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

2026-08-18 12:41:08 +00:00

76 lines

8.5 KiB

Plaintext

Raw Permalink Blame History

 # Shared-state manifest for the native test suites.
 #
 # Every suite runs inside its own scratch $HOME (bin/pio-test-isolate.sh), so leftovers cannot reach
 # the next suite. This file is not what makes that safe - it is what makes each suite's intent
 # reviewable. A suite listed here is declaring "I mutate persisted state on purpose, and here is
 # why"; a suite that is not listed is expected to leave the sandbox as it found it.
 #
 # The invariant behind the flags: mutation *inside* a suite is free, carrying state *across* a suite
 # boundary is never permitted. No flag grants cross-suite carry - a suite that needs another suite's
 # output needs an explicit fixture, not inheritance. That is the whole bug, and per-suite isolation
 # keeps it impossible by construction rather than by policy.
 #
 # Format - three tab-separated columns, the same shape as an allowlist entry:
 #
 #   <suite>	<flags>	<reason>
 #
 # The reason column is mandatory and is reviewed on change. One central file rather than a file per
 # suite, so every opt-out is visible in one diffable list and attracts review pressure; per-suite
 # files hide growth. bin/run-tests.sh prints how many suites declare non-default handling, so the
 # number creeping upward is visible without anyone auditing this file.
 #
 # Flags (space separated; the default is no entry at all):
 #
 #   writes=<a,b>      files this suite mutates inside its sandbox. Matched against the path relative
 #                     to the scratch $HOME or just the basename, so `nodes.proto` is enough. A
 #                     declared write makes the change CLEAN instead of DIRTY - the list is the
 #                     documentation. A declared file that does NOT change is reported as MISSING,
 #                     which catches silently-broken persistence.
 #   state=per-suite   state persists across this suite's own test cases; the default is per-test.
 #                     Use for persistence round-trips, migration ladders, anything where test N must
 #                     observe test N-1's write. With this set, only the suite boundary is checked -
 #                     per-test checking would flag every test by design.
 #
 # state=per-suite is the one to watch. It is legitimate, and it is also the pattern that let
 # test_nodedb_blocked accumulate 198 protected nodes across its test cases. Requiring the flag makes
 # that an explicit, defended choice instead of an accident of setUp().
 #
 # To propose entries from a real run: ./bin/run-tests.sh --write-manifest prints the lines it would
 # add, for a human to paste and justify. It never applies them itself, and CI never applies them at
 # all - an auto-accepted baseline is the same rot as an auto-updated snapshot.
 #
 # errors=<max> | <min>..<max> | <min>..   caps a suite's LOG_ERROR lines, default 100. A range, not a
 # ceiling: for a fuzz suite the floor is the half that matters. test_fuzz_decode logging ~100k
 # rejections is the suite working; the same suite logging none means it stopped feeding malformed
 # input, and every case would still pass. Bounds are wide on purpose - they catch a path that has
 # stopped running, not a drift of a few hundred lines.
 #
 # suite	flags	reason
 test_admin_radio	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs errors=400	per-test NodeDB fixture, and the admin handlers under test persist config, channels and node metadata
 test_admin_session_repro	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	constructs a NodeDB, whose constructor persists a default set when the prefs directory is empty
 test_event_channel_phone_api	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	constructs a NodeDB, whose constructor persists a default set when the prefs directory is empty
 test_event_channel_router	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto errors=200	subclasses NodeDB for the event-channel fixtures; the base constructor persists a default set when the prefs directory is empty
 test_firmware_edition	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	persists an event firmware_edition in devicestate, then reboots a NodeDB to prove a vanilla build resets it
 test_fuzz_decode	errors=20000..250000	fuzzes protobuf decode; every rejection logs. A collapse to near zero means the corpus stopped reaching the decoder
 test_fuzz_packets	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs errors=5000..60000	drives decode of fuzzed packets through the real NodeDB and message store
 test_hop_scaling	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	constructs a NodeDB to hold the hop-distance fixtures
 test_hop_start_policy	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	constructs a NodeDB (isFromUs needs nodeDB->getNodeNum()), whose constructor persists a default set when the prefs directory is empty
 test_mesh_beacon	writes=module.proto	exercises the beacon's module-config save path
 test_mesh_module	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat	module framework tests construct a NodeDB
 test_mqtt	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto errors=1000..12000	constructs a NodeDB for node lookups in the MQTT paths
 test_nexthop_routing	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	next-hop selection reads and updates the node DB
 test_nodedb_blocked	state=per-suite writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat	saturates the DB with MAX_NUM_NODES-2 favourited nodes to test the protected cap; a later test's removeNodeByNum() persists that state, and the cap test depends on the fill from the test before it
 test_nodedb_boot_recovery	state=per-suite writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	deliberate boot-recovery ladder: corrupts/deletes/restores the pref files and reboots a NodeDB per test to pin the DECODE_FAILED identity freeze, so each test observes the previous test's on-disk state
 test_nodedb_identity_hygiene	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs	constructs a NodeDB; addFromContact persists the node DB after every merge, the reboot test proves the key-erasure guard survives a reload, and the should_ignore path rewrites the message store
 test_nodedb_legacy_migration	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat	each test hand-writes a v24-format nodes.proto fixture and cold-boots a NodeDB, whose constructor persists the migrated v25 database (warm.dat via the over-cap eviction absorb)
 test_nodedb_v25_roundtrip	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat	v25 persistence round-trips: every test saves nodes.proto and cold-boots a NodeDB whose constructor persists the default segments; warm.dat on the node-DB save cadence
 test_packet_signing	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat errors=300	needs a NodeDB holding both peers' keys for the PKI encode/decode paths
 test_phone_api_config_dump	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	per-test NodeDB fixture backing full PhoneAPI want_config dumps; the constructor persists a default config/channel/node set in a fresh sandbox
 test_pki_admin_fallback	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	needs a NodeDB holding admin keys for the fallback paths
 test_reliable_ack_matrix	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	constructs a NodeDB (whose constructor persists a default set when the prefs directory is empty) for the sender-key lookups in the ACK/NAK matrix
 test_stream_api	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	drives real PhoneAPI handshakes, which read and persist config and the node DB
 test_traceroute_nexthop	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto	traceroute route selection reads the node DB
 test_traffic_management	writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat errors=3000..12000	constructs a NodeDB for the per-node rate-limit and dedup state; test_tm_fuzz_nodenum_blitz feeds malformed payloads, and each rejection logs (measured 7985)
 test_transmit_history	writes=transmit_history.dat	persistence round-trip: what it asserts is that retransmission state survives a save/load
 test_warm_store	writes=warm.dat	persistence round-trip of the warm-tier snapshot, which is the tier's whole contract

76 lines 8.5 KiB Plaintext Raw Permalink Blame History

76 lines

8.5 KiB

Plaintext

Raw Permalink Blame History