mirror of
https://github.com/alexhopeoconnor/firmware.git
synced 2026-10-04 03:18:10 +10:00
Ben MeadorsandClaude Opus 5
83fd62b756
* test(native): add 14 suites for routing, persistence, parsing and identity gaps Coverage audit of the native test tree; adds the highest-value untested logic as 11 new suites and extends 3 existing ones (200 test functions). New: test_stream_framing, test_nodedb_boot_recovery, test_nodedb_legacy_migration, test_nodedb_v25_roundtrip, test_nodedb_identity_hygiene, test_channel_keys, test_reliable_ack_matrix, test_hop_start_policy, test_routing_response_hops, test_phone_api_config_dump, test_observer. Extended: test_rtc, test_mqtt, test_xmodem. Two source changes the audit produced: - StreamAPI::handleRecStream copied stream->read()'s `cInt < 0` EOF check into the buffer-fed path, where there is no EOF sentinel; with signed char any byte >= 0x80 (START1 is 0x94) aborted the parse. Read the byte as uint8_t directly. Latent on develop (no callers), pinned by test_stream_framing. - Extract the post-decode pre-hop predicate from Router::handleReceived into shouldSkipHandleForPostDecodeHop() (NodeDB.h) so test_hop_start_policy drives the exact expression the router calls. No behavior change. test/state-manifest.tsv declares the suites that construct a NodeDB. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): address review - harden observer dispatch, trim comments Review follow-ups on the coverage-audit suites: - Observable::notifyObservers() erased list nodes while holding an iterator into them, so an observer that unobserves itself from onNotify corrupted the dispatch. Today the only self-detacher (PhoneAPI::onNotify -> checkConnectionTimeout -> close -> unobserve) survives solely because it returns -1 and aborts the chain before the increment; that unwritten contract is now gone. Removal during a dispatch nulls the entry and the outermost notify sweeps afterwards, which keeps self-detach, next-detach and destruction-during-notify all safe without an allocation. Hoisting the next iterator instead would have inverted the hazard and broken the existing next-detach case. Two regression tests added. - Correct the documented caller of shouldSkipHandleForPostDecodeHop: the call is in Router::dispatchReceived, not handleReceived. - Cast hop fields to unsigned at the %u call site in test_hop_start_policy. - Trim the new suites' file headers to the one-or-two-line rule in AGENTS.md. - Rename eight test functions whose names were exactly `test_` + 35 chars: that is the shape of a Lob API key, so trufflehog flagged them as secrets and failed the Trunk CI check. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): revert the observer dispatch change, keep the contract test Backs out the notifyObservers() deferred-removal hardening from the previous commit. It was reviewer-driven scope creep: nothing in the coverage audit needed it, no test required it, and it changes dispatch semantics in a header with ~76 observe() call sites on native verification alone. The hazard it addressed is not reachable today. The only observer that unobserves itself from onNotify is PhoneAPI (onNotify -> checkConnectionTimeout -> close -> unobserve), and it returns -1, which aborts the chain before the iterator is advanced past the erased node. test_self_detach_with_abort_during_notify stays: it passes against the unmodified dispatch and pins that the -1 is load-bearing, so a later cleanup that "simplifies" it away goes red. The unsafe variant (self-detach returning 0) is documented in a comment rather than tested, since asserting it would be asserting UB. * fix(serial): recover the frame behind a stray framing marker A byte that failed the START2 check was discarded rather than re-tested as a possible START1, so 0x94 0x94 0xc3 ... lost the real frame: one corrupted byte on a noisy UART silently dropped the frame behind it. Re-test the byte in place instead. Applied to both copies of the receive state machine. readStream() is the one that matters in the field - it is the serial path every phone client uses - while handleRecStream() still has no callers on develop. Strictly widens what the parser accepts; no frame that parsed before parses differently. test_stream_framing covers it on both receive paths, plus a run of stray markers and a START1-then-unrelated-byte resync. This was originally documented as a known gap in the framing suite. Fixing it instead was NomDeTom's call on review: a passing test asserting the bad behavior is what makes it hard to change later, and it is the same defect shape as the signedness fix three functions away. Also: use Throttle::deadlinePassed() in test_reliable_ack_matrix rather than a bare millis() compare, matching the house deadline rule. * test(native): cover the stray-marker resync on the buffer path too The stray-marker fix went into both copies of the receive state machine, but only test_stray_start1_before_frame_still_delivers drove both. The repeated- marker and unrelated-byte cases drove readStream() alone, so a regression in handleRecStream() would have gone unnoticed by two of the three. Verified load-bearing: reverting only the handleRecStream() half of the fix turns test_repeated_stray_start1_before_frame_still_delivers red on the new assertion. test_start1_then_unrelated_byte_resyncs stays green under that mutation by design - its failing byte is 0x00, where both branches reset to 0 - and covers the other half of the ternary. Also drops the stale header on test_stray_start1_before_frame_still_delivers, which still described the gap as pinned-as-is after the fix landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(native): make the hop-start truth table assert the rows it prints test_truth_table_summary was six TEST_MESSAGE lines and no assertion, so it reported as a case that could not fail - the anti-pattern #11517 names in its unfinished assertion-presence lint, and the one exception to NomDeTom's "no RUN_TEST without an assertion" pass over this PR. The printed row and the checked expectation now come from one struct, so the summary cannot narrate a table the predicates no longer implement. It also covers the consequence columns the per-row tests do not assert together: classifyHopStart, shouldDropPacketForPreHop and shouldSkipHandleForPostDecodeHop for the same packet, with the expectations gated on MESHTASTIC_PREHOP_DROP. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
8.5 KiB
8.5 KiB
| 1 | # Shared-state manifest for the native test suites. | ||
|---|---|---|---|
| 2 | # | ||
| 3 | # Every suite runs inside its own scratch $HOME (bin/pio-test-isolate.sh), so leftovers cannot reach | ||
| 4 | # the next suite. This file is not what makes that safe - it is what makes each suite's intent | ||
| 5 | # | ||
| 6 | # The invariant behind the flags: mutation *inside* a suite is free, carrying state *across* a suite | ||
| 7 | # boundary is never permitted. No flag grants cross-suite carry - a suite that needs another suite's | ||
| 8 | # output needs an explicit fixture, not inheritance. That is the whole bug, and per-suite isolation | ||
| 9 | # keeps it impossible by construction rather than by policy. | ||
| 10 | # | ||
| 11 | # Format - three tab-separated columns, the same shape as an allowlist entry: | ||
| 12 | # | ||
| 13 | # <suite> | <flags> | <reason> |
| 14 | # | ||
| 15 | # The reason column is mandatory and is reviewed on change. One central file rather than a file per | ||
| 16 | # suite, so every opt-out is visible in one diffable list and attracts review pressure; per-suite | ||
| 17 | # files hide growth. bin/run-tests.sh prints how many suites declare non-default handling, so the | ||
| 18 | # number creeping upward is visible without anyone auditing this file. | ||
| 19 | # | ||
| 20 | # Flags (space separated; the default is no entry at all): | ||
| 21 | # | ||
| 22 | # writes=<a,b> files this suite mutates inside its sandbox. Matched against the path relative | ||
| 23 | # to the scratch $HOME or just the basename, so `nodes.proto` is enough. A | ||
| 24 | # declared write makes the change CLEAN instead of DIRTY - the list is the | ||
| 25 | # documentation. A declared file that does NOT change is reported as MISSING, | ||
| 26 | # which catches silently-broken persistence. | ||
| 27 | # state=per-suite state persists across this suite's own test cases; the default is per-test. | ||
| 28 | # Use for persistence round-trips, migration ladders, anything where test N must | ||
| 29 | # observe test N-1's write. With this set, only the suite boundary is checked - | ||
| 30 | # per-test checking would flag every test by design. | ||
| 31 | # | ||
| 32 | # state=per-suite is the one to watch. It is legitimate, and it is also the pattern that let | ||
| 33 | # test_nodedb_blocked accumulate 198 protected nodes across its test cases. Requiring the flag makes | ||
| 34 | # that an explicit, defended choice instead of an accident of setUp(). | ||
| 35 | # | ||
| 36 | # To propose entries from a real run: ./bin/run-tests.sh --write-manifest prints the lines it would | ||
| 37 | # add, for a human to paste and justify. It never applies them itself, and CI never applies them at | ||
| 38 | # all - an auto-accepted baseline is the same rot as an auto-updated snapshot. | ||
| 39 | # | ||
| 40 | # errors=<max> | <min>..<max> | <min>.. caps a suite's LOG_ERROR lines, default 100. A range, not a | ||
| 41 | # ceiling: for a fuzz suite the floor is the half that matters. test_fuzz_decode logging ~100k | ||
| 42 | # rejections is the suite working; the same suite logging none means it stopped feeding malformed | ||
| 43 | # input, and every case would still pass. Bounds are wide on purpose - they catch a path that has | ||
| 44 | # stopped running, not a drift of a few hundred lines. | ||
| 45 | # | ||
| 46 | # suite | flags | reason |
| 47 | test_admin_radio | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs errors=400 | per-test NodeDB fixture, and the admin handlers under test persist config, channels and node metadata |
| 48 | test_admin_session_repro | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | constructs a NodeDB, whose constructor persists a default set when the prefs directory is empty |
| 49 | test_event_channel_phone_api | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | constructs a NodeDB, whose constructor persists a default set when the prefs directory is empty |
| 50 | test_event_channel_router | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto errors=200 | subclasses NodeDB for the event-channel fixtures; the base constructor persists a default set when the prefs directory is empty |
| 51 | test_firmware_edition | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | persists an event firmware_edition in devicestate, then reboots a NodeDB to prove a vanilla build resets it |
| 52 | test_fuzz_decode | errors=20000..250000 | fuzzes protobuf decode; every rejection logs. A collapse to near zero means the corpus stopped reaching the decoder |
| 53 | test_fuzz_packets | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs errors=5000..60000 | drives decode of fuzzed packets through the real NodeDB and message store |
| 54 | test_hop_scaling | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | constructs a NodeDB to hold the hop-distance fixtures |
| 55 | test_hop_start_policy | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | constructs a NodeDB (isFromUs needs nodeDB->getNodeNum()), whose constructor persists a default set when the prefs directory is empty |
| 56 | test_mesh_beacon | writes=module.proto | exercises the beacon's module-config save path |
| 57 | test_mesh_module | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat | module framework tests construct a NodeDB |
| 58 | test_mqtt | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto errors=1000..12000 | constructs a NodeDB for node lookups in the MQTT paths |
| 59 | test_nexthop_routing | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | next-hop selection reads and updates the node DB |
| 60 | test_nodedb_blocked | state=per-suite writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat | saturates the DB with MAX_NUM_NODES-2 favourited nodes to test the protected cap; a later test's removeNodeByNum() persists that state, and the cap test depends on the fill from the test before it |
| 61 | test_nodedb_boot_recovery | state=per-suite writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | deliberate boot-recovery ladder: corrupts/deletes/restores the pref files and reboots a NodeDB per test to pin the DECODE_FAILED identity freeze, so each test observes the previous test's on-disk state |
| 62 | test_nodedb_identity_hygiene | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat,Messages_default.msgs | constructs a NodeDB; addFromContact persists the node DB after every merge, the reboot test proves the key-erasure guard survives a reload, and the should_ignore path rewrites the message store |
| 63 | test_nodedb_legacy_migration | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat | each test hand-writes a v24-format nodes.proto fixture and cold-boots a NodeDB, whose constructor persists the migrated v25 database (warm.dat via the over-cap eviction absorb) |
| 64 | test_nodedb_v25_roundtrip | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat | v25 persistence round-trips: every test saves nodes.proto and cold-boots a NodeDB whose constructor persists the default segments; warm.dat on the node-DB save cadence |
| 65 | test_packet_signing | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat errors=300 | needs a NodeDB holding both peers' keys for the PKI encode/decode paths |
| 66 | test_phone_api_config_dump | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | per-test NodeDB fixture backing full PhoneAPI want_config dumps; the constructor persists a default config/channel/node set in a fresh sandbox |
| 67 | test_pki_admin_fallback | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | needs a NodeDB holding admin keys for the fallback paths |
| 68 | test_reliable_ack_matrix | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | constructs a NodeDB (whose constructor persists a default set when the prefs directory is empty) for the sender-key lookups in the ACK/NAK matrix |
| 69 | test_stream_api | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | drives real PhoneAPI handshakes, which read and persist config and the node DB |
| 70 | test_traceroute_nexthop | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto | traceroute route selection reads the node DB |
| 71 | test_traffic_management | writes=config.proto,module.proto,device.proto,channels.proto,nodes.proto,warm.dat errors=3000..12000 | constructs a NodeDB for the per-node rate-limit and dedup state; test_tm_fuzz_nodenum_blitz feeds malformed payloads, and each rejection logs (measured 7985) |
| 72 | test_transmit_history | writes=transmit_history.dat | persistence round-trip: what it asserts is that retransmission state survives a save/load |
| 73 | test_warm_store | writes=warm.dat | persistence round-trip of the warm-tier snapshot, which is the tier's whole contract |