mirror of
https://github.com/alexhopeoconnor/firmware.git
synced 2026-10-21 10:42:35 +10:00
Replace native-suite-count file with dynamic test discovery (#11413)
* Derive the native suite count on the fly instead of registering it in a file test/native-suite-count was a manually-maintained register of the test_* directory count, reconciled against the actual directories by bin/run-tests.sh (as an AMBER verdict) and by a dedicated suite-count-check CI job. The reconciliation only ever guarded the file itself: the check that matters - suites that actually ran vs. the test_* directories on disk - already derives its expected count from a directory walk, so the file added a bookkeeping step to every suite addition/removal without adding signal. Remove the file and everything that existed to keep it honest: - bin/run-tests.sh: drop the canonical-count file read, the count-mismatch AMBER verdict, and the [canonical: x/y] suffix; the verdict lines already carry ran/expected from the directory walk. The shuffle seed suffix stays. - test_native.yml: delete the suite-count-check job and its needs: edges. - Docs (copilot-instructions.md, AGENTS.md, test/README.md) and the test-script comments now describe the count as derived from test/test_* at run time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Add suite-shrinkage-check: fail a PR that silently loses a test_* suite With test/native-suite-count gone, nothing in CI noticed the suite set shrinking: platformio test discovers and runs whatever test_* directories exist, and bin/run-tests.sh derives its expected count from the same walk, so a suite directory lost in a bad rebase or an overzealous cleanup just means fewer suites run - every remaining check stays green. Restore that tripwire git-aware instead of file-based: on pull_request runs, compare the test_* directory list at the PR's merge base against the PR result. A vanished suite fails the job unless its name appears in the PR title, PR body, or a commit message in the PR's range - a deliberate removal satisfies that by stating what it removes; an accidental loss cannot. Other events skip: they have no natural base, and PRs are where accidents arrive. No job depends on this one (a skipped job would skip its dependents). Incidentally: test/ currently holds 47 test_* directories while the deleted count file said 46 - the manual register had already drifted, which is exactly the bookkeeping failure mode this replaces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Re-pad the verdict table after shortening the AMBER row Shrinking the AMBER cell left the table's column padding inconsistent, which trunk (prettier + markdownlint MD060) rejects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
d765bd99ca
commit
af56a11f00
+7
-6
@@ -467,8 +467,8 @@ Unity suite, because what it asserts - the exit status and printed report of
|
||||
`meshtasticd --check`, and the fact that a normal run still refuses a bad config - are
|
||||
properties of the process, not of a linkable function. Fixtures live in
|
||||
`test/fixtures/portduino-config/` (see the README there); CI runs it in
|
||||
`test_native.yml`. It is not counted in `native-suite-count`, which only tracks `test_*`
|
||||
directories.
|
||||
`test_native.yml`. It is not a `test_*` directory, so it sits outside the suite count the
|
||||
harness derives from `test/`.
|
||||
|
||||
```bash
|
||||
pio run -e native && ./bin/test-config-check.sh
|
||||
@@ -476,10 +476,11 @@ pio run -e native && ./bin/test-config-check.sh
|
||||
|
||||
## Existing Test Suites
|
||||
|
||||
**This table is a description, not an inventory.** The canonical suite total lives in
|
||||
`test/native-suite-count`, is machine-checked against `test/test_*` on every full run and in CI
|
||||
(`test_native.yml`), and is the only number that should be trusted or quoted. Entries below carry
|
||||
per-suite descriptions the count cannot; do not infer completeness from the row count.
|
||||
**This table is a description, not an inventory.** The canonical suite total is the number of
|
||||
`test_*` directories under `test/`, detected on the fly by `bin/run-tests.sh` on every full run
|
||||
and cross-checked against the suites that actually ran. That derived count is the only number
|
||||
that should be trusted or quoted. Entries below carry per-suite descriptions the count cannot;
|
||||
do not infer completeness from the row count.
|
||||
|
||||
| Suite | Module Under Test |
|
||||
| ---------------------------- | ----------------------------- |
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
47
|
||||
Reference in New Issue
Block a user