diff --git a/.claude/commands/diagnose.md b/.claude/commands/diagnose.md index 749668956..d664f6312 100644 --- a/.claude/commands/diagnose.md +++ b/.claude/commands/diagnose.md @@ -49,11 +49,17 @@ Call the meshtastic MCP tool bundle and format a structured health report for on - Do the LoRa configs match? (region, channel_num, modem_preset should all agree; mismatch = no mesh) - Do the primary channel NAMES match? Mismatch = different PSK = no decode. -7. **Suggest next actions only for specific, recognisable failure modes**: +7. **Recorder slice (cheap, always available).** The mcp-server runs an autouse log recorder that's been collecting from every connected device. Pull two short slices to surface anything weird that's already happened: + - `mcp__meshtastic__logs_window(start="-2m", level="WARN|ERROR|CRIT", max_lines=20)` — recent firmware errors. If empty, say "no recent errors"; don't manufacture concern. + - `mcp__meshtastic__telemetry_timeline(window="1h", field="free_heap", max_points=60)` — heap trend. If `slope_per_min < -50`, flag it and recommend `/leakhunt window=6h` for a deeper read; otherwise just note the current free heap. + - If `recorder_status` shows `running:false` or `files.telemetry.last_ts` is null, note "recorder has no telemetry yet — enable `set_debug_log_api(True)` to populate" and skip this step gracefully. + +8. **Suggest next actions only for specific, recognisable failure modes**: - Stale PKI pubkey one-way → "run `/test tests/mesh/test_direct_with_ack.py` — the retry + nodeinfo-ping heals this in the test path." - Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`." - Device unreachable, reachable via DFU → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds AND the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`. - CP2102-wedged-driver on macOS → see the note in `run-tests.sh`. + - Heap slope strongly negative → "run `/leakhunt window=6h` for a full timeline + classification." ## What NOT to do diff --git a/.claude/commands/leakhunt.md b/.claude/commands/leakhunt.md new file mode 100644 index 000000000..ef90b133e --- /dev/null +++ b/.claude/commands/leakhunt.md @@ -0,0 +1,103 @@ +--- +description: Hunt for memory leaks (and other slow degradations) by reading the persistent recorder's heap timeline + log slice over a window +argument-hint: [window=1h] [field=free_heap] [variant=local] +--- + + + +# `/leakhunt` — read the recorder, classify a memory leak + +Use the always-on recorder (`mcp-server/.mtlog/`) to read a heap timeline plus the matching log slice and produce a one-page verdict: **steady / slow leak / fragmentation / OOM-imminent**. No firmware changes, no special build flags — the LocalStats telemetry packet that the firmware already broadcasts every ~60 s carries `heap_free_bytes` and `heap_total_bytes`. + +## Two signal paths — pick the right one + +| Path | Build flag | Cadence | Per-thread attribution | Cost | +| --------------------- | ---------------- | -------------- | ---------------------- | ------------------------- | +| LocalStats packet | (default) | ~60 s | No | Free — always on | +| `[heap N]` log prefix | `-DDEBUG_HEAP=1` | every log line | Yes (Thread X leaked) | Bigger flash + log volume | + +Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_HEAP is on, the recorder synthesizes telemetry rows from log prefixes (tagged `source: debug_heap`), so a single timeline call gets whichever signal is available. **For a slow leak diagnosis, the default path is plenty** (60 s cadence over 6 h = 360 points; linear regression over that nails sub-100-byte/min slopes). **DEBUG_HEAP is for attribution** — when the slope is real and you need to know which thread is leaking. + +## What to do + +1. **Parse `$ARGUMENTS`**: optional `window` (default `1h`, accepts `30m`/`6h`/`-3d`/etc.), optional `field` (default `free_heap`; alternates: `total_heap`, `battery_level`, anything in the LocalStats variant), optional `variant` (default `local`; alternates: `device`, `environment`, `power`, `airQuality`, `health`). + +2. **Verify the recorder is alive** — call `mcp__meshtastic__recorder_status`. Check: + - `running == True` + - `files.telemetry.lines > 0` (at least one telemetry packet recorded — if zero, the device hasn't broadcast LocalStats yet OR `set_debug_log_api` has never been on; tell the operator to run `mcp__meshtastic__set_debug_log_api(enabled=True)` and wait one device-update interval) + - `files.telemetry.last_ts` within the last 5 minutes (if older, the device is silent — log that, not "leak detected") + +3. **Detect whether DEBUG_HEAP is active** — `mcp__meshtastic__logs_window(start="-2m", grep=r"\\[heap \\d+\\]", max_lines=3)`. If any line matches, the firmware has the prefix → DEBUG_HEAP is on, expect higher-cadence data and `heap_event` rows. If zero matches over the last 2 minutes, you're on the LocalStats-only path. + +4. **Pull the timeline** — `mcp__meshtastic__telemetry_timeline(window=$window, variant=$variant, field=$field, max_points=200)`. Read: + - `samples` — how many raw points contributed + - `min`, `max` — total swing + - `slope_per_min` — units per minute (linear regression over the whole window) + +5. **Pull the log context for the same window** — `mcp__meshtastic__logs_window(start="-${window}", grep="Heap status|leaked heap|freed heap|out of memory|Alloc an err|panic|abort", max_lines=200)`. These are the strings the firmware emits when something memory-related happens (`DEBUG_HEAP` builds emit `"Heap status:"` and `"leaked heap"` lines; production builds emit `"Alloc an err"` on failure and `"out of memory"` on OOM). + +6. **Pull marker events** so we know if the operator labeled phases — `mcp__meshtastic__events_window(start="-${window}", kind="mark|connection_lost|connection_established")`. If a `connection_lost` overlaps a sharp drop, that's not a leak; that's a reboot. + +6a. **(DEBUG_HEAP only) Per-thread attribution** — `mcp__meshtastic__logs_window(start="-${window}", grep="leaked heap", max_lines=200)`. Each row has a structured `heap_event` field with `{kind, thread, before, after, delta}`. Aggregate by thread: sum the `delta` over the window per thread name. The thread with the largest cumulative negative delta is your suspect. Note the count too — a thread with 50× small leaks is different from 1× big leak. + +7. **Classify** based on what the data says, NOT on what you wish it said. Use these rules in order: + - **Insufficient data** (< 5 samples): say so. Suggest a longer window or longer wait. Stop. + - **Reboot mid-window**: if any `connection_lost` event is present AND `free_heap` jumped UP at that timestamp, the device rebooted. Note it; pre-reboot trend may be a leak but you only have part of the curve. + - **OOM-imminent**: any `Alloc an err=` or `out of memory` line in the log slice. This trumps everything; flag urgently. + - **Slow leak**: `slope_per_min < -50` AND `max - min > 1000` AND no reboot. The heap is monotonically (or near-monotonically) declining. Estimate time-to-zero: `min / -slope_per_min` minutes. Surface it. + - **Fragmentation suspect**: `slope_per_min` close to zero (|x| < 50) BUT min trends down across the window AND the log slice shows `Alloc an err` warnings WITHOUT total OOM. Means free total is OK but largest contiguous block is shrinking. Recommend a `DEBUG_HEAP` build to confirm. + - **Steady**: |slope_per_min| < 50, no error lines. Heap is fine. + - **Recovery curve**: slope is POSITIVE — heap recovered. Either a workload completed or GC fired. Note it; not a leak. + +8. **Report**: + + ```text + /leakhunt window=6h field=free_heap variant=local + ──────────────────────────────────────────────────── + recorder : running, telem last_ts 8s ago + build : DEBUG_HEAP=ON (per-line prefix detected) + samples : 14,200 over 6h (cadence ~1.5s, log-line synth) + free_heap : min 92,344 / max 124,008 / range 31,664 + slope : -82 bytes/min (negative — heap declining) + reboots : none in window + OOM events : none + error lines : 3× "Alloc an err=ESP_ERR_NO_MEM" at +4h12m, +5h08m, +5h44m + thread leaks : (DEBUG_HEAP) MeshPacket -3,124 B over 18 events + Router -1,408 B over 4 events + others -240 B + verdict : SLOW LEAK — primary suspect MeshPacket thread + est. time-to-OOM: ~1,127 min (~18.8 h) at current slope + evidence : (3 log line citations with uptimes) + ``` + + Then: **what to do next.** + - SLOW LEAK, **DEBUG_HEAP off** → recommend rebuilding with the flag and re-running this skill. Concrete one-liner the operator can copy: + ```text + mcp__meshtastic__build(env="", build_flags={"DEBUG_HEAP": 1}) + mcp__meshtastic__pio_flash(env="", port="", confirm=True) + ``` + After flash, set debug_log_api back on and wait one window; re-run `/leakhunt`. + - SLOW LEAK, **DEBUG_HEAP on** → cite the top-leaking thread name from step 6a. Point at the corresponding source file (`grep -rn "ThreadName(\"\")" src/`); the operator decides what to fix. + - FRAGMENTATION SUSPECT → propose pre-allocating any per-packet buffers; or rebuilding with `CONFIG_HEAP_TASK_TRACKING=y` on ESP32 to see who's holding the largest blocks. + - OOM-IMMINENT → flag for immediate attention; don't wait for the next telemetry interval. + - STEADY → say so; stop. Don't invent problems. + +## What NOT to do + +- Don't assume a leak from a single dip. LocalStats fires every ~60 s and the firmware naturally allocates+frees on each broadcast cycle; one packet sees the trough. Look at the slope, not the deltas. +- Don't recommend code changes. This skill diagnoses; the operator decides what to fix. +- Don't enable `set_debug_log_api` automatically — if it's off, telemetry isn't reaching pubsub anyway, and the recorder will be empty. Tell the operator to flip it on and wait, then re-run. +- Don't run heavy workloads to "trigger the leak." The recorder is passive; we read what's there. + +## Companion: `mark_event` for stress runs + +If the operator wants to test under stimulus (e.g. blast 50 broadcasts and see what the heap does), they can frame the experiment with markers: + +```text +mark_event("burst-start") +… run the workload … +mark_event("burst-end") +/leakhunt window=15m +``` + +The markers land in both `events.jsonl` and `logs.jsonl`, so the report can show "free_heap dipped 8 KB during the burst window, recovered to baseline within 2 LocalStats cycles" → not a leak. diff --git a/.claude/commands/repro.md b/.claude/commands/repro.md index c5f466ce6..84513e45b 100644 --- a/.claude/commands/repro.md +++ b/.claude/commands/repro.md @@ -3,6 +3,8 @@ description: Re-run a specific test N times in isolation to triage flakes, diff argument-hint: [count=5] --- + + # `/repro` — flakiness triage for one test Re-run a single pytest node ID N times in isolation, track pass rate, and surface what's _different_ in the firmware logs between the passing attempts and the failing ones. Turns "it's flaky, I guess" into "it fails when X, passes when Y." @@ -40,6 +42,8 @@ Re-run a single pytest node ID N times in isolation, track pass rate, and surfac Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs — pull specific lines with uptime timestamps. +5a. **Archive recorder slices per attempt** (no extra device interaction; the recorder runs autouse). Right after each attempt finishes, capture its `(start_ts, end_ts)` and call `mcp__meshtastic__recorder_export(start=, end=, dest_dir="mcp-server/tests/repro_artifacts//attempt_/")`. This drops a `logs.jsonl`, `telemetry.jsonl`, `packets.jsonl`, and `events.jsonl` snapshot scoped to the attempt window. Use these for cross-attempt diffs in step 5: `jq '.line' logs.jsonl` is faster than re-running the test, and the telemetry slice lets you compare heap behavior across attempts. + 6. **Classify the flake** into one of: - **LoRa airtime collision** → pass rate improves with fewer concurrent transmitters; propose a `time.sleep` gap or retry bump in the test body. - **PKI key staleness** → fails on first attempt, passes after self-heal; existing retry loop in `test_direct_with_ack.py` handles this. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index fe9af4359..bac47853d 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -135,6 +135,95 @@ On top of authorization, any remote admin message that **mutates** state (not a - **Channel 0 PSK change** → every peer must re-learn the channel hash; cached NodeInfo becomes temporarily unreachable until the next broadcast. - **`security.private_key` blanked via admin** → regenerates both halves (unless in Ham mode) and propagates the new public key via NodeInfo. +## NodeDB Layout (v25) + +`DEVICESTATE_CUR_VER = 25`, `DEVICESTATE_MIN_VER = 24`. The on-device NodeDB was split in v25 into a slim header table plus four optional satellite stores. Older v24 saves auto-migrate at boot. Old training-data instincts (`node->user.long_name`, `node->position.latitude_i`, `node->is_favorite`, `node->device_metrics.battery_level`) are wrong now — the fields aren't there. Read this section before touching anything that walks `nodeDB->meshNodes`. + +### Slim `NodeInfoLite` + +`UserLite` is flattened onto `NodeInfoLite` (no nested sub-message); `position` and `device_metrics` are removed entirely (tags reserved). MAC address is dropped. Long names are capped at 25 chars (`max_size:25` in `deviceonly.options`); `hw_model` and `role` are `int_size:8`. Encoded size dropped from ~166 B → ~105 B per node. + +Booleans are bit-packed into `NodeInfoLite.bitfield`. **Do not read or write the bits directly** — use the inline helpers in `src/mesh/NodeDB.h`: + +```cpp +nodeInfoLiteHasUser(n) // bit 5 — user fields populated +nodeInfoLiteIsFavorite(n) // bit 3 +nodeInfoLiteIsIgnored(n) // bit 4 +nodeInfoLiteIsMuted(n) // bit 1 +nodeInfoLiteIsLicensed(n) // bit 6 — Ham mode peer +nodeInfoLiteIsKeyManuallyVerified(n) // bit 0 +nodeInfoLiteHasIsUnmessagable(n) // bit 8 — "is_unmessagable was sent" +nodeInfoLiteIsUnmessagable(n) // bit 7 +// via_mqtt is bit 2 (mask exposed; predicate uses the mask directly) + +nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); // setter +``` + +### Satellite stores + +Four `std::unordered_map` members on `NodeDB`, each gated by its own build flag: + +| Map | Value type | Build flag | +| ----------------- | ------------------------------- | ---------------------------------- | +| `nodePositions` | `meshtastic_PositionLite` | `MESHTASTIC_EXCLUDE_POSITIONDB` | +| `nodeTelemetry` | `meshtastic_DeviceMetrics` | `MESHTASTIC_EXCLUDE_TELEMETRYDB` | +| `nodeEnvironment` | `meshtastic_EnvironmentMetrics` | `MESHTASTIC_EXCLUDE_ENVIRONMENTDB` | +| `nodeStatus` | `meshtastic_StatusMessage` | `MESHTASTIC_EXCLUDE_STATUSDB` | + +Defaults are ON (i.e., maps **excluded**) for STM32WL only — see `src/mesh/mesh-pb-constants.h`. On every other arch all four maps are present. When excluded, the map member is absent and the corresponding accessors return `false`. + +All four maps are guarded by **`mutable concurrency::Lock satelliteMutex`** — concurrent access from receive threads, the phone API state machine, and the renderer is the rule, not the exception. + +### Accessor convention + +**Never hand out pointers into the maps.** Use the copy-out accessors on `NodeDB`: + +```cpp +bool copyNodePosition(NodeNum, meshtastic_PositionLite &out) const; +bool copyNodeTelemetry(NodeNum, meshtastic_DeviceMetrics &out) const; +bool copyNodeEnvironment(NodeNum, meshtastic_EnvironmentMetrics &out) const; +bool copyNodeStatus(NodeNum, meshtastic_StatusMessage &out) const; +``` + +Each takes the lock, copies the value if present, returns `false` if the entry is absent or the DB is excluded. Pass-by-out-param is deliberate — pointer-style accessors would invite UAF and lock-leak bugs across the renderer. The "has any X" convenience predicates (`hasValidPosition` etc.) are implemented in terms of these. + +Writers go through `setNodeStatus`, `updatePosition`, `updateTelemetry` (which dispatches on `which_variant` for device vs environment metrics) — these own the lock and the eviction hooks. + +### Eviction + +Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly. + +### Gradient sync (opt-in via special nonces) + +`client_capabilities` is **not** a thing in this branch. Phone clients opt into the new sync flow by sending one of two values in the `ToRadio.want_config_id`: + +- `SPECIAL_NONCE_GRADIENT_SYNC` (69422) — full config + thin NodeInfo + replay phases. +- `SPECIAL_NONCE_GRADIENT_ONLY_NODES` (69423) — skip config segments, NodeInfo + replay only. + +`PhoneAPI::clientWantsGradientSync()` is the single switch. When true, `STATE_SEND_OTHER_NODEINFOS` is followed by: + +```text +STATE_REPLAY_POSITIONS → STATE_REPLAY_TELEMETRY → STATE_REPLAY_ENVIRONMENT → STATE_REPLAY_STATUS +``` + +Each replay phase walks the corresponding satellite map and emits synthetic `MeshPacket`s on the matching portnum (`POSITION_APP`, `TELEMETRY_APP` for both device + environment variants, `STATUS_MESSAGE_APP`). Legacy clients (no special nonce) get the bundled-NodeInfo path with position/device_metrics joined back in by `ConvertToNodeInfo(lite, pos*, dm*)` — wire bytes are byte-identical to pre-v25 for them. + +`ConvertToNodeInfoThin(lite)` is the gradient-sync emitter (no position/telemetry). + +### v24 → v25 migration + +The legacy migration code lives in **`src/mesh/NodeDBLegacyMigration.cpp`**, not in `NodeDB.cpp`. It owns the `meshtastic_NodeDatabase_Legacy` callback and `NodeDB::migrateLegacyNodeDatabase()`. The legacy proto descriptor is `protobufs/meshtastic/deviceonly_legacy.proto` (only included by the migration TU). The boot path peeks the file's leading version tag, runs the migration if `version < 25`, then re-saves in v25 layout. The legacy descriptor is scheduled for removal once `DEVICESTATE_MIN_VER` is bumped. + +### Read-site rules of thumb + +- Never `node->position.X` / `node->device_metrics.X` — those fields no longer exist. Pull from the satellite map via `copyNodePosition` / `copyNodeTelemetry`. +- Never `node->user.long_name` — `long_name`, `short_name`, `public_key`, `hw_model`, `role`, `macaddr` (gone), `is_licensed`, `is_unmessagable` are flat on `NodeInfoLite`. +- Never `node->is_favorite` / `node->is_ignored` / `node->via_mqtt` / `node->is_key_manually_verified` — use the bitfield helpers. +- Never assume `nodeDB->getMeshNode(num)->position.time` — call `copyNodePosition` and check the return. +- Don't lock `satelliteMutex` yourself in renderer code; the copy-out accessors already do. + +Unit tests for the conversion layer live in `test/test_type_conversions/test_main.cpp` (Unity) — bitfield round-trips, `long_name` truncation, thin-vs-full conversions. Add cases there when extending the schema. + ## Project Structure ``` @@ -196,6 +285,8 @@ firmware/ - Prefer `LOG_DEBUG`, `LOG_INFO`, `LOG_WARN`, `LOG_ERROR` for logging - Use `assert()` for invariants that should never fail - C++17 features are available (`std::optional`, structured bindings, `if constexpr`, etc.) +- **Keep code comments minimal — one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. +- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check — raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. ### Naming Conventions diff --git a/.trunk/trunk.yaml b/.trunk/trunk.yaml index d2ccc60a4..00649f7ff 100644 --- a/.trunk/trunk.yaml +++ b/.trunk/trunk.yaml @@ -4,11 +4,11 @@ cli: plugins: sources: - id: trunk - ref: v1.8.0 + ref: v1.9.0 uri: https://github.com/trunk-io/plugins lint: enabled: - - checkov@3.2.526 + - checkov@3.2.528 - renovate@43.150.0 - prettier@3.8.3 - trufflehog@3.95.2 diff --git a/AGENTS.md b/AGENTS.md index cdccda1f4..82912f252 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,8 @@ Key rotation to never trigger casually: only the **full** factory reset (`factor - **Don't speculate about firmware root causes.** When evidence doesn't support a classification, say "unknown" and list what would disambiguate. - **Run `trunk fmt` before proposing a commit.** The `trunk_check` CI gate will reject unformatted code. - **`confirm=True` on destructive MCP tools is a real gate, not a formality.** Don't bypass it via auto-approve settings. +- **Keep code comments minimal — one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. +- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check — raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. ## Typical agent workflows diff --git a/bin/org.meshtastic.meshtasticd.metainfo.xml b/bin/org.meshtastic.meshtasticd.metainfo.xml index a1690186b..ed5338af6 100644 --- a/bin/org.meshtastic.meshtasticd.metainfo.xml +++ b/bin/org.meshtastic.meshtasticd.metainfo.xml @@ -87,6 +87,9 @@ + + https://github.com/meshtastic/firmware/releases?q=tag%3Av2.7.24 + https://github.com/meshtastic/firmware/releases?q=tag%3Av2.7.23 diff --git a/debian/changelog b/debian/changelog index c3f1424a5..6b9d0668e 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +meshtasticd (2.7.24.0) unstable; urgency=medium + + * Version 2.7.24 + + -- GitHub Actions Fri, 08 May 2026 10:44:12 +0000 + meshtasticd (2.7.23.0) unstable; urgency=medium * Version 2.7.23 diff --git a/mcp-server/.gitignore b/mcp-server/.gitignore index 4cc892b2a..744a4401d 100644 --- a/mcp-server/.gitignore +++ b/mcp-server/.gitignore @@ -7,6 +7,12 @@ __pycache__/ dist/ build/ +# Persistent device-log capture (recorder + Datadog cursor). +# Cross-session JSONL streams written by the autouse Recorder singleton +# (see src/meshtastic_mcp/recorder/). Lives outside tests/ so the pytest +# fixture truncate doesn't touch it. +.mtlog/ + # Test harness artifacts tests/report.html tests/junit.xml diff --git a/mcp-server/scripts/datadog-dashboard.json b/mcp-server/scripts/datadog-dashboard.json new file mode 100644 index 000000000..73aa35201 --- /dev/null +++ b/mcp-server/scripts/datadog-dashboard.json @@ -0,0 +1,217 @@ +{ + "title": "Meshtastic Firmware — Recorder Stream", + "description": "Live view of `.mtlog/` streams shipped by `mtlog_to_datadog.py`. Heap, packet volume, log levels, errors. One row per port.", + "widgets": [ + { + "definition": { + "title": "Free heap (bytes)", + "type": "timeseries", + "show_legend": true, + "requests": [ + { + "queries": [ + { + "name": "free_heap", + "data_source": "metrics", + "query": "avg:mesh.local.heap_free_bytes{service:meshtastic-firmware} by {port}" + } + ], + "response_format": "timeseries", + "display_type": "line" + } + ], + "yaxis": { "label": "bytes" } + } + }, + { + "definition": { + "title": "Heap slope (bytes/min) — last 1h", + "type": "query_value", + "precision": 0, + "requests": [ + { + "queries": [ + { + "name": "slope", + "data_source": "metrics", + "query": "derivative(avg:mesh.local.heap_free_bytes{service:meshtastic-firmware})", + "aggregator": "avg" + } + ], + "response_format": "scalar" + } + ], + "conditional_formats": [ + { "comparator": "<", "value": -100, "palette": "white_on_red" }, + { "comparator": "<", "value": 0, "palette": "white_on_yellow" }, + { "comparator": ">=", "value": 0, "palette": "white_on_green" } + ] + } + }, + { + "definition": { + "title": "Total heap (bytes)", + "type": "timeseries", + "requests": [ + { + "queries": [ + { + "name": "total_heap", + "data_source": "metrics", + "query": "avg:mesh.local.heap_total_bytes{service:meshtastic-firmware} by {port}" + } + ], + "response_format": "timeseries", + "display_type": "line" + } + ] + } + }, + { + "definition": { + "title": "Battery level (%)", + "type": "timeseries", + "requests": [ + { + "queries": [ + { + "name": "battery", + "data_source": "metrics", + "query": "avg:mesh.device.battery_level{service:meshtastic-firmware} by {port}" + } + ], + "response_format": "timeseries", + "display_type": "line" + } + ], + "yaxis": { "min": "0", "max": "105" } + } + }, + { + "definition": { + "title": "Air utilization (TX %)", + "type": "timeseries", + "requests": [ + { + "queries": [ + { + "name": "airutil", + "data_source": "metrics", + "query": "avg:mesh.device.air_util_tx{service:meshtastic-firmware} by {port}" + } + ], + "response_format": "timeseries", + "display_type": "line" + } + ] + } + }, + { + "definition": { + "title": "Channel utilization (%)", + "type": "timeseries", + "requests": [ + { + "queries": [ + { + "name": "chutil", + "data_source": "metrics", + "query": "avg:mesh.device.channel_utilization{service:meshtastic-firmware} by {port}" + } + ], + "response_format": "timeseries", + "display_type": "line" + } + ] + } + }, + { + "definition": { + "title": "Log volume by level", + "type": "timeseries", + "show_legend": true, + "requests": [ + { + "response_format": "timeseries", + "display_type": "bars", + "queries": [ + { + "name": "log_count", + "data_source": "logs", + "indexes": ["*"], + "compute": { "aggregation": "count" }, + "search": { "query": "service:meshtastic-firmware" }, + "group_by": [ + { + "facet": "@level", + "limit": 10, + "sort": { "order": "desc", "aggregation": "count" } + } + ] + } + ] + } + ] + } + }, + { + "definition": { + "title": "Recent ERROR / CRIT firmware logs", + "type": "list_stream", + "requests": [ + { + "response_format": "event_list", + "query": { + "data_source": "logs_stream", + "query_string": "service:meshtastic-firmware (status:error OR @level:ERROR OR @level:CRIT)", + "indexes": [], + "sort": { "column": "timestamp", "order": "desc" } + }, + "columns": [ + { "field": "timestamp", "width": "auto" }, + { "field": "host", "width": "auto" }, + { "field": "@port", "width": "auto" }, + { "field": "@level", "width": "auto" }, + { "field": "@thread", "width": "auto" }, + { "field": "message", "width": "stretch" } + ] + } + ] + } + }, + { + "definition": { + "title": "Recorder marker events", + "type": "list_stream", + "requests": [ + { + "response_format": "event_list", + "query": { + "data_source": "logs_stream", + "query_string": "service:meshtastic-firmware @level:MARK", + "indexes": [], + "sort": { "column": "timestamp", "order": "desc" } + }, + "columns": [ + { "field": "timestamp", "width": "auto" }, + { "field": "host", "width": "auto" }, + { "field": "message", "width": "stretch" } + ] + } + ] + } + } + ], + "template_variables": [ + { + "name": "port", + "prefix": "port", + "available_values": [], + "default": "*" + }, + { "name": "host", "prefix": "host", "available_values": [], "default": "*" } + ], + "layout_type": "ordered", + "notify_list": [], + "reflow_type": "auto" +} diff --git a/mcp-server/scripts/mtlog_to_datadog.py b/mcp-server/scripts/mtlog_to_datadog.py new file mode 100755 index 000000000..51496adc4 --- /dev/null +++ b/mcp-server/scripts/mtlog_to_datadog.py @@ -0,0 +1,389 @@ +#!/usr/bin/env python3 +"""Forward selected recorder JSONL streams to Datadog. + +Reads `.mtlog/logs.jsonl` and `.mtlog/telemetry.jsonl`, ships logs to the +Logs Intake API and telemetry numerics to the Metrics v2 series API. +Resumes from `.mtlog/.dd-cursor.json` so a daemon restart doesn't +duplicate rows already shipped from the current live files. + +This forwarder does not currently backfill rotated `.jsonl.gz` archives. +If the recorder rotates before this process drains the live file, or the +forwarder is down across a rotation, those older rows are skipped. + +Usage: + DD_API_KEY=... ./scripts/mtlog_to_datadog.py --tail + ./scripts/mtlog_to_datadog.py --once # catch up + exit + ./scripts/mtlog_to_datadog.py --since 3600 # backfill last hour from start + +Default `DD_SITE` is `us5.datadoghq.com` — the team's Datadog instance. +Override via `DD_SITE=...` env var or `--site` flag for one-offs. + +The forwarder is a separate process by design — a Datadog outage or +auth failure must not backpressure the recorder. We exit non-zero on +fatal config errors (missing API key) and keep retrying on transient +network/HTTP errors. +""" + +from __future__ import annotations + +import argparse +import json +import os +import socket +import sys +import time +from pathlib import Path +from typing import Any, Iterator + +try: + import requests +except ImportError: + print( + "requests is required. Install it in the mcp-server venv: " + "uv pip install requests", + file=sys.stderr, + ) + sys.exit(2) + + +_DEFAULT_LOG_DIR = Path(__file__).resolve().parents[1] / ".mtlog" +_LOG_INTAKE_TPL = "https://http-intake.logs.{site}/api/v2/logs" +_METRICS_TPL = "https://api.{site}/api/v2/series" +_LOG_BATCH = 50 +_METRICS_BATCH = 100 +_MAX_RETRIES = 5 +_RETRY_BASE_S = 1.5 + + +# --- streaming JSONL with byte-position cursor ------------------------- + + +class _StreamReader: + """Reads a single rotating JSONL with cursor-based resume. + + This tails only the live `.jsonl` file. The recorder rotates files + (live `.jsonl` → `.YYYYMMDD-HHMMSS-uuuuuu-NNNNN.jsonl.gz`), which means + the live file shrinks abruptly. We detect that via inode change OR live + size < cursor position, and reset the live-file cursor to 0. + """ + + def __init__(self, path: Path, cursor: dict[str, Any]): + self.path = path + self.cursor = cursor + + def _state(self) -> tuple[int, int]: + """Return (inode, size) for the live file. (0, 0) if missing.""" + try: + st = self.path.stat() + return (st.st_ino, st.st_size) + except FileNotFoundError: + return (0, 0) + + def iter_new_records(self) -> Iterator[dict[str, Any]]: + ino, size = self._state() + last_ino = self.cursor.get("ino") + last_pos = int(self.cursor.get("pos") or 0) + if ino == 0: + return + if last_ino is not None and last_ino != ino: + # Rotation happened. Start over. + last_pos = 0 + if last_pos > size: + # Live file truncated/shrunk under us — recorder rotated. + last_pos = 0 + try: + with self.path.open("r", encoding="utf-8") as fh: + fh.seek(last_pos) + for line in fh: + line = line.rstrip("\n") + if not line: + continue + try: + yield json.loads(line) + except json.JSONDecodeError: + continue + last_pos = fh.tell() + except FileNotFoundError: + return + self.cursor["ino"] = ino + self.cursor["pos"] = last_pos + + +def _load_cursor(path: Path) -> dict[str, Any]: + if not path.exists(): + return {} + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return {} + + +def _save_cursor(path: Path, data: dict[str, Any]) -> None: + tmp = path.with_suffix(".json.tmp") + tmp.write_text(json.dumps(data, separators=(",", ":"))) + tmp.replace(path) + + +# --- Datadog clients --------------------------------------------------- + + +class _DDSession: + """Pool one HTTPS session, share retry logic.""" + + def __init__(self, api_key: str, site: str, hostname: str) -> None: + self.api_key = api_key + self.site = site + self.hostname = hostname + self.session = requests.Session() + self.session.headers.update( + { + "DD-API-KEY": api_key, + "Content-Type": "application/json", + } + ) + + def _post(self, url: str, payload: Any) -> bool: + for attempt in range(_MAX_RETRIES): + try: + resp = self.session.post(url, json=payload, timeout=30) + except requests.RequestException as e: + _wait_retry(attempt, f"network error: {e}") + continue + if 200 <= resp.status_code < 300: + return True + if resp.status_code in (408, 429, 500, 502, 503, 504): + _wait_retry( + attempt, + f"HTTP {resp.status_code} retrying", + ) + continue + print( + f"datadog refused: {resp.status_code} {resp.text[:200]}", + file=sys.stderr, + ) + return False + return False + + def send_logs(self, records: list[dict[str, Any]]) -> int: + if not records: + return 0 + url = _LOG_INTAKE_TPL.format(site=self.site) + sent = 0 + for i in range(0, len(records), _LOG_BATCH): + batch = records[i : i + _LOG_BATCH] + if self._post(url, batch): + sent += len(batch) + return sent + + def send_metrics(self, series: list[dict[str, Any]]) -> int: + if not series: + return 0 + url = _METRICS_TPL.format(site=self.site) + sent = 0 + for i in range(0, len(series), _METRICS_BATCH): + batch = series[i : i + _METRICS_BATCH] + if self._post(url, {"series": batch}): + sent += len(batch) + return sent + + +def _wait_retry(attempt: int, reason: str) -> None: + wait = _RETRY_BASE_S * (2**attempt) + print( + f" retry {attempt + 1}/{_MAX_RETRIES} in {wait:.1f}s ({reason})", + file=sys.stderr, + ) + time.sleep(wait) + + +# --- record → datadog payload ------------------------------------------ + + +def _log_record_to_dd(rec: dict[str, Any], host: str) -> dict[str, Any]: + line = rec.get("line") or "" + tags = [ + f"role:{rec.get('role')}", + f"port:{rec.get('port')}", + ] + level = rec.get("level") + if level: + tags.append(f"level:{level}") + tag = rec.get("tag") + if tag: + tags.append(f"thread:{tag}") + return { + "ddsource": "meshtastic-firmware", + "service": "meshtastic-firmware", + "hostname": host, + "message": line, + "ddtags": ",".join(t for t in tags if t and "None" not in t), + "timestamp": int((rec.get("ts") or time.time()) * 1000), + "level": level, + } + + +def _telemetry_record_to_metrics( + rec: dict[str, Any], host: str +) -> list[dict[str, Any]]: + fields = rec.get("fields") or {} + if not isinstance(fields, dict): + return [] + variant = rec.get("variant") or "unknown" + ts = int(rec.get("ts") or time.time()) + out: list[dict[str, Any]] = [] + tags = [] + if rec.get("port"): + tags.append(f"port:{rec['port']}") + if rec.get("role"): + tags.append(f"role:{rec['role']}") + if rec.get("from_node"): + tags.append(f"from_node:{rec['from_node']}") + tags.append(f"variant:{variant}") + for field, value in fields.items(): + if not isinstance(value, (int, float)) or isinstance(value, bool): + continue + metric = f"mesh.{variant}.{_metric_safe(field)}" + out.append( + { + "metric": metric, + "type": 3, # GAUGE + "points": [{"timestamp": ts, "value": float(value)}], + "tags": tags, + "resources": [{"type": "host", "name": host}], + } + ) + return out + + +def _metric_safe(name: str) -> str: + # Lowercase, replace non-alnum with underscore for safe metric names. + return "".join(c.lower() if c.isalnum() else "_" for c in name) + + +# --- main loop --------------------------------------------------------- + + +def run( + log_dir: Path, + *, + once: bool, + since_seconds: float | None, + poll_interval: float, + dd: _DDSession, +) -> int: + cursor_path = log_dir / ".dd-cursor.json" + cursors = _load_cursor(cursor_path) + + # `--since` overrides cursor: rewind to (now-since) timestamp. + # We can't seek by timestamp directly (cursor is byte position), so + # we just reset cursors to 0 and let the time filter in iter_new + # drop older records. + cutoff_ts: float | None = None + if since_seconds is not None: + cursors = {} + cutoff_ts = time.time() - since_seconds + + sent_total = {"logs": 0, "telemetry": 0} + + while True: + # logs.jsonl → DD logs + log_cursor = cursors.setdefault("logs", {}) + log_batch: list[dict[str, Any]] = [] + for rec in _StreamReader(log_dir / "logs.jsonl", log_cursor).iter_new_records(): + if cutoff_ts and (rec.get("ts") or 0) < cutoff_ts: + continue + log_batch.append(_log_record_to_dd(rec, dd.hostname)) + if log_batch: + n = dd.send_logs(log_batch) + sent_total["logs"] += n + print(f"logs: sent {n}/{len(log_batch)}") + + # telemetry.jsonl → DD metrics + telem_cursor = cursors.setdefault("telemetry", {}) + metric_series: list[dict[str, Any]] = [] + for rec in _StreamReader( + log_dir / "telemetry.jsonl", telem_cursor + ).iter_new_records(): + if cutoff_ts and (rec.get("ts") or 0) < cutoff_ts: + continue + metric_series.extend(_telemetry_record_to_metrics(rec, dd.hostname)) + if metric_series: + n = dd.send_metrics(metric_series) + sent_total["telemetry"] += n + print(f"telemetry: sent {n}/{len(metric_series)} metric points") + + _save_cursor(cursor_path, cursors) + + if once: + print(f"done. logs={sent_total['logs']} metrics={sent_total['telemetry']}") + return 0 + time.sleep(poll_interval) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--log-dir", + default=str(_DEFAULT_LOG_DIR), + help="Path to .mtlog/ (default: mcp-server/.mtlog)", + ) + mode = parser.add_mutually_exclusive_group() + mode.add_argument("--once", action="store_true", help="Catch up then exit") + mode.add_argument( + "--tail", + action="store_true", + help="Daemon: poll forever (default)", + ) + parser.add_argument( + "--since", + type=float, + default=None, + help="Backfill last N seconds. Resets cursor.", + ) + parser.add_argument( + "--poll-interval", + type=float, + default=5.0, + help="Seconds between tail polls (default 5)", + ) + parser.add_argument( + "--site", + default=os.environ.get("DD_SITE", "us5.datadoghq.com"), + help=( + "Datadog site. Default is the team's instance (us5.datadoghq.com). " + "Override via DD_SITE env var or this flag." + ), + ) + parser.add_argument( + "--host", + default=socket.gethostname(), + help="Hostname tag (default: socket.gethostname())", + ) + args = parser.parse_args(argv) + + api_key = os.environ.get("DD_API_KEY") + if not api_key: + print("DD_API_KEY env var required.", file=sys.stderr) + return 2 + + log_dir = Path(args.log_dir) + if not log_dir.exists(): + print( + f"log dir {log_dir} does not exist — start the mcp-server first.", + file=sys.stderr, + ) + return 2 + + dd = _DDSession(api_key=api_key, site=args.site, hostname=args.host) + once = args.once and not args.tail + return run( + log_dir, + once=once, + since_seconds=args.since, + poll_interval=args.poll_interval, + dd=dd, + ) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/mcp-server/src/meshtastic_mcp/flash.py b/mcp-server/src/meshtastic_mcp/flash.py index e11197d5f..debc2cf91 100644 --- a/mcp-server/src/meshtastic_mcp/flash.py +++ b/mcp-server/src/meshtastic_mcp/flash.py @@ -108,18 +108,33 @@ def build( env: str, with_manifest: bool = True, userprefs_overrides: dict[str, Any] | None = None, + build_flags: dict[str, Any] | None = None, ) -> dict[str, Any]: """Run `pio run -e ` and return artifact paths. `userprefs_overrides` (optional): dict of `USERPREFS_: value` to inject into userPrefs.jsonc for this build only. File is restored byte-for-byte on exit. Use `userprefs_set()` for persistent changes. + + `build_flags` (optional): dict of `-D=` macros to set for + this build only via `PLATFORMIO_BUILD_FLAGS`. Common useful flag: + `{"DEBUG_HEAP": 1}` enables per-thread leak detection + `[heap N]` + prefix on every log line. Combines with the recorder so heap shows + up at log cadence (much higher resolution than the ~60 s LocalStats + packet) — see `recorder/parsers.py:_HEAP_PREFIX_RE`. Bool values + expand to bare `-D` (presence-only flags). """ args = ["run", "-e", env] if with_manifest: args.extend(["-t", "mtjson"]) + extra_env = _build_flags_env(build_flags) if build_flags else None with userprefs.temporary_overrides(userprefs_overrides) as effective: - result = pio.run(args, timeout=pio.TIMEOUT_BUILD, check=False) + result = pio.run( + args, + timeout=pio.TIMEOUT_BUILD, + check=False, + extra_env=extra_env, + ) return { "exit_code": result.returncode, "artifacts": [str(p) for p in _artifacts_for(env)], @@ -127,9 +142,27 @@ def build( "stderr_tail": pio.tail_lines(result.stderr, 200), "duration_s": round(result.duration_s, 2), "userprefs": _userprefs_summary(effective), + "build_flags": dict(build_flags) if build_flags else None, } +def _build_flags_env(build_flags: dict[str, Any]) -> dict[str, str]: + """Translate `{"DEBUG_HEAP": 1, "FOO": "bar"}` → `{"PLATFORMIO_BUILD_FLAGS": + "-DDEBUG_HEAP=1 -DFOO=bar"}`. Bool True → bare `-D`; False/None drop + the flag entirely. Other types stringify.""" + parts: list[str] = [] + for key, value in build_flags.items(): + if value is False or value is None: + continue + if value is True: + parts.append(f"-D{key}") + else: + parts.append(f"-D{key}={value}") + if not parts: + return {} + return {"PLATFORMIO_BUILD_FLAGS": " ".join(parts)} + + def clean(env: str) -> dict[str, Any]: """Run `pio run -e -t clean`.""" result = pio.run(["run", "-e", env, "-t", "clean"], timeout=120, check=False) @@ -146,20 +179,29 @@ def flash( port: str, confirm: bool = False, userprefs_overrides: dict[str, Any] | None = None, + build_flags: dict[str, Any] | None = None, ) -> dict[str, Any]: """`pio run -e -t upload --upload-port `. All architectures. `userprefs_overrides` (optional): see `build()` — the rebuild-before-upload that pio performs will pick up the injected values. + + `build_flags` (optional): same shape as `build()` — `PLATFORMIO_BUILD_FLAGS` + is exported for the rebuild-before-upload, so the uploaded firmware + actually carries the flags. Without this propagation, `pio run -t upload` + would relink without the env var and silently drop them. Common use: + `build_flags={"DEBUG_HEAP": 1}` for the leak-hunt path. """ _require_confirm(confirm, "flash") _reject_native_env(env, "flash") connection.reject_if_tcp(port, "flash") + extra_env = _build_flags_env(build_flags) if build_flags else None with userprefs.temporary_overrides(userprefs_overrides) as effective: result = pio.run( ["run", "-e", env, "-t", "upload", "--upload-port", port], timeout=pio.TIMEOUT_UPLOAD, check=False, + extra_env=extra_env, ) return { "exit_code": result.returncode, @@ -167,6 +209,7 @@ def flash( "stderr_tail": pio.tail_lines(result.stderr, 200), "duration_s": round(result.duration_s, 2), "userprefs": _userprefs_summary(effective), + "build_flags": dict(build_flags) if build_flags else None, } diff --git a/mcp-server/src/meshtastic_mcp/log_query.py b/mcp-server/src/meshtastic_mcp/log_query.py new file mode 100644 index 000000000..0f3ad7b69 --- /dev/null +++ b/mcp-server/src/meshtastic_mcp/log_query.py @@ -0,0 +1,410 @@ +"""Read-side queries over the recorder's JSONL streams. + +Pure functions over `mcp-server/.mtlog/`. Streaming JSONL reader: never +loads a whole file. Time-bound queries short-circuit as soon as `ts` +exceeds the requested end. The recorder writes monotonically, so a +forward scan is cheap; we don't need an index. + +All time arguments accept: + - epoch seconds (int/float) + - relative strings: "-15m", "-2h", "-3d", "now" + - ISO-ish absolute strings: "2026-05-07T14:30:00" (naive timestamps are + treated as UTC) + +Tools that return data ALWAYS cap their output (max_lines / max_points +/ max), and report whether more matched than was returned. +""" + +from __future__ import annotations + +import gzip +import json +import re +import statistics +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Iterator + +from .recorder.recorder import get_recorder + +_REL_RE = re.compile(r"^\s*-\s*(\d+(?:\.\d+)?)\s*([smhd])\s*$") +_REGEX_PREVIEW_MAX = 100 +_REGEX_PREVIEW_TRUNCATE = 97 + + +def _parse_time(value: Any, *, now: float | None = None) -> float: + """Coerce to epoch seconds. Defaults `now` to `time.time()`.""" + if value is None: + return time.time() + if isinstance(value, (int, float)): + return float(value) + if not isinstance(value, str): + raise ValueError(f"invalid time: {value!r}") + s = value.strip().lower() + if s in ("", "now"): + return time.time() if now is None else now + m = _REL_RE.match(s) + if m: + n = float(m.group(1)) + unit = m.group(2) + secs = n * {"s": 1, "m": 60, "h": 3600, "d": 86400}[unit] + base = time.time() if now is None else now + return base - secs + # Try ISO 8601. Accept naive (assume UTC) and Z-suffixed. + try: + if s.endswith("z"): + s = s[:-1] + "+00:00" + dt = datetime.fromisoformat(s) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return dt.timestamp() + except ValueError as e: + raise ValueError(f"unparseable time: {value!r}") from e + + +def _iter_jsonl(path: Path, *, since: float, until: float) -> Iterator[dict[str, Any]]: + """Stream records in chronological order: rotated archives first + (oldest → newest by lex sort, which is chronological for our + `YYYYMMDD-HHMMSS-uuuuuu-NNNNN` archive naming), then the live file + last. The "keep last N" pop-front logic in the window queries + relies on records arriving in time order across files. + """ + files: list[Path] = [] + # Gzipped archives are named ".YYYYMMDD-HHMMSS-uuuuuu-NNNNN.jsonl.gz". + for archive in sorted(path.parent.glob(f"{path.stem}.*.jsonl.gz")): + files.append(archive) + if path.exists(): + files.append(path) + for f in files: + opener = gzip.open if f.suffix == ".gz" else open + try: + with opener(f, "rt", encoding="utf-8") as fh: # type: ignore[arg-type] + for line in fh: + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except json.JSONDecodeError: + continue + ts = rec.get("ts") + if not isinstance(ts, (int, float)): + continue + if ts < since: + continue + if ts > until: + # Records are append-monotonic within a file, so + # the rest of this file is also past `until`. + # Archives can still overlap each other, so only + # short-circuit this file, not the whole scan. + break + yield rec + except (FileNotFoundError, OSError): + continue + + +# -- queries ------------------------------------------------------------ + + +def logs_window( + start: Any = "-15m", + end: Any = "now", + *, + grep: str | None = None, + level: str | None = None, + tag: str | None = None, + port: str | None = None, + max_lines: int = 200, +) -> dict[str, Any]: + """Recent firmware log lines, filtered. + + `level` accepts a single level name or pipe-separated set + ("WARN|ERROR|CRIT"). `grep` is a regex (Python re) over the raw + `line` field. Returns the last `max_lines` matches. + """ + s = _parse_time(start) + e = _parse_time(end) + levels = _split_set(level) + if grep: + try: + grep_re = re.compile(grep) + except re.error as exc: + preview = ( + grep + if len(grep) <= _REGEX_PREVIEW_MAX + else f"{grep[:_REGEX_PREVIEW_TRUNCATE]}..." + ) + raise ValueError(f"invalid grep regex {preview!r}: {exc}") from exc + else: + grep_re = None + + base = get_recorder().base_dir + matched = 0 + out: list[dict[str, Any]] = [] + for rec in _iter_jsonl(base / "logs.jsonl", since=s, until=e): + if levels and rec.get("level") not in levels: + continue + if tag and rec.get("tag") != tag: + continue + if port and rec.get("port") != port: + continue + if grep_re and not grep_re.search(rec.get("line") or ""): + continue + matched += 1 + out.append(rec) + if len(out) > max_lines: + out.pop(0) # keep the most recent N + return { + "lines": out, + "total_matched": matched, + "dropped": max(0, matched - max_lines), + "window": {"start": s, "end": e}, + } + + +def telemetry_timeline( + window: Any = "1h", + *, + variant: str = "local", + field: str = "free_heap", + port: str | None = None, + max_points: int = 200, +) -> dict[str, Any]: + """Timeseries of one telemetry field, downsampled. + + `field` matches both the protobuf snake_case name (`free_heap`, + `heap_free_bytes`, `battery_level`) and camelCase (`freeHeap`). + Server-side bucket-mean downsamples to ≤ `max_points`. Returns + `slope_per_min` (linear regression slope, units/min) so a leak + detector can read one number. + """ + end = time.time() + if isinstance(window, (int, float)): + # Numeric `window` is a duration in seconds — "last N seconds". + # Without this branch, `_parse_time(-N)` would treat -N as an + # absolute epoch timestamp (i.e., Jan 1 1970 minus N seconds), + # producing a wildly negative `start` and matching nothing. + start = end - float(window) + elif isinstance(window, str) and not window.startswith("-"): + # Bare string like "1h" is sugar for "-1h". + start = _parse_time(f"-{window}", now=end) + else: + start = _parse_time(window, now=end) + + base = get_recorder().base_dir + raw: list[tuple[float, float]] = [] + field_aliases = _field_aliases(field) + for rec in _iter_jsonl(base / "telemetry.jsonl", since=start, until=end): + if rec.get("variant") != variant: + continue + if port and rec.get("port") != port: + continue + fields = rec.get("fields") or {} + value: Any = None + for alias in field_aliases: + if alias in fields: + value = fields[alias] + break + if not isinstance(value, (int, float)): + continue + raw.append((float(rec["ts"]), float(value))) + + if not raw: + return { + "points": [], + "samples": 0, + "min": None, + "max": None, + "slope_per_min": None, + "window": {"start": start, "end": end, "variant": variant, "field": field}, + } + + points = _downsample(raw, max_points=max_points) + values = [v for _, v in raw] + return { + "points": [{"ts": ts, "value": v} for ts, v in points], + "samples": len(raw), + "min": min(values), + "max": max(values), + "slope_per_min": _slope_per_min(raw), + "window": {"start": start, "end": end, "variant": variant, "field": field}, + } + + +def packets_window( + start: Any = "-5m", + end: Any = "now", + *, + portnum: str | None = None, + from_node: str | None = None, + to_node: str | None = None, + max: int = 200, +) -> dict[str, Any]: + s = _parse_time(start) + e = _parse_time(end) + portnums = _split_set(portnum) + base = get_recorder().base_dir + matched = 0 + out: list[dict[str, Any]] = [] + for rec in _iter_jsonl(base / "packets.jsonl", since=s, until=e): + if portnums and rec.get("portnum") not in portnums: + continue + if from_node and str(rec.get("from_node")) != str(from_node): + continue + if to_node and str(rec.get("to_node")) != str(to_node): + continue + matched += 1 + out.append(rec) + if len(out) > max: + out.pop(0) + return { + "packets": out, + "total_matched": matched, + "dropped": matched - max if matched > max else 0, + "window": {"start": s, "end": e}, + } + + +def events_window( + start: Any = "-1h", + end: Any = "now", + *, + kind: str | None = None, + max: int = 200, +) -> dict[str, Any]: + s = _parse_time(start) + e = _parse_time(end) + kinds = _split_set(kind) + base = get_recorder().base_dir + matched = 0 + out: list[dict[str, Any]] = [] + for rec in _iter_jsonl(base / "events.jsonl", since=s, until=e): + if kinds and rec.get("kind") not in kinds: + continue + matched += 1 + out.append(rec) + if len(out) > max: + out.pop(0) + return { + "events": out, + "total_matched": matched, + "dropped": matched - max if matched > max else 0, + "window": {"start": s, "end": e}, + } + + +def export( + start: Any, + end: Any, + dest_dir: str, + *, + streams: list[str] | None = None, +) -> dict[str, Any]: + """Bundle a slice of each requested stream into `dest_dir`. + + For a notebook, a bug report, or a Datadog backfill. Output files + are uncompressed JSONL (callers gzip themselves if they want to). + """ + s = _parse_time(start) + e = _parse_time(end) + selected = streams or ["logs", "telemetry", "packets", "events"] + dest = Path(dest_dir) + dest.mkdir(parents=True, exist_ok=True) + + base = get_recorder().base_dir + paths: dict[str, str] = {} + for stream in selected: + src = base / f"{stream}.jsonl" + if not src.exists() and not list(base.glob(f"{stream}.*.jsonl.gz")): + continue + out_path = dest / f"{stream}.jsonl" + n = 0 + with out_path.open("w", encoding="utf-8") as fh: + for rec in _iter_jsonl(src, since=s, until=e): + fh.write(json.dumps(rec, separators=(",", ":")) + "\n") + n += 1 + paths[stream] = str(out_path) + paths[f"{stream}_count"] = str(n) + return {"dest_dir": str(dest), "paths": paths, "window": {"start": s, "end": e}} + + +# -- helpers ------------------------------------------------------------ + + +def _split_set(value: str | None) -> set[str] | None: + if not value: + return None + return {v.strip() for v in value.split("|") if v.strip()} + + +def _field_aliases(field: str) -> list[str]: + """Accept snake_case OR camelCase, plus a few legacy aliases.""" + snake = field + camel = _snake_to_camel(field) + aliases = {snake, camel} + # Old protobuf fields (pre-LocalStats) used different names + legacy = { + "free_heap": ["free_heap", "freeHeap", "heap_free_bytes", "heapFreeBytes"], + "heap_free_bytes": [ + "heap_free_bytes", + "heapFreeBytes", + "free_heap", + "freeHeap", + ], + "total_heap": ["total_heap", "totalHeap", "heap_total_bytes", "heapTotalBytes"], + "heap_total_bytes": [ + "heap_total_bytes", + "heapTotalBytes", + "total_heap", + "totalHeap", + ], + } + if field in legacy: + aliases.update(legacy[field]) + return list(aliases) + + +def _snake_to_camel(name: str) -> str: + parts = name.split("_") + return parts[0] + "".join(p.title() for p in parts[1:]) + + +def _downsample( + points: list[tuple[float, float]], *, max_points: int +) -> list[tuple[float, float]]: + if len(points) <= max_points: + return points + # Even-bucket mean. Preserves shape better than nth-sample picking. + n = len(points) + bucket = n / max_points + out: list[tuple[float, float]] = [] + i = 0 + for k in range(max_points): + end = int((k + 1) * bucket) + end = min(end, n) + if end <= i: + continue + chunk = points[i:end] + ts = chunk[len(chunk) // 2][0] + val = statistics.fmean(v for _, v in chunk) + out.append((ts, val)) + i = end + return out + + +def _slope_per_min(points: list[tuple[float, float]]) -> float | None: + """Least-squares slope (units per minute). None if too few points.""" + if len(points) < 2: + return None + xs = [t for t, _ in points] + ys = [v for _, v in points] + n = len(xs) + mean_x = sum(xs) / n + mean_y = sum(ys) / n + num = sum((xs[i] - mean_x) * (ys[i] - mean_y) for i in range(n)) + den = sum((x - mean_x) ** 2 for x in xs) + if den == 0: + return None + slope_per_sec = num / den + return slope_per_sec * 60.0 diff --git a/mcp-server/src/meshtastic_mcp/pio.py b/mcp-server/src/meshtastic_mcp/pio.py index c0c23f9bb..c984d7a42 100644 --- a/mcp-server/src/meshtastic_mcp/pio.py +++ b/mcp-server/src/meshtastic_mcp/pio.py @@ -92,6 +92,7 @@ def _run_capturing( cwd: Path | None = None, timeout: float | None = None, tee_header: str | None = None, + extra_env: dict[str, str] | None = None, ) -> tuple[int, str, str, float]: """Run a subprocess, capture stdout+stderr, optionally tee to the flash log. @@ -99,6 +100,9 @@ def _run_capturing( `subprocess.TimeoutExpired` on timeout (callers map this to their own domain-specific error). + `extra_env` merges into the subprocess environment (parent env stays + intact). Used for `PLATFORMIO_BUILD_FLAGS=-DDEBUG_HEAP=1` and similar. + Fast path: `subprocess.run(capture_output=True)` when no flash log is configured (unchanged behavior). @@ -110,6 +114,9 @@ def _run_capturing( """ log_path = _flash_log_path() t0 = time.monotonic() + env = None + if extra_env: + env = {**os.environ, **extra_env} if log_path is None: # Fast path — unchanged. @@ -119,6 +126,7 @@ def _run_capturing( capture_output=True, text=True, timeout=timeout, + env=env, ) return ( proc.returncode, @@ -145,6 +153,7 @@ def _run_capturing( stderr=subprocess.PIPE, text=True, bufsize=1, # line-buffered + env=env, ) stdout_chunks: list[str] = [] stderr_chunks: list[str] = [] @@ -232,12 +241,17 @@ def run( cwd: Path | None = None, timeout: float | None = TIMEOUT_DEFAULT, check: bool = True, + extra_env: dict[str, str] | None = None, ) -> PioResult: """Invoke `pio ` and return captured output. `cwd` defaults to the firmware root. `check=True` raises `PioError` on non-zero exit; set `check=False` to inspect `returncode` manually. + `extra_env` merges into the subprocess environment — used for + `PLATFORMIO_BUILD_FLAGS=-DDEBUG_HEAP=1` and similar build-time + toggles that can't be expressed as command-line args. + If `MESHTASTIC_MCP_FLASH_LOG` is set, output is also tee'd to that file line-by-line as it arrives (for live flash progress in the TUI). """ @@ -250,6 +264,7 @@ def run( cwd=work_dir, timeout=timeout, tee_header=f"pio {' '.join(args)}", + extra_env=extra_env, ) except subprocess.TimeoutExpired as exc: raise PioTimeout(f"pio {' '.join(args)} timed out after {timeout}s") from exc diff --git a/mcp-server/src/meshtastic_mcp/recorder/__init__.py b/mcp-server/src/meshtastic_mcp/recorder/__init__.py new file mode 100644 index 000000000..874d59d0e --- /dev/null +++ b/mcp-server/src/meshtastic_mcp/recorder/__init__.py @@ -0,0 +1,19 @@ +"""Persistent device-log capture. + +Singleton `Recorder` subscribes once to the meshtastic pubsub fan-out +(`meshtastic.log.line`, `meshtastic.receive.*`, `meshtastic.connection.*`) +and appends to four JSONL files under `mcp-server/.mtlog/`. Pubsub is +process-global so a single subscription captures every active interface +(serial / TCP / BLE) without any per-connection bookkeeping. + +The recorder is opt-in-by-import: importing this package is a no-op; call +`get_recorder().start()` (which `server.py` does at FastMCP app init) to +begin writing. `pause()` / `resume()` exist for the rare case the user +wants a clean stretch of file (e.g. capturing a known-good baseline). +""" + +from __future__ import annotations + +from .recorder import Recorder, get_recorder + +__all__ = ["Recorder", "get_recorder"] diff --git a/mcp-server/src/meshtastic_mcp/recorder/parsers.py b/mcp-server/src/meshtastic_mcp/recorder/parsers.py new file mode 100644 index 000000000..1936ccbda --- /dev/null +++ b/mcp-server/src/meshtastic_mcp/recorder/parsers.py @@ -0,0 +1,309 @@ +"""Best-effort parsers for log lines and telemetry packets. + +Two flavors of log line cross our pubsub subscription: + 1. Text-mode path (debug_log_api disabled): the meshtastic Python lib + accumulates bytes between protobuf frames and emits the full + firmware-formatted line, e.g. + "INFO | 12:34:56 12345 [Main] Booting" + — level, HH:MM:SS, uptime seconds, thread bracket, then message. + 2. LogRecord protobuf path (debug_log_api enabled): the lib calls + `_handleLogLine(record.message)` with ONLY the message body. The + level/source/time fields on the LogRecord are dropped before + pubsub fan-out. We get e.g. just "Booting". + +Both arrive on `meshtastic.log.line`. The parser tries to recover a +level + thread when the prefix is present and falls back to level=None +otherwise. Consumers who want level filtering on protobuf-mode hosts +should grep the raw `line` field instead. + +Telemetry: `meshtastic.receive.telemetry` packets carry one of several +metric variants in `packet["decoded"]["telemetry"]`. We flatten the +chosen variant into a {field: value} dict so callers don't have to +know the protobuf shape. +""" + +from __future__ import annotations + +import re +from typing import Any + +# Match: LEVEL | HH:MM:SS UPTIME [Thread] message +# HH:MM:SS may be ??:??:?? when RTC isn't valid. The level alternation +# below is the canonical list — DebugConfiguration.h's MESHTASTIC_LOG_LEVEL_* +# macros must stay in sync with these strings. +_LINE_RE = re.compile( + r""" + ^ + (?PDEBUG|INFO\ |WARN\ |ERROR|CRIT\ |TRACE|HEAP\ ) + \s*\|\s* + (?P(?:\d{2}:\d{2}:\d{2})|(?:\?{2}:\?{2}:\?{2})) + \s+ + (?P\d+) + \s+ + (?:\[(?P[^\]]+)\]\s+)? + (?P.*) + $ + """, + re.VERBOSE, +) + +# DEBUG_HEAP build prepends `[heap N] ` to every message body, AFTER the +# thread bracket. See src/RedirectablePrint.cpp:175. +_HEAP_PREFIX_RE = re.compile(r"^\[heap\s+(?P\d+)\]\s+(?P.*)$") + +# OSThread leak/free detection. See src/concurrency/OSThread.cpp:89-91. +# Format: "------ Thread NAME leaked heap A -> B (delta) ------" +# "++++++ Thread NAME freed heap A -> B (delta) ++++++" +_THREAD_HEAP_RE = re.compile( + r""" + ^[\-+]+\s* + Thread\s+(?P\S+)\s+ + (?Pleaked|freed)\s+heap\s+ + (?P-?\d+)\s*->\s*(?P-?\d+)\s+ + \((?P-?\d+)\) + """, + re.VERBOSE, +) + +# Power.cpp:908 periodic heap status (DEBUG_HEAP only). +# Format: "Heap status: FREE/TOTAL bytes free (DELTA), running R/N threads" +_HEAP_STATUS_RE = re.compile( + r""" + Heap\s+status:\s+ + (?P\d+)\s*/\s*(?P\d+)\s+bytes\s+free + (?:\s+\((?P-?\d+)\))? + """, + re.VERBOSE, +) + + +_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]") +_HEAP_BRACKET_RE = re.compile(r"^heap\s+(?P\d+)$") + + +def parse_log_line(line: str) -> dict[str, Any]: + """Best-effort decompose a raw firmware log line. + + Returns a dict with at least `line` (the original, unmodified — ANSI + codes preserved for fidelity). Adds `level`, `tag`, `clock`, + `uptime_s`, and `msg` when the full prefix is present. + + Handles two firmware quirks: + - LogRecord.message can carry ANSI color escapes from RedirectablePrint + (the BLE/StreamAPI path inherited the colored body in some builds). + We strip ANSI before regex matching so the prefix survives. + - DEBUG_HEAP injects `[heap N]` after the thread bracket. When NO + thread name is set, the heap takes the thread bracket position — + looks like `[heap 12345] msg`. We detect that shape and move it + out of `tag` and into `heap_free`. + + DEBUG_HEAP-build extras (when `[heap N]` is injected): `heap_free` + (bytes), and when a `Thread X leaked|freed heap` line is recognized, + `heap_event` = {kind, thread, before, after, delta}. + + Never raises. + """ + out: dict[str, Any] = {"line": line} + if not line: + return out + + # Strip ANSI escapes BEFORE any regex matching. The original `line` + # stays in `out["line"]` for fidelity / future grep. + clean = _ANSI_RE.sub("", line) + + m = _LINE_RE.match(clean) + msg: str | None = None + if m: + level = m.group("level").rstrip() + out["level"] = level + out["clock"] = m.group("clock") + try: + out["uptime_s"] = int(m.group("uptime")) + except (TypeError, ValueError): + out["uptime_s"] = None + thread = m.group("thread") + if thread: + # If "thread" is actually the heap prefix taking the bracket + # position (DEBUG_HEAP build, no thread set), capture heap + # and leave tag unset. + hb = _HEAP_BRACKET_RE.match(thread.strip()) + if hb: + try: + out["heap_free"] = int(hb.group("heap")) + except (TypeError, ValueError): + pass + else: + out["tag"] = thread + msg = m.group("msg") + out["msg"] = msg + else: + # No prefix — bare LogRecord.message body. Inspect the whole + # line for DEBUG_HEAP-style content; the heap-prefix and + # thread-leak patterns can survive on either path. + msg = clean + + # DEBUG_HEAP per-line heap prefix: `[heap 92344] message`. + # Sits AFTER the thread bracket and BEFORE the message body, but + # for bare LogRecord lines it's at the start. Match it at the + # head of `msg`. + if msg: + hp = _HEAP_PREFIX_RE.match(msg) + if hp: + try: + out["heap_free"] = int(hp.group("heap")) + except (TypeError, ValueError): + pass + else: + # Strip the prefix from `msg` so a grep on the message + # body doesn't have to know about it. + out["msg"] = hp.group("rest") + msg = hp.group("rest") + + # Thread-level leak/free detection. + thr = _THREAD_HEAP_RE.search(msg) + if thr: + try: + out["heap_event"] = { + "kind": thr.group("kind"), + "thread": thr.group("thread"), + "before": int(thr.group("before")), + "after": int(thr.group("after")), + "delta": int(thr.group("delta")), + } + except (TypeError, ValueError): + pass + + # Power.cpp periodic "Heap status: F/T bytes free (D), running ..." + hs = _HEAP_STATUS_RE.search(msg) + if hs: + try: + out["heap_free"] = int(hs.group("free")) + out["heap_total"] = int(hs.group("total")) + if hs.group("delta") is not None: + out["heap_delta"] = int(hs.group("delta")) + except (TypeError, ValueError): + pass + + return out + + +# -- Telemetry ---------------------------------------------------------- + +# Order matters: meshtastic-python decoded packets use the protobuf +# `oneof variant` field name (snake_case) as the dict key. +_TELEMETRY_VARIANTS = ( + ("device_metrics", "device"), + ("local_stats", "local"), + ("environment_metrics", "environment"), + ("power_metrics", "power"), + ("air_quality_metrics", "airQuality"), + ("health_metrics", "health"), + ("host_metrics", "host"), +) + + +def extract_telemetry(packet: dict[str, Any]) -> dict[str, Any] | None: + """Pull the telemetry variant + flat fields out of a `meshtastic.receive.telemetry` + packet. Returns None when the shape isn't what we expect — so the + caller can fall back to a generic packets.jsonl row. + """ + if not isinstance(packet, dict): + return None + decoded = packet.get("decoded") + if not isinstance(decoded, dict): + return None + telem = decoded.get("telemetry") + if not isinstance(telem, dict): + return None + # The Python lib produces dict-of-camelCase keys via MessageToDict. + # Try both camelCase and snake_case to be robust to lib version drift. + for snake, label in _TELEMETRY_VARIANTS: + camel = _snake_to_camel(snake) + for key in (snake, camel): + value = telem.get(key) + if isinstance(value, dict): + return { + "variant": label, + "fields": {k: _scalarize(v) for k, v in value.items()}, + "time": telem.get("time"), + } + return None + + +def _snake_to_camel(name: str) -> str: + parts = name.split("_") + return parts[0] + "".join(p.title() for p in parts[1:]) + + +def _scalarize(value: Any) -> Any: + """Keep telemetry fields JSON-friendly. Lists/dicts pass through + untouched; bytes -> hex string; protobuf enums occasionally arrive + as ints (fine) or strings (also fine).""" + if isinstance(value, (bytes, bytearray, memoryview)): + return bytes(value).hex() + return value + + +# -- Generic packet summary --------------------------------------------- + + +def summarize_packet( + packet: dict[str, Any], *, payload_hex_len: int = 64 +) -> dict[str, Any]: + """Reduce a packet dict to a stable, queryable summary. Drops the + full payload bytes — the recorder records summaries, not pcaps. + """ + if not isinstance(packet, dict): + return {"raw_type": type(packet).__name__} + decoded = packet.get("decoded") if isinstance(packet.get("decoded"), dict) else {} + portnum = decoded.get("portnum") if isinstance(decoded, dict) else None + payload = decoded.get("payload") if isinstance(decoded, dict) else None + payload_hex = None + payload_size = None + if isinstance(payload, (bytes, bytearray, memoryview)): + b = bytes(payload) + payload_size = len(b) + payload_hex = b[:payload_hex_len].hex() if b else "" + elif isinstance(payload, str): + # Some decoded payloads (text messages) come as decoded strings. + payload_size = len(payload) + payload_hex = None # not bytes + return { + "from_node": packet.get("fromId") or packet.get("from"), + "to_node": packet.get("toId") or packet.get("to"), + "portnum": portnum, + "hop_limit": packet.get("hopLimit"), + "want_ack": packet.get("wantAck"), + "rx_rssi": packet.get("rxRssi"), + "rx_snr": packet.get("rxSnr"), + "channel": packet.get("channel"), + "id": packet.get("id"), + "payload_size": payload_size, + "payload_hex_prefix": payload_hex, + } + + +# -- Interface identification ------------------------------------------ + + +def interface_label(interface: Any) -> dict[str, Any]: + """Stable identifier for the meshtastic interface that emitted an event. + + Used as the `port`/`role` tag on every recorded row. SerialInterface + has `devPath`; TCPInterface has `hostname`+`portNumber`; BLEInterface + has `address`. Falls back to the class name when none of those exist. + """ + if interface is None: + return {"port": None, "role": None} + dev_path = getattr(interface, "devPath", None) + if dev_path: + return {"port": str(dev_path), "role": "serial"} + hostname = getattr(interface, "hostname", None) + if hostname: + port_num = getattr(interface, "portNumber", None) + endpoint = f"tcp://{hostname}:{port_num}" if port_num else f"tcp://{hostname}" + return {"port": endpoint, "role": "tcp"} + address = getattr(interface, "address", None) + if address: + return {"port": str(address), "role": "ble"} + return {"port": type(interface).__name__, "role": None} diff --git a/mcp-server/src/meshtastic_mcp/recorder/recorder.py b/mcp-server/src/meshtastic_mcp/recorder/recorder.py new file mode 100644 index 000000000..2b8a5b481 --- /dev/null +++ b/mcp-server/src/meshtastic_mcp/recorder/recorder.py @@ -0,0 +1,467 @@ +"""Process-global recorder singleton. + +Subscribes once to the meshtastic pubsub fan-out and writes four append-only +JSONL streams under `mcp-server/.mtlog/`. The pubsub fan-out is +process-global — a single subscription captures every active interface +without per-connection bookkeeping. + +Files: + logs.jsonl — every `meshtastic.log.line` event (best-effort prefix + parsed for level/tag/uptime; raw `line` always preserved) + telemetry.jsonl — `meshtastic.receive.telemetry` packets, flattened by + variant (device / local / environment / power / etc.) + packets.jsonl — every other `meshtastic.receive.*` packet, summarized + (portnum, hops, RSSI/SNR, payload size + 64-byte hex) + events.jsonl — connection lifecycle, node-DB updates, and manual + `mark_event` rows. Lower volume; useful for aligning + timelines. + +Pause/resume: `pause()` flips a flag; subscriptions stay registered. The +write methods short-circuit when paused, so we don't lose ordering when +resumed (we just have a gap). No queueing. +""" + +from __future__ import annotations + +import logging +import os +import threading +import time +from pathlib import Path +from typing import Any + +from . import parsers +from .rotating import _RotatingJsonl + +_DEFAULT_DIR = Path(__file__).resolve().parents[3] / ".mtlog" +log = logging.getLogger(__name__) + + +class Recorder: + """Singleton write-side of the persistent log capture system.""" + + def __init__(self, base_dir: Path | None = None) -> None: + self.base_dir = Path(base_dir) if base_dir else _DEFAULT_DIR + self._lock = threading.RLock() + self._started = False + self._paused = False + self._pause_reason: str | None = None + self._started_at: float | None = None + self._handlers: list[tuple[str, Any]] = [] + self._files: dict[str, _RotatingJsonl] = {} + + # -- lifecycle ---------------------------------------------------- + + def start(self) -> None: + """Idempotent. Safe to call from FastMCP app startup.""" + with self._lock: + if self._started: + return + self.base_dir.mkdir(parents=True, exist_ok=True) + self._files = { + "logs": _RotatingJsonl(self.base_dir / "logs.jsonl"), + "telemetry": _RotatingJsonl(self.base_dir / "telemetry.jsonl"), + "packets": _RotatingJsonl(self.base_dir / "packets.jsonl"), + "events": _RotatingJsonl(self.base_dir / "events.jsonl"), + } + self._wire_pubsub() + self._started = True + self._started_at = time.time() + # Write the recorder_start marker after the initialization block. + # `_write_event()` re-checks recorder state via `_files_snapshot()`, + # so keeping this out of the setup block avoids nested lifecycle work. + self._write_event(kind="recorder_start", label="recorder_started") + + def stop(self) -> None: + with self._lock: + if not self._started: + return + self._unwire_pubsub() + for f in self._files.values(): + f.close() + self._files = {} + self._started = False + + def pause(self, reason: str | None = None) -> None: + # Write the pause marker BEFORE flipping the flag — `_write_event` + # short-circuits when paused, so the order matters for this event + # to actually land in events.jsonl. + self._write_event( + kind="recorder_pause", + label="paused", + note=reason, + ) + with self._lock: + self._paused = True + self._pause_reason = reason + + def resume(self) -> None: + # Mirror of `pause()`: clear the flag first, then write the marker + # so it isn't suppressed by the still-paused short-circuit. + with self._lock: + self._paused = False + self._pause_reason = None + self._write_event(kind="recorder_resume", label="resumed") + + # -- pubsub wiring ------------------------------------------------ + + def _wire_pubsub(self) -> None: + from pubsub import pub # type: ignore[import-untyped] + + # Subscribers — one per topic. Each pubsub publisher sends + # keyword args matching its handler's signature; pubsub + # introspects the function signature to route args. + bindings = [ + ("meshtastic.log.line", self._on_log_line), + ("meshtastic.serial.line", self._on_serial_line), + ("meshtastic.receive", self._on_receive), + ("meshtastic.receive.telemetry", self._on_telemetry), + ("meshtastic.connection.established", self._on_connection_established), + ("meshtastic.connection.lost", self._on_connection_lost), + ("meshtastic.node.updated", self._on_node_updated), + ] + for topic, handler in bindings: + try: + pub.subscribe(handler, topic) + self._handlers.append((topic, handler)) + except Exception as exc: + # If pubsub refuses one binding (signature mismatch on + # an old lib version), log it and keep the rest. + log.warning("Recorder failed to subscribe to %s: %s", topic, exc) + + def _unwire_pubsub(self) -> None: + from pubsub import pub # type: ignore[import-untyped] + + for topic, handler in self._handlers: + try: + pub.unsubscribe(handler, topic) + except Exception: + pass + self._handlers.clear() + + # -- handlers ----------------------------------------------------- + # + # Pubsub callbacks must never raise. Every handler is wrapped in a + # try/except that swallows so a bug here can't take down the + # SerialInterface receive thread. + # + # Threading: handlers fire on whatever thread the meshtastic library + # dispatches from (varies by interface), while `stop()` clears + # `self._files` under `self._lock`. We snapshot `_files` under the + # lock at the top of each handler so a concurrent stop can't + # KeyError us mid-write. The actual file write goes through + # `_RotatingJsonl` which has its own lock. + + def _files_snapshot(self) -> dict[str, _RotatingJsonl] | None: + """Atomic-ish view of `self._files`. Returns None when the recorder + is paused or stopped, so handlers can early-exit cleanly without + racing `stop()`'s clear.""" + with self._lock: + if not self._started or self._paused: + return None + return dict(self._files) + + def _on_log_line(self, line: str, interface: Any = None) -> None: + files = self._files_snapshot() + if files is None: + return + try: + tags = parsers.interface_label(interface) + parsed = parsers.parse_log_line(str(line)) + ts = time.time() + record: dict[str, Any] = { + "ts": ts, + "port": tags["port"], + "role": tags["role"], + "level": parsed.get("level"), + "tag": parsed.get("tag"), + "uptime_s": parsed.get("uptime_s"), + "line": parsed["line"], + } + # DEBUG_HEAP enrichments (only present when the firmware + # was built with -DDEBUG_HEAP=1). Surface as first-class + # fields so logs_window can grep/filter on them and so + # heap_free synthesizes a telemetry point below. + if "heap_free" in parsed: + record["heap_free"] = parsed["heap_free"] + if "heap_total" in parsed: + record["heap_total"] = parsed["heap_total"] + if "heap_delta" in parsed: + record["heap_delta"] = parsed["heap_delta"] + heap_event = parsed.get("heap_event") + if heap_event: + record["heap_event"] = heap_event + files["logs"].write(record) + + # If the line carried a heap snapshot, also write it as a + # synthesized LocalStats-shaped row so telemetry_timeline + # picks it up at log cadence (much higher resolution than + # the ~60 s LocalStats packet). Tagged source=debug_heap so + # consumers can filter if mixing scales is unwanted. + heap_free = parsed.get("heap_free") + if isinstance(heap_free, int): + fields: dict[str, Any] = {"heap_free_bytes": heap_free} + heap_total = parsed.get("heap_total") + if isinstance(heap_total, int): + fields["heap_total_bytes"] = heap_total + files["telemetry"].write( + { + "ts": ts, + "port": tags["port"], + "role": tags["role"], + "from_node": None, + "variant": "local", + "fields": fields, + "source": "debug_heap", + } + ) + except Exception: + pass + + def _on_serial_line(self, line: str, port: str | None = None) -> None: + """Text-mode passive tap. Fired from `serial_session._drain` when a + `pio device monitor` subprocess is running. + + Same parse + heap-synthesis path as `_on_log_line`, but receives + the raw text-formatted line (full level/clock/uptime/thread/`[heap N]`/ + body). On DEBUG_HEAP builds in text mode this gives us per-log-line + heap data — far higher cadence than LocalStats, and works without + protobuf API mode (no SerialInterface required). + """ + files = self._files_snapshot() + if files is None: + return + try: + parsed = parsers.parse_log_line(str(line)) + ts = time.time() + record: dict[str, Any] = { + "ts": ts, + "port": port, + "role": "serial_session", + "level": parsed.get("level"), + "tag": parsed.get("tag"), + "uptime_s": parsed.get("uptime_s"), + "line": parsed["line"], + } + if "heap_free" in parsed: + record["heap_free"] = parsed["heap_free"] + if "heap_total" in parsed: + record["heap_total"] = parsed["heap_total"] + if "heap_delta" in parsed: + record["heap_delta"] = parsed["heap_delta"] + heap_event = parsed.get("heap_event") + if heap_event: + record["heap_event"] = heap_event + files["logs"].write(record) + + # Synthesize a heap_free telemetry sample whenever the line + # carries one — same logic as _on_log_line, tagged source so + # consumers can distinguish text-mode tap from protobuf path. + heap_free = parsed.get("heap_free") + if isinstance(heap_free, int): + fields: dict[str, Any] = {"heap_free_bytes": heap_free} + heap_total = parsed.get("heap_total") + if isinstance(heap_total, int): + fields["heap_total_bytes"] = heap_total + files["telemetry"].write( + { + "ts": ts, + "port": port, + "role": "serial_session", + "from_node": None, + "variant": "local", + "fields": fields, + "source": "debug_heap_serial", + } + ) + except Exception: + pass + + def _on_telemetry(self, packet: dict[str, Any], interface: Any = None) -> None: + files = self._files_snapshot() + if files is None: + return + try: + tags = parsers.interface_label(interface) + extracted = parsers.extract_telemetry(packet) + if extracted is None: + # Couldn't extract a known variant — fall through to the + # generic `_on_receive` path, which will still fire for + # this packet via the parent topic. + return + record = { + "ts": time.time(), + "port": tags["port"], + "role": tags["role"], + "from_node": packet.get("fromId") or packet.get("from"), + "variant": extracted["variant"], + "fields": extracted["fields"], + "device_time": extracted.get("time"), + } + files["telemetry"].write(record) + except Exception: + pass + + def _on_receive(self, packet: dict[str, Any], interface: Any = None) -> None: + # Generic-receive fires for EVERY packet. Telemetry packets get + # recorded twice (here and in _on_telemetry) — that's intentional: + # packets.jsonl is the universal record, telemetry.jsonl is the + # structured timeseries view. + files = self._files_snapshot() + if files is None: + return + try: + tags = parsers.interface_label(interface) + summary = parsers.summarize_packet(packet) + record = { + "ts": time.time(), + "port": tags["port"], + "role": tags["role"], + **summary, + } + files["packets"].write(record) + except Exception: + pass + + def _on_connection_established(self, interface: Any = None) -> None: + self._write_event( + kind="connection_established", + interface=interface, + ) + + def _on_connection_lost(self, interface: Any = None) -> None: + self._write_event( + kind="connection_lost", + interface=interface, + ) + + def _on_node_updated( + self, node: dict[str, Any] | None = None, interface: Any = None + ) -> None: + # Lower-volume than packets but informative — node ID, hops away, + # last heard. Skip the user dict if absent. + try: + user = (node or {}).get("user") if isinstance(node, dict) else None + self._write_event( + kind="node_updated", + interface=interface, + data={ + "num": (node or {}).get("num"), + "id": (user or {}).get("id"), + "short": (user or {}).get("shortName"), + "long": (user or {}).get("longName"), + "hops_away": (node or {}).get("hopsAway"), + "snr": (node or {}).get("snr"), + "last_heard": (node or {}).get("lastHeard"), + }, + ) + except Exception: + pass + + # -- public write helpers ----------------------------------------- + + def mark_event( + self, + label: str, + note: str | None = None, + data: dict[str, Any] | None = None, + ) -> dict[str, Any]: + """User-facing marker. Writes to events.jsonl AND emits a + synthetic logs.jsonl row tagged level=MARK so timelines align. + """ + ts = self._write_event(kind="mark", label=label, note=note, data=data) + # Mirror into logs so a single logs_window grep finds it. + files = self._files_snapshot() + if files is not None: + try: + files["logs"].write( + { + "ts": ts, + "port": None, + "role": "marker", + "level": "MARK", + "tag": "mark_event", + "line": f"[mark] {label}" + (f" — {note}" if note else ""), + } + ) + except Exception: + pass + return {"ts": ts, "label": label} + + def _write_event( + self, + *, + kind: str, + label: str | None = None, + note: str | None = None, + interface: Any = None, + data: dict[str, Any] | None = None, + ) -> float: + ts = time.time() + # Lifecycle markers (recorder_start, recorder_pause, recorder_resume) + # arrive at choreographed moments — `pause()` writes BEFORE flipping + # the flag and `resume()` writes AFTER clearing it, so those calls + # see _paused=False here. Other event kinds short-circuit when + # paused via the snapshot guard below. + files = self._files_snapshot() + if files is None: + return ts + try: + tags = parsers.interface_label(interface) + files["events"].write( + { + "ts": ts, + "kind": kind, + "label": label, + "note": note, + "port": tags["port"], + "role": tags["role"], + "data": data, + } + ) + except Exception: + pass + return ts + + # -- introspection ------------------------------------------------ + + def status(self) -> dict[str, Any]: + with self._lock: + return { + "running": self._started, + "paused": self._paused, + "pause_reason": self._pause_reason, + "started_at": self._started_at, + "base_dir": str(self.base_dir), + "files": {name: f.status() for name, f in self._files.items()}, + } + + def force_rotate_all(self) -> dict[str, Any]: + """Test/admin hook: rotate every stream right now.""" + with self._lock: + files = list(self._files.values()) + for f in files: + f.force_rotate() + # `status()` re-acquires `self._lock`; release before calling it. + return self.status() + + +# -- module-level singleton accessor ------------------------------------ + +_INSTANCE_LOCK = threading.Lock() +_INSTANCE: Recorder | None = None + + +def get_recorder() -> Recorder: + """Return the process-global Recorder. Created on first call. + + Honors `MESHTASTIC_MCP_LOG_DIR` env var for the base directory + (used by tests to redirect to a tmpdir). + """ + global _INSTANCE + with _INSTANCE_LOCK: + if _INSTANCE is None: + override = os.environ.get("MESHTASTIC_MCP_LOG_DIR") + base = Path(override) if override else None + _INSTANCE = Recorder(base_dir=base) + return _INSTANCE diff --git a/mcp-server/src/meshtastic_mcp/recorder/rotating.py b/mcp-server/src/meshtastic_mcp/recorder/rotating.py new file mode 100644 index 000000000..631317790 --- /dev/null +++ b/mcp-server/src/meshtastic_mcp/recorder/rotating.py @@ -0,0 +1,163 @@ +"""Append-only JSONL writer with size-capped rotation. + +A `_RotatingJsonl` owns one live `.jsonl` file. Writes are line-delimited +JSON objects (one row per call). When the live file exceeds `max_bytes`, +it is closed, gzipped to `.YYYYMMDD-HHMMSS-uuuuuu-NNNNN.jsonl.gz`, +and the live file resets to empty. Old archives past `keep_archives` are +unlinked oldest-first. + +Size check is amortized — `os.fstat` runs every `check_every` writes, +not per-write, so the hot path stays at one `fh.write` + one `fh.flush`. + +Threading: every public method acquires `self._lock`. The recorder runs +several pubsub handlers on whatever thread the meshtastic library +dispatches from (varies by interface), and queries from MCP tool calls +arrive on the FastMCP request thread, so this lock is not optional. +""" + +from __future__ import annotations + +import gzip +import json +import os +import shutil +import threading +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +class _RotatingJsonl: + """Append-only JSONL with size rotation. Thread-safe.""" + + def __init__( + self, + path: Path, + *, + max_bytes: int = 100 * 1024 * 1024, + keep_archives: int = 5, + check_every: int = 1000, + ) -> None: + self.path = path + self.max_bytes = max_bytes + self.keep_archives = keep_archives + self.check_every = check_every + self._lock = threading.Lock() + self._fh: Any = None + self._writes_since_check = 0 + self._rotations = 0 + self._lines_written = 0 + self._last_ts: float | None = None + self._open() + + # -- lifecycle ---------------------------------------------------- + + def _open(self) -> None: + self.path.parent.mkdir(parents=True, exist_ok=True) + self._fh = self.path.open("a", encoding="utf-8") + + def close(self) -> None: + with self._lock: + if self._fh is not None: + try: + self._fh.close() + finally: + self._fh = None + + # -- write -------------------------------------------------------- + + def write(self, record: dict[str, Any]) -> None: + """Append one JSON object as a line. Triggers rotation if oversized.""" + line = json.dumps(record, separators=(",", ":"), default=str) + "\n" + with self._lock: + if self._fh is None: + return + try: + self._fh.write(line) + self._fh.flush() + except Exception: + # Best-effort: a failed write must not crash the pubsub + # handler. Caller has no way to react anyway. + return + self._lines_written += 1 + ts = record.get("ts") + if isinstance(ts, (int, float)): + self._last_ts = float(ts) + self._writes_since_check += 1 + if self._writes_since_check >= self.check_every: + self._writes_since_check = 0 + self._maybe_rotate() + + # -- rotation ----------------------------------------------------- + + def _maybe_rotate(self) -> None: + # Caller holds self._lock. + try: + size = os.fstat(self._fh.fileno()).st_size + except OSError: + return + if size < self.max_bytes: + return + self._rotate_locked() + + def _rotate_locked(self) -> None: + # Close, gzip-rename, reopen empty, prune oldest archives. + try: + self._fh.close() + except Exception: + pass + self._fh = None + # Microsecond-resolution timestamp + per-instance counter so back- + # to-back rotations (small max_bytes, repeated `force_rotate()`, + # or chatty test loops) get unique archive filenames. The lex + # sort order of `YYYYMMDD-HHMMSS-uuuuuu-NNNNN` is chronological, + # which `_prune_archives()` and `log_query._iter_jsonl()` both + # rely on. + stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S-%f") + archive = self.path.with_suffix(f".{stamp}-{self._rotations:05d}.jsonl.gz") + try: + with self.path.open("rb") as src, gzip.open(archive, "wb") as dst: + shutil.copyfileobj(src, dst, length=1024 * 1024) + self.path.unlink() + except Exception: + # Rotation is best-effort. If gzip fails, leave the file + # in place and re-open it; we'll try again next check. + pass + self._open() + self._rotations += 1 + self._prune_archives() + + def _prune_archives(self) -> None: + # Match siblings of self.path.name with `.jsonl.gz` suffix. + prefix = self.path.stem # "logs" for "logs.jsonl" + # Archive filenames are already lexicographically chronological. + # Prune by name, not mtime, so copied/restored files don't reorder. + archives = sorted(self.path.parent.glob(f"{prefix}.*.jsonl.gz")) + excess = len(archives) - self.keep_archives + for old in archives[: max(0, excess)]: + try: + old.unlink() + except OSError: + pass + + def force_rotate(self) -> None: + """Test/admin hook: rotate immediately regardless of size.""" + with self._lock: + if self._fh is not None: + self._rotate_locked() + + # -- introspection ------------------------------------------------ + + def status(self) -> dict[str, Any]: + with self._lock: + try: + size = os.fstat(self._fh.fileno()).st_size if self._fh else 0 + except OSError: + size = 0 + return { + "path": str(self.path), + "size": size, + "lines": self._lines_written, + "last_ts": self._last_ts, + "rotations": self._rotations, + } diff --git a/mcp-server/src/meshtastic_mcp/serial_session.py b/mcp-server/src/meshtastic_mcp/serial_session.py index 43537323f..fe1a452d0 100644 --- a/mcp-server/src/meshtastic_mcp/serial_session.py +++ b/mcp-server/src/meshtastic_mcp/serial_session.py @@ -46,7 +46,23 @@ class SerialSession: def _drain(session: SerialSession) -> None: - """Reader thread: line-by-line pull stdout into buffer.""" + """Reader thread: line-by-line pull stdout into buffer. + + Each line is also published to the `meshtastic.serial.line` pubsub + topic so the persistent recorder can capture it without holding its + own port. This is the text-mode tap path: when no SerialInterface is + open, the firmware emits full formatted lines (level + clock + uptime + + thread + `[heap N]` prefix on DEBUG_HEAP builds + body), and we + fan them out to whoever is listening. Pubsub is best-effort — + publish failures must never block the reader. + """ + # Lazy import: pubsub isn't required just to import this module + # (e.g., during static analysis), and we want a clean test surface. + try: + from pubsub import pub # type: ignore[import-untyped] + except Exception: # pragma: no cover - defensive + pub = None + assert session.proc.stdout is not None try: for line in session.proc.stdout: @@ -54,6 +70,16 @@ def _drain(session: SerialSession) -> None: with session.lock: session.buffer.append(line_stripped) session.total_lines += 1 + if pub is not None: + try: + pub.sendMessage( + "meshtastic.serial.line", + line=line_stripped, + port=session.port, + ) + except Exception: + # A subscriber raising must not break the reader. + pass except Exception: # pragma: no cover - defensive pass finally: diff --git a/mcp-server/src/meshtastic_mcp/server.py b/mcp-server/src/meshtastic_mcp/server.py index 83aa80c45..573765e26 100644 --- a/mcp-server/src/meshtastic_mcp/server.py +++ b/mcp-server/src/meshtastic_mcp/server.py @@ -6,6 +6,7 @@ etc.). Business logic does not live here. from __future__ import annotations +import logging from typing import Any from mcp.server.fastmcp import FastMCP @@ -17,14 +18,34 @@ from . import ( flash, hw_tools, info, + log_query, registry, serial_session, ) from . import userprefs as userprefs_mod +from .recorder import get_recorder + +log = logging.getLogger(__name__) app = FastMCP("meshtastic-mcp") +def _start_recorder() -> None: + # Persistent device-log capture. Starts on first import — pubsub fan-out + # is process-global, so subscribing here captures every active interface + # (whether opened by an MCP tool, a pytest fixture, or a serial_session). + # Files land in mcp-server/.mtlog/ (gitignored). See recorder/recorder.py + # for the full design. Recorder startup is best-effort: an unwritable + # log dir or pubsub mismatch should not take the MCP server down. + try: + get_recorder().start() + except Exception as exc: + log.warning("Failed to start persistent recorder: %s", exc) + + +_start_recorder() + + # ---------- Discovery & metadata ------------------------------------------ @@ -75,6 +96,7 @@ def build( env: str, with_manifest: bool = True, userprefs: dict[str, Any] | None = None, + build_flags: dict[str, Any] | None = None, ) -> dict[str, Any]: """Build firmware for one env via `pio run -e `. @@ -86,8 +108,21 @@ def build( build via userPrefs.jsonc injection. The file is restored after the build completes. Use `userprefs_manifest` to discover available keys. Use `userprefs_set` for persistent changes. + + `build_flags` (optional): dict of `-D=` macros for this build + only, injected via `PLATFORMIO_BUILD_FLAGS`. Common pattern: + `build_flags={"DEBUG_HEAP": 1}` enables per-thread leak detection + a + `[heap N]` prefix on every log line. The recorder picks the prefix up + automatically and synthesizes a high-resolution heap timeline that + `telemetry_timeline(field="free_heap")` can read alongside the normal + ~60 s LocalStats packets. Pair with `/leakhunt` for classification. """ - return flash.build(env, with_manifest=with_manifest, userprefs_overrides=userprefs) + return flash.build( + env, + with_manifest=with_manifest, + userprefs_overrides=userprefs, + build_flags=build_flags, + ) @app.tool() @@ -105,6 +140,7 @@ def pio_flash( port: str, confirm: bool = False, userprefs: dict[str, Any] | None = None, + build_flags: dict[str, Any] | None = None, ) -> dict[str, Any]: """Flash firmware via `pio run -e -t upload --upload-port `. @@ -114,8 +150,19 @@ def pio_flash( `userprefs` (optional): dict of `USERPREFS_: value` baked into this build via userPrefs.jsonc injection; restored after upload. + + `build_flags` (optional): dict of `-D=` macros for the + rebuild-before-upload, e.g. `{"DEBUG_HEAP": 1}`. Required for the flags + to actually land in the uploaded firmware — without it, the implicit + rebuild relinks without the env var and silently drops them. """ - return flash.flash(env, port, confirm=confirm, userprefs_overrides=userprefs) + return flash.flash( + env, + port, + confirm=confirm, + userprefs_overrides=userprefs, + build_flags=build_flags, + ) @app.tool() @@ -734,3 +781,185 @@ def picotool_load(uf2_path: str, confirm: bool = False) -> dict[str, Any]: def picotool_raw(args: list[str], confirm: bool = False) -> dict[str, Any]: """Pass-through to `picotool`. load/reboot/save/erase require confirm=True.""" return hw_tools.picotool_raw(args, confirm=confirm) + + +# ---------- Persistent device-log capture (recorder) ---------------------- +# +# The recorder is autouse — it starts at server import and continuously +# writes every meshtastic pubsub event to JSONL files under .mtlog/. These +# tools are query-only over those files, plus a few lifecycle controls. + + +@app.tool() +def logs_window( + start: str = "-15m", + end: str = "now", + grep: str | None = None, + level: str | None = None, + tag: str | None = None, + port: str | None = None, + max_lines: int = 200, +) -> dict[str, Any]: + """Recent firmware log lines from the persistent recorder. + + Filters by time window, regex over the line, level (single or + pipe-separated set like "WARN|ERROR|CRIT"), thread-name tag, and + interface port. Returns up to max_lines most-recent matches. + + Time strings: "-15m", "-2h", "-3d", "now", or ISO 8601. + + Note: lines arriving via the LogRecord protobuf path (when + set_debug_log_api(True) is on) come without level prefix — the + meshtastic Python lib drops record.level before fan-out. For those, + `level` filter won't match; use `grep` instead. + """ + return log_query.logs_window( + start=start, + end=end, + grep=grep, + level=level, + tag=tag, + port=port, + max_lines=max_lines, + ) + + +@app.tool() +def telemetry_timeline( + window: str = "1h", + variant: str = "local", + field: str = "free_heap", + port: str | None = None, + max_points: int = 200, +) -> dict[str, Any]: + """Time series of one telemetry field, downsampled to <= max_points. + + `variant` ∈ device, local, environment, power, airQuality, health, host. + `field` accepts snake_case or camelCase; common aliases (free_heap ↔ + heap_free_bytes) are normalized. + + Returns slope_per_min (linear-regression slope, units/minute) so a + leak detector can read one number — negative slope on free_heap over + a long window indicates a real leak. + + LocalStats variant ("local") cadence is ~60 s (whatever the device's + `device_update_interval` is set to), so a 1 h window gives ~60 raw + points. Bucket-mean downsampling preserves shape. + """ + return log_query.telemetry_timeline( + window=window, + variant=variant, + field=field, + port=port, + max_points=max_points, + ) + + +@app.tool() +def packets_window( + start: str = "-5m", + end: str = "now", + portnum: str | None = None, + from_node: str | None = None, + to_node: str | None = None, + max: int = 200, +) -> dict[str, Any]: + """Recent mesh packets recorded by the recorder. + + Each row is a summary (portnum, from/to, hop_limit, RSSI/SNR, payload + size + first 64 bytes hex) — full payload bytes are not stored. + `portnum` accepts a pipe-separated set like "TEXT_MESSAGE_APP|POSITION_APP". + """ + return log_query.packets_window( + start=start, + end=end, + portnum=portnum, + from_node=from_node, + to_node=to_node, + max=max, + ) + + +@app.tool() +def events_window( + start: str = "-1h", + end: str = "now", + kind: str | None = None, + max: int = 200, +) -> dict[str, Any]: + """Return recorder events: connection lifecycle, node updates, and `mark_event` markers. + + `kind` ∈ recorder_start, recorder_pause, recorder_resume, + connection_established, connection_lost, node_updated, mark. + Pipe-separated sets ("connection_lost|connection_established") work. + """ + return log_query.events_window(start=start, end=end, kind=kind, max=max) + + +@app.tool() +def mark_event( + label: str, + note: str | None = None, + data: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Drop a named marker into events.jsonl AND logs.jsonl. + + Useful for aligning a timeline around a known stimulus: call before + and after a stress workload, then query telemetry_timeline / + logs_window with the markers' timestamps as bounds. + + The marker also lands in logs.jsonl with level=MARK so a single + grep over logs picks it up. + """ + return get_recorder().mark_event(label=label, note=note, data=data) + + +@app.tool() +def recorder_status() -> dict[str, Any]: + """Return recorder runtime info: running, paused, file sizes, last_ts per stream. + + Use this to sanity-check that capture is working before you trust a + `logs_window` / `telemetry_timeline` result. + """ + return get_recorder().status() + + +@app.tool() +def recorder_pause(reason: str | None = None) -> dict[str, Any]: + """Pause writes to all four streams. Pubsub subscriptions stay active — + we just drop events on the floor while paused. Resume with `recorder_resume`. + + Use when capturing a known-good baseline that you don't want to + pollute with pre-test noise. Default state is recording; this is + rarely needed. + """ + get_recorder().pause(reason=reason) + return {"ok": True, "paused": True, "reason": reason} + + +@app.tool() +def recorder_resume() -> dict[str, Any]: + """Resume writes after `recorder_pause`. No-op if already running.""" + get_recorder().resume() + return {"ok": True, "paused": False} + + +@app.tool() +def recorder_export( + start: str, + end: str, + dest_dir: str, + streams: list[str] | None = None, +) -> dict[str, Any]: + """Bundle a slice of the recorder's streams into `dest_dir`. + + Writes one uncompressed JSONL per requested stream (logs / telemetry / + packets / events). Useful for: attaching to a bug report, feeding a + notebook, or backfilling Datadog after the fact. + """ + return log_query.export( + start=start, + end=end, + dest_dir=dest_dir, + streams=streams, + ) diff --git a/mcp-server/tests/unit/test_build_flags.py b/mcp-server/tests/unit/test_build_flags.py new file mode 100644 index 000000000..bace81fae --- /dev/null +++ b/mcp-server/tests/unit/test_build_flags.py @@ -0,0 +1,88 @@ +"""Unit tests for the `build_flags` injection on `flash.build()`. + +We don't actually run pio here — too slow, requires hardware-aware envs. +We test the translation layer (`_build_flags_env`) and that the env vars +are threaded through pio.run correctly via mock. +""" + +from __future__ import annotations + +from unittest.mock import patch + +from meshtastic_mcp import flash, pio + + +class TestBuildFlagsEnv: + def test_simple_value(self) -> None: + out = flash._build_flags_env({"DEBUG_HEAP": 1}) + assert out == {"PLATFORMIO_BUILD_FLAGS": "-DDEBUG_HEAP=1"} + + def test_string_value(self) -> None: + out = flash._build_flags_env({"FOO": "bar"}) + assert out == {"PLATFORMIO_BUILD_FLAGS": "-DFOO=bar"} + + def test_bool_true_is_bare_flag(self) -> None: + out = flash._build_flags_env({"DEBUG_HEAP": True}) + assert out == {"PLATFORMIO_BUILD_FLAGS": "-DDEBUG_HEAP"} + + def test_bool_false_dropped(self) -> None: + out = flash._build_flags_env({"DEBUG_HEAP": False, "OTHER": 1}) + assert out == {"PLATFORMIO_BUILD_FLAGS": "-DOTHER=1"} + + def test_none_dropped(self) -> None: + out = flash._build_flags_env({"DEBUG_HEAP": None}) + assert out == {} + + def test_multiple_combined(self) -> None: + out = flash._build_flags_env({"DEBUG_HEAP": 1, "FOO": "x", "BAR": True}) + # Order isn't guaranteed in dict iteration, so check membership. + flags = out["PLATFORMIO_BUILD_FLAGS"].split() + assert set(flags) == {"-DDEBUG_HEAP=1", "-DFOO=x", "-DBAR"} + + +class TestBuildPropagatesFlags: + def test_extra_env_passed_to_pio_run(self) -> None: + # Mock pio.run so we don't actually invoke pio. Capture extra_env. + captured = {} + + class _StubResult: + returncode = 0 + stdout = "" + stderr = "" + duration_s = 0.1 + + def _stub(args, **kwargs): + captured["args"] = args + captured["kwargs"] = kwargs + return _StubResult() + + with patch.object(pio, "run", side_effect=_stub): + with patch.object(flash, "_artifacts_for", return_value=[]): + out = flash.build( + "fake-env", + with_manifest=False, + build_flags={"DEBUG_HEAP": 1}, + ) + assert captured["args"] == ["run", "-e", "fake-env"] + assert captured["kwargs"]["extra_env"] == { + "PLATFORMIO_BUILD_FLAGS": "-DDEBUG_HEAP=1" + } + assert out["build_flags"] == {"DEBUG_HEAP": 1} + + def test_no_flags_means_no_extra_env(self) -> None: + captured = {} + + class _StubResult: + returncode = 0 + stdout = "" + stderr = "" + duration_s = 0.1 + + def _stub(args, **kwargs): + captured["kwargs"] = kwargs + return _StubResult() + + with patch.object(pio, "run", side_effect=_stub): + with patch.object(flash, "_artifacts_for", return_value=[]): + flash.build("fake-env", with_manifest=False) + assert captured["kwargs"]["extra_env"] is None diff --git a/mcp-server/tests/unit/test_recorder.py b/mcp-server/tests/unit/test_recorder.py new file mode 100644 index 000000000..a7d93d78f --- /dev/null +++ b/mcp-server/tests/unit/test_recorder.py @@ -0,0 +1,548 @@ +"""Unit tests for the persistent device-log recorder. + +Hardware-free: drives the Recorder through its `_on_*` handlers with +synthetic packet/line dicts, then queries via log_query. Validates +prefix parsing, telemetry variant dispatch, marker round-trip, time +window filtering, downsampling, slope estimation, and gzip rotation ++ archive pruning. +""" + +from __future__ import annotations + +import gzip +import json +import logging +import os +import time +from pathlib import Path + +import pubsub +import pytest +from meshtastic_mcp import log_query +from meshtastic_mcp.recorder.parsers import ( + extract_telemetry, + interface_label, + parse_log_line, + summarize_packet, +) +from meshtastic_mcp.recorder.recorder import Recorder +from meshtastic_mcp.recorder.rotating import _RotatingJsonl + +# -- isolation: every test gets a fresh Recorder + tmp dir ----------- + + +@pytest.fixture +def recorder(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Recorder: + # Redirect both the Recorder and the module-level singleton lookup + # to the same tmp dir so log_query queries the same files we write. + monkeypatch.setenv("MESHTASTIC_MCP_LOG_DIR", str(tmp_path)) + monkeypatch.setattr( + "meshtastic_mcp.recorder.recorder._INSTANCE", None, raising=False + ) + r = Recorder(base_dir=tmp_path) + r.start() + monkeypatch.setattr("meshtastic_mcp.recorder.recorder._INSTANCE", r, raising=False) + yield r + r.stop() + + +class _FakeIface: + devPath = "/dev/cu.fake" + + +# -- parsers --------------------------------------------------------- + + +class TestParseLogLine: + def test_full_prefix(self) -> None: + out = parse_log_line("INFO | 12:34:56 12345 [Main] Booting") + assert out["level"] == "INFO" + assert out["tag"] == "Main" + assert out["uptime_s"] == 12345 + assert out["msg"] == "Booting" + assert out["clock"] == "12:34:56" + + def test_invalid_clock(self) -> None: + out = parse_log_line("WARN | ??:??:?? 7 [SerialConsole] Boot") + assert out["level"] == "WARN" + assert out["clock"] == "??:??:??" + assert out["uptime_s"] == 7 + + def test_no_thread_bracket(self) -> None: + out = parse_log_line("DEBUG | 00:00:00 0 raw message body") + assert out["level"] == "DEBUG" + assert out.get("tag") is None + assert out["msg"] == "raw message body" + + def test_bare_message(self) -> None: + # LogRecord.message path — no level prefix at all. + out = parse_log_line("just a bare message") + assert "level" not in out or out.get("level") is None + assert out["line"] == "just a bare message" + + def test_empty(self) -> None: + assert parse_log_line("") == {"line": ""} + + def test_debug_heap_prefix_extracted(self) -> None: + out = parse_log_line("INFO | 12:34:56 12345 [Main] [heap 92344] Booting") + assert out["level"] == "INFO" + assert out["tag"] == "Main" + assert out["heap_free"] == 92344 + assert out["msg"] == "Booting" + + def test_debug_heap_prefix_on_bare_line(self) -> None: + # LogRecord.message path: no level prefix but still has [heap N]. + out = parse_log_line("[heap 12345] some message") + assert out["heap_free"] == 12345 + assert out["msg"] == "some message" + + def test_thread_leak_event(self) -> None: + out = parse_log_line( + "HEAP | 00:00:01 100 [Power] [heap 90000] " + "------ Thread MeshPacket leaked heap 92344 -> 90000 (-2344) ------" + ) + assert out["level"] == "HEAP" + assert out["heap_free"] == 90000 + ev = out["heap_event"] + assert ev["kind"] == "leaked" + assert ev["thread"] == "MeshPacket" + assert ev["before"] == 92344 + assert ev["after"] == 90000 + assert ev["delta"] == -2344 + + def test_thread_freed_event(self) -> None: + out = parse_log_line( + "++++++ Thread Router freed heap 1000 -> 1500 (500) ++++++" + ) + ev = out["heap_event"] + assert ev["kind"] == "freed" + assert ev["thread"] == "Router" + assert ev["delta"] == 500 + + def test_heap_status_periodic(self) -> None: + out = parse_log_line( + "HEAP | 00:00:30 30 [Power] " + "Heap status: 92344/200000 bytes free (-128), running 8/12 threads" + ) + assert out["heap_free"] == 92344 + assert out["heap_total"] == 200000 + assert out["heap_delta"] == -128 + + +class TestRecorderDebugHeapSynthesis: + def test_log_with_heap_writes_telemetry(self, recorder: "Recorder") -> None: + # When a log line carries [heap N], the recorder should also + # emit a synthesized telemetry row tagged source=debug_heap. + recorder._on_log_line( + "INFO | 00:00:00 1 [Main] [heap 88888] hello", + _FakeIface(), + ) + telem = (recorder.base_dir / "telemetry.jsonl").read_text().splitlines() + synth = [json.loads(r) for r in telem if '"source":"debug_heap"' in r] + assert len(synth) == 1 + assert synth[0]["fields"]["heap_free_bytes"] == 88888 + assert synth[0]["variant"] == "local" + + def test_heap_status_writes_total_too(self, recorder: "Recorder") -> None: + recorder._on_log_line( + "HEAP | 00:00:30 30 [Power] " + "Heap status: 50000/200000 bytes free (-100), running 8/12 threads", + _FakeIface(), + ) + telem = (recorder.base_dir / "telemetry.jsonl").read_text().splitlines() + synth = [json.loads(r) for r in telem if '"source":"debug_heap"' in r] + assert synth[-1]["fields"]["heap_free_bytes"] == 50000 + assert synth[-1]["fields"]["heap_total_bytes"] == 200000 + + def test_no_heap_no_synthesis(self, recorder: "Recorder") -> None: + # Plain log line (no [heap N], no Heap status) — telemetry.jsonl + # should NOT gain a synth row. + before = (recorder.base_dir / "telemetry.jsonl").read_text().count("\n") + recorder._on_log_line("INFO | 00:00:00 1 [Main] just a message", _FakeIface()) + after = (recorder.base_dir / "telemetry.jsonl").read_text().count("\n") + assert after == before + + def test_thread_leak_event_persists_on_log_row(self, recorder: "Recorder") -> None: + recorder._on_log_line( + "HEAP | 00:00:01 100 [Power] [heap 90000] " + "------ Thread MeshPacket leaked heap 92344 -> 90000 (-2344) ------", + _FakeIface(), + ) + rows = [ + json.loads(r) + for r in (recorder.base_dir / "logs.jsonl").read_text().splitlines() + if r + ] + evt_rows = [r for r in rows if r.get("heap_event")] + assert len(evt_rows) == 1 + assert evt_rows[0]["heap_event"]["thread"] == "MeshPacket" + assert evt_rows[0]["heap_event"]["delta"] == -2344 + + +class TestSerialTap: + def test_serial_line_records_log_and_synthesizes_heap( + self, recorder: "Recorder" + ) -> None: + recorder._on_serial_line( + "INFO | 00:00:00 5 [Main] [heap 88888] tap-line", + port="/dev/cu.tap", + ) + logs = (recorder.base_dir / "logs.jsonl").read_text().splitlines() + telem = (recorder.base_dir / "telemetry.jsonl").read_text().splitlines() + log_rows = [json.loads(r) for r in logs if r] + # Find the row from this call (port=/dev/cu.tap, role=serial_session) + tap_rows = [r for r in log_rows if r.get("port") == "/dev/cu.tap"] + assert len(tap_rows) == 1 + assert tap_rows[0]["role"] == "serial_session" + assert tap_rows[0]["level"] == "INFO" + assert tap_rows[0]["tag"] == "Main" + assert tap_rows[0]["heap_free"] == 88888 + synth = [json.loads(r) for r in telem if '"source":"debug_heap_serial"' in r] + assert len(synth) == 1 + assert synth[0]["fields"]["heap_free_bytes"] == 88888 + assert synth[0]["role"] == "serial_session" + + def test_serial_line_thread_leak_event(self, recorder: "Recorder") -> None: + recorder._on_serial_line( + "HEAP | 00:00:30 30 [Power] [heap 53484] " + "------ Thread Router leaked heap 53612 -> 53484 (-128) ------", + port="/dev/cu.tap", + ) + rows = [ + json.loads(r) + for r in (recorder.base_dir / "logs.jsonl").read_text().splitlines() + if r + ] + evt = [r for r in rows if r.get("heap_event")] + assert len(evt) == 1 + assert evt[0]["heap_event"]["thread"] == "Router" + assert evt[0]["heap_event"]["delta"] == -128 + # Heap also synthesized. + telem = (recorder.base_dir / "telemetry.jsonl").read_text() + assert '"source":"debug_heap_serial"' in telem + + def test_serial_line_pause(self, recorder: "Recorder") -> None: + recorder.pause("baseline") + recorder._on_serial_line( + "INFO | 00:00:00 1 [t] [heap 1000] dropped", + port="/dev/cu.tap", + ) + # Only the pause event row should exist; no tap row. + logs = (recorder.base_dir / "logs.jsonl").read_text() + assert "dropped" not in logs + + def test_serial_line_handler_swallows_exceptions( + self, recorder: "Recorder" + ) -> None: + # Hostile input — should not raise. + recorder._on_serial_line(None, port="/dev/cu.tap") # type: ignore[arg-type] + recorder._on_serial_line(b"\x00\x01\x02\x03", port="/dev/cu.tap") # type: ignore[arg-type] + # Survived. + + +class TestExtractTelemetry: + def test_local_stats_camel(self) -> None: + pkt = { + "decoded": { + "telemetry": { + "localStats": {"heap_total_bytes": 1000, "heap_free_bytes": 600} + } + } + } + out = extract_telemetry(pkt) + assert out is not None + assert out["variant"] == "local" + assert out["fields"]["heap_free_bytes"] == 600 + + def test_device_metrics_snake(self) -> None: + pkt = { + "decoded": { + "telemetry": {"device_metrics": {"battery_level": 88, "voltage": 4.1}} + } + } + out = extract_telemetry(pkt) + assert out is not None + assert out["variant"] == "device" + assert out["fields"]["battery_level"] == 88 + + def test_unknown_variant_returns_none(self) -> None: + assert extract_telemetry({"decoded": {"telemetry": {"weird": {}}}}) is None + assert extract_telemetry({}) is None + assert extract_telemetry({"decoded": "not-a-dict"}) is None + + +class TestSummarizePacket: + def test_text_with_payload(self) -> None: + pkt = { + "fromId": "!abc", + "toId": "!def", + "decoded": {"portnum": "TEXT_MESSAGE_APP", "payload": b"hello"}, + "hopLimit": 3, + } + out = summarize_packet(pkt) + assert out["from_node"] == "!abc" + assert out["portnum"] == "TEXT_MESSAGE_APP" + assert out["payload_size"] == 5 + assert out["payload_hex_prefix"] == "68656c6c6f" + + def test_no_decoded(self) -> None: + out = summarize_packet({"fromId": "!abc"}) + assert out["from_node"] == "!abc" + assert out["portnum"] is None + + +class TestInterfaceLabel: + def test_serial(self) -> None: + assert interface_label(_FakeIface()) == { + "port": "/dev/cu.fake", + "role": "serial", + } + + def test_tcp(self) -> None: + class T: + hostname = "node.lan" + portNumber = 4403 + + assert interface_label(T()) == {"port": "tcp://node.lan:4403", "role": "tcp"} + + def test_unknown(self) -> None: + assert interface_label(object()) == {"port": "object", "role": None} + + def test_none(self) -> None: + assert interface_label(None) == {"port": None, "role": None} + + +# -- recorder write side --------------------------------------------- + + +class TestRecorderWrites: + def test_log_line_is_recorded(self, recorder: Recorder) -> None: + recorder._on_log_line("INFO | 12:34:56 99 [T] hi", _FakeIface()) + path = recorder.base_dir / "logs.jsonl" + rows = [json.loads(line) for line in path.read_text().splitlines() if line] + # First row is recorder_start_event mirror? No — that's events.jsonl only. + assert any(r.get("level") == "INFO" and r.get("tag") == "T" for r in rows) + + def test_telemetry_recorded_and_packet_double(self, recorder: Recorder) -> None: + # _on_telemetry alone — only telemetry.jsonl + recorder._on_telemetry( + { + "fromId": "!abc", + "decoded": {"telemetry": {"localStats": {"heap_free_bytes": 600}}}, + }, + _FakeIface(), + ) + telem_rows = (recorder.base_dir / "telemetry.jsonl").read_text().splitlines() + assert any('"variant":"local"' in r for r in telem_rows) + + def test_packets_summary(self, recorder: Recorder) -> None: + recorder._on_receive( + { + "fromId": "!abc", + "toId": "!def", + "decoded": {"portnum": "TEXT_MESSAGE_APP", "payload": b"hi"}, + }, + _FakeIface(), + ) + rows = (recorder.base_dir / "packets.jsonl").read_text().splitlines() + assert any('"portnum":"TEXT_MESSAGE_APP"' in r for r in rows) + + def test_mark_event_round_trip(self, recorder: Recorder) -> None: + out = recorder.mark_event("checkpoint", note="midpoint") + assert "ts" in out + events = (recorder.base_dir / "events.jsonl").read_text().splitlines() + logs = (recorder.base_dir / "logs.jsonl").read_text().splitlines() + assert any('"label":"checkpoint"' in r and '"kind":"mark"' in r for r in events) + assert any('"level":"MARK"' in r and "checkpoint" in r for r in logs) + + def test_pause_drops_writes(self, recorder: Recorder) -> None: + before = len((recorder.base_dir / "logs.jsonl").read_text().splitlines()) + recorder.pause(reason="baseline") + recorder._on_log_line("INFO | 00:00:00 1 [t] swallowed", _FakeIface()) + after = len((recorder.base_dir / "logs.jsonl").read_text().splitlines()) + assert after == before + recorder.resume() + recorder._on_log_line("INFO | 00:00:00 2 [t] kept", _FakeIface()) + post_resume = (recorder.base_dir / "logs.jsonl").read_text() + assert "kept" in post_resume + + def test_pubsub_handler_swallows_exceptions(self, recorder: Recorder) -> None: + # If the writer dies, the pubsub callback must NOT raise — that + # would crash the meshtastic receive thread. + bad_packet = object() # not a dict + recorder._on_receive(bad_packet, _FakeIface()) # type: ignore[arg-type] + recorder._on_telemetry(bad_packet, _FakeIface()) # type: ignore[arg-type] + recorder._on_log_line(None, _FakeIface()) # type: ignore[arg-type] + # No assertion needed — survival is the test. + + +# -- log_query read side --------------------------------------------- + + +class TestLogQuery: + def test_logs_window_grep_and_level(self, recorder: Recorder) -> None: + recorder._on_log_line("INFO | 12:00:00 1 [A] alpha", _FakeIface()) + recorder._on_log_line("WARN | 12:00:01 2 [B] bravo failed", _FakeIface()) + recorder._on_log_line("ERROR | 12:00:02 3 [C] charlie failed", _FakeIface()) + + out = log_query.logs_window(start="-1m", level="WARN|ERROR", max_lines=10) + assert out["total_matched"] == 2 + levels = {r["level"] for r in out["lines"]} + assert levels == {"WARN", "ERROR"} + + out2 = log_query.logs_window(start="-1m", grep=r"failed$", max_lines=10) + assert out2["total_matched"] == 2 + + def test_logs_window_invalid_regex(self, recorder: Recorder) -> None: + recorder._on_log_line("INFO | 12:00:00 1 [A] alpha", _FakeIface()) + with pytest.raises(ValueError, match="invalid grep regex"): + log_query.logs_window(start="-1m", grep="(") + + def test_telemetry_timeline_slope_and_downsample(self, recorder: Recorder) -> None: + # Synthesize a downward leak: 100 points, free_heap drops 1 byte/sample. + base_ts = time.time() - 60 + for i in range(100): + recorder._files["telemetry"].write( + { + "ts": base_ts + i * 0.5, + "port": "/dev/cu.fake", + "role": "serial", + "from_node": "!abc", + "variant": "local", + "fields": {"heap_free_bytes": 10000 - i}, + } + ) + + out = log_query.telemetry_timeline( + window="2m", variant="local", field="free_heap", max_points=10 + ) + assert out["samples"] == 100 + assert len(out["points"]) <= 10 + # Negative slope (heap dropping). Magnitude: 1 byte every 0.5s = 120/min. + assert out["slope_per_min"] is not None + assert out["slope_per_min"] < -100 + + def test_export_bundles_slice(self, recorder: Recorder, tmp_path: Path) -> None: + recorder._on_log_line("INFO | 00:00:00 1 [t] one", _FakeIface()) + recorder._on_log_line("INFO | 00:00:00 2 [t] two", _FakeIface()) + dest = tmp_path / "bundle" + out = log_query.export(start="-1m", end="now", dest_dir=str(dest)) + assert (dest / "logs.jsonl").exists() + assert "logs" in out["paths"] + + +# -- time parser ----------------------------------------------------- + + +class TestParseTime: + def test_relative(self) -> None: + now = 1_000_000.0 + assert log_query._parse_time("-15m", now=now) == now - 900 + assert log_query._parse_time("-2h", now=now) == now - 7200 + assert log_query._parse_time("-1d", now=now) == now - 86400 + + def test_now_and_epoch(self) -> None: + now = 1_000_000.0 + assert log_query._parse_time("now", now=now) == now + assert log_query._parse_time(now) == now + + def test_iso(self) -> None: + ts = log_query._parse_time("2026-01-01T00:00:00Z") + assert isinstance(ts, float) and ts > 1_700_000_000 + + def test_naive_iso_assumes_utc(self) -> None: + assert log_query._parse_time("2026-01-01T00:00:00") == log_query._parse_time( + "2026-01-01T00:00:00Z" + ) + + def test_invalid(self) -> None: + with pytest.raises(ValueError): + log_query._parse_time("not a time") + + +# -- rotation -------------------------------------------------------- + + +class TestRotation: + def test_size_cap_rotates_and_gzips(self, tmp_path: Path) -> None: + path = tmp_path / "rot.jsonl" + r = _RotatingJsonl(path, max_bytes=512, keep_archives=5, check_every=1) + for i in range(100): + r.write({"ts": float(i), "i": i, "pad": "x" * 40}) + r.close() + archives = sorted(tmp_path.glob("rot.*.jsonl.gz")) + assert archives, "expected at least one rotation" + # Archive content is valid gzip + valid JSONL + with gzip.open(archives[0], "rt") as fh: + first = json.loads(fh.readline()) + assert "ts" in first + + def test_archive_pruning(self, tmp_path: Path) -> None: + path = tmp_path / "rot.jsonl" + r = _RotatingJsonl(path, max_bytes=200, keep_archives=2, check_every=1) + # Force several rotations. + for _ in range(8): + for i in range(20): + r.write({"ts": float(i), "pad": "x" * 30}) + r.force_rotate() + r.close() + archives = sorted(tmp_path.glob("rot.*.jsonl.gz")) + assert len(archives) <= 2, f"expected ≤2 kept archives, got {len(archives)}" + + def test_archive_pruning_uses_filename_order(self, tmp_path: Path) -> None: + path = tmp_path / "rot.jsonl" + r = _RotatingJsonl(path, keep_archives=2) + old = tmp_path / "rot.20260101-000000-000000-00000.jsonl.gz" + mid = tmp_path / "rot.20260101-000001-000000-00000.jsonl.gz" + new = tmp_path / "rot.20260101-000002-000000-00000.jsonl.gz" + for archive in (old, mid, new): + with gzip.open(archive, "wt", encoding="utf-8") as fh: + fh.write('{"ts":1}\n') + # Deliberately scramble mtimes so lexicographic filename order is + # the only stable chronological signal. + os.utime(old, (300, 300)) + os.utime(mid, (100, 100)) + os.utime(new, (200, 200)) + + r._prune_archives() + r.close() + + archives = sorted(p.name for p in tmp_path.glob("rot.*.jsonl.gz")) + assert archives == [mid.name, new.name] + + def test_force_rotate_when_below_threshold(self, tmp_path: Path) -> None: + path = tmp_path / "rot.jsonl" + r = _RotatingJsonl(path, max_bytes=10_000_000, check_every=999_999) + r.write({"ts": 1.0, "msg": "tiny"}) + r.force_rotate() + r.write({"ts": 2.0, "msg": "after-rotate"}) + r.close() + archives = sorted(tmp_path.glob("rot.*.jsonl.gz")) + assert len(archives) == 1 + assert path.exists() + assert "after-rotate" in path.read_text() + + +class TestRecorderLocks: + def test_force_rotate_all_returns_status(self, recorder: Recorder) -> None: + out = recorder.force_rotate_all() + assert out["running"] is True + assert out["files"] + + def test_wire_pubsub_logs_subscription_failure( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, + ) -> None: + class FailingPubSubMock: + def subscribe(self, callback: object, topic: str) -> None: + raise RuntimeError("boom") + + monkeypatch.setattr(pubsub, "pub", FailingPubSubMock()) + recorder = Recorder(base_dir=tmp_path) + with caplog.at_level(logging.WARNING): + recorder._wire_pubsub() + assert ( + "Recorder failed to subscribe to meshtastic.log.line: boom" in caplog.text + ) diff --git a/protobufs b/protobufs index 1d6f1a71f..559f3c12d 160000 --- a/protobufs +++ b/protobufs @@ -1 +1 @@ -Subproject commit 1d6f1a71ff329fa52ad8bb7899951e96f8280a1f +Subproject commit 559f3c12dd2cfdb9a80153546d1755d6dac3005b diff --git a/src/DebugConfiguration.cpp b/src/DebugConfiguration.cpp index a236d85a6..b96d0a9f1 100644 --- a/src/DebugConfiguration.cpp +++ b/src/DebugConfiguration.cpp @@ -26,6 +26,8 @@ SOFTWARE.*/ #include "DebugConfiguration.h" +#include + #ifdef ARCH_PORTDUINO #include "platform/portduino/PortduinoGlue.h" #endif @@ -119,27 +121,22 @@ bool Syslog::vlogf(uint16_t pri, const char *fmt, va_list args) bool Syslog::vlogf(uint16_t pri, const char *appName, const char *fmt, va_list args) { - char *message; - size_t initialLen; - size_t len; - bool result; + // First measure the formatted length using a copy of args; passing args directly + // to vsnprintf consumes it, and reusing a consumed va_list is undefined behavior. + va_list args_measure; + va_copy(args_measure, args); + int needed = vsnprintf(nullptr, 0, fmt, args_measure); + va_end(args_measure); - initialLen = strlen(fmt); + if (needed < 0) + return false; // encoding error - message = new char[initialLen + 1]; + auto message = std::unique_ptr(new char[static_cast(needed) + 1]); + int written = vsnprintf(message.get(), static_cast(needed) + 1, fmt, args); + if (written < 0) + return false; - len = vsnprintf(message, initialLen + 1, fmt, args); - if (len > initialLen) { - delete[] message; - message = new char[len + 1]; - - vsnprintf(message, len + 1, fmt, args); - } - - result = this->_sendLog(pri, appName, message); - - delete[] message; - return result; + return this->_sendLog(pri, appName, message.get()); } inline bool Syslog::_sendLog(uint16_t pri, const char *appName, const char *message) diff --git a/src/GPSStatus.h b/src/GPSStatus.h index a1a9f2c56..25c7b1039 100644 --- a/src/GPSStatus.h +++ b/src/GPSStatus.h @@ -53,8 +53,7 @@ class GPSStatus : public Status int32_t getLatitude() const { if (config.position.fixed_position) { - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); - return node->position.latitude_i; + return localPosition.latitude_i; } else { return p.latitude_i; } @@ -63,8 +62,7 @@ class GPSStatus : public Status int32_t getLongitude() const { if (config.position.fixed_position) { - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); - return node->position.longitude_i; + return localPosition.longitude_i; } else { return p.longitude_i; } @@ -73,8 +71,7 @@ class GPSStatus : public Status int32_t getAltitude() const { if (config.position.fixed_position) { - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); - return node->position.altitude; + return localPosition.altitude; } else { return p.altitude; } diff --git a/src/concurrency/Lock.cpp b/src/concurrency/Lock.cpp index 0fe80e455..4596e0edf 100644 --- a/src/concurrency/Lock.cpp +++ b/src/concurrency/Lock.cpp @@ -14,6 +14,11 @@ Lock::Lock() : handle(xSemaphoreCreateBinary()) } } +Lock::~Lock() +{ + vSemaphoreDelete(handle); +} + void Lock::lock() { if (xSemaphoreTake(handle, portMAX_DELAY) == false) { @@ -30,6 +35,8 @@ void Lock::unlock() #else Lock::Lock() {} +Lock::~Lock() {} + void Lock::lock() {} void Lock::unlock() {} diff --git a/src/concurrency/Lock.h b/src/concurrency/Lock.h index 1b9ea20d5..a51248117 100644 --- a/src/concurrency/Lock.h +++ b/src/concurrency/Lock.h @@ -12,6 +12,7 @@ class Lock { public: Lock(); + ~Lock(); Lock(const Lock &) = delete; Lock &operator=(const Lock &) = delete; diff --git a/src/detect/LoRaRadioType.h b/src/detect/LoRaRadioType.h index a059a3668..f741b929c 100644 --- a/src/detect/LoRaRadioType.h +++ b/src/detect/LoRaRadioType.h @@ -11,7 +11,8 @@ enum LoRaRadioType { SX1280_RADIO, LR1110_RADIO, LR1120_RADIO, - LR1121_RADIO + LR1121_RADIO, + LR2021_RADIO }; extern LoRaRadioType radioType; \ No newline at end of file diff --git a/src/gps/GPS.cpp b/src/gps/GPS.cpp index 814983d2f..e581eb0ef 100644 --- a/src/gps/GPS.cpp +++ b/src/gps/GPS.cpp @@ -1025,10 +1025,13 @@ void GPS::up() setPowerState(GPS_ACTIVE); } -// We've got a GPS lock. Enter a low power state, potentially. +// We've finished a GPS search cycle (lock or timeout). Enter a low power state, potentially. void GPS::down() { - scheduling.informGotLock(); + if (hasValidLocation) + scheduling.informGotLock(); + else + scheduling.informSearchFailed(); uint32_t predictedSearchDuration = scheduling.predictedSearchDurationMs(); uint32_t sleepTime = scheduling.msUntilNextSearch(); uint32_t updateInterval = Default::getConfiguredOrDefaultMs(config.position.gps_update_interval); diff --git a/src/gps/GPSUpdateScheduling.cpp b/src/gps/GPSUpdateScheduling.cpp index 53d6c833f..07920e967 100644 --- a/src/gps/GPSUpdateScheduling.cpp +++ b/src/gps/GPSUpdateScheduling.cpp @@ -15,6 +15,19 @@ void GPSUpdateScheduling::informGotLock() searchEndedMs = millis(); LOG_DEBUG("Took %us to get lock", (searchEndedMs - searchStartedMs) / 1000); updateLockTimePrediction(); + consecutiveFailures = 0; // Drop back to fast cadence as soon as we acquire any fix +} + +// Search finished without obtaining a fix. We still need to mark the end time so +// the next sleep is timed correctly, but we must not feed the timeout duration +// into predictedMsToGetLock — doing so poisons msUntilNextSearch() and causes +// down() to fall into GPS_IDLE, leaving the chip awake on subsequent indoor cycles. +void GPSUpdateScheduling::informSearchFailed() +{ + searchEndedMs = millis(); + consecutiveFailures++; + LOG_DEBUG("GPS search ended without fix after %us (consecutive failures: %u)", (searchEndedMs - searchStartedMs) / 1000, + consecutiveFailures); } // Clear old lock-time prediction data. @@ -25,6 +38,7 @@ void GPSUpdateScheduling::reset() searchEndedMs = 0; searchCount = 0; predictedMsToGetLock = 0; + consecutiveFailures = 0; } // How many milliseconds before we should next search for GPS position @@ -36,6 +50,20 @@ uint32_t GPSUpdateScheduling::msUntilNextSearch() // Target interval (seconds), between GPS updates uint32_t updateInterval = Default::getConfiguredOrDefaultMs(config.position.gps_update_interval, default_gps_update_interval); + // After a failed search, back off: indoors / no-sky environments will keep failing, + // so wake at most once per broadcast interval rather than once per gps_update_interval. + // Capped at 1 hour so a user-configured very-long broadcast interval still retries + // periodically (in case conditions change). Reset on any successful lock. + if (consecutiveFailures > 0) { + constexpr uint32_t failureRetryCapMs = 60UL * 60UL * 1000UL; // 1 hour cap + uint32_t failureSleepMs = + Default::getConfiguredOrDefaultMs(config.position.position_broadcast_secs, default_broadcast_interval_secs); + if (failureSleepMs > failureRetryCapMs) + failureSleepMs = failureRetryCapMs; + if (updateInterval < failureSleepMs) + updateInterval = failureSleepMs; + } + // Check how long until we should start searching, to hopefully hit our target interval uint32_t dueAtMs = searchEndedMs + updateInterval; uint32_t compensatedStart = dueAtMs - predictedMsToGetLock; @@ -71,14 +99,18 @@ bool GPSUpdateScheduling::isUpdateDue() bool GPSUpdateScheduling::searchedTooLong() { constexpr uint32_t oneMinuteMs = 60UL * 1000UL; - constexpr uint32_t maxSearchClampMs = 15UL * oneMinuteMs; // Hard cap: 15 minutes is always too long + constexpr uint32_t maxSearchClampMs = 15UL * oneMinuteMs; // Hard cap: 15 minutes is always too long + constexpr uint32_t postFailureSearchMs = 5UL * oneMinuteMs; // Tighter dwell once we know the environment is hostile uint32_t elapsed = elapsedSearchMs(); // Anything over 15 minutes is too long, regardless of the broadcast interval. - // TODO: Make a smarter algorithm that backs off the search dwell time when not getting a lock. if (elapsed > maxSearchClampMs) return true; + // After a prior failed search, shorten the dwell + if (consecutiveFailures > 0 && elapsed > postFailureSearchMs) + return true; + uint32_t minimumOrConfiguredSecs = Default::getConfiguredOrMinimumValue(config.position.position_broadcast_secs, default_broadcast_interval_secs); uint32_t maxSearchMs = Default::getConfiguredOrDefaultMs(minimumOrConfiguredSecs, default_broadcast_interval_secs); diff --git a/src/gps/GPSUpdateScheduling.h b/src/gps/GPSUpdateScheduling.h index 7e121c9b6..120605c4e 100644 --- a/src/gps/GPSUpdateScheduling.h +++ b/src/gps/GPSUpdateScheduling.h @@ -8,7 +8,8 @@ class GPSUpdateScheduling public: // Marks the time of these events, for calculation use void informSearching(); - void informGotLock(); // Predicted lock-time is recalculated here + void informGotLock(); // Predicted lock-time is recalculated here + void informSearchFailed(); // Search ended without a fix; prediction is left untouched void reset(); // Reset the prediction - after GPS::disable() / GPS::enable() bool isUpdateDue(); // Is it time to begin searching for a GPS position? @@ -24,6 +25,7 @@ class GPSUpdateScheduling uint32_t searchEndedMs = 0; uint32_t searchCount = 0; uint32_t predictedMsToGetLock = 0; + uint32_t consecutiveFailures = 0; // Count of search cycles that ended without a fix; reset on lock const float weighting = 0.2; // Controls exponential smoothing of lock-times prediction. 20% weighting of "latest lock-time". }; \ No newline at end of file diff --git a/src/graphics/Screen.cpp b/src/graphics/Screen.cpp index d02938df9..205e71bcf 100644 --- a/src/graphics/Screen.cpp +++ b/src/graphics/Screen.cpp @@ -1285,7 +1285,7 @@ void Screen::setFrames(FrameFocus focus) for (size_t i = 0; i < nodeDB->getNumMeshNodes(); i++) { const meshtastic_NodeInfoLite *n = nodeDB->getMeshNodeByIndex(i); - if (n && n->num != nodeDB->getNodeNum() && n->is_favorite) { + if (n && n->num != nodeDB->getNodeNum() && nodeInfoLiteIsFavorite(n)) { favoriteFrames.push_back(graphics::UIRenderer::drawFavoriteNode); } } @@ -1670,7 +1670,7 @@ int Screen::handleTextMessage(const meshtastic_MeshPacket *packet) const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(packet->from); const meshtastic_Channel channel = channels.getByIndex(packet->channel ? packet->channel : channels.getPrimaryIndex()); - const char *longName = (node && node->has_user) ? node->user.long_name : nullptr; + const char *longName = nodeInfoLiteHasUser(node) ? node->long_name : nullptr; const char *msgRaw = reinterpret_cast(packet->decoded.payload.bytes); diff --git a/src/graphics/draw/MenuHandler.cpp b/src/graphics/draw/MenuHandler.cpp index f31cb405b..382b89f2f 100644 --- a/src/graphics/draw/MenuHandler.cpp +++ b/src/graphics/draw/MenuHandler.cpp @@ -845,10 +845,10 @@ void menuHandler::messageViewModeMenu() // Encode peers for (size_t i = 0; i < uniquePeers.size(); ++i) { uint32_t peer = uniquePeers[i]; - auto node = nodeDB->getMeshNode(peer); + const auto *node = nodeDB->getMeshNode(peer); std::string name; - if (node && node->has_user) - name = sanitizeString(node->user.long_name).substr(0, 15); + if (nodeInfoLiteHasUser(node)) + name = sanitizeString(node->long_name).substr(0, 15); else { char buf[20]; snprintf(buf, sizeof(buf), "Node %08X", peer); @@ -1355,14 +1355,14 @@ void menuHandler::manageNodeMenu() static int optionsEnumArray[enumEnd] = {Back}; int options = 1; - if (node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { optionsArray[options] = "Unfavorite"; } else { optionsArray[options] = "Favorite"; } optionsEnumArray[options++] = Favorite; - bool isMuted = (node->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0; + bool isMuted = nodeInfoLiteIsMuted(node); if (isMuted) { optionsArray[options] = "Unmute Notifications"; } else { @@ -1376,7 +1376,7 @@ void menuHandler::manageNodeMenu() optionsArray[options] = "Key Verification"; optionsEnumArray[options++] = KeyVerification; - if (node->is_ignored) { + if (nodeInfoLiteIsIgnored(node)) { optionsArray[options] = "Unignore Node"; } else { optionsArray[options] = "Ignore Node"; @@ -1386,8 +1386,8 @@ void menuHandler::manageNodeMenu() BannerOverlayOptions bannerOptions; std::string title = ""; - if (node->has_user && node->user.long_name && node->user.long_name[0]) { - title += sanitizeString(node->user.long_name).substr(0, 15); + if (nodeInfoLiteHasUser(node) && node->long_name[0]) { + title += sanitizeString(node->long_name).substr(0, 15); } else { char buf[20]; snprintf(buf, sizeof(buf), "%08X", (unsigned int)node->num); @@ -1409,7 +1409,7 @@ void menuHandler::manageNodeMenu() if (!n) { return; } - if (n->is_favorite) { + if (nodeInfoLiteIsFavorite(n)) { LOG_INFO("Removing node %08X from favorites", menuHandler::pickedNodeNum); nodeDB->set_favorite(false, menuHandler::pickedNodeNum); } else { @@ -1426,13 +1426,9 @@ void menuHandler::manageNodeMenu() return; } - if (n->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) { - n->bitfield &= ~NODEINFO_BITFIELD_IS_MUTED_MASK; - LOG_INFO("Unmuted node %08X", menuHandler::pickedNodeNum); - } else { - n->bitfield |= NODEINFO_BITFIELD_IS_MUTED_MASK; - LOG_INFO("Muted node %08X", menuHandler::pickedNodeNum); - } + const bool wasMuted = nodeInfoLiteIsMuted(n); + nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_MUTED_MASK, !wasMuted); + LOG_INFO(wasMuted ? "Unmuted node %08X" : "Muted node %08X", menuHandler::pickedNodeNum); nodeDB->notifyObservers(true); nodeDB->saveToDisk(); screen->setFrames(graphics::Screen::FOCUS_PRESERVE); @@ -1461,11 +1457,11 @@ void menuHandler::manageNodeMenu() return; } - if (n->is_ignored) { - n->is_ignored = false; + if (nodeInfoLiteIsIgnored(n)) { + nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, false); LOG_INFO("Unignoring node %08X", menuHandler::pickedNodeNum); } else { - n->is_ignored = true; + nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); LOG_INFO("Ignoring node %08X", menuHandler::pickedNodeNum); } nodeDB->notifyObservers(true); @@ -2079,9 +2075,9 @@ void menuHandler::removeFavoriteMenu() static const char *optionsArray[] = {"Back", "Yes"}; BannerOverlayOptions bannerOptions; std::string message = "Unfavorite This Node?\n"; - auto node = nodeDB->getMeshNode(graphics::UIRenderer::currentFavoriteNodeNum); - if (node && node->has_user) { - message += sanitizeString(node->user.long_name).substr(0, 15); + const auto *node = nodeDB->getMeshNode(graphics::UIRenderer::currentFavoriteNodeNum); + if (nodeInfoLiteHasUser(node)) { + message += sanitizeString(node->long_name).substr(0, 15); } bannerOptions.message = message.c_str(); bannerOptions.optionsArrayPtr = optionsArray; diff --git a/src/graphics/draw/MessageRenderer.cpp b/src/graphics/draw/MessageRenderer.cpp index 2260c57df..a42ef1e08 100644 --- a/src/graphics/draw/MessageRenderer.cpp +++ b/src/graphics/draw/MessageRenderer.cpp @@ -462,8 +462,8 @@ void drawTextMessageFrame(OLEDDisplay *display, OLEDDisplayUiState *state, int16 } case ThreadMode::DIRECT: { meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(currentPeer); - if (node && node->has_user && node->user.short_name[0]) { - snprintf(titleStr, sizeof(titleStr), "@%s", node->user.short_name); + if (nodeInfoLiteHasUser(node) && node->short_name[0]) { + snprintf(titleStr, sizeof(titleStr), "@%s", node->short_name); } else { snprintf(titleStr, sizeof(titleStr), "@%08x", currentPeer); } @@ -585,11 +585,11 @@ void drawTextMessageFrame(OLEDDisplay *display, OLEDDisplayUiState *state, int16 meshtastic_NodeInfoLite *node_recipient = nodeDB->getMeshNode(m.dest); char senderName[64] = ""; - if (node && node->has_user) { - if (node->user.long_name[0]) { - strncpy(senderName, node->user.long_name, sizeof(senderName) - 1); - } else if (node->user.short_name[0]) { - strncpy(senderName, node->user.short_name, sizeof(senderName) - 1); + if (nodeInfoLiteHasUser(node)) { + if (node->long_name[0]) { + strncpy(senderName, node->long_name, sizeof(senderName) - 1); + } else if (node->short_name[0]) { + strncpy(senderName, node->short_name, sizeof(senderName) - 1); } senderName[sizeof(senderName) - 1] = '\0'; } @@ -599,18 +599,17 @@ void drawTextMessageFrame(OLEDDisplay *display, OLEDDisplayUiState *state, int16 // If this is *our own* message, override senderName to who the recipient was bool mine = (m.sender == nodeDB->getNodeNum()); - if (mine && node_recipient && node_recipient->has_user) { - if (node_recipient->user.long_name[0]) { - strncpy(senderName, node_recipient->user.long_name, sizeof(senderName) - 1); + if (mine && nodeInfoLiteHasUser(node_recipient)) { + if (node_recipient->long_name[0]) { + strncpy(senderName, node_recipient->long_name, sizeof(senderName) - 1); senderName[sizeof(senderName) - 1] = '\0'; - } else if (node_recipient->user.short_name[0]) { - strncpy(senderName, node_recipient->user.short_name, sizeof(senderName) - 1); + } else if (node_recipient->short_name[0]) { + strncpy(senderName, node_recipient->short_name, sizeof(senderName) - 1); senderName[sizeof(senderName) - 1] = '\0'; } } // If recipient info is missing/empty, prefer a recipient identifier for outbound messages. - if (mine && (!node_recipient || !node_recipient->has_user || - (!node_recipient->user.long_name[0] && !node_recipient->user.short_name[0]))) { + if (mine && (!nodeInfoLiteHasUser(node_recipient) || (!node_recipient->long_name[0] && !node_recipient->short_name[0]))) { snprintf(senderName, sizeof(senderName), "(%08x)", m.dest); } @@ -1073,12 +1072,12 @@ void handleNewMessage(OLEDDisplay *display, const StoredMessage &sm, const mesht // Banner logic const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(packet.from); char longName[64] = "?"; - if (node && node->has_user) { - if (node->user.long_name[0]) { - strncpy(longName, node->user.long_name, sizeof(longName) - 1); + if (nodeInfoLiteHasUser(node)) { + if (node->long_name[0]) { + strncpy(longName, node->long_name, sizeof(longName) - 1); longName[sizeof(longName) - 1] = '\0'; - } else if (node->user.short_name[0]) { - strncpy(longName, node->user.short_name, sizeof(longName) - 1); + } else if (node->short_name[0]) { + strncpy(longName, node->short_name, sizeof(longName) - 1); longName[sizeof(longName) - 1] = '\0'; } } diff --git a/src/graphics/draw/NodeListRenderer.cpp b/src/graphics/draw/NodeListRenderer.cpp index 00ae74b58..d0b027c13 100644 --- a/src/graphics/draw/NodeListRenderer.cpp +++ b/src/graphics/draw/NodeListRenderer.cpp @@ -98,29 +98,23 @@ std::string getSafeNodeName(OLEDDisplay *display, meshtastic_NodeInfoLite *node, // 1) Choose target candidate (long vs short) only if present const char *raw = nullptr; -#if !MESHTASTIC_EXCLUDE_STATUS +#if !MESHTASTIC_EXCLUDE_STATUS && !MESHTASTIC_EXCLUDE_STATUSDB // If long-name mode is enabled, and we have a recent status for this node, - // prefer "(short_name) statusText" as the raw candidate. + // prefer "(short_name) statusText" as the raw candidate. Pull straight out + // of NodeDB's per-NodeNum cache instead of scanning a FIFO. std::string composedFromStatus; - if (config.display.use_long_node_name && node && node->has_user && statusMessageModule) { - const auto &recent = statusMessageModule->getRecentReceived(); - const StatusMessageModule::RecentStatus *found = nullptr; - for (auto it = recent.rbegin(); it != recent.rend(); ++it) { - if (it->fromNodeId == node->num && !it->statusText.empty()) { - found = &(*it); - break; - } - } - - if (found) { - const char *shortName = node->user.short_name; - composedFromStatus.reserve(4 + (shortName ? std::strlen(shortName) : 0) + 1 + found->statusText.size()); + if (config.display.use_long_node_name && nodeInfoLiteHasUser(node) && nodeDB) { + meshtastic_StatusMessage cachedStatus; + if (nodeDB->copyNodeStatus(node->num, cachedStatus) && cachedStatus.status[0]) { + const char *shortName = node->short_name; + const size_t statusLen = std::strlen(cachedStatus.status); + composedFromStatus.reserve(4 + (shortName ? std::strlen(shortName) : 0) + 1 + statusLen); composedFromStatus += "("; if (shortName && *shortName) { composedFromStatus += shortName; } composedFromStatus += ") "; - composedFromStatus += found->statusText; + composedFromStatus += cachedStatus.status; raw = composedFromStatus.c_str(); // safe for now; we'll sanitize immediately into std::string } @@ -129,8 +123,8 @@ std::string getSafeNodeName(OLEDDisplay *display, meshtastic_NodeInfoLite *node, // If we didn't compose from status, use normal long/short selection if (!raw) { - if (node && node->has_user) { - raw = config.display.use_long_node_name ? node->user.long_name : node->user.short_name; + if (nodeInfoLiteHasUser(node)) { + raw = config.display.use_long_node_name ? node->long_name : node->short_name; } } @@ -218,7 +212,7 @@ void drawScrollbar(OLEDDisplay *display, int visibleNodeRows, int totalEntries, static inline void applyFavoriteNodeNameColor(OLEDDisplay *display, const meshtastic_NodeInfoLite *node, const char *nodeName, int16_t nameX, int16_t y, int nameMaxWidth) { - if (!display || !node || !node->is_favorite || !isTFTColoringEnabled() || !nodeName) { + if (!display || !node || !nodeInfoLiteIsFavorite(node) || !isTFTColoringEnabled() || !nodeName) { return; } @@ -259,7 +253,7 @@ void drawEntryLastHeard(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int #if GRAPHICS_TFT_COLORING_ENABLED applyFavoriteNodeNameColor(display, node, nodeName, nameX, y, nameMaxWidth); #endif - bool isMuted = (node->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0; + bool isMuted = nodeInfoLiteIsMuted(node); char timeStr[10]; uint32_t seconds = sinceLastSeen(node); @@ -279,14 +273,14 @@ void drawEntryLastHeard(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int display->setTextAlignment(TEXT_ALIGN_LEFT); display->setFont(FONT_SMALL); UIRenderer::drawStringWithEmotes(display, nameX, y, nodeName, FONT_HEIGHT_SMALL, 1, false); - if (node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { if (currentResolution == ScreenResolution::High) { drawScaledXBitmap16x16(x, y + 6, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint, display); } else { display->drawXbm(x, y + 5, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint); } } - if (node->is_ignored || isMuted) { + if (nodeInfoLiteIsIgnored(node) || isMuted) { if (currentResolution == ScreenResolution::High) { display->drawLine(x + 8, y + 8, (isLeftCol ? 0 : x - 4) + nameMaxWidth - 17, y + 8); } else { @@ -321,20 +315,20 @@ void drawEntryHopSignal(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int #if GRAPHICS_TFT_COLORING_ENABLED applyFavoriteNodeNameColor(display, node, nodeName, nameX, y, nameMaxWidth); #endif - bool isMuted = (node->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0; + bool isMuted = nodeInfoLiteIsMuted(node); display->setTextAlignment(TEXT_ALIGN_LEFT); display->setFont(FONT_SMALL); UIRenderer::drawStringWithEmotes(display, nameX, y, nodeName, FONT_HEIGHT_SMALL, 1, false); - if (node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { if (currentResolution == ScreenResolution::High) { drawScaledXBitmap16x16(x, y + 6, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint, display); } else { display->drawXbm(x, y + 5, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint); } } - if (node->is_ignored || isMuted) { + if (nodeInfoLiteIsIgnored(node) || isMuted) { if (currentResolution == ScreenResolution::High) { display->drawLine(x + 8, y + 8, (isLeftCol ? 0 : x - 4) + nameMaxWidth - 17, y + 8); } else { @@ -401,15 +395,19 @@ void drawNodeDistance(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int16 #if GRAPHICS_TFT_COLORING_ENABLED applyFavoriteNodeNameColor(display, node, nodeName, nameX, y, nameMaxWidth); #endif - bool isMuted = (node->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0; + bool isMuted = nodeInfoLiteIsMuted(node); char distStr[10] = ""; - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); - if (nodeDB->hasValidPosition(ourNode) && nodeDB->hasValidPosition(node)) { - double lat1 = ourNode->position.latitude_i * 1e-7; - double lon1 = ourNode->position.longitude_i * 1e-7; - double lat2 = node->position.latitude_i * 1e-7; - double lon2 = node->position.longitude_i * 1e-7; + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + meshtastic_PositionLite ourPos; + meshtastic_PositionLite theirPos; + const bool haveOurPos = ourNode && nodeDB->copyNodePosition(ourNode->num, ourPos); + const bool haveTheirPos = nodeDB->copyNodePosition(node->num, theirPos); + if (nodeDB->hasValidPosition(ourNode) && nodeDB->hasValidPosition(node) && haveOurPos && haveTheirPos) { + double lat1 = ourPos.latitude_i * 1e-7; + double lon1 = ourPos.longitude_i * 1e-7; + double lat2 = theirPos.latitude_i * 1e-7; + double lon2 = theirPos.longitude_i * 1e-7; double earthRadiusKm = 6371.0; double dLat = (lat2 - lat1) * DEG_TO_RAD; @@ -455,14 +453,14 @@ void drawNodeDistance(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int16 display->setTextAlignment(TEXT_ALIGN_LEFT); display->setFont(FONT_SMALL); UIRenderer::drawStringWithEmotes(display, nameX, y, nodeName, FONT_HEIGHT_SMALL, 1, false); - if (node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { if (currentResolution == ScreenResolution::High) { drawScaledXBitmap16x16(x, y + 6, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint, display); } else { display->drawXbm(x, y + 5, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint); } } - if (node->is_ignored || isMuted) { + if (nodeInfoLiteIsIgnored(node) || isMuted) { if (currentResolution == ScreenResolution::High) { display->drawLine(x + 8, y + 8, (isLeftCol ? 0 : x - 4) + nameMaxWidth - 17, y + 8); } else { @@ -509,19 +507,19 @@ void drawEntryCompass(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int16 #if GRAPHICS_TFT_COLORING_ENABLED applyFavoriteNodeNameColor(display, node, nodeName, nameX, y, nameMaxWidth); #endif - bool isMuted = (node->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0; + bool isMuted = nodeInfoLiteIsMuted(node); display->setTextAlignment(TEXT_ALIGN_LEFT); display->setFont(FONT_SMALL); UIRenderer::drawStringWithEmotes(display, nameX, y, nodeName, FONT_HEIGHT_SMALL, 1, false); - if (node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { if (currentResolution == ScreenResolution::High) { drawScaledXBitmap16x16(x, y + 6, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint, display); } else { display->drawXbm(x, y + 5, smallbulletpoint_width, smallbulletpoint_height, smallbulletpoint); } } - if (node->is_ignored || isMuted) { + if (nodeInfoLiteIsIgnored(node) || isMuted) { if (currentResolution == ScreenResolution::High) { display->drawLine(x + 8, y + 8, (isLeftCol ? 0 : x - 4) + nameMaxWidth - 17, y + 8); } else { @@ -542,8 +540,11 @@ void drawCompassArrow(OLEDDisplay *display, meshtastic_NodeInfoLite *node, int16 int centerX = x + columnWidth - arrowXOffset; int centerY = y + FONT_HEIGHT_SMALL / 2; - double nodeLat = node->position.latitude_i * 1e-7; - double nodeLon = node->position.longitude_i * 1e-7; + meshtastic_PositionLite nodePos; + if (!nodeDB->copyNodePosition(node->num, nodePos)) + return; + double nodeLat = nodePos.latitude_i * 1e-7; + double nodeLon = nodePos.longitude_i * 1e-7; float bearing = GeoCoord::bearing(userLat, userLon, nodeLat, nodeLon); float relativeBearing = CompassRenderer::adjustBearingForCompassMode(bearing, myHeadingRadian); float relativeBearingDeg = CompassRenderer::radiansToDegrees360(relativeBearing); @@ -652,7 +653,7 @@ void drawNodeListScreen(OLEDDisplay *display, OLEDDisplayUiState *state, int16_t continue; if (n->num == nodeDB->getNodeNum()) continue; - if (locationScreen && !n->has_position) + if (locationScreen && !nodeDB->hasNodePosition(n->num)) continue; drawList.push_back(n->num); @@ -883,14 +884,15 @@ void drawDistanceScreen(OLEDDisplay *display, OLEDDisplayUiState *state, int16_t void drawNodeListWithCompasses(OLEDDisplay *display, OLEDDisplayUiState *state, int16_t x, int16_t y) { float headingRadian = 0.0f; - auto ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); - if (!ourNode || !nodeDB->hasValidPosition(ourNode)) { + const auto *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + meshtastic_PositionLite ourSelfPos; + if (!ourNode || !nodeDB->hasValidPosition(ourNode) || !nodeDB->copyNodePosition(ourNode->num, ourSelfPos)) { drawNodeListScreen(display, state, x, y, "Bearings", drawEntryCompass, drawCompassUnknown, headingRadian, 0.0, 0.0); return; } - double lat = DegD(ourNode->position.latitude_i); - double lon = DegD(ourNode->position.longitude_i); + double lat = DegD(ourSelfPos.latitude_i); + double lon = DegD(ourSelfPos.longitude_i); #if defined(M5STACK_UNITC6L) display->clear(); diff --git a/src/graphics/draw/NotificationRenderer.cpp b/src/graphics/draw/NotificationRenderer.cpp index cca60d1e2..8d031cf73 100644 --- a/src/graphics/draw/NotificationRenderer.cpp +++ b/src/graphics/draw/NotificationRenderer.cpp @@ -317,12 +317,12 @@ void NotificationRenderer::drawNodePicker(OLEDDisplay *display, OLEDDisplayUiSta for (int i = firstOptionToShow; i < alertBannerOptions && linesShown < visibleTotalLines; i++, linesShown++) { char tempName[48] = {0}; meshtastic_NodeInfoLite *node = nodeDB->getMeshNodeByIndex(i + 1); - if (node && node->has_user) { + if (nodeInfoLiteHasUser(node)) { const char *rawName = nullptr; - if (node->user.long_name[0]) { - rawName = node->user.long_name; - } else if (node->user.short_name[0]) { - rawName = node->user.short_name; + if (node->long_name[0]) { + rawName = node->long_name; + } else if (node->short_name[0]) { + rawName = node->short_name; } if (rawName) { const int arrowWidth = (currentResolution == ScreenResolution::High) diff --git a/src/graphics/draw/UIRenderer.cpp b/src/graphics/draw/UIRenderer.cpp index b75bcd17b..cfde10124 100644 --- a/src/graphics/draw/UIRenderer.cpp +++ b/src/graphics/draw/UIRenderer.cpp @@ -464,11 +464,11 @@ static bool computeBottomCompassPlacement(OLEDDisplay *display, int16_t xOffset, return true; } -static void drawTruncatedStatusLine(OLEDDisplay *display, int16_t x, int16_t y, const std::string &statusText) +static void drawTruncatedStatusLine(OLEDDisplay *display, int16_t x, int16_t y, const char *statusText) { // Fixed-buffer truncate helper replaces iterative std::string chopping to keep code size down. char rawStatus[96]; - snprintf(rawStatus, sizeof(rawStatus), " Status: %s", statusText.c_str()); + snprintf(rawStatus, sizeof(rawStatus), " Status: %s", statusText ? statusText : ""); char clippedStatus[96]; UIRenderer::truncateStringWithEmotes(display, rawStatus, clippedStatus, sizeof(clippedStatus), display->getWidth()); @@ -495,7 +495,7 @@ void graphics::UIRenderer::rebuildFavoritedNodes() meshtastic_NodeInfoLite *n = nodeDB->getMeshNodeByIndex(i); if (!n || n->num == nodeDB->getNodeNum()) continue; - if (n->is_favorite) + if (nodeInfoLiteIsFavorite(n)) favoritedNodes.push_back(n); } @@ -750,7 +750,7 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat return; meshtastic_NodeInfoLite *node = favoritedNodes[nodeIndex]; - if (!node || node->num == nodeDB->getNodeNum() || !node->is_favorite) + if (!node || node->num == nodeDB->getNodeNum() || !nodeInfoLiteIsFavorite(node)) return; display->clear(); #if defined(M5STACK_UNITC6L) @@ -763,7 +763,7 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat #endif currentFavoriteNodeNum = node->num; // === Create the shortName and title string === - const char *shortName = (node->has_user && node->user.short_name[0]) ? node->user.short_name : "Node"; + const char *shortName = (nodeInfoLiteHasUser(node) && node->short_name[0]) ? node->short_name : "Node"; char titlestr[40]; snprintf(titlestr, sizeof(titlestr), "*%s*", shortName); @@ -781,9 +781,9 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat // === 1. Long Name (always try to show first) === const char *username; if (currentResolution == ScreenResolution::UltraLow) { - username = (node->has_user && node->user.long_name[0]) ? node->user.short_name : nullptr; + username = (nodeInfoLiteHasUser(node) && node->long_name[0]) ? node->short_name : nullptr; } else { - username = (node->has_user && node->user.long_name[0]) ? node->user.long_name : nullptr; + username = (nodeInfoLiteHasUser(node) && node->long_name[0]) ? node->long_name : nullptr; } // Print node's long name (e.g. "Backpack Node") @@ -796,23 +796,14 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat UIRenderer::drawStringWithEmotes(display, x, getTextPositions(display)[line++], username, FONT_HEIGHT_SMALL, 1, false); } -#if !MESHTASTIC_EXCLUDE_STATUS +#if !MESHTASTIC_EXCLUDE_STATUS && !MESHTASTIC_EXCLUDE_STATUSDB // === Optional: Last received StatusMessage line for this node === - // Display it directly under the username line (if we have one). - if (statusMessageModule) { - const auto &recent = statusMessageModule->getRecentReceived(); - const StatusMessageModule::RecentStatus *found = nullptr; - - // Search newest-to-oldest - for (auto it = recent.rbegin(); it != recent.rend(); ++it) { - if (it->fromNodeId == node->num && !it->statusText.empty()) { - found = &(*it); - break; - } - } - - if (found) { - drawTruncatedStatusLine(display, x, getTextPositions(display)[line++], found->statusText); + // Display it directly under the username line (if we have one). The cache + // lives on NodeDB now, keyed by NodeNum, so this is an O(1) lookup. + if (nodeDB) { + meshtastic_StatusMessage cachedStatus; + if (nodeDB->copyNodeStatus(node->num, cachedStatus) && cachedStatus.status[0]) { + drawTruncatedStatusLine(display, x, getTextPositions(display)[line++], cachedStatus.status); } } #endif @@ -971,25 +962,30 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat #if !defined(M5STACK_UNITC6L) // === 4. Uptime (only show if metric is present) === char uptimeStr[32] = ""; - if (node->has_device_metrics && node->device_metrics.has_uptime_seconds) { + meshtastic_DeviceMetrics nodeMetrics; + const bool haveNodeMetrics = nodeDB->copyNodeTelemetry(node->num, nodeMetrics); + if (haveNodeMetrics && nodeMetrics.has_uptime_seconds) { char upPrefix[12]; // enough for leftSideSpacing + "Up:" snprintf(upPrefix, sizeof(upPrefix), "%sUp:", leftSideSpacing); - getUptimeStr(node->device_metrics.uptime_seconds * 1000, upPrefix, uptimeStr, sizeof(uptimeStr)); + getUptimeStr(nodeMetrics.uptime_seconds * 1000, upPrefix, uptimeStr, sizeof(uptimeStr)); } if (uptimeStr[0]) { display->drawString(x, getTextPositions(display)[line++], uptimeStr); } // === 5. Distance (only if both nodes have GPS position) === - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); char distStr[24] = ""; // Make buffer big enough for any string bool haveDistance = false; - if (nodeDB->hasValidPosition(ourNode) && nodeDB->hasValidPosition(node)) { + meshtastic_PositionLite nodePos; + meshtastic_PositionLite ourPos; + const bool haveNodePos = nodeDB->copyNodePosition(node->num, nodePos); + const bool haveOurPos = ourNode && nodeDB->copyNodePosition(ourNode->num, ourPos); + if (nodeDB->hasValidPosition(ourNode) && nodeDB->hasValidPosition(node) && haveNodePos && haveOurPos) { // Use shared meter conversion, then format display units with lightweight integer rounding. - const float distanceMeters = - GeoCoord::latLongToMeter(DegD(node->position.latitude_i), DegD(node->position.longitude_i), - DegD(ourNode->position.latitude_i), DegD(ourNode->position.longitude_i)); + const float distanceMeters = GeoCoord::latLongToMeter(DegD(nodePos.latitude_i), DegD(nodePos.longitude_i), + DegD(ourPos.latitude_i), DegD(ourPos.longitude_i)); if (config.display.units == meshtastic_Config_DisplayConfig_DisplayUnits_IMPERIAL) { const int feet = static_cast((distanceMeters * METERS_TO_FEET) + 0.5f); if (feet > 0 && feet < 1000) { @@ -1024,19 +1020,19 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat char batLine[32] = ""; bool haveBatLine = false; - if (node->has_device_metrics) { - bool hasPct = node->device_metrics.has_battery_level; - bool hasVolt = node->device_metrics.has_voltage && node->device_metrics.voltage > 0.001f; + if (haveNodeMetrics) { + bool hasPct = nodeMetrics.has_battery_level; + bool hasVolt = nodeMetrics.has_voltage && nodeMetrics.voltage > 0.001f; int pct = 0; float volt = 0.0f; if (hasPct) { - pct = (int)node->device_metrics.battery_level; + pct = (int)nodeMetrics.battery_level; } if (hasVolt) { - volt = node->device_metrics.voltage; + volt = nodeMetrics.voltage; } if (hasPct && pct > 0 && pct <= 100) { @@ -1076,11 +1072,11 @@ void UIRenderer::drawFavoriteNode(OLEDDisplay *display, OLEDDisplayUiState *stat const bool hasNodePositionFix = nodeDB->hasValidPosition(node); const char *statusLine1 = nullptr; const char *statusLine2 = nullptr; - if (hasOwnPositionFix && hasNodePositionFix) { - const auto &op = ourNode->position; + if (hasOwnPositionFix && hasNodePositionFix && haveOurPos && haveNodePos) { + const auto &op = ourPos; showCompass = CompassRenderer::getHeadingRadians(DegD(op.latitude_i), DegD(op.longitude_i), myHeading); if (showCompass) { - const auto &p = node->position; + const auto &p = nodePos; bearing = GeoCoord::bearing(DegD(op.latitude_i), DegD(op.longitude_i), DegD(p.latitude_i), DegD(p.longitude_i)); bearing = CompassRenderer::adjustBearingForCompassMode(bearing, myHeading); } else { @@ -1130,7 +1126,7 @@ void UIRenderer::drawDeviceFocused(OLEDDisplay *display, OLEDDisplayUiState *sta display->setTextAlignment(TEXT_ALIGN_LEFT); display->setFont(FONT_SMALL); int line = 1; - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); // === Header === if (currentResolution == ScreenResolution::UltraLow) { @@ -1270,7 +1266,7 @@ void UIRenderer::drawDeviceFocused(OLEDDisplay *display, OLEDDisplayUiState *sta int textWidth = 0; int nameX = 0; int yOffset = (currentResolution == ScreenResolution::High) ? 0 : 5; - const char *longName = (ourNode && ourNode->has_user && ourNode->user.long_name[0]) ? ourNode->user.long_name : ""; + const char *longName = (nodeInfoLiteHasUser(ourNode) && ourNode->long_name[0]) ? ourNode->long_name : ""; const char *shortName = owner.short_name ? owner.short_name : ""; char combinedName[96]; if (longName[0] && shortName[0]) { @@ -1597,7 +1593,7 @@ void UIRenderer::drawCompassAndLocationScreen(OLEDDisplay *display, OLEDDisplayU geoCoord.updateCoords(int32_t(gpsStatus->getLatitude()), int32_t(gpsStatus->getLongitude()), int32_t(gpsStatus->getAltitude())); - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); const bool hasOwnPositionFix = (ourNode && nodeDB->hasValidPosition(ourNode)); const bool hasLiveGpsFix = (gpsStatus && gpsStatus->getHasLock() && (gpsStatus->getLatitude() != 0 || gpsStatus->getLongitude() != 0)); @@ -1613,9 +1609,11 @@ void UIRenderer::drawCompassAndLocationScreen(OLEDDisplay *display, OLEDDisplayU headingLat = DegD(gpsStatus->getLatitude()); headingLon = DegD(gpsStatus->getLongitude()); } else if (hasOwnPositionFix) { - const auto &op = ourNode->position; - headingLat = DegD(op.latitude_i); - headingLon = DegD(op.longitude_i); + meshtastic_PositionLite ownPos; + if (nodeDB->copyNodePosition(ourNode->num, ownPos)) { + headingLat = DegD(ownPos.latitude_i); + headingLon = DegD(ownPos.longitude_i); + } } validHeading = CompassRenderer::getHeadingRadians(headingLat, headingLon, heading); } diff --git a/src/graphics/niche/InkHUD/Applet.cpp b/src/graphics/niche/InkHUD/Applet.cpp index 0a9cd3add..94039d519 100644 --- a/src/graphics/niche/InkHUD/Applet.cpp +++ b/src/graphics/niche/InkHUD/Applet.cpp @@ -362,8 +362,8 @@ std::string InkHUD::Applet::parseShortName(meshtastic_NodeInfoLite *node) assert(node); // Use the true shortname if known, and doesn't contain any unprintable characters (emoji, etc.) - if (node->has_user) { - std::string parsed = parse(node->user.short_name); + if (nodeInfoLiteHasUser(node)) { + std::string parsed = parse(node->short_name); if (isPrintable(parsed)) return parsed; } diff --git a/src/graphics/niche/InkHUD/Applets/Bases/Map/MapApplet.cpp b/src/graphics/niche/InkHUD/Applets/Bases/Map/MapApplet.cpp index 63ccaa216..881371e2d 100644 --- a/src/graphics/niche/InkHUD/Applets/Bases/Map/MapApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/Bases/Map/MapApplet.cpp @@ -136,9 +136,10 @@ void InkHUD::MapApplet::onRender(bool full) printAt(vertBarX + (bottomLabelW / 2) + 1, bottomLabelY + (bottomLabelH / 2), vertBottomLabel, CENTER, MIDDLE); // Draw our node LAST with full white fill + outline - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); - if (ourNode && nodeDB->hasValidPosition(ourNode)) { - Marker self = calculateMarker(ourNode->position.latitude_i * 1e-7, ourNode->position.longitude_i * 1e-7, false, 0); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + meshtastic_PositionLite ourSelfPos; + if (ourNode && nodeDB->hasValidPosition(ourNode) && nodeDB->copyNodePosition(ourNode->num, ourSelfPos)) { + Marker self = calculateMarker(ourSelfPos.latitude_i * 1e-7, ourSelfPos.longitude_i * 1e-7, false, 0); int16_t centerX = X(0.5) + (self.eastMeters * metersToPx); int16_t centerY = Y(0.5) - (self.northMeters * metersToPx); @@ -168,10 +169,11 @@ void InkHUD::MapApplet::onRender(bool full) void InkHUD::MapApplet::getMapCenter(float *lat, float *lng) { // If we have a valid position for our own node, use that as the anchor - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); - if (ourNode && nodeDB->hasValidPosition(ourNode)) { - *lat = ourNode->position.latitude_i * 1e-7; - *lng = ourNode->position.longitude_i * 1e-7; + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + meshtastic_PositionLite ourSelfPos; + if (ourNode && nodeDB->hasValidPosition(ourNode) && nodeDB->copyNodePosition(ourNode->num, ourSelfPos)) { + *lat = ourSelfPos.latitude_i * 1e-7; + *lng = ourSelfPos.longitude_i * 1e-7; } else { // Find mean lat long coords // ============================ @@ -201,9 +203,13 @@ void InkHUD::MapApplet::getMapCenter(float *lat, float *lng) if (!shouldDrawNode(node)) continue; + meshtastic_PositionLite pos; + if (!nodeDB->copyNodePosition(node->num, pos)) + continue; + // Latitude and Longitude of node, in radians - float latRad = node->position.latitude_i * (1e-7) * DEG_TO_RAD; - float lngRad = node->position.longitude_i * (1e-7) * DEG_TO_RAD; + float latRad = pos.latitude_i * (1e-7) * DEG_TO_RAD; + float lngRad = pos.longitude_i * (1e-7) * DEG_TO_RAD; // Convert to cartesian points, with center of earth at 0, 0, 0 // Exact distance from center is irrelevant, as we're only interested in the vector @@ -300,13 +306,17 @@ void InkHUD::MapApplet::getMapCenter(float *lat, float *lng) if (!shouldDrawNode(node)) continue; + meshtastic_PositionLite pos; + if (!nodeDB->copyNodePosition(node->num, pos)) + continue; + // Check for a new top or bottom latitude - float latNode = node->position.latitude_i * 1e-7; + float latNode = pos.latitude_i * 1e-7; northernmost = max(northernmost, latNode); southernmost = min(southernmost, latNode); // Longitude is trickier - float lngNode = node->position.longitude_i * 1e-7; + float lngNode = pos.longitude_i * 1e-7; float degEastward = fmod(((lngNode - lngCenter) + 360), 360); // Degrees traveled east from lngCenter to reach node float degWestward = abs(fmod(((lngNode - lngCenter) - 360), 360)); // Degrees traveled west from lngCenter to reach node if (degEastward < degWestward) @@ -372,10 +382,13 @@ void InkHUD::MapApplet::drawLabeledMarker(meshtastic_NodeInfoLite *node) { // Find x and y position based on node's position in nodeDB assert(nodeDB->hasValidPosition(node)); - Marker m = calculateMarker(node->position.latitude_i * 1e-7, // Lat, converted from Meshtastic's internal int32 style - node->position.longitude_i * 1e-7, // Long, converted from Meshtastic's internal int32 style - node->has_hops_away, // Is the hopsAway number valid - node->hops_away // Hops away + meshtastic_PositionLite pos; + const bool hasPos = nodeDB->copyNodePosition(node->num, pos); + assert(hasPos); + Marker m = calculateMarker(pos.latitude_i * 1e-7, // Lat, converted from Meshtastic's internal int32 style + pos.longitude_i * 1e-7, // Long, converted from Meshtastic's internal int32 style + node->has_hops_away, // Is the hopsAway number valid + node->hops_away // Hops away ); // Convert to pixel coords @@ -516,13 +529,16 @@ void InkHUD::MapApplet::calculateAllMarkers() if (node->num == nodeDB->getNodeNum()) continue; + meshtastic_PositionLite pos; + if (!nodeDB->copyNodePosition(node->num, pos)) + continue; + // Calculate marker and store it - markers.push_back( - calculateMarker(node->position.latitude_i * 1e-7, // Lat, converted from Meshtastic's internal int32 style - node->position.longitude_i * 1e-7, // Long, converted from Meshtastic's internal int32 style - node->has_hops_away, // Is the hopsAway number valid - node->hops_away // Hops away - )); + markers.push_back(calculateMarker(pos.latitude_i * 1e-7, // Lat, converted from Meshtastic's internal int32 style + pos.longitude_i * 1e-7, // Long, converted from Meshtastic's internal int32 style + node->has_hops_away, // Is the hopsAway number valid + node->hops_away // Hops away + )); } } diff --git a/src/graphics/niche/InkHUD/Applets/Bases/NodeList/NodeListApplet.cpp b/src/graphics/niche/InkHUD/Applets/Bases/NodeList/NodeListApplet.cpp index 607fd4ef7..3da912a78 100644 --- a/src/graphics/niche/InkHUD/Applets/Bases/NodeList/NodeListApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/Bases/NodeList/NodeListApplet.cpp @@ -50,21 +50,23 @@ ProcessMessage InkHUD::NodeListApplet::handleReceived(const meshtastic_MeshPacke c.signal = getSignalStrength(mp.rx_snr, mp.rx_rssi); // Assemble info: from nodeDB (needed to detect changes) - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(c.nodeNum); - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(c.nodeNum); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); if (node) { if (node->has_hops_away) c.hopsAway = node->hops_away; if (nodeDB->hasValidPosition(node) && nodeDB->hasValidPosition(ourNode)) { - // Get lat and long as float - // Meshtastic stores these as integers internally - float ourLat = ourNode->position.latitude_i * 1e-7; - float ourLong = ourNode->position.longitude_i * 1e-7; - float theirLat = node->position.latitude_i * 1e-7; - float theirLong = node->position.longitude_i * 1e-7; + meshtastic_PositionLite ourPos; + meshtastic_PositionLite theirPos; + if (nodeDB->copyNodePosition(ourNode->num, ourPos) && nodeDB->copyNodePosition(node->num, theirPos)) { + float ourLat = ourPos.latitude_i * 1e-7; + float ourLong = ourPos.longitude_i * 1e-7; + float theirLat = theirPos.latitude_i * 1e-7; + float theirLong = theirPos.longitude_i * 1e-7; - c.distanceMeters = (int32_t)GeoCoord::latLongToMeter(theirLat, theirLong, ourLat, ourLong); + c.distanceMeters = (int32_t)GeoCoord::latLongToMeter(theirLat, theirLong, ourLat, ourLong); + } } } @@ -179,8 +181,8 @@ void InkHUD::NodeListApplet::onRender(bool full) // -- Longname -- // Parse special chars in long name // Use node id if unknown - if (node && node->has_user) - longName = parse(node->user.long_name); // Found in nodeDB + if (nodeInfoLiteHasUser(node)) + longName = parse(node->long_name); // Found in nodeDB else { // Not found in nodeDB, show a hex nodeid instead longName = hexifyNodeNum(nodeNum); diff --git a/src/graphics/niche/InkHUD/Applets/System/Menu/MenuApplet.cpp b/src/graphics/niche/InkHUD/Applets/System/Menu/MenuApplet.cpp index dfef4d085..79ac1e701 100644 --- a/src/graphics/niche/InkHUD/Applets/System/Menu/MenuApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/System/Menu/MenuApplet.cpp @@ -2090,7 +2090,7 @@ void InkHUD::MenuApplet::populateRecipientPage() // Count favorites for (uint32_t i = 0; i < nodeCount; i++) { - if (nodeDB->getMeshNodeByIndex(i)->is_favorite) + if (nodeInfoLiteIsFavorite(nodeDB->getMeshNodeByIndex(i))) favoriteCount++; } @@ -2098,10 +2098,10 @@ void InkHUD::MenuApplet::populateRecipientPage() // Don't want some monstrous list that takes 100 clicks to reach exit if (favoriteCount < 20) { for (uint32_t i = 0; i < nodeCount; i++) { - meshtastic_NodeInfoLite *node = nodeDB->getMeshNodeByIndex(i); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNodeByIndex(i); // Skip node if not a favorite - if (!node->is_favorite) + if (!nodeInfoLiteIsFavorite(node)) continue; CannedMessages::RecipientItem r; @@ -2111,8 +2111,8 @@ void InkHUD::MenuApplet::populateRecipientPage() // Set a label for the menu item r.label = "DM: "; - if (node->has_user) - r.label += parse(node->user.long_name); + if (nodeInfoLiteHasUser(node)) + r.label += parse(node->long_name); else r.label += hexifyNodeNum(node->num); // Unsure if it's possible to favorite a node without NodeInfo? diff --git a/src/graphics/niche/InkHUD/Applets/System/Notification/NotificationApplet.cpp b/src/graphics/niche/InkHUD/Applets/System/Notification/NotificationApplet.cpp index 6c8069c8b..44d0a8089 100644 --- a/src/graphics/niche/InkHUD/Applets/System/Notification/NotificationApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/System/Notification/NotificationApplet.cpp @@ -241,7 +241,7 @@ std::string InkHUD::NotificationApplet::getNotificationText(uint16_t widthAvaila text += msgIsBroadcast ? "From:" : "DM: "; // Sender id - if (node && node->has_user) + if (nodeInfoLiteHasUser(node)) text += parseShortName(node); else text += hexifyNodeNum(message->sender); @@ -255,7 +255,7 @@ std::string InkHUD::NotificationApplet::getNotificationText(uint16_t widthAvaila text += msgIsBroadcast ? "Msg from " : "DM from "; // Sender id - if (node && node->has_user) + if (nodeInfoLiteHasUser(node)) text += parseShortName(node); else text += hexifyNodeNum(message->sender); diff --git a/src/graphics/niche/InkHUD/Applets/User/AllMessage/AllMessageApplet.cpp b/src/graphics/niche/InkHUD/Applets/User/AllMessage/AllMessageApplet.cpp index 96c519599..bbbe826b1 100644 --- a/src/graphics/niche/InkHUD/Applets/User/AllMessage/AllMessageApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/User/AllMessage/AllMessageApplet.cpp @@ -70,10 +70,10 @@ void InkHUD::AllMessageApplet::onRender(bool full) // - short name and long name, if available, or // - node id meshtastic_NodeInfoLite *sender = nodeDB->getMeshNode(message->sender); - if (sender && sender->has_user) { + if (nodeInfoLiteHasUser(sender)) { header += parseShortName(sender); // May be last-four of node if unprintable (emoji, etc) header += " ("; - header += parse(sender->user.long_name); + header += parse(sender->long_name); header += ")"; } else header += hexifyNodeNum(message->sender); diff --git a/src/graphics/niche/InkHUD/Applets/User/DM/DMApplet.cpp b/src/graphics/niche/InkHUD/Applets/User/DM/DMApplet.cpp index 189a56cab..c15152d3a 100644 --- a/src/graphics/niche/InkHUD/Applets/User/DM/DMApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/User/DM/DMApplet.cpp @@ -66,10 +66,10 @@ void InkHUD::DMApplet::onRender(bool full) // - shortname and long name, if available, or // - node id meshtastic_NodeInfoLite *sender = nodeDB->getMeshNode(latestMessage->dm.sender); - if (sender && sender->has_user) { + if (nodeInfoLiteHasUser(sender)) { header += parseShortName(sender); // May be last-four of node if unprintable (emoji, etc) header += " ("; - header += parse(sender->user.long_name); + header += parse(sender->long_name); header += ")"; } else header += hexifyNodeNum(latestMessage->dm.sender); diff --git a/src/graphics/niche/InkHUD/Applets/User/FavoritesMap/FavoritesMapApplet.cpp b/src/graphics/niche/InkHUD/Applets/User/FavoritesMap/FavoritesMapApplet.cpp index 520070d72..290546a6f 100644 --- a/src/graphics/niche/InkHUD/Applets/User/FavoritesMap/FavoritesMapApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/User/FavoritesMap/FavoritesMapApplet.cpp @@ -8,7 +8,7 @@ using namespace NicheGraphics; bool InkHUD::FavoritesMapApplet::shouldDrawNode(meshtastic_NodeInfoLite *node) { // Keep our own node available as map anchor/center; all others must be favorited. - return node && (node->num == nodeDB->getNodeNum() || node->is_favorite); + return node && (node->num == nodeDB->getNodeNum() || nodeInfoLiteIsFavorite(node)); } void InkHUD::FavoritesMapApplet::onRender(bool full) @@ -25,7 +25,7 @@ void InkHUD::FavoritesMapApplet::onRender(bool full) // Draw our latest "node of interest" as a special marker. meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(lastFrom); - if (node && node->is_favorite && nodeDB->hasValidPosition(node) && enoughMarkers()) + if (node && nodeInfoLiteIsFavorite(node) && nodeDB->hasValidPosition(node) && enoughMarkers()) drawLabeledMarker(node); } @@ -74,7 +74,7 @@ ProcessMessage InkHUD::FavoritesMapApplet::handleReceived(const meshtastic_MeshP } else { // For non-local packets, this applet only reacts to favorited nodes. const meshtastic_NodeInfoLite *sender = nodeDB->getMeshNode(mp.from); - if (!sender || !sender->is_favorite) + if (!nodeInfoLiteIsFavorite(sender)) return ProcessMessage::CONTINUE; // Check if this position is from someone different than our previous position packet. diff --git a/src/graphics/niche/InkHUD/Applets/User/Heard/HeardApplet.cpp b/src/graphics/niche/InkHUD/Applets/User/Heard/HeardApplet.cpp index a7fd094e6..94a87d23d 100644 --- a/src/graphics/niche/InkHUD/Applets/User/Heard/HeardApplet.cpp +++ b/src/graphics/niche/InkHUD/Applets/User/Heard/HeardApplet.cpp @@ -80,8 +80,8 @@ void InkHUD::HeardApplet::populateFromNodeDB() ordered.resize(maxCards()); // Create card info for these (stale) node observations - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); - for (meshtastic_NodeInfoLite *node : ordered) { + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + for (const meshtastic_NodeInfoLite *node : ordered) { CardInfo c; c.nodeNum = node->num; @@ -89,14 +89,16 @@ void InkHUD::HeardApplet::populateFromNodeDB() c.hopsAway = node->hops_away; if (nodeDB->hasValidPosition(node) && nodeDB->hasValidPosition(ourNode)) { - // Get lat and long as float - // Meshtastic stores these as integers internally - float ourLat = ourNode->position.latitude_i * 1e-7; - float ourLong = ourNode->position.longitude_i * 1e-7; - float theirLat = node->position.latitude_i * 1e-7; - float theirLong = node->position.longitude_i * 1e-7; + meshtastic_PositionLite ourPos; + meshtastic_PositionLite theirPos; + if (nodeDB->copyNodePosition(ourNode->num, ourPos) && nodeDB->copyNodePosition(node->num, theirPos)) { + float ourLat = ourPos.latitude_i * 1e-7; + float ourLong = ourPos.longitude_i * 1e-7; + float theirLat = theirPos.latitude_i * 1e-7; + float theirLong = theirPos.longitude_i * 1e-7; - c.distanceMeters = (int32_t)GeoCoord::latLongToMeter(theirLat, theirLong, ourLat, ourLong); + c.distanceMeters = (int32_t)GeoCoord::latLongToMeter(theirLat, theirLong, ourLat, ourLong); + } } // Insert into the card collection (member of base class) @@ -122,4 +124,4 @@ std::string InkHUD::HeardApplet::getHeaderText() return text; } -#endif \ No newline at end of file +#endif diff --git a/src/input/RotaryEncoderImpl.cpp b/src/input/RotaryEncoderImpl.cpp index cc1222595..dcdbf0d36 100644 --- a/src/input/RotaryEncoderImpl.cpp +++ b/src/input/RotaryEncoderImpl.cpp @@ -13,7 +13,6 @@ RotaryEncoderImpl *rotaryEncoderImpl; RotaryEncoderImpl::RotaryEncoderImpl() { - rotary = nullptr; #ifdef ARCH_ESP32 isFirstInit = true; #endif @@ -23,11 +22,6 @@ RotaryEncoderImpl::~RotaryEncoderImpl() { LOG_DEBUG("RotaryEncoderImpl destructor"); detachRotaryEncoderInterrupts(); - - if (rotary != nullptr) { - delete rotary; - rotary = nullptr; - } } bool RotaryEncoderImpl::init() @@ -43,8 +37,9 @@ bool RotaryEncoderImpl::init() eventPressed = static_cast(moduleConfig.canned_message.inputbroker_event_press); if (rotary == nullptr) { - rotary = new RotaryEncoder(moduleConfig.canned_message.inputbroker_pin_a, moduleConfig.canned_message.inputbroker_pin_b, - moduleConfig.canned_message.inputbroker_pin_press); + rotary.reset(new RotaryEncoder(moduleConfig.canned_message.inputbroker_pin_a, + moduleConfig.canned_message.inputbroker_pin_b, + moduleConfig.canned_message.inputbroker_pin_press)); } attachRotaryEncoderInterrupts(); diff --git a/src/input/RotaryEncoderImpl.h b/src/input/RotaryEncoderImpl.h index ec8a064bd..58b4e3450 100644 --- a/src/input/RotaryEncoderImpl.h +++ b/src/input/RotaryEncoderImpl.h @@ -5,6 +5,7 @@ #include "InputBroker.h" #include "concurrency/OSThread.h" #include "mesh/NodeDB.h" +#include class RotaryEncoder; @@ -28,7 +29,7 @@ class RotaryEncoderImpl final : public InputPollable input_broker_event eventCcw = INPUT_BROKER_NONE; input_broker_event eventPressed = INPUT_BROKER_NONE; - RotaryEncoder *rotary; + std::unique_ptr rotary; private: #ifdef ARCH_ESP32 diff --git a/src/main.cpp b/src/main.cpp index 702787b6b..2f4b12437 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -731,8 +731,15 @@ void setup() #elif defined(USE_SH1107_128_64) screen_model = meshtastic_Config_DisplayConfig_OledType_OLED_SH1107; // keep dimension of 128x64 #else - if (config.display.oled != meshtastic_Config_DisplayConfig_OledType_OLED_AUTO) + if (config.display.oled != meshtastic_Config_DisplayConfig_OledType_OLED_AUTO) { screen_model = config.display.oled; + + // Fix: update geometry for SH1107 128x128 selected via menu + if (screen_model == meshtastic_Config_DisplayConfig_OledType_OLED_SH1107_128_128) { + screen_geometry = GEOMETRY_128_128; + screen_model = meshtastic_Config_DisplayConfig_OledType_OLED_SH1107; // normalize + } + } #endif #endif diff --git a/src/mesh/CryptoEngine.cpp b/src/mesh/CryptoEngine.cpp index daa7a3a75..59c6701c9 100644 --- a/src/mesh/CryptoEngine.cpp +++ b/src/mesh/CryptoEngine.cpp @@ -112,7 +112,7 @@ bool CryptoEngine::ensurePkiKeys(meshtastic_Config_SecurityConfig &security, mes * @param bytes Buffer containing plaintext input. * @param bytesOut Output buffer to be populated with encrypted ciphertext. */ -bool CryptoEngine::encryptCurve25519(uint32_t toNode, uint32_t fromNode, meshtastic_UserLite_public_key_t remotePublic, +bool CryptoEngine::encryptCurve25519(uint32_t toNode, uint32_t fromNode, meshtastic_NodeInfoLite_public_key_t remotePublic, uint64_t packetNum, size_t numBytes, const uint8_t *bytes, uint8_t *bytesOut) { uint8_t *auth; @@ -152,7 +152,7 @@ bool CryptoEngine::encryptCurve25519(uint32_t toNode, uint32_t fromNode, meshtas * @param bytes Buffer containing ciphertext input. * @param bytesOut Output buffer to be populated with decrypted plaintext. */ -bool CryptoEngine::decryptCurve25519(uint32_t fromNode, meshtastic_UserLite_public_key_t remotePublic, uint64_t packetNum, +bool CryptoEngine::decryptCurve25519(uint32_t fromNode, meshtastic_NodeInfoLite_public_key_t remotePublic, uint64_t packetNum, size_t numBytes, const uint8_t *bytes, uint8_t *bytesOut) { const uint8_t *auth = bytes + numBytes - 12; // set to last 8 bytes of text? diff --git a/src/mesh/CryptoEngine.h b/src/mesh/CryptoEngine.h index f40400331..9410b63e7 100644 --- a/src/mesh/CryptoEngine.h +++ b/src/mesh/CryptoEngine.h @@ -40,9 +40,12 @@ class CryptoEngine #endif void setDHPrivateKey(uint8_t *_private_key); - virtual bool encryptCurve25519(uint32_t toNode, uint32_t fromNode, meshtastic_UserLite_public_key_t remotePublic, + // The remotePublic key parameter takes the public_key bytes container from + // a stored node header. NodeInfoLite is the on-device storage type since + // the slim refactor flattened UserLite into it. + virtual bool encryptCurve25519(uint32_t toNode, uint32_t fromNode, meshtastic_NodeInfoLite_public_key_t remotePublic, uint64_t packetNum, size_t numBytes, const uint8_t *bytes, uint8_t *bytesOut); - virtual bool decryptCurve25519(uint32_t fromNode, meshtastic_UserLite_public_key_t remotePublic, uint64_t packetNum, + virtual bool decryptCurve25519(uint32_t fromNode, meshtastic_NodeInfoLite_public_key_t remotePublic, uint64_t packetNum, size_t numBytes, const uint8_t *bytes, uint8_t *bytesOut); virtual bool setDHPublicKey(uint8_t *publicKey); virtual void hash(uint8_t *bytes, size_t numBytes); diff --git a/src/mesh/InterfacesTemplates.cpp b/src/mesh/InterfacesTemplates.cpp index c246141dc..6f6b2d2a1 100644 --- a/src/mesh/InterfacesTemplates.cpp +++ b/src/mesh/InterfacesTemplates.cpp @@ -1,5 +1,9 @@ +#include "configuration.h" + #include "LR11x0Interface.cpp" #include "LR11x0Interface.h" +#include "LR20x0Interface.cpp" +#include "LR20x0Interface.h" #include "SX126xInterface.cpp" #include "SX126xInterface.h" #include "SX128xInterface.cpp" @@ -21,6 +25,9 @@ template class LR11x0Interface; template class LR11x0Interface; template class LR11x0Interface; #endif +#if defined(USE_LR2021) && RADIOLIB_EXCLUDE_LR2021 != 1 +template class LR20x0Interface; +#endif #ifdef ARCH_STM32WL template class SX126xInterface; #endif diff --git a/src/mesh/LR11x0Interface.cpp b/src/mesh/LR11x0Interface.cpp index 7a193f7f3..1d1616ed6 100644 --- a/src/mesh/LR11x0Interface.cpp +++ b/src/mesh/LR11x0Interface.cpp @@ -57,16 +57,19 @@ template bool LR11x0Interface::init() #if ARCH_PORTDUINO float tcxoVoltage = (float)portduino_config.dio3_tcxo_voltage / 1000; // FIXME: correct logic to default to not using TCXO if no voltage is specified for LR11x0_DIO3_TCXO_VOLTAGE -#elif !defined(LR11X0_DIO3_TCXO_VOLTAGE) +#elif defined(LR11X0_DIO3_TCXO_VOLTAGE) + float tcxoVoltage = LR11X0_DIO3_TCXO_VOLTAGE; + LOG_DEBUG("LR11X0_DIO3_TCXO_VOLTAGE defined, using DIO3 as TCXO reference voltage at %f V", LR11X0_DIO3_TCXO_VOLTAGE); + // (DIO3 is not free to be used as an IRQ) +#elif defined(TCXO_OPTIONAL) + float tcxoVoltage = 1.6f; // TCXO_OPTIONAL: try default 1.6 V first, fall back to XTAL on failure + LOG_DEBUG("TCXO_OPTIONAL: no LR11X0_DIO3_TCXO_VOLTAGE defined, trying default TCXO Vref 1.6 V first"); +#else float tcxoVoltage = 0; // "TCXO reference voltage to be set on DIO3. Defaults to 1.6 V, set to 0 to skip." per // https://github.com/jgromes/RadioLib/blob/690a050ebb46e6097c5d00c371e961c1caa3b52e/src/modules/LR11x0/LR11x0.h#L471C26-L471C104 // (DIO3 is free to be used as an IRQ) LOG_DEBUG("LR11X0_DIO3_TCXO_VOLTAGE not defined, not using DIO3 as TCXO reference voltage"); -#else - float tcxoVoltage = LR11X0_DIO3_TCXO_VOLTAGE; - LOG_DEBUG("LR11X0_DIO3_TCXO_VOLTAGE defined, using DIO3 as TCXO reference voltage at %f V", LR11X0_DIO3_TCXO_VOLTAGE); - // (DIO3 is not free to be used as an IRQ) #endif RadioLibInterface::init(); @@ -101,6 +104,17 @@ template bool LR11x0Interface::init() res = lora.begin(getFreq(), bw, sf, cr, syncWord, power, preambleLength, tcxoVoltage); } +#if defined(TCXO_OPTIONAL) + // If init failed for any reason other than chip not found, retry without TCXO (XTAL mode) + if (res != RADIOLIB_ERR_NONE && res != RADIOLIB_ERR_CHIP_NOT_FOUND && tcxoVoltage > 0) { + LOG_WARN("LR11x0 init failed with TCXO Vref %f V (err %d), retrying without TCXO", tcxoVoltage, res); + tcxoVoltage = 0; + res = lora.begin(getFreq(), bw, sf, cr, syncWord, power, preambleLength, tcxoVoltage); + if (res == RADIOLIB_ERR_NONE) + LOG_INFO("LR11x0 init success without TCXO (XTAL mode)"); + } +#endif + // \todo Display actual typename of the adapter, not just `LR11x0` LOG_INFO("LR11x0 init result %d", res); if (res == RADIOLIB_ERR_CHIP_NOT_FOUND || res == RADIOLIB_ERR_SPI_CMD_FAILED) @@ -198,7 +212,7 @@ template bool LR11x0Interface::reconfigure() startReceive(); // restart receiving - return RADIOLIB_ERR_NONE; + return true; } template void LR11x0Interface::disableInterrupt() diff --git a/src/mesh/LR2021Interface.cpp b/src/mesh/LR2021Interface.cpp new file mode 100644 index 000000000..9aa4d5f1a --- /dev/null +++ b/src/mesh/LR2021Interface.cpp @@ -0,0 +1,18 @@ +#include "configuration.h" + +#if defined(USE_LR2021) && RADIOLIB_EXCLUDE_LR2021 != 1 + +#include "LR2021Interface.h" +#include "error.h" + +LR2021Interface::LR2021Interface(LockingArduinoHal *hal, RADIOLIB_PIN_TYPE cs, RADIOLIB_PIN_TYPE irq, RADIOLIB_PIN_TYPE rst, + RADIOLIB_PIN_TYPE busy) + : LR20x0Interface(hal, cs, irq, rst, busy) +{ +} + +bool LR2021Interface::wideLora() +{ + return true; +} +#endif diff --git a/src/mesh/LR2021Interface.h b/src/mesh/LR2021Interface.h new file mode 100644 index 000000000..52c04ee90 --- /dev/null +++ b/src/mesh/LR2021Interface.h @@ -0,0 +1,15 @@ +#pragma once +#if RADIOLIB_EXCLUDE_LR2021 != 1 +#include "LR20x0Interface.h" + +/** + * Our adapter for LR2021 radios + */ +class LR2021Interface : public LR20x0Interface +{ + public: + LR2021Interface(LockingArduinoHal *hal, RADIOLIB_PIN_TYPE cs, RADIOLIB_PIN_TYPE irq, RADIOLIB_PIN_TYPE rst, + RADIOLIB_PIN_TYPE busy); + bool wideLora() override; +}; +#endif diff --git a/src/mesh/LR20x0Interface.cpp b/src/mesh/LR20x0Interface.cpp new file mode 100644 index 000000000..699663cf0 --- /dev/null +++ b/src/mesh/LR20x0Interface.cpp @@ -0,0 +1,399 @@ +#include "configuration.h" + +#if defined(USE_LR2021) && RADIOLIB_EXCLUDE_LR2021 != 1 +#include "LR20x0Interface.h" +#include "error.h" +#include "mesh/NodeDB.h" + +// Keep LR20x0 naming while RadioLib exposes LR2021 symbols. +#ifndef LR20x0 +#define LR20x0 LR2021 +#endif + +#ifdef LR2021_DIO_AS_RF_SWITCH +#include "rfswitch.h" +#elif ARCH_PORTDUINO +#include "PortduinoGlue.h" +#define lr20x0_rfswitch_dio_pins portduino_config.rfswitch_dio_pins +#define lr20x0_rfswitch_table portduino_config.rfswitch_table +#else +static const uint32_t lr20x0_rfswitch_dio_pins[] = {RADIOLIB_NC, RADIOLIB_NC, RADIOLIB_NC, RADIOLIB_NC, RADIOLIB_NC}; +static const Module::RfSwitchMode_t lr20x0_rfswitch_table[] = { + {LR20x0::MODE_STBY, {}}, {LR20x0::MODE_RX, {}}, {LR20x0::MODE_TX, {}}, + {LR20x0::MODE_RX_HF, {}}, {LR20x0::MODE_TX_HF, {}}, END_OF_MODE_TABLE, +}; +#endif + +// Particular boards might define a different max power based on what their hardware can do, default to max power output if not +// specified (may be dangerous if using external PA and LR20x0 power config forgotten) +#if ARCH_PORTDUINO +#define LR2021_MAX_POWER portduino_config.lr2021_max_power +#endif +#ifndef LR2021_MAX_POWER +#define LR2021_MAX_POWER 22 +#endif + +// the 2.4G part maxes at 12dBm + +#if ARCH_PORTDUINO +#define LR2021_MAX_POWER_HF portduino_config.lr2021_max_power_hf +#endif +#ifndef LR2021_MAX_POWER_HF +#define LR2021_MAX_POWER_HF 12 +#endif + +template +LR20x0Interface::LR20x0Interface(LockingArduinoHal *hal, RADIOLIB_PIN_TYPE cs, RADIOLIB_PIN_TYPE irq, RADIOLIB_PIN_TYPE rst, + RADIOLIB_PIN_TYPE busy) + : RadioLibInterface(hal, cs, irq, rst, busy, &lora), lora(&module) +{ + LOG_WARN("LR20x0Interface(cs=%d, irq=%d, rst=%d, busy=%d)", cs, irq, rst, busy); +} + +/// Initialise the Driver transport hardware and software. +/// Make sure the Driver is properly configured before calling init(). +/// \return true if initialisation succeeded. +template bool LR20x0Interface::init() +{ +#ifdef LR2021_POWER_EN + pinMode(LR2021_POWER_EN, OUTPUT); + digitalWrite(LR2021_POWER_EN, HIGH); +#endif + +#if ARCH_PORTDUINO + float tcxoVoltage = (float)portduino_config.dio3_tcxo_voltage / 1000; +// FIXME: correct logic to default to not using TCXO if no voltage is specified for LR20x0_DIO3_TCXO_VOLTAGE +#elif defined(LR2021_DIO3_TCXO_VOLTAGE) + float tcxoVoltage = LR2021_DIO3_TCXO_VOLTAGE; + LOG_DEBUG("LR2021_DIO3_TCXO_VOLTAGE defined, using DIO3 as TCXO reference voltage at %f V", LR2021_DIO3_TCXO_VOLTAGE); + // (DIO3 is not free to be used as an IRQ) +#elif defined(TCXO_OPTIONAL) + float tcxoVoltage = 1.6f; // TCXO_OPTIONAL: try default 1.6 V first, fall back to XTAL on failure + LOG_DEBUG("TCXO_OPTIONAL: no LR2021_DIO3_TCXO_VOLTAGE defined, trying default TCXO Vref 1.6 V first"); +#else + float tcxoVoltage = + 0; // "TCXO reference voltage to be set on DIO3. Defaults to 1.6 V, set to 0 to skip." per + // https://github.com/jgromes/RadioLib/blob/690a050ebb46e6097c5d00c371e961c1caa3b52e/src/modules/LR11x0/LR11x0.h#L471C26-L471C104 + // (DIO3 is free to be used as an IRQ) + LOG_DEBUG("LR2021_DIO3_TCXO_VOLTAGE not defined, not using DIO3 as TCXO reference voltage"); +#endif + + RadioLibInterface::init(); + +#ifdef LR2021_IRQ_DIO_NUM + lora.irqDioNum = LR2021_IRQ_DIO_NUM; + LOG_DEBUG("Set irqDioNum %d", lora.irqDioNum); +#elif defined(IRQ_DIO_NUM) + lora.irqDioNum = IRQ_DIO_NUM; + LOG_DEBUG("Set irqDioNum %d", lora.irqDioNum); +#else + LOG_DEBUG("Use default irqDioNum %d", lora.irqDioNum); +#endif + + if (config.lora.region == meshtastic_Config_LoRaConfig_RegionCode_LORA_24) { // clamp if wide freq range + limitPower(LR2021_MAX_POWER_HF); + } else { + limitPower(LR2021_MAX_POWER); // default clamp for non-wide freq range + } + +#ifdef LR2021_RF_SWITCH_SUBGHZ + pinMode(LR2021_RF_SWITCH_SUBGHZ, OUTPUT); + digitalWrite(LR2021_RF_SWITCH_SUBGHZ, getFreq() < 1e9 ? HIGH : LOW); + LOG_DEBUG("Set RF0 switch to %s", getFreq() < 1e9 ? "SubGHz" : "2.4GHz"); +#endif + +#ifdef LR2021_RF_SWITCH_2_4GHZ + pinMode(LR2021_RF_SWITCH_2_4GHZ, OUTPUT); + digitalWrite(LR2021_RF_SWITCH_2_4GHZ, getFreq() < 1e9 ? LOW : HIGH); + LOG_DEBUG("Set RF1 switch to %s", getFreq() < 1e9 ? "SubGHz" : "2.4GHz"); +#endif + + // Allow extra time for TCXO to stabilize after power-on + delay(10); + + int res = lora.begin(getFreq(), bw, sf, cr, syncWord, power, preambleLength, tcxoVoltage); + + // Retry if we get SPI command failed - some units need extra TCXO stabilization time + if (res == RADIOLIB_ERR_SPI_CMD_FAILED) { + LOG_WARN("LR20x0 init failed with %d (SPI_CMD_FAILED), retrying after delay...", res); + delay(100); + res = lora.begin(getFreq(), bw, sf, cr, syncWord, power, preambleLength, tcxoVoltage); + } + +#if defined(TCXO_OPTIONAL) + // If init failed for any reason other than chip not found, retry without TCXO (XTAL mode) + if (res != RADIOLIB_ERR_NONE && res != RADIOLIB_ERR_CHIP_NOT_FOUND && tcxoVoltage > 0) { + LOG_WARN("LR20x0 init failed with TCXO Vref %f V (err %d), retrying without TCXO", tcxoVoltage, res); + tcxoVoltage = 0; + res = lora.begin(getFreq(), bw, sf, cr, syncWord, power, preambleLength, tcxoVoltage); + if (res == RADIOLIB_ERR_NONE) + LOG_INFO("LR20x0 init success without TCXO (XTAL mode)"); + } +#endif + + // \todo Display actual typename of the adapter, not just `LR20x0` + LOG_INFO("LR20x0 init result %d", res); + if (res == RADIOLIB_ERR_CHIP_NOT_FOUND || res == RADIOLIB_ERR_SPI_CMD_FAILED) + return false; + + // Some basic info about the module's explicit firmware version - no other info available + // Currently requires radiolib godmode + +#if RADIOLIB_GODMODE + uint8_t fwMajor = 0; + uint8_t fwMinor = 0; + int versionRes = lora.getVersion(&fwMajor, &fwMinor); + if (versionRes == RADIOLIB_ERR_NONE) + LOG_DEBUG("LR20x0 FW %d.%d", fwMajor, fwMinor); +#endif + + LOG_INFO("Frequency set to %f", getFreq()); + LOG_INFO("Bandwidth set to %f", bw); + LOG_INFO("Power output set to %d", power); + + if (res == RADIOLIB_ERR_NONE) + res = lora.setCRC(2); + + // Standard DCDC ramp timing from RadioLib workarounds (register 0x00F20024) + // Currently requires radiolib godmode +#if RADIOLIB_GODMODE + if (res == RADIOLIB_ERR_NONE) { + uint8_t rampTimes[4] = {15, 15, 15, 15}; // Standard case for all conditions + res = lora.setRegMode(RADIOLIB_LR2021_REG_MODE_SIMO_NORMAL, rampTimes); + if (res != RADIOLIB_ERR_NONE) + LOG_WARN("LR2021 setRegMode failed: %d", res); + } +#endif + +#ifdef LR2021_DIO_AS_RF_SWITCH + bool dioAsRfSwitch = true; +#elif defined(ARCH_PORTDUINO) + bool dioAsRfSwitch = portduino_config.has_rfswitch_table; +#else + bool dioAsRfSwitch = false; +#endif + + if (dioAsRfSwitch) { + lora.setRfSwitchTable(lr20x0_rfswitch_dio_pins, lr20x0_rfswitch_table); + LOG_DEBUG("Set DIO RF switch"); + } + + if (res == RADIOLIB_ERR_NONE) { + if (config.lora.sx126x_rx_boosted_gain) { // the name is unfortunate but historically accurate + res = lora.setRxBoostedGainMode(true); + LOG_INFO("Set RX gain to boosted mode; result: %d", res); + } else { + res = lora.setRxBoostedGainMode(false); + LOG_INFO("Set RX gain to power saving mode (boosted mode off); result: %d", res); + } + } + + if (res == RADIOLIB_ERR_NONE) + startReceive(); // start receiving + + return res == RADIOLIB_ERR_NONE; +} + +template bool LR20x0Interface::reconfigure() +{ + RadioLibInterface::reconfigure(); + + // set mode to standby + setStandby(); + + // configure publicly accessible settings + int err = lora.setSpreadingFactor(sf); + if (err != RADIOLIB_ERR_NONE) + RECORD_CRITICALERROR(meshtastic_CriticalErrorCode_INVALID_RADIO_SETTING); + + err = lora.setBandwidth(bw); // different form than LR11xx + if (err != RADIOLIB_ERR_NONE) + RECORD_CRITICALERROR(meshtastic_CriticalErrorCode_INVALID_RADIO_SETTING); + + err = lora.setCodingRate(cr, cr != 7); // use long interleaving except if CR is 4/7 which doesn't support it + if (err != RADIOLIB_ERR_NONE) + RECORD_CRITICALERROR(meshtastic_CriticalErrorCode_INVALID_RADIO_SETTING); + + err = lora.setSyncWord(syncWord); + assert(err == RADIOLIB_ERR_NONE); + + if (config.lora.region == meshtastic_Config_LoRaConfig_RegionCode_LORA_24) { // clamp if wide freq range + limitPower(LR2021_MAX_POWER_HF); + } else { + limitPower(LR2021_MAX_POWER); // default clamp for non-wide freq range + } + + err = lora.setPreambleLength(preambleLength); + assert(err == RADIOLIB_ERR_NONE); + + err = lora.setFrequency(getFreq()); + if (err != RADIOLIB_ERR_NONE) + RECORD_CRITICALERROR(meshtastic_CriticalErrorCode_INVALID_RADIO_SETTING); + + err = lora.setOutputPower(power); + assert(err == RADIOLIB_ERR_NONE); + + // Apply RX gain mode — valid in STDBY, matches resetAGC() pattern + err = lora.setRxBoostedGainMode(config.lora.sx126x_rx_boosted_gain); + if (err != RADIOLIB_ERR_NONE) + LOG_WARN("LR20x0 setRxBoostedGainMode %s%d", radioLibErr, err); + + startReceive(); // restart receiving + + return true; +} + +template void LR20x0Interface::disableInterrupt() +{ + lora.clearIrqAction(); +} + +template void LR20x0Interface::setStandby() +{ + checkNotification(); // handle any pending interrupts before we force standby + + int err = lora.standby(); + + if (err != RADIOLIB_ERR_NONE) { + LOG_DEBUG("LR20x0 standby failed with error %d", err); + } + + assert(err == RADIOLIB_ERR_NONE); + + isReceiving = false; // If we were receiving, not any more + activeReceiveStart = 0; + disableInterrupt(); + completeSending(); // If we were sending, not anymore + RadioLibInterface::setStandby(); +} + +/** + * Add SNR data to received messages + */ +template void LR20x0Interface::addReceiveMetadata(meshtastic_MeshPacket *mp) +{ + // LOG_DEBUG("PacketStatus %x", lora.getPacketStatus()); + mp->rx_snr = lora.getSNR(); + mp->rx_rssi = lround(lora.getRSSI()); + // LOG_DEBUG("Corrected frequency offset: %f", lora.getFrequencyError()); // not implemented for LR20x0, but noop for LR11x0 + // too(!) +} + +/** We override to turn on transmitter power as needed. + */ +template void LR20x0Interface::configHardwareForSend() +{ + RadioLibInterface::configHardwareForSend(); +} + +// For power draw measurements, helpful to force radio to stay sleeping +// #define SLEEP_ONLY + +template void LR20x0Interface::startReceive() +{ +#ifdef SLEEP_ONLY + sleep(); +#else + + setStandby(); + + lora.setPreambleLength(preambleLength); // Solve RX ack fail after direct message sent. Not sure why this is needed. + + // We use a 16 bit preamble so this should save some power by letting radio sit in standby mostly. + int err = + lora.startReceive(RADIOLIB_LR2021_RX_TIMEOUT_INF, MESHTASTIC_RADIOLIB_IRQ_RX_FLAGS, RADIOLIB_IRQ_RX_DEFAULT_MASK, 0); + if (err) + LOG_ERROR("StartReceive error: %d", err); + assert(err == RADIOLIB_ERR_NONE); + + RadioLibInterface::startReceive(); + + // Must be done AFTER starting receive, because startReceive clears (possibly stale) interrupt pending register bits + enableInterrupt(isrRxLevel0); + checkRxDoneIrqFlag(); +#endif +} + +/** Is the channel currently active? */ +template bool LR20x0Interface::isChannelActive() +{ + // check if we can detect a LoRa preamble on the current channel + ChannelScanConfig_t cfg = {.cad = {.symNum = NUM_SYM_CAD, + .detPeak = RADIOLIB_LR2021_CAD_PARAM_DEFAULT, + .detMin = RADIOLIB_LR2021_CAD_PARAM_DEFAULT, + .exitMode = RADIOLIB_LR2021_CAD_PARAM_DEFAULT, + .timeout = 0, + .irqFlags = RADIOLIB_IRQ_CAD_DEFAULT_FLAGS, + .irqMask = RADIOLIB_IRQ_CAD_DEFAULT_MASK}}; + int16_t result; + + setStandby(); + result = lora.scanChannel(cfg); + if (result == RADIOLIB_LORA_DETECTED) + return true; + + assert(result != RADIOLIB_ERR_WRONG_MODEM); + + return false; +} + +/** Could we send right now (i.e. either not actively receiving or transmitting)? */ +template bool LR20x0Interface::isActivelyReceiving() +{ + // The IRQ status will be cleared when we start our read operation. Check if we've started a header, but haven't yet + // received and handled the interrupt for reading the packet/handling errors. + return receiveDetected(lora.getIrqStatus(), RADIOLIB_LR2021_IRQ_LORA_HEADER_VALID, RADIOLIB_LR2021_IRQ_PREAMBLE_DETECTED); +} + +#ifdef LR20X0_AGC_RESET +template void LR20x0Interface::resetAGC() +{ + // Safety: don't reset mid-packet + if (sendingPacket != NULL || (isReceiving && isActivelyReceiving())) + return; + + LOG_DEBUG("LR20x0 AGC reset: warm sleep + Calibrate(0x3F)"); + + // 1. Warm sleep — powers down the analog frontend, resetting AGC state + lora.sleep(true, 0); + + // 2. Wake to RC standby for stable calibration + lora.standby(RADIOLIB_LR20X0_STANDBY_RC, true); + + // 3. Calibrate all blocks (PLL, ADC, image, RC oscillators) + // calibrate() is protected on LR20x0, so use raw SPI (same as internal implementation) + uint8_t calData = RADIOLIB_LR20X0_CALIBRATE_ALL; + module.SPIwriteStream(RADIOLIB_LR20X0_CMD_CALIBRATE, &calData, 1, true, true); + + // 4. Re-calibrate image rejection for actual operating frequency + // Calibrate(0x3F) defaults to 902-928 MHz which is wrong for other regions. + lora.calibrateImageRejection(getFreq() - 4.0f, getFreq() + 4.0f); + + // 5. Re-apply RX boosted gain mode + lora.setRxBoostedGainMode(config.lora.sx126x_rx_boosted_gain); + + // 6. Resume receiving + startReceive(); +} +#endif + +template bool LR20x0Interface::sleep() +{ + // \todo Display actual typename of the adapter, not just `LR20x0` + LOG_DEBUG("LR20x0 entering sleep mode"); + setStandby(); // Stop any pending operations + + // turn off TCXO if it was powered + lora.setTCXO(0); + + // put chipset into sleep mode (we've already disabled interrupts by now) + bool keepConfig = false; + lora.sleep(keepConfig, 0); // Note: we do not keep the config, full reinit will be needed + +#ifdef LR2021_POWER_EN + digitalWrite(LR2021_POWER_EN, LOW); +#endif + + return true; +} +#endif diff --git a/src/mesh/LR20x0Interface.h b/src/mesh/LR20x0Interface.h new file mode 100644 index 000000000..6176cd0b8 --- /dev/null +++ b/src/mesh/LR20x0Interface.h @@ -0,0 +1,75 @@ +#pragma once +#if RADIOLIB_EXCLUDE_LR2021 != 1 +#include "RadioLibInterface.h" + +/** + * \brief Adapter for LR20x0 radio family. Implements common logic for child classes. + * \tparam T RadioLib module type for LR20x0, e.g. LR2021. + */ +template class LR20x0Interface : public RadioLibInterface +{ + public: + LR20x0Interface(LockingArduinoHal *hal, RADIOLIB_PIN_TYPE cs, RADIOLIB_PIN_TYPE irq, RADIOLIB_PIN_TYPE rst, + RADIOLIB_PIN_TYPE busy); + + /// Initialise the Driver transport hardware and software. + /// Make sure the Driver is properly configured before calling init(). + /// \return true if initialisation succeeded. + virtual bool init() override; + + /// Apply any radio provisioning changes + /// Make sure the Driver is properly configured before calling init(). + /// \return true if initialisation succeeded. + virtual bool reconfigure() override; + + /// Prepare hardware for sleep. Call this _only_ for deep sleep, not needed for light sleep. + virtual bool sleep() override; + + bool isIRQPending() override { return lora.getIrqFlags() != 0; } + +#ifdef LR20X0_AGC_RESET + void resetAGC() override; +#endif + + protected: + /** + * Specific module instance + */ + T lora; + + /** + * Glue functions called from ISR land + */ + virtual void disableInterrupt() override; + + /** + * Enable a particular ISR callback glue function + */ + virtual void enableInterrupt(void (*callback)()) { lora.setIrqAction(callback); } + + /** can we detect a LoRa preamble on the current channel? */ + virtual bool isChannelActive() override; + + /** are we actively receiving a packet (only called during receiving state) */ + virtual bool isActivelyReceiving() override; + + /** + * Start waiting to receive a message + */ + virtual void startReceive() override; + + /** + * We override to turn on transmitter power as needed. + */ + virtual void configHardwareForSend() override; + + /** + * Add SNR data to received messages + */ + virtual void addReceiveMetadata(meshtastic_MeshPacket *mp) override; + + virtual void setStandby() override; + + uint32_t getPacketTime(uint32_t pl, bool received) override { return computePacketTime(lora, pl, received); } +}; +#endif diff --git a/src/mesh/MeshService.cpp b/src/mesh/MeshService.cpp index 952a6d2be..151831b96 100644 --- a/src/mesh/MeshService.cpp +++ b/src/mesh/MeshService.cpp @@ -94,8 +94,9 @@ int MeshService::handleFromRadio(const meshtastic_MeshPacket *mp) mp->decoded.portnum == meshtastic_PortNum_TELEMETRY_APP && mp->decoded.request_id > 0) { LOG_DEBUG("Received telemetry response. Skip sending our NodeInfo"); // ignore our request for its NodeInfo - } else if (mp->which_payload_variant == meshtastic_MeshPacket_decoded_tag && !nodeDB->getMeshNode(mp->from)->has_user && - nodeInfoModule && !isPreferredRebroadcaster && !nodeDB->isFull()) { + } else if (mp->which_payload_variant == meshtastic_MeshPacket_decoded_tag && + !nodeInfoLiteHasUser(nodeDB->getMeshNode(mp->from)) && nodeInfoModule && !isPreferredRebroadcaster && + !nodeDB->isFull()) { if (airTime->isTxAllowedChannelUtil(true)) { const int8_t hopsUsed = getHopsAway(*mp, config.lora.hop_limit); if (hopsUsed > (int32_t)(config.lora.hop_limit + 2)) { @@ -392,19 +393,16 @@ meshtastic_NodeInfoLite *MeshService::refreshLocalMeshNode() meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); assert(node); - // We might not have a position yet for our local node, in that case, at least try to send the time - if (!node->has_position) { - memset(&node->position, 0, sizeof(node->position)); - node->has_position = true; - } - - meshtastic_PositionLite &position = node->position; - // Update our local node info with our time (even if we don't decide to update anyone else) node->last_heard = getValidTime(RTCQualityFromNet); // This nodedb timestamp might be stale, so update it if our clock is kinda valid - position.time = getValidTime(RTCQualityFromNet); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + // Make sure our own NodeNum has a slot in the position map so subsequent + // updates (and the bundled NodeInfo emission to the phone) have somewhere + // to read from. Insert a default-zero entry on first call. + nodeDB->touchNodePositionTime(node->num, getValidTime(RTCQualityFromNet)); +#endif if (powerStatus->getHasBattery() == 1) { updateBatteryLevel(powerStatus->getBatteryChargePercent()); @@ -432,7 +430,9 @@ int MeshService::onGPSChanged(const meshtastic::GPSStatus *newStatus) // Used fixed position if configured regardless of GPS lock if (config.position.fixed_position) { LOG_WARN("Use fixed position"); - pos = TypeConversions::ConvertToPosition(node->position); + meshtastic_PositionLite fixedSlot; + if (nodeDB->copyNodePosition(node->num, fixedSlot)) + pos = TypeConversions::ConvertToPosition(fixedSlot); } // Add a fresh timestamp diff --git a/src/mesh/NodeDB.cpp b/src/mesh/NodeDB.cpp index ac6880ade..46aa61b28 100644 --- a/src/mesh/NodeDB.cpp +++ b/src/mesh/NodeDB.cpp @@ -22,6 +22,7 @@ #include "error.h" #include "main.h" #include "mesh-pb-constants.h" +#include "mesh/generated/meshtastic/deviceonly_legacy.pb.h" #include "meshUtils.h" #include "modules/NeighborInfoModule.h" #include @@ -151,24 +152,103 @@ uint32_t get_st7789_id(uint8_t cs, uint8_t sck, uint8_t mosi, uint8_t dc, uint8_ #endif -bool meshtastic_NodeDatabase_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field) +bool meshtastic_NodeDatabase_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_t *field) { - if (ostream) { - std::vector const *vec = (std::vector *)field->pData; - for (auto item : *vec) { - if (!pb_encode_tag_for_field(ostream, field)) - return false; - pb_encode_submessage(ostream, meshtastic_NodeInfoLite_fields, &item); + const auto *iter = reinterpret_cast(field); + switch (iter->tag) { + case meshtastic_NodeDatabase_nodes_tag: { + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodeInfoLite_fields, &item)) + return false; + } } + if (istream) { + meshtastic_NodeInfoLite node; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodeInfoLite_fields, &node)) + vec->push_back(node); + } + return true; } - if (istream) { - meshtastic_NodeInfoLite node; // this gets good data - std::vector *vec = (std::vector *)field->pData; - - if (istream->bytes_left && pb_decode(istream, meshtastic_NodeInfoLite_fields, &node)) - vec->push_back(node); + case meshtastic_NodeDatabase_positions_tag: { + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodePositionEntry_fields, &item)) + return false; + } + } + if (istream) { + meshtastic_NodePositionEntry entry; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodePositionEntry_fields, &entry)) + vec->push_back(entry); + } + return true; + } + case meshtastic_NodeDatabase_telemetry_tag: { + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodeTelemetryEntry_fields, &item)) + return false; + } + } + if (istream) { + meshtastic_NodeTelemetryEntry entry; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodeTelemetryEntry_fields, &entry)) + vec->push_back(entry); + } + return true; + } + case meshtastic_NodeDatabase_status_tag: { + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodeStatusEntry_fields, &item)) + return false; + } + } + if (istream) { + meshtastic_NodeStatusEntry entry; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodeStatusEntry_fields, &entry)) + vec->push_back(entry); + } + return true; + } + case meshtastic_NodeDatabase_environment_tag: { + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodeEnvironmentEntry_fields, &item)) + return false; + } + } + if (istream) { + meshtastic_NodeEnvironmentEntry entry; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodeEnvironmentEntry_fields, &entry)) + vec->push_back(entry); + } + return true; + } + default: + return true; } - return true; } /** The current change # for radio settings. Starts at 0 on boot and any time the radio settings @@ -298,8 +378,7 @@ NodeDB::NodeDB() #endif // Include our owner in the node db under our nodenum meshtastic_NodeInfoLite *info = getOrCreateMeshNode(getNodeNum()); - info->user = TypeConversions::ConvertToUserLite(owner); - info->has_user = true; + TypeConversions::CopyUserToNodeInfoLite(info, owner); // If node database has not been saved for the first time, save it now #ifdef FSCom @@ -442,8 +521,12 @@ NodeDB::NodeDB() #if defined(USERPREFS_FIXED_GPS_LAT) && defined(USERPREFS_FIXED_GPS_LON) fixedGPS.location_source = meshtastic_Position_LocSource_LOC_MANUAL; config.has_position = true; - info->has_position = true; - info->position = TypeConversions::ConvertToPositionLite(fixedGPS); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + { + concurrency::LockGuard guard(&satelliteMutex); + nodePositions[info->num] = TypeConversions::ConvertToPositionLite(fixedGPS); + } +#endif nodeDB->setLocalPosition(fixedGPS); config.position.fixed_position = true; #endif @@ -479,6 +562,29 @@ bool isBroadcast(uint32_t dest) return dest == NODENUM_BROADCAST || dest == NODENUM_BROADCAST_NO_LORA; } +namespace +{ +template bool copySatelliteEntry(const Map &map, NodeNum n, Value &out) +{ + auto it = map.find(n); + if (it == map.end()) + return false; + out = it->second; + return true; +} + +template std::vector snapshotSatelliteNodeNums(const Map &map, NodeNum exclude) +{ + std::vector result; + result.reserve(map.size()); + for (const auto &kv : map) { + if (kv.first != exclude) + result.push_back(kv.first); + } + return result; +} +} // namespace + void NodeDB::resetRadioConfig(bool is_fresh_install) { if (is_fresh_install) { @@ -548,6 +654,19 @@ void NodeDB::installDefaultNodeDatabase() nodeDatabase.nodes = std::vector(MAX_NUM_NODES); numMeshNodes = 0; meshNodes = &nodeDatabase.nodes; + concurrency::LockGuard satelliteGuard(&satelliteMutex); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + nodePositions.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + nodeTelemetry.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + nodeEnvironment.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_STATUSDB + nodeStatus.clear(); +#endif } void NodeDB::installDefaultConfig(bool preserveKey = false) @@ -1019,12 +1138,14 @@ void NodeDB::resetNodes(bool keepFavorites) { if (!config.position.fixed_position) clearLocalPosition(); + NodeNum ourNum = getNodeNum(); numMeshNodes = 1; if (keepFavorites) { LOG_INFO("Clearing node database - preserving favorites"); for (size_t i = 0; i < meshNodes->size(); i++) { meshtastic_NodeInfoLite &node = meshNodes->at(i); - if (i > 0 && !node.is_favorite) { + if (i > 0 && !nodeInfoLiteIsFavorite(&node)) { + eraseNodeSatellites(node.num); node = meshtastic_NodeInfoLite(); } else { numMeshNodes += 1; @@ -1032,8 +1153,14 @@ void NodeDB::resetNodes(bool keepFavorites) }; } else { LOG_INFO("Clearing node database - removing favorites"); + for (size_t i = 1; i < meshNodes->size(); i++) { + const NodeNum gone = meshNodes->at(i).num; + if (gone) + eraseNodeSatellites(gone); + } std::fill(nodeDatabase.nodes.begin() + 1, nodeDatabase.nodes.end(), meshtastic_NodeInfoLite()); } + (void)ourNum; devicestate.has_rx_text_message = false; devicestate.has_rx_waypoint = false; saveNodeDatabaseToDisk(); @@ -1054,36 +1181,173 @@ void NodeDB::removeNodeByNum(NodeNum nodeNum) numMeshNodes -= removed; std::fill(nodeDatabase.nodes.begin() + numMeshNodes, nodeDatabase.nodes.begin() + numMeshNodes + 1, meshtastic_NodeInfoLite()); + if (removed) + eraseNodeSatellites(nodeNum); LOG_DEBUG("NodeDB::removeNodeByNum purged %d entries. Save changes", removed); saveNodeDatabaseToDisk(); } void NodeDB::clearLocalPosition() { - meshtastic_NodeInfoLite *node = getMeshNode(nodeDB->getNodeNum()); - node->position.latitude_i = 0; - node->position.longitude_i = 0; - node->position.altitude = 0; - node->position.time = 0; +#if !MESHTASTIC_EXCLUDE_POSITIONDB + concurrency::LockGuard guard(&satelliteMutex); + nodePositions.erase(getNodeNum()); +#endif setLocalPosition(meshtastic_Position_init_default); localPositionUpdatedSinceBoot = false; } +bool NodeDB::copyNodePosition(NodeNum n, meshtastic_PositionLite &out) const +{ +#if MESHTASTIC_EXCLUDE_POSITIONDB + (void)n; + (void)out; + return false; +#else + concurrency::LockGuard guard(&satelliteMutex); + return copySatelliteEntry(nodePositions, n, out); +#endif +} + +bool NodeDB::copyNodeTelemetry(NodeNum n, meshtastic_DeviceMetrics &out) const +{ +#if MESHTASTIC_EXCLUDE_TELEMETRYDB + (void)n; + (void)out; + return false; +#else + concurrency::LockGuard guard(&satelliteMutex); + return copySatelliteEntry(nodeTelemetry, n, out); +#endif +} + +bool NodeDB::copyNodeEnvironment(NodeNum n, meshtastic_EnvironmentMetrics &out) const +{ +#if MESHTASTIC_EXCLUDE_ENVIRONMENTDB + (void)n; + (void)out; + return false; +#else + concurrency::LockGuard guard(&satelliteMutex); + return copySatelliteEntry(nodeEnvironment, n, out); +#endif +} + +bool NodeDB::copyNodeStatus(NodeNum n, meshtastic_StatusMessage &out) const +{ +#if MESHTASTIC_EXCLUDE_STATUSDB + (void)n; + (void)out; + return false; +#else + concurrency::LockGuard guard(&satelliteMutex); + return copySatelliteEntry(nodeStatus, n, out); +#endif +} + +std::vector NodeDB::snapshotPositionNodeNums(NodeNum exclude) const +{ +#if MESHTASTIC_EXCLUDE_POSITIONDB + (void)exclude; + return {}; +#else + concurrency::LockGuard guard(&satelliteMutex); + return snapshotSatelliteNodeNums(nodePositions, exclude); +#endif +} + +std::vector NodeDB::snapshotTelemetryNodeNums(NodeNum exclude) const +{ +#if MESHTASTIC_EXCLUDE_TELEMETRYDB + (void)exclude; + return {}; +#else + concurrency::LockGuard guard(&satelliteMutex); + return snapshotSatelliteNodeNums(nodeTelemetry, exclude); +#endif +} + +std::vector NodeDB::snapshotEnvironmentNodeNums(NodeNum exclude) const +{ +#if MESHTASTIC_EXCLUDE_ENVIRONMENTDB + (void)exclude; + return {}; +#else + concurrency::LockGuard guard(&satelliteMutex); + return snapshotSatelliteNodeNums(nodeEnvironment, exclude); +#endif +} + +std::vector NodeDB::snapshotStatusNodeNums(NodeNum exclude) const +{ +#if MESHTASTIC_EXCLUDE_STATUSDB + (void)exclude; + return {}; +#else + concurrency::LockGuard guard(&satelliteMutex); + return snapshotSatelliteNodeNums(nodeStatus, exclude); +#endif +} + +void NodeDB::setNodeStatus(NodeNum n, const meshtastic_StatusMessage &status) +{ +#if MESHTASTIC_EXCLUDE_STATUSDB + (void)n; + (void)status; +#else + concurrency::LockGuard guard(&satelliteMutex); + nodeStatus[n] = status; +#endif +} + +void NodeDB::touchNodePositionTime(NodeNum n, uint32_t time) +{ +#if MESHTASTIC_EXCLUDE_POSITIONDB + (void)n; + (void)time; +#else + concurrency::LockGuard guard(&satelliteMutex); + nodePositions[n].time = time; +#endif +} + +void NodeDB::eraseNodeSatellites(NodeNum n) +{ + concurrency::LockGuard guard(&satelliteMutex); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + nodePositions.erase(n); +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + nodeTelemetry.erase(n); +#endif +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + nodeEnvironment.erase(n); +#endif +#if !MESHTASTIC_EXCLUDE_STATUSDB + nodeStatus.erase(n); +#endif +} + void NodeDB::cleanupMeshDB() { int newPos = 0, removed = 0; for (int i = 0; i < numMeshNodes; i++) { - if (meshNodes->at(i).has_user) { - if (meshNodes->at(i).user.public_key.size > 0) { - if (memfll(meshNodes->at(i).user.public_key.bytes, 0, meshNodes->at(i).user.public_key.size)) { - meshNodes->at(i).user.public_key.size = 0; + meshtastic_NodeInfoLite &n = meshNodes->at(i); + if (nodeInfoLiteHasUser(&n)) { + if (n.public_key.size > 0) { + if (memfll(n.public_key.bytes, 0, n.public_key.size)) { + n.public_key.size = 0; } } if (newPos != i) - meshNodes->at(newPos++) = meshNodes->at(i); + meshNodes->at(newPos++) = n; else newPos++; } else { + // No user info - drop this node and its satellites + const NodeNum gone = n.num; + if (gone) + eraseNodeSatellites(gone); removed++; } } @@ -1141,14 +1405,22 @@ void NodeDB::pickNewNodeNum() nodeNum = (ourMacAddr[2] << 24) | (ourMacAddr[3] << 16) | (ourMacAddr[4] << 8) | ourMacAddr[5]; } + // Identity check via public key (or "empty slot?" when no keys yet); + // macaddr no longer lives on the slim header. + auto isOurOwnEntry = [&](const meshtastic_NodeInfoLite *n) -> bool { + if (!n) + return false; + if (owner.public_key.size == 32 && n->public_key.size == 32) + return memcmp(n->public_key.bytes, owner.public_key.bytes, 32) == 0; + return !nodeInfoLiteHasUser(n); + }; + meshtastic_NodeInfoLite *found; - while (((found = getMeshNode(nodeNum)) && memcmp(found->user.macaddr, ourMacAddr, sizeof(ourMacAddr)) != 0) || + while (((found = getMeshNode(nodeNum)) && !isOurOwnEntry(found)) || (nodeNum == NODENUM_BROADCAST || nodeNum < NUM_RESERVED)) { NodeNum candidate = random(NUM_RESERVED, LONG_MAX); // try a new random choice if (found) - LOG_WARN("NOTE! Our desired nodenum 0x%x is invalid or in use, by MAC ending in 0x%02x%02x vs our 0x%02x%02x, so " - "trying for 0x%x", - nodeNum, found->user.macaddr[4], found->user.macaddr[5], ourMacAddr[4], ourMacAddr[5], candidate); + LOG_WARN("NOTE! Our desired nodenum 0x%x is invalid or in use, picking 0x%x", nodeNum, candidate); nodeNum = candidate; } LOG_DEBUG("Use nodenum 0x%x ", nodeNum); @@ -1169,7 +1441,8 @@ LoadFileResult NodeDB::loadProto(const char *filename, size_t protoSize, size_t if (f) { LOG_INFO("Load %s", filename); pb_istream_t stream = {&readcb, &f, protoSize}; - if (fields != &meshtastic_NodeDatabase_msg) // contains a vector object + if (fields != &meshtastic_NodeDatabase_msg && + fields != &meshtastic_NodeDatabase_Legacy_msg) // both NodeDatabase descriptors contain std::vector members memset(dest_struct, 0, objSize); if (!pb_decode(&stream, fields, dest_struct)) { LOG_ERROR("Error: can't decode protobuf %s", PB_GET_ERROR(&stream)); @@ -1242,9 +1515,57 @@ void NodeDB::loadFromDisk() if (nodeDatabase.version < DEVICESTATE_MIN_VER) { LOG_WARN("NodeDatabase %d is old, discard", nodeDatabase.version); installDefaultNodeDatabase(); + } else if (nodeDatabase.version < DEVICESTATE_CUR_VER) { + if (migrateLegacyNodeDatabase()) + saveNodeDatabaseToDisk(); + else + installDefaultNodeDatabase(); } else { meshNodes = &nodeDatabase.nodes; numMeshNodes = nodeDatabase.nodes.size(); + // Hydrate the satellite maps; the on-disk vectors stay empty in steady + // state and are repopulated only at save time. + concurrency::LockGuard guard(&satelliteMutex); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + nodePositions.clear(); + nodePositions.reserve(nodeDatabase.positions.size()); + for (const auto &entry : nodeDatabase.positions) { + if (entry.has_position) + nodePositions[entry.num] = entry.position; + } + nodeDatabase.positions.clear(); + nodeDatabase.positions.shrink_to_fit(); +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + nodeTelemetry.clear(); + nodeTelemetry.reserve(nodeDatabase.telemetry.size()); + for (const auto &entry : nodeDatabase.telemetry) { + if (entry.has_device_metrics) + nodeTelemetry[entry.num] = entry.device_metrics; + } + nodeDatabase.telemetry.clear(); + nodeDatabase.telemetry.shrink_to_fit(); +#endif +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + nodeEnvironment.clear(); + nodeEnvironment.reserve(nodeDatabase.environment.size()); + for (const auto &entry : nodeDatabase.environment) { + if (entry.has_environment_metrics) + nodeEnvironment[entry.num] = entry.environment_metrics; + } + nodeDatabase.environment.clear(); + nodeDatabase.environment.shrink_to_fit(); +#endif +#if !MESHTASTIC_EXCLUDE_STATUSDB + nodeStatus.clear(); + nodeStatus.reserve(nodeDatabase.status.size()); + for (const auto &entry : nodeDatabase.status) { + if (entry.has_status) + nodeStatus[entry.num] = entry.status; + } + nodeDatabase.status.clear(); + nodeDatabase.status.shrink_to_fit(); +#endif LOG_INFO("Loaded saved nodedatabase version %d, with nodes count: %d", nodeDatabase.version, nodeDatabase.nodes.size()); } @@ -1272,16 +1593,16 @@ void NodeDB::loadFromDisk() // Attempt recovery of owner fields from our own NodeDB entry if available. meshtastic_NodeInfoLite *us = getMeshNode(getNodeNum()); - if (us && us->has_user) { + if (nodeInfoLiteHasUser(us)) { LOG_WARN("Restoring owner fields (long_name/short_name/is_licensed/is_unmessagable) from NodeDB for our node 0x%08x", us->num); - memcpy(owner.long_name, us->user.long_name, sizeof(owner.long_name)); + memcpy(owner.long_name, us->long_name, sizeof(owner.long_name)); owner.long_name[sizeof(owner.long_name) - 1] = '\0'; - memcpy(owner.short_name, us->user.short_name, sizeof(owner.short_name)); + memcpy(owner.short_name, us->short_name, sizeof(owner.short_name)); owner.short_name[sizeof(owner.short_name) - 1] = '\0'; - owner.is_licensed = us->user.is_licensed; - owner.has_is_unmessagable = us->user.has_is_unmessagable; - owner.is_unmessagable = us->user.is_unmessagable; + owner.is_licensed = nodeInfoLiteIsLicensed(us); + owner.has_is_unmessagable = nodeInfoLiteHasIsUnmessagable(us); + owner.is_unmessagable = nodeInfoLiteIsUnmessagable(us); // Save the recovered owner to device state on disk saveToDisk(SEGMENT_DEVICESTATE); @@ -1528,9 +1849,75 @@ bool NodeDB::saveNodeDatabaseToDisk() FSCom.mkdir("/prefs"); spiLock->unlock(); #endif + // Project the maps into the on-disk vectors just before encoding; cleared + // again on the way out so we don't carry duplicate state. + concurrency::LockGuard guard(&satelliteMutex); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + nodeDatabase.positions.clear(); + nodeDatabase.positions.reserve(nodePositions.size()); + for (const auto &kv : nodePositions) { + meshtastic_NodePositionEntry entry = meshtastic_NodePositionEntry_init_default; + entry.num = kv.first; + entry.has_position = true; + entry.position = kv.second; + nodeDatabase.positions.push_back(entry); + } +#else + nodeDatabase.positions.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + nodeDatabase.telemetry.clear(); + nodeDatabase.telemetry.reserve(nodeTelemetry.size()); + for (const auto &kv : nodeTelemetry) { + meshtastic_NodeTelemetryEntry entry = meshtastic_NodeTelemetryEntry_init_default; + entry.num = kv.first; + entry.has_device_metrics = true; + entry.device_metrics = kv.second; + nodeDatabase.telemetry.push_back(entry); + } +#else + nodeDatabase.telemetry.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + nodeDatabase.environment.clear(); + nodeDatabase.environment.reserve(nodeEnvironment.size()); + for (const auto &kv : nodeEnvironment) { + meshtastic_NodeEnvironmentEntry entry = meshtastic_NodeEnvironmentEntry_init_default; + entry.num = kv.first; + entry.has_environment_metrics = true; + entry.environment_metrics = kv.second; + nodeDatabase.environment.push_back(entry); + } +#else + nodeDatabase.environment.clear(); +#endif +#if !MESHTASTIC_EXCLUDE_STATUSDB + nodeDatabase.status.clear(); + nodeDatabase.status.reserve(nodeStatus.size()); + for (const auto &kv : nodeStatus) { + meshtastic_NodeStatusEntry entry = meshtastic_NodeStatusEntry_init_default; + entry.num = kv.first; + entry.has_status = true; + entry.status = kv.second; + nodeDatabase.status.push_back(entry); + } +#else + nodeDatabase.status.clear(); +#endif + size_t nodeDatabaseSize; pb_get_encoded_size(&nodeDatabaseSize, meshtastic_NodeDatabase_fields, &nodeDatabase); - return saveProto(nodeDatabaseFileName, nodeDatabaseSize, &meshtastic_NodeDatabase_msg, &nodeDatabase, false); + bool ok = saveProto(nodeDatabaseFileName, nodeDatabaseSize, &meshtastic_NodeDatabase_msg, &nodeDatabase, false); + + nodeDatabase.positions.clear(); + nodeDatabase.positions.shrink_to_fit(); + nodeDatabase.telemetry.clear(); + nodeDatabase.telemetry.shrink_to_fit(); + nodeDatabase.environment.clear(); + nodeDatabase.environment.shrink_to_fit(); + nodeDatabase.status.clear(); + nodeDatabase.status.shrink_to_fit(); + return ok; } bool NodeDB::saveToDiskNoRetry(int saveWhat) @@ -1699,7 +2086,7 @@ size_t NodeDB::getNumOnlineMeshNodes(bool localOnly) // FIXME this implementation is kinda expensive for (int i = 0; i < numMeshNodes; i++) { - if (localOnly && meshNodes->at(i).via_mqtt) + if (localOnly && nodeInfoLiteViaMqtt(&meshNodes->at(i))) continue; if (sinceLastSeen(&meshNodes->at(i)) < NUM_ONLINE_SECS) numseen++; @@ -1736,62 +2123,94 @@ void NodeDB::updatePosition(uint32_t nodeId, const meshtastic_Position &p, RxSou return; } +#if MESHTASTIC_EXCLUDE_POSITIONDB + // Build flag opted out: header still tracks last_heard via updateFrom; we + // simply don't cache the position payload anywhere on this device. if (src == RX_SRC_LOCAL) { - // Local packet, fully authoritative - LOG_INFO("updatePosition LOCAL pos@%x time=%u lat=%d lon=%d alt=%d", p.timestamp, p.time, p.latitude_i, p.longitude_i, + LOG_INFO("updatePosition LOCAL (PositionDB excluded) time=%u lat=%d lon=%d alt=%d", p.time, p.latitude_i, p.longitude_i, p.altitude); - setLocalPosition(p); - info->position = TypeConversions::ConvertToPositionLite(p); - } else if ((p.time > 0) && !p.latitude_i && !p.longitude_i && !p.timestamp && !p.location_source) { - // FIXME SPECIAL TIME SETTING PACKET FROM EUD TO RADIO - // (stop-gap fix for issue #900) - LOG_DEBUG("updatePosition SPECIAL time setting time=%u", p.time); - info->position.time = p.time; - } else { - // Be careful to only update fields that have been set by the REMOTE sender - // A lot of position reports don't have time populated. In that case, be careful to not blow away the time we - // recorded based on the packet rxTime - // - // FIXME perhaps handle RX_SRC_USER separately? - LOG_INFO("updatePosition REMOTE node=0x%x time=%u lat=%d lon=%d", nodeId, p.time, p.latitude_i, p.longitude_i); - - // First, back up fields that we want to protect from overwrite - uint32_t tmp_time = info->position.time; - - // Next, update atomically - info->position = TypeConversions::ConvertToPositionLite(p); - - // Last, restore any fields that may have been overwritten - if (!info->position.time) - info->position.time = tmp_time; } - info->has_position = true; + (void)nodeId; + updateGUIforNode = info; + notifyObservers(true); +#else + { + concurrency::LockGuard guard(&satelliteMutex); + meshtastic_PositionLite &slot = nodePositions[nodeId]; // creates default-zero entry if missing + + if (src == RX_SRC_LOCAL) { + // Local packet, fully authoritative + LOG_INFO("updatePosition LOCAL pos@%x time=%u lat=%d lon=%d alt=%d", p.timestamp, p.time, p.latitude_i, p.longitude_i, + p.altitude); + + setLocalPosition(p); + slot = TypeConversions::ConvertToPositionLite(p); + } else if ((p.time > 0) && !p.latitude_i && !p.longitude_i && !p.timestamp && !p.location_source) { + // FIXME SPECIAL TIME SETTING PACKET FROM EUD TO RADIO + // (stop-gap fix for issue #900) + LOG_DEBUG("updatePosition SPECIAL time setting time=%u", p.time); + slot.time = p.time; + } else { + // Be careful to only update fields that have been set by the REMOTE sender + // A lot of position reports don't have time populated. In that case, be careful to not blow away the time we + // recorded based on the packet rxTime + // + // FIXME perhaps handle RX_SRC_USER separately? + LOG_INFO("updatePosition REMOTE node=0x%x time=%u lat=%d lon=%d", nodeId, p.time, p.latitude_i, p.longitude_i); + + // First, back up fields that we want to protect from overwrite + uint32_t tmp_time = slot.time; + + // Next, update atomically + slot = TypeConversions::ConvertToPositionLite(p); + + // Last, restore any fields that may have been overwritten + if (!slot.time) + slot.time = tmp_time; + } + } updateGUIforNode = info; notifyObservers(true); // Force an update whether or not our node counts have changed +#endif } -/** Update telemetry info for this node based on received metrics - * We only care about device telemetry here +/** Update telemetry info for this node based on received metrics. Stores + * device_metrics and environment_metrics into their respective satellite + * maps; other variants (air_quality, power, local_stats, health) are + * intentionally not retained per-node. */ void NodeDB::updateTelemetry(uint32_t nodeId, const meshtastic_Telemetry &t, RxSource src) { meshtastic_NodeInfoLite *info = getOrCreateMeshNode(nodeId); - // Environment metrics should never go to NodeDb but we'll safegaurd anyway - if (!info || t.which_variant != meshtastic_Telemetry_device_metrics_tag) { + if (!info) return; - } - if (src == RX_SRC_LOCAL) { - // Local packet, fully authoritative - LOG_DEBUG("updateTelemetry LOCAL"); + if (t.which_variant == meshtastic_Telemetry_device_metrics_tag) { + if (src == RX_SRC_LOCAL) { + LOG_DEBUG("updateTelemetry LOCAL device"); + } else { + LOG_DEBUG("updateTelemetry REMOTE device node=0x%x", nodeId); + } +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + concurrency::LockGuard guard(&satelliteMutex); + nodeTelemetry[nodeId] = t.variant.device_metrics; +#endif + } else if (t.which_variant == meshtastic_Telemetry_environment_metrics_tag) { + if (src == RX_SRC_LOCAL) { + LOG_DEBUG("updateTelemetry LOCAL env"); + } else { + LOG_DEBUG("updateTelemetry REMOTE env node=0x%x", nodeId); + } +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + concurrency::LockGuard guard(&satelliteMutex); + nodeEnvironment[nodeId] = t.variant.environment_metrics; +#endif } else { - LOG_DEBUG("updateTelemetry REMOTE node=0x%x ", nodeId); + return; // air_quality / power / local_stats / health: not stored per-node } - info->device_metrics = t.variant.device_metrics; - info->has_device_metrics = true; updateGUIforNode = info; - notifyObservers(true); // Force an update whether or not our node counts have changed + notifyObservers(true); } /** @@ -1806,24 +2225,22 @@ void NodeDB::addFromContact(meshtastic_SharedContact contact) // If the local node has this node marked as manually verified // and the client does not, do not allow the client to update the // saved public key. - if ((info->bitfield & NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK) && !contact.manually_verified) { - if (contact.user.public_key.size != info->user.public_key.size || - memcmp(contact.user.public_key.bytes, info->user.public_key.bytes, info->user.public_key.size) != 0) { + if (nodeInfoLiteIsKeyManuallyVerified(info) && !contact.manually_verified) { + if (contact.user.public_key.size != info->public_key.size || + memcmp(contact.user.public_key.bytes, info->public_key.bytes, info->public_key.size) != 0) { return; } } info->num = contact.node_num; - info->has_user = true; - info->user = TypeConversions::ConvertToUserLite(contact.user); + TypeConversions::CopyUserToNodeInfoLite(info, contact.user); if (contact.should_ignore) { // If should_ignore is set, // we need to clear the public key and other cruft, in addition to setting the node as ignored - info->is_ignored = true; - info->is_favorite = false; - info->has_device_metrics = false; - info->has_position = false; - info->user.public_key.size = 0; - memset(info->user.public_key.bytes, 0, sizeof(info->user.public_key.bytes)); + nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); + nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_FAVORITE_MASK, false); + eraseNodeSatellites(contact.node_num); + info->public_key.size = 0; + memset(info->public_key.bytes, 0, sizeof(info->public_key.bytes)); } else { /* Clients are sending add_contact before every text message DM (because clients may hold a larger node database with * public keys than the radio holds). However, we don't want to update last_heard just because we sent someone a DM! @@ -1842,12 +2259,12 @@ void NodeDB::addFromContact(meshtastic_SharedContact contact) } else { // Normal case: set is_favorite to prevent expiration. // last_heard will remain as-is (or remain 0 if this entry wasn't in the nodeDB). - info->is_favorite = true; + nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); } // As the clients will begin sending the contact with DMs, we want to strictly check if the node is manually verified if (contact.manually_verified) { - info->bitfield |= NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; + nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK, true); } // Mark the node's key as manually verified to indicate trustworthiness. updateGUIforNode = info; @@ -1881,15 +2298,15 @@ bool NodeDB::updateUser(uint32_t nodeId, meshtastic_User &p, uint8_t channelInde meshtastic_ClientNotification *cn = clientNotificationPool.allocZeroed(); cn->level = meshtastic_LogRecord_Level_WARNING; cn->time = getValidTime(RTCQualityFromNet); - sprintf(cn->message, warning, p.long_name); + snprintf(cn->message, sizeof(cn->message), warning, p.long_name); service->sendClientNotification(cn); } return false; } } - if (info->user.public_key.size == 32) { // if we have a key for this user already, don't overwrite with a new one + if (info->public_key.size == 32) { // if we have a key for this user already, don't overwrite with a new one // if the key doesn't match, don't update nodeDB at all. - if (p.public_key.size != 32 || (memcmp(p.public_key.bytes, info->user.public_key.bytes, 32) != 0)) { + if (p.public_key.size != 32 || (memcmp(p.public_key.bytes, info->public_key.bytes, 32) != 0)) { LOG_WARN("Public Key mismatch, dropping NodeInfo"); return false; } @@ -1902,19 +2319,22 @@ bool NodeDB::updateUser(uint32_t nodeId, meshtastic_User &p, uint8_t channelInde // Always ensure user.id is derived from nodeId, regardless of what was received snprintf(p.id, sizeof(p.id), "!%08x", nodeId); - // Both of info->user and p start as filled with zero so I think this is okay - auto lite = TypeConversions::ConvertToUserLite(p); - bool changed = memcmp(&info->user, &lite, sizeof(info->user)) || (info->channel != channelIndex); + meshtastic_NodeInfoLite before = *info; + TypeConversions::CopyUserToNodeInfoLite(info, p); + bool changed = + (memcmp(before.long_name, info->long_name, sizeof(info->long_name)) != 0) || + (memcmp(before.short_name, info->short_name, sizeof(info->short_name)) != 0) || (before.hw_model != info->hw_model) || + (before.role != info->role) || (before.public_key.size != info->public_key.size) || + (info->public_key.size > 0 && memcmp(before.public_key.bytes, info->public_key.bytes, info->public_key.size) != 0) || + (before.bitfield != info->bitfield) || (info->channel != channelIndex); - info->user = lite; - if (info->user.public_key.size == 32) { - printBytes("Saved Pubkey: ", info->user.public_key.bytes, 32); + if (info->public_key.size == 32) { + printBytes("Saved Pubkey: ", info->public_key.bytes, 32); } if (nodeId != getNodeNum()) info->channel = channelIndex; // Set channel we need to use to reach this node (but don't set our own channel) - LOG_DEBUG("Update changed=%d user %s/%s, id=0x%08x, channel=%d", changed, info->user.long_name, info->user.short_name, nodeId, + LOG_DEBUG("Update changed=%d user %s/%s, id=0x%08x, channel=%d", changed, info->long_name, info->short_name, nodeId, info->channel); - info->has_user = true; if (changed) { updateGUIforNode = info; @@ -1956,7 +2376,8 @@ void NodeDB::updateFrom(const meshtastic_MeshPacket &mp) if (mp.rx_snr) info->snr = mp.rx_snr; // keep the most recent SNR we received for this node. - info->via_mqtt = mp.via_mqtt; // Store if we received this packet via MQTT + nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_VIA_MQTT_MASK, + mp.via_mqtt); // Store if we received this packet via MQTT // If hopStart was set and there wasn't someone messing with the limit in the middle, add hopsAway const int8_t hopsAway = getHopsAway(mp); @@ -1971,8 +2392,8 @@ void NodeDB::updateFrom(const meshtastic_MeshPacket &mp) void NodeDB::set_favorite(bool is_favorite, uint32_t nodeId) { meshtastic_NodeInfoLite *lite = getMeshNode(nodeId); - if (lite && lite->is_favorite != is_favorite) { - lite->is_favorite = is_favorite; + if (lite && nodeInfoLiteIsFavorite(lite) != is_favorite) { + nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_IS_FAVORITE_MASK, is_favorite); sortMeshDB(); saveNodeDatabaseToDisk(); } @@ -1986,10 +2407,10 @@ bool NodeDB::isFavorite(uint32_t nodeId) if (nodeId == NODENUM_BROADCAST) return false; - meshtastic_NodeInfoLite *lite = getMeshNode(nodeId); + const meshtastic_NodeInfoLite *lite = getMeshNode(nodeId); if (lite) { - return lite->is_favorite; + return nodeInfoLiteIsFavorite(lite); } return false; } @@ -2014,14 +2435,14 @@ bool NodeDB::isFromOrToFavoritedNode(const meshtastic_MeshPacket &p) lite = &meshNodes->at(i); if (lite->num == p.from) { - if (lite->is_favorite) + if (nodeInfoLiteIsFavorite(lite)) return true; seenFrom = true; } if (lite->num == p.to) { - if (lite->is_favorite) + if (nodeInfoLiteIsFavorite(lite)) return true; seenTo = true; @@ -2057,10 +2478,10 @@ void NodeDB::sortMeshDB() // TODO: Look for at(i-1) also matching own node num, and throw the DB in the trash std::swap(meshNodes->at(i), meshNodes->at(i - 1)); changed = true; - } else if (meshNodes->at(i).is_favorite && !meshNodes->at(i - 1).is_favorite) { + } else if (nodeInfoLiteIsFavorite(&meshNodes->at(i)) && !nodeInfoLiteIsFavorite(&meshNodes->at(i - 1))) { std::swap(meshNodes->at(i), meshNodes->at(i - 1)); changed = true; - } else if (!meshNodes->at(i).is_favorite && meshNodes->at(i - 1).is_favorite) { + } else if (!nodeInfoLiteIsFavorite(&meshNodes->at(i)) && nodeInfoLiteIsFavorite(&meshNodes->at(i - 1))) { // noop } else if (meshNodes->at(i).last_heard > meshNodes->at(i - 1).last_heard) { std::swap(meshNodes->at(i), meshNodes->at(i - 1)); @@ -2120,17 +2541,18 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) int oldestIndex = -1; int oldestBoringIndex = -1; for (int i = 1; i < numMeshNodes; i++) { + const meshtastic_NodeInfoLite *cand = &meshNodes->at(i); + const bool isFavoriteNode = nodeInfoLiteIsFavorite(cand); + const bool isIgnored = nodeInfoLiteIsIgnored(cand); + const bool isVerified = nodeInfoLiteIsKeyManuallyVerified(cand); // Simply the oldest non-favorite, non-ignored, non-verified node - if (!meshNodes->at(i).is_favorite && !meshNodes->at(i).is_ignored && - !(meshNodes->at(i).bitfield & NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK) && - meshNodes->at(i).last_heard < oldest) { - oldest = meshNodes->at(i).last_heard; + if (!isFavoriteNode && !isIgnored && !isVerified && cand->last_heard < oldest) { + oldest = cand->last_heard; oldestIndex = i; } // The oldest "boring" node - if (!meshNodes->at(i).is_favorite && !meshNodes->at(i).is_ignored && meshNodes->at(i).user.public_key.size == 0 && - meshNodes->at(i).last_heard < oldestBoring) { - oldestBoring = meshNodes->at(i).last_heard; + if (!isFavoriteNode && !isIgnored && cand->public_key.size == 0 && cand->last_heard < oldestBoring) { + oldestBoring = cand->last_heard; oldestBoringIndex = i; } } @@ -2140,6 +2562,7 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) } if (oldestIndex != -1) { + eraseNodeSatellites(meshNodes->at(oldestIndex).num); // Shove the remaining nodes down the chain for (int i = oldestIndex; i < numMeshNodes - 1; i++) { meshNodes->at(i) = meshNodes->at(i + 1); @@ -2163,18 +2586,23 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) /// valid lat/lon bool NodeDB::hasValidPosition(const meshtastic_NodeInfoLite *n) { - return n->has_position && (n->position.latitude_i != 0 || n->position.longitude_i != 0); + if (!n) + return false; + if (n->num == getNodeNum()) { + return localPosition.latitude_i != 0 || localPosition.longitude_i != 0; + } + meshtastic_PositionLite pos; + return copyNodePosition(n->num, pos) && (pos.latitude_i != 0 || pos.longitude_i != 0); } /// If we have a node / user and they report is_licensed = true /// we consider them licensed UserLicenseStatus NodeDB::getLicenseStatus(uint32_t nodeNum) { - meshtastic_NodeInfoLite *info = getMeshNode(nodeNum); - if (!info || !info->has_user) { + const meshtastic_NodeInfoLite *info = getMeshNode(nodeNum); + if (!nodeInfoLiteHasUser(info)) return UserLicenseStatus::NotKnown; - } - return info->user.is_licensed ? UserLicenseStatus::Licensed : UserLicenseStatus::NotLicensed; + return nodeInfoLiteIsLicensed(info) ? UserLicenseStatus::Licensed : UserLicenseStatus::NotLicensed; } #if !defined(MESHTASTIC_EXCLUDE_PKI) diff --git a/src/mesh/NodeDB.h b/src/mesh/NodeDB.h index f6be963c1..090f3bb9a 100644 --- a/src/mesh/NodeDB.h +++ b/src/mesh/NodeDB.h @@ -6,10 +6,12 @@ #include #include #include +#include #include #include "MeshTypes.h" #include "NodeStatus.h" +#include "concurrency/Lock.h" #include "configuration.h" #include "mesh-pb-constants.h" #include "mesh/generated/meshtastic/mesh.pb.h" // For CriticalErrorCode @@ -82,7 +84,9 @@ DeviceState versions used to be defined in the .proto file but really only this #define SEGMENT_CHANNELS 8 #define SEGMENT_NODEDATABASE 16 -#define DEVICESTATE_CUR_VER 24 +#define DEVICESTATE_CUR_VER 25 +// Lowest on-disk version we still know how to load. v24 saves are migrated +// at boot via the parallel deviceonly_legacy descriptor and re-saved as v25. #define DEVICESTATE_MIN_VER 24 extern meshtastic_DeviceState devicestate; @@ -166,6 +170,21 @@ class NodeDB Observable newStatus; pb_size_t numMeshNodes; + // Satellite per-NodeNum maps for data we used to inline into NodeInfoLite, + // gated by MESHTASTIC_EXCLUDE_*DB so STM32WL can omit them. +#if !MESHTASTIC_EXCLUDE_POSITIONDB + std::unordered_map nodePositions; +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + std::unordered_map nodeTelemetry; +#endif +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + std::unordered_map nodeEnvironment; +#endif +#if !MESHTASTIC_EXCLUDE_STATUSDB + std::unordered_map nodeStatus; +#endif + bool keyIsLowEntropy = false; bool hasWarned = false; @@ -277,6 +296,43 @@ class NodeDB virtual meshtastic_NodeInfoLite *getMeshNode(NodeNum n); size_t getNumMeshNodes() { return numMeshNodes; } + // Thread-safe satellite-map accessors. Return false if absent or the + // corresponding DB is compiled out. + bool copyNodePosition(NodeNum n, meshtastic_PositionLite &out) const; + bool copyNodeTelemetry(NodeNum n, meshtastic_DeviceMetrics &out) const; + bool copyNodeEnvironment(NodeNum n, meshtastic_EnvironmentMetrics &out) const; + bool copyNodeStatus(NodeNum n, meshtastic_StatusMessage &out) const; + std::vector snapshotPositionNodeNums(NodeNum exclude) const; + std::vector snapshotTelemetryNodeNums(NodeNum exclude) const; + std::vector snapshotEnvironmentNodeNums(NodeNum exclude) const; + std::vector snapshotStatusNodeNums(NodeNum exclude) const; + + void setNodeStatus(NodeNum n, const meshtastic_StatusMessage &status); + void touchNodePositionTime(NodeNum n, uint32_t time); + + bool hasNodePosition(NodeNum n) const + { + meshtastic_PositionLite scratch; + return copyNodePosition(n, scratch); + } + bool hasNodeTelemetry(NodeNum n) const + { + meshtastic_DeviceMetrics scratch; + return copyNodeTelemetry(n, scratch); + } + bool hasNodeEnvironment(NodeNum n) const + { + meshtastic_EnvironmentMetrics scratch; + return copyNodeEnvironment(n, scratch); + } + bool hasNodeStatus(NodeNum n) const + { + meshtastic_StatusMessage scratch; + return copyNodeStatus(n, scratch); + } + + void eraseNodeSatellites(NodeNum n); + UserLicenseStatus getLicenseStatus(uint32_t nodeNum); size_t getMaxNodesAllocatedSize() @@ -285,7 +341,11 @@ class NodeDB emptyNodeDatabase.version = DEVICESTATE_CUR_VER; size_t nodeDatabaseSize; pb_get_encoded_size(&nodeDatabaseSize, meshtastic_NodeDatabase_fields, &emptyNodeDatabase); - return nodeDatabaseSize + (MAX_NUM_NODES * meshtastic_NodeInfoLite_size); + // Always include satellite slots so backups from higher-cap peers + // decode without truncation, even when our build excludes the DBs. + return nodeDatabaseSize + (MAX_NUM_NODES * meshtastic_NodeInfoLite_size) + + (MAX_NUM_NODES * meshtastic_NodePositionEntry_size) + (MAX_NUM_NODES * meshtastic_NodeTelemetryEntry_size) + + (MAX_NUM_NODES * meshtastic_NodeEnvironmentEntry_size) + (MAX_NUM_NODES * meshtastic_NodeStatusEntry_size); } // returns true if the maximum number of nodes is reached or we are running low on memory @@ -329,6 +389,7 @@ class NodeDB } private: + mutable concurrency::Lock satelliteMutex; bool duplicateWarned = false; bool localPositionUpdatedSinceBoot = false; uint32_t lastNodeDbSave = 0; // when we last saved our db to flash @@ -363,6 +424,11 @@ class NodeDB bool saveDeviceStateToDisk(); bool saveNodeDatabaseToDisk(); void sortMeshDB(); + + // Defined in NodeDBLegacyMigration.cpp. Decodes /prefs/nodes.proto via + // the legacy descriptor and copies entries into the v25 layout. Caller + // is responsible for save / install-default on the result. + bool migrateLegacyNodeDatabase(); }; extern NodeDB *nodeDB; @@ -397,10 +463,74 @@ extern meshtastic_CriticalErrorCode error_code; * A numeric error address (nonzero if available) */ extern uint32_t error_address; +// Bit assignments for meshtastic_NodeInfoLite.bitfield. #define NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_SHIFT 0 -#define NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK (1 << NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_SHIFT) +#define NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK (1u << NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_SHIFT) #define NODEINFO_BITFIELD_IS_MUTED_SHIFT 1 -#define NODEINFO_BITFIELD_IS_MUTED_MASK (1 << NODEINFO_BITFIELD_IS_MUTED_SHIFT) +#define NODEINFO_BITFIELD_IS_MUTED_MASK (1u << NODEINFO_BITFIELD_IS_MUTED_SHIFT) +#define NODEINFO_BITFIELD_VIA_MQTT_SHIFT 2 +#define NODEINFO_BITFIELD_VIA_MQTT_MASK (1u << NODEINFO_BITFIELD_VIA_MQTT_SHIFT) +#define NODEINFO_BITFIELD_IS_FAVORITE_SHIFT 3 +#define NODEINFO_BITFIELD_IS_FAVORITE_MASK (1u << NODEINFO_BITFIELD_IS_FAVORITE_SHIFT) +#define NODEINFO_BITFIELD_IS_IGNORED_SHIFT 4 +#define NODEINFO_BITFIELD_IS_IGNORED_MASK (1u << NODEINFO_BITFIELD_IS_IGNORED_SHIFT) +#define NODEINFO_BITFIELD_HAS_USER_SHIFT 5 +#define NODEINFO_BITFIELD_HAS_USER_MASK (1u << NODEINFO_BITFIELD_HAS_USER_SHIFT) +#define NODEINFO_BITFIELD_IS_LICENSED_SHIFT 6 +#define NODEINFO_BITFIELD_IS_LICENSED_MASK (1u << NODEINFO_BITFIELD_IS_LICENSED_SHIFT) +#define NODEINFO_BITFIELD_IS_UNMESSAGABLE_SHIFT 7 +#define NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK (1u << NODEINFO_BITFIELD_IS_UNMESSAGABLE_SHIFT) +#define NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_SHIFT 8 +#define NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK (1u << NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_SHIFT) +// Bits 9..31 reserved for future single-bit flags. + +// Convenience accessors so call sites read like the old struct fields. +inline bool nodeInfoLiteHasUser(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_HAS_USER_MASK); +} +inline bool nodeInfoLiteViaMqtt(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_VIA_MQTT_MASK); +} +inline bool nodeInfoLiteIsFavorite(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_FAVORITE_MASK); +} +inline bool nodeInfoLiteIsIgnored(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_IGNORED_MASK); +} +inline bool nodeInfoLiteIsLicensed(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_LICENSED_MASK); +} +inline bool nodeInfoLiteHasIsUnmessagable(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK); +} +inline bool nodeInfoLiteIsUnmessagable(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK); +} +inline bool nodeInfoLiteIsMuted(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK); +} +inline bool nodeInfoLiteIsKeyManuallyVerified(const meshtastic_NodeInfoLite *n) +{ + return n && (n->bitfield & NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK); +} + +inline void nodeInfoLiteSetBit(meshtastic_NodeInfoLite *n, uint32_t mask, bool value) +{ + if (!n) + return; + if (value) + n->bitfield |= mask; + else + n->bitfield &= ~mask; +} #define Module_Config_size \ (ModuleConfig_CannedMessageConfig_size + ModuleConfig_ExternalNotificationConfig_size + ModuleConfig_MQTTConfig_size + \ diff --git a/src/mesh/NodeDBLegacyMigration.cpp b/src/mesh/NodeDBLegacyMigration.cpp new file mode 100644 index 000000000..18d6daa4d --- /dev/null +++ b/src/mesh/NodeDBLegacyMigration.cpp @@ -0,0 +1,129 @@ +// Migration from the legacy (pre-v25) NodeDatabase shape to the slim header + +// satellite maps layout. Lives in its own translation unit so the bulk +// fixed-shape decode + field-by-field copy doesn't clutter NodeDB.cpp. +// +// Caller (NodeDB::loadFromDisk) decides what to do with the result: +// - true -> persist via saveNodeDatabaseToDisk() +// - false -> reset via installDefaultNodeDatabase() +// +// This file (and the deviceonly_legacy proto) can be removed once +// DEVICESTATE_MIN_VER advances past 24. + +#include "NodeDB.h" +#include "concurrency/LockGuard.h" +#include "configuration.h" +#include "mesh-pb-constants.h" +#include "mesh/generated/meshtastic/deviceonly_legacy.pb.h" + +#include +#include +#include +#include + +bool meshtastic_NodeDatabase_Legacy_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_t *field) +{ + const auto *iter = reinterpret_cast(field); + if (ostream) { + const auto *vec = static_cast *>(iter->pData); + for (auto item : *vec) { + if (!pb_encode_tag_for_field(ostream, iter)) + return false; + if (!pb_encode_submessage(ostream, meshtastic_NodeInfoLite_Legacy_fields, &item)) + return false; + } + } + if (istream) { + meshtastic_NodeInfoLite_Legacy node; + auto *vec = static_cast *>(iter->pData); + if (istream->bytes_left && pb_decode(istream, meshtastic_NodeInfoLite_Legacy_fields, &node)) + vec->push_back(node); + } + return true; +} + +bool NodeDB::migrateLegacyNodeDatabase() +{ + LOG_WARN("NodeDatabase v%u: migrating to v%u", nodeDatabase.version, DEVICESTATE_CUR_VER); + + // _init_zero brace-inits the embedded std::vector via its explicit + // (size_type, allocator) ctor, so default-construct instead. + meshtastic_NodeDatabase_Legacy legacyDb{}; + legacyDb.version = 0; + meshtastic_NodeDatabase_Legacy legacyEmpty{}; + legacyEmpty.version = DEVICESTATE_CUR_VER; + size_t legacyEmptyEncoded = 0; + pb_get_encoded_size(&legacyEmptyEncoded, meshtastic_NodeDatabase_Legacy_fields, &legacyEmpty); + const size_t legacyBufSize = legacyEmptyEncoded + (MAX_NUM_NODES * meshtastic_NodeInfoLite_Legacy_size); + auto legacyState = loadProto(nodeDatabaseFileName, legacyBufSize, sizeof(meshtastic_NodeDatabase_Legacy), + &meshtastic_NodeDatabase_Legacy_msg, &legacyDb); + if (legacyState != LoadFileResult::LOAD_SUCCESS) { + LOG_ERROR("Failed to load NodeDatabase via legacy descriptor; installing default"); + return false; + } + + nodeDatabase.nodes.clear(); + const size_t maxToMigrate = std::min(legacyDb.nodes.size(), MAX_NUM_NODES); + nodeDatabase.nodes.reserve(maxToMigrate); + size_t posCount = 0, telCount = 0; + { + concurrency::LockGuard guard(&satelliteMutex); + for (size_t i = 0; i < maxToMigrate; ++i) { + const auto &legacy = legacyDb.nodes[i]; + meshtastic_NodeInfoLite slim = meshtastic_NodeInfoLite_init_default; + slim.num = legacy.num; + slim.snr = legacy.snr; + slim.last_heard = legacy.last_heard; + slim.channel = legacy.channel; + slim.has_hops_away = legacy.has_hops_away; + slim.hops_away = legacy.hops_away; + slim.next_hop = legacy.next_hop; + slim.bitfield = legacy.bitfield; + if (legacy.via_mqtt) + slim.bitfield |= NODEINFO_BITFIELD_VIA_MQTT_MASK; + if (legacy.is_favorite) + slim.bitfield |= NODEINFO_BITFIELD_IS_FAVORITE_MASK; + if (legacy.is_ignored) + slim.bitfield |= NODEINFO_BITFIELD_IS_IGNORED_MASK; + if (legacy.has_user) { + slim.bitfield |= NODEINFO_BITFIELD_HAS_USER_MASK; + strncpy(slim.long_name, legacy.user.long_name, sizeof(slim.long_name)); + slim.long_name[sizeof(slim.long_name) - 1] = '\0'; + strncpy(slim.short_name, legacy.user.short_name, sizeof(slim.short_name)); + slim.short_name[sizeof(slim.short_name) - 1] = '\0'; + slim.hw_model = legacy.user.hw_model; + slim.role = legacy.user.role; + if (legacy.user.is_licensed) + slim.bitfield |= NODEINFO_BITFIELD_IS_LICENSED_MASK; + slim.public_key.size = legacy.user.public_key.size; + memcpy(slim.public_key.bytes, legacy.user.public_key.bytes, sizeof(slim.public_key.bytes)); + if (legacy.user.has_is_unmessagable) { + slim.bitfield |= NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK; + if (legacy.user.is_unmessagable) + slim.bitfield |= NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK; + } + // macaddr deprecated since 1.2.11 - dropped from slim header. + } + nodeDatabase.nodes.push_back(slim); +#if !MESHTASTIC_EXCLUDE_POSITIONDB + if (legacy.has_position) + nodePositions[legacy.num] = legacy.position; +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + if (legacy.has_device_metrics) + nodeTelemetry[legacy.num] = legacy.device_metrics; +#endif + } +#if !MESHTASTIC_EXCLUDE_POSITIONDB + posCount = nodePositions.size(); +#endif +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + telCount = nodeTelemetry.size(); +#endif + } + nodeDatabase.version = DEVICESTATE_CUR_VER; + meshNodes = &nodeDatabase.nodes; + numMeshNodes = nodeDatabase.nodes.size(); + LOG_INFO("Migrated %u nodes from legacy -> v%u (positions: %u, telemetry: %u)", (unsigned)numMeshNodes, DEVICESTATE_CUR_VER, + (unsigned)posCount, (unsigned)telCount); + return true; +} diff --git a/src/mesh/PacketHistory.cpp b/src/mesh/PacketHistory.cpp index 8289f0078..b56ce57c9 100644 --- a/src/mesh/PacketHistory.cpp +++ b/src/mesh/PacketHistory.cpp @@ -17,7 +17,7 @@ #define VERBOSE_PACKET_HISTORY 0 // Set to 1 for verbose logging, 2 for heavy debugging #define PACKET_HISTORY_TRACE_AGING 1 // Set to 1 to enable logging of the age of re/used history slots -PacketHistory::PacketHistory(uint32_t size) : recentPacketsCapacity(0), recentPackets(NULL) // Initialize members +PacketHistory::PacketHistory(uint32_t size) : recentPacketsCapacity(0) // Initialize members { if (size < 4 || size > PACKETHISTORY_MAX) { // Copilot suggested - makes sense LOG_WARN("Packet History - Invalid size %d, using default %d", size, PACKETHISTORY_MAX); @@ -34,7 +34,7 @@ PacketHistory::PacketHistory(uint32_t size) : recentPacketsCapacity(0), recentPa // Allocate memory for the recent packets array recentPacketsCapacity = size; - recentPackets = new PacketRecord[recentPacketsCapacity]; + recentPackets.reset(new PacketRecord[recentPacketsCapacity]); if (!recentPackets) { // No logging here, console/log probably uninitialized yet. LOG_ERROR("Packet History - Memory allocation failed for size=%d entries / %d Bytes", size, sizeof(PacketRecord) * recentPacketsCapacity); @@ -43,33 +43,20 @@ PacketHistory::PacketHistory(uint32_t size) : recentPacketsCapacity(0), recentPa } // Initialize the recent packets array to zero - memset(recentPackets, 0, sizeof(PacketRecord) * recentPacketsCapacity); + memset(recentPackets.get(), 0, sizeof(PacketRecord) * recentPacketsCapacity); #if !MESHTASTIC_EXCLUDE_PKT_HISTORY_HASH // Allocate hash index with load factor <= 0.5 for short probe chains hashCapacity = nextPowerOf2(recentPacketsCapacity * 2); hashMask = hashCapacity - 1; - hashIndex = new uint16_t[hashCapacity]; + hashIndex.reset(new uint16_t[hashCapacity]); if (!hashIndex) { LOG_ERROR("Packet History - Hash index allocation failed for %d entries", hashCapacity); hashCapacity = 0; hashMask = 0; return; } - memset(hashIndex, 0xFF, sizeof(uint16_t) * hashCapacity); // Fill with HASH_EMPTY (0xFFFF) -#endif -} - -PacketHistory::~PacketHistory() -{ - recentPacketsCapacity = 0; - delete[] recentPackets; - recentPackets = NULL; -#if !MESHTASTIC_EXCLUDE_PKT_HISTORY_HASH - delete[] hashIndex; - hashIndex = NULL; - hashCapacity = 0; - hashMask = 0; + memset(hashIndex.get(), 0xFF, sizeof(uint16_t) * hashCapacity); // Fill with HASH_EMPTY (0xFFFF) #endif } @@ -285,7 +272,7 @@ void PacketHistory::hashRebuild() { if (!hashIndex) return; - memset(hashIndex, 0xFF, sizeof(uint16_t) * hashCapacity); + memset(hashIndex.get(), 0xFF, sizeof(uint16_t) * hashCapacity); for (uint32_t i = 0; i < recentPacketsCapacity; i++) { if (recentPackets[i].rxTimeMsec != 0) hashInsert(recentPackets[i].sender, recentPackets[i].id, (uint16_t)i); @@ -332,7 +319,8 @@ PacketHistory::PacketRecord *PacketHistory::find(NodeNum sender, PacketId id) #endif // Linear scan (sole path when hash excluded, fallback when hash allocation failed) - for (PacketRecord *it = recentPackets; it < (recentPackets + recentPacketsCapacity); ++it) { + PacketRecord *base = recentPackets.get(); + for (PacketRecord *it = base; it < (base + recentPacketsCapacity); ++it) { if (it->id == id && it->sender == sender) { return it; } @@ -346,39 +334,38 @@ void PacketHistory::insert(const PacketRecord &r) { uint32_t now_millis = millis(); // Should not jump with time changes uint32_t OldtrxTimeMsec = 0; + PacketRecord *base = recentPackets.get(); PacketRecord *tu = NULL; // Will insert here. PacketRecord *it = NULL; // Find a free, matching or oldest used slot in the recentPackets array - for (it = recentPackets; it < (recentPackets + recentPacketsCapacity); ++it) { + for (it = base; it < (base + recentPacketsCapacity); ++it) { if (it->id == 0 && it->sender == 0 /*&& rxTimeMsec == 0*/) { // Record is empty tu = it; // Remember the free slot #if VERBOSE_PACKET_HISTORY >= 2 - LOG_DEBUG("Packet History - insert: Free slot@ %d/%d", tu - recentPackets, recentPacketsCapacity); + LOG_DEBUG("Packet History - insert: Free slot@ %d/%d", tu - base, recentPacketsCapacity); #endif // We have that, Exit the loop - it = (recentPackets + recentPacketsCapacity); + it = (base + recentPacketsCapacity); } else if (it->id == r.id && it->sender == r.sender) { // Record matches the packet we want to insert tu = it; // Remember the matching slot OldtrxTimeMsec = now_millis - it->rxTimeMsec; // ..and save current entry's age #if VERBOSE_PACKET_HISTORY >= 2 - LOG_DEBUG("Packet History - insert: Matched slot@ %d/%d age=%d", tu - recentPackets, recentPacketsCapacity, - OldtrxTimeMsec); + LOG_DEBUG("Packet History - insert: Matched slot@ %d/%d age=%d", tu - base, recentPacketsCapacity, OldtrxTimeMsec); #endif // We have that, Exit the loop - it = (recentPackets + recentPacketsCapacity); + it = (base + recentPacketsCapacity); } else { if (it->rxTimeMsec == 0) { LOG_WARN( "Packet History - insert: Found packet s=%08x id=%08x with rxTimeMsec = 0, slot %d/%d. Should never happen!", - it->sender, it->id, it - recentPackets, recentPacketsCapacity); + it->sender, it->id, it - base, recentPacketsCapacity); } if ((now_millis - it->rxTimeMsec) > OldtrxTimeMsec) { // 49.7 days rollover friendly OldtrxTimeMsec = now_millis - it->rxTimeMsec; tu = it; // remember the oldest packet #if VERBOSE_PACKET_HISTORY >= 2 - LOG_DEBUG("Packet History - insert: Older slot@ %d/%d age=%d", tu - recentPackets, recentPacketsCapacity, - OldtrxTimeMsec); + LOG_DEBUG("Packet History - insert: Older slot@ %d/%d age=%d", tu - base, recentPacketsCapacity, OldtrxTimeMsec); #endif } // keep looking for oldest till entire array is checked @@ -393,13 +380,11 @@ void PacketHistory::insert(const PacketRecord &r) #if VERBOSE_PACKET_HISTORY if (tu->id == 0 && tu->sender == 0) { - LOG_DEBUG("Packet History - insert: slot@ %d/%d is NEW", tu - recentPackets, recentPacketsCapacity); + LOG_DEBUG("Packet History - insert: slot@ %d/%d is NEW", tu - base, recentPacketsCapacity); } else if (tu->id == r.id && tu->sender == r.sender) { - LOG_DEBUG("Packet History - insert: slot@ %d/%d MATCHED, age=%d", tu - recentPackets, recentPacketsCapacity, - OldtrxTimeMsec); + LOG_DEBUG("Packet History - insert: slot@ %d/%d MATCHED, age=%d", tu - base, recentPacketsCapacity, OldtrxTimeMsec); } else { - LOG_DEBUG("Packet History - insert: slot@ %d/%d REUSE OLDEST, age=%d", tu - recentPackets, recentPacketsCapacity, - OldtrxTimeMsec); + LOG_DEBUG("Packet History - insert: slot@ %d/%d REUSE OLDEST, age=%d", tu - base, recentPacketsCapacity, OldtrxTimeMsec); } #endif @@ -432,9 +417,9 @@ void PacketHistory::insert(const PacketRecord &r) #endif #if VERBOSE_PACKET_HISTORY - LOG_DEBUG("Packet History - insert: Store slot@ %d/%d s=%08x id=%08x nh=%02x rby=%02x %02x %02x rxT=%d BEFORE", - tu - recentPackets, recentPacketsCapacity, tu->sender, tu->id, tu->next_hop, tu->relayed_by[0], tu->relayed_by[1], - tu->relayed_by[2], tu->rxTimeMsec); + LOG_DEBUG("Packet History - insert: Store slot@ %d/%d s=%08x id=%08x nh=%02x rby=%02x %02x %02x rxT=%d BEFORE", tu - base, + recentPacketsCapacity, tu->sender, tu->id, tu->next_hop, tu->relayed_by[0], tu->relayed_by[1], tu->relayed_by[2], + tu->rxTimeMsec); #endif if (r.rxTimeMsec == 0) { @@ -454,16 +439,16 @@ void PacketHistory::insert(const PacketRecord &r) *tu = r; // store the packet if (!isMatchingSlot) { - hashInsert(r.sender, r.id, (uint16_t)(tu - recentPackets)); + hashInsert(r.sender, r.id, (uint16_t)(tu - base)); } #else *tu = r; // store the packet #endif #if VERBOSE_PACKET_HISTORY - LOG_DEBUG("Packet History - insert: Store slot@ %d/%d s=%08x id=%08x nh=%02x rby=%02x %02x %02x rxT=%d AFTER", - tu - recentPackets, recentPacketsCapacity, tu->sender, tu->id, tu->next_hop, tu->relayed_by[0], tu->relayed_by[1], - tu->relayed_by[2], tu->rxTimeMsec); + LOG_DEBUG("Packet History - insert: Store slot@ %d/%d s=%08x id=%08x nh=%02x rby=%02x %02x %02x rxT=%d AFTER", tu - base, + recentPacketsCapacity, tu->sender, tu->id, tu->next_hop, tu->relayed_by[0], tu->relayed_by[1], tu->relayed_by[2], + tu->rxTimeMsec); #endif } diff --git a/src/mesh/PacketHistory.h b/src/mesh/PacketHistory.h index a11e2d038..e45df8474 100644 --- a/src/mesh/PacketHistory.h +++ b/src/mesh/PacketHistory.h @@ -1,6 +1,7 @@ #pragma once #include "NodeDB.h" +#include // Number of relayers we keep track of. Use 6 to be efficient with memory alignment of PacketRecord to 20 bytes #define NUM_RELAYERS 6 @@ -26,7 +27,7 @@ class PacketHistory uint32_t recentPacketsCapacity = 0; // Can be set in constructor, no need to recompile. Used to allocate memory for mx_recentPackets. - PacketRecord *recentPackets = NULL; // Simple and fixed in size. Debloat. + std::unique_ptr recentPackets; // Simple and fixed in size. Debloat. #if !MESHTASTIC_EXCLUDE_PKT_HISTORY_HASH // Open-addressing hash table for O(1) lookup in find(), replacing the O(N) linear scan. @@ -34,7 +35,7 @@ class PacketHistory // The load factor invariant holds permanently: hashCapacity = 2 * nextPowerOf2(recentPacketsCapacity), // and at most recentPacketsCapacity entries can ever be live (one per recentPackets[] slot). static constexpr uint16_t HASH_EMPTY = 0xFFFF; - uint16_t *hashIndex = NULL; + std::unique_ptr hashIndex; uint32_t hashCapacity = 0; // Always a power of 2 uint32_t hashMask = 0; // hashCapacity - 1, for fast modular indexing @@ -64,11 +65,8 @@ class PacketHistory uint8_t getOurTxHopLimit(const PacketRecord &r); void setOurTxHopLimit(PacketRecord &r, uint8_t hopLimit); - PacketHistory(const PacketHistory &); // non construction-copyable - PacketHistory &operator=(const PacketHistory &); // non copyable public: explicit PacketHistory(uint32_t size = -1); // Constructor with size parameter, default is PACKETHISTORY_MAX - ~PacketHistory(); /** * Update recentBroadcasts and return true if we have already seen this packet @@ -100,5 +98,5 @@ class PacketHistory void removeRelayer(const uint8_t relayer, const uint32_t id, const NodeNum sender); // To check if the PacketHistory was initialized correctly by constructor - bool initOk(void) { return recentPackets != NULL && recentPacketsCapacity != 0; } + bool initOk(void) { return recentPackets != nullptr && recentPacketsCapacity != 0; } }; diff --git a/src/mesh/PhoneAPI.cpp b/src/mesh/PhoneAPI.cpp index 5c3ab486a..ffe413056 100644 --- a/src/mesh/PhoneAPI.cpp +++ b/src/mesh/PhoneAPI.cpp @@ -62,7 +62,7 @@ void PhoneAPI::handleStartConfig() onConfigStart(); // even if we were already connected - restart our state machine - if (config_nonce == SPECIAL_NONCE_ONLY_NODES) { + if (config_nonce == SPECIAL_NONCE_ONLY_NODES || config_nonce == SPECIAL_NONCE_GRADIENT_ONLY_NODES) { // If client only wants node info, jump directly to sending nodes state = STATE_SEND_OWN_NODEINFO; LOG_INFO("Client only wants node info, skipping other config"); @@ -81,6 +81,15 @@ void PhoneAPI::handleStartConfig() concurrency::LockGuard guard(&nodeInfoMutex); nodeInfoForPhone = {}; nodeInfoQueue.clear(); + replayQueue.clear(); + replayPositionOrder.clear(); + replayTelemetryOrder.clear(); + replayEnvironmentOrder.clear(); + replayStatusOrder.clear(); + replayPositionIndex = 0; + replayTelemetryIndex = 0; + replayEnvironmentIndex = 0; + replayStatusIndex = 0; } resetReadIndex(); } @@ -120,6 +129,15 @@ void PhoneAPI::close() concurrency::LockGuard guard(&nodeInfoMutex); nodeInfoForPhone = {}; nodeInfoQueue.clear(); + replayQueue.clear(); + replayPositionOrder.clear(); + replayTelemetryOrder.clear(); + replayEnvironmentOrder.clear(); + replayStatusOrder.clear(); + replayPositionIndex = 0; + replayTelemetryIndex = 0; + replayEnvironmentIndex = 0; + replayStatusIndex = 0; } packetForPhone = NULL; filesManifest.clear(); @@ -302,7 +320,7 @@ size_t PhoneAPI::getFromRadio(uint8_t *buf) nodeInfoForPhone.num = 0; } } - if (config_nonce == SPECIAL_NONCE_ONLY_NODES) { + if (config_nonce == SPECIAL_NONCE_ONLY_NODES || config_nonce == SPECIAL_NONCE_GRADIENT_ONLY_NODES) { // If client only wants node info, jump directly to sending nodes state = STATE_SEND_OTHER_NODEINFOS; onNowHasData(0); @@ -529,8 +547,124 @@ size_t PhoneAPI::getFromRadio(uint8_t *buf) LOG_DEBUG("Done sending %d of %d nodeinfos millis=%u", readIndex, nodeDB->getNumMeshNodes(), millis()); concurrency::LockGuard guard(&nodeInfoMutex); nodeInfoQueue.clear(); + // Replay states no-op for legacy clients / excluded DBs. + state = STATE_REPLAY_POSITIONS; + return getFromRadio(buf); + } + break; + } + + case STATE_REPLAY_POSITIONS: { + if (replayPositionOrder.empty() && replayPositionIndex == 0) + beginReplayPositions(); + prefetchReplayPositions(); + + meshtastic_MeshPacket pkt = {}; + bool havePkt = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + if (!replayQueue.empty()) { + pkt = replayQueue.front(); + replayQueue.pop_front(); + havePkt = true; + } + } + + if (havePkt) { + fromRadioScratch.which_payload_variant = meshtastic_FromRadio_packet_tag; + fromRadioScratch.packet = pkt; + } else { + LOG_DEBUG("Done replaying positions count=%u millis=%u", (unsigned)replayPositionIndex, millis()); + state = STATE_REPLAY_TELEMETRY; + return getFromRadio(buf); + } + break; + } + + case STATE_REPLAY_TELEMETRY: { + if (replayTelemetryOrder.empty() && replayTelemetryIndex == 0) + beginReplayTelemetry(); + prefetchReplayTelemetry(); + + meshtastic_MeshPacket pkt = {}; + bool havePkt = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + if (!replayQueue.empty()) { + pkt = replayQueue.front(); + replayQueue.pop_front(); + havePkt = true; + } + } + + if (havePkt) { + fromRadioScratch.which_payload_variant = meshtastic_FromRadio_packet_tag; + fromRadioScratch.packet = pkt; + } else { + LOG_DEBUG("Done replaying telemetry count=%u millis=%u", (unsigned)replayTelemetryIndex, millis()); + state = STATE_REPLAY_ENVIRONMENT; + return getFromRadio(buf); + } + break; + } + + case STATE_REPLAY_ENVIRONMENT: { + if (replayEnvironmentOrder.empty() && replayEnvironmentIndex == 0) + beginReplayEnvironment(); + prefetchReplayEnvironment(); + + meshtastic_MeshPacket pkt = {}; + bool havePkt = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + if (!replayQueue.empty()) { + pkt = replayQueue.front(); + replayQueue.pop_front(); + havePkt = true; + } + } + + if (havePkt) { + fromRadioScratch.which_payload_variant = meshtastic_FromRadio_packet_tag; + fromRadioScratch.packet = pkt; + } else { + LOG_DEBUG("Done replaying environment count=%u millis=%u", (unsigned)replayEnvironmentIndex, millis()); + state = STATE_REPLAY_STATUS; + return getFromRadio(buf); + } + break; + } + + case STATE_REPLAY_STATUS: { + if (replayStatusOrder.empty() && replayStatusIndex == 0) + beginReplayStatus(); + prefetchReplayStatus(); + + meshtastic_MeshPacket pkt = {}; + bool havePkt = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + if (!replayQueue.empty()) { + pkt = replayQueue.front(); + replayQueue.pop_front(); + havePkt = true; + } + } + + if (havePkt) { + fromRadioScratch.which_payload_variant = meshtastic_FromRadio_packet_tag; + fromRadioScratch.packet = pkt; + } else { + LOG_DEBUG("Done replaying status count=%u millis=%u", (unsigned)replayStatusIndex, millis()); + replayPositionOrder.clear(); + replayPositionOrder.shrink_to_fit(); + replayTelemetryOrder.clear(); + replayTelemetryOrder.shrink_to_fit(); + replayEnvironmentOrder.clear(); + replayEnvironmentOrder.shrink_to_fit(); + replayStatusOrder.clear(); + replayStatusOrder.shrink_to_fit(); state = STATE_SEND_FILEMANIFEST; - // Go ahead and send that ID right now return getFromRadio(buf); } break; @@ -538,9 +672,9 @@ size_t PhoneAPI::getFromRadio(uint8_t *buf) case STATE_SEND_FILEMANIFEST: { LOG_DEBUG("FromRadio=STATE_SEND_FILEMANIFEST"); - // last element - if (config_state == filesManifest.size() || - config_nonce == SPECIAL_NONCE_ONLY_NODES) { // also handles an empty filesManifest + // ONLY_NODES variants skip the manifest. + if (config_state == filesManifest.size() || config_nonce == SPECIAL_NONCE_ONLY_NODES || + config_nonce == SPECIAL_NONCE_GRADIENT_ONLY_NODES) { config_state = 0; filesManifest.clear(); // Skip to complete packet @@ -653,6 +787,7 @@ void PhoneAPI::prefetchNodeInfos() { bool added = false; bool wasEmpty = false; + const bool gradient = clientWantsGradientSync(); // Keep the queue topped up so BLE reads stay responsive even if DB fetches take a moment. { concurrency::LockGuard guard(&nodeInfoMutex); @@ -662,7 +797,8 @@ void PhoneAPI::prefetchNodeInfos() if (!nextNode) break; - auto info = TypeConversions::ConvertToNodeInfo(nextNode); + auto info = + gradient ? TypeConversions::ConvertToNodeInfoThin(nextNode) : TypeConversions::ConvertToNodeInfo(nextNode); bool isUs = info.num == nodeDB->getNodeNum(); info.hops_away = isUs ? 0 : info.hops_away; info.last_heard = isUs ? getValidTime(RTCQualityFromNet) : info.last_heard; @@ -682,6 +818,257 @@ void PhoneAPI::prefetchNodeInfos() onNowHasData(0); } +meshtastic_MeshPacket PhoneAPI::makeReplayPositionPacket(NodeNum num, const meshtastic_PositionLite &pos) +{ + meshtastic_MeshPacket pkt = meshtastic_MeshPacket_init_default; + pkt.from = num; + pkt.to = nodeDB->getNodeNum(); + pkt.id = generatePacketId(); + pkt.rx_time = pos.time; + pkt.which_payload_variant = meshtastic_MeshPacket_decoded_tag; + pkt.decoded.portnum = meshtastic_PortNum_POSITION_APP; + meshtastic_Position fullPos = TypeConversions::ConvertToPosition(pos); + size_t len = + pb_encode_to_bytes(pkt.decoded.payload.bytes, sizeof(pkt.decoded.payload.bytes), &meshtastic_Position_msg, &fullPos); + pkt.decoded.payload.size = (pb_size_t)len; + return pkt; +} + +meshtastic_MeshPacket PhoneAPI::makeReplayTelemetryPacket(NodeNum num, const meshtastic_DeviceMetrics &metrics) +{ + meshtastic_MeshPacket pkt = meshtastic_MeshPacket_init_default; + pkt.from = num; + pkt.to = nodeDB->getNodeNum(); + pkt.id = generatePacketId(); + // No native timestamp on telemetry packets here; use last_heard. + const meshtastic_NodeInfoLite *header = nodeDB->getMeshNode(num); + pkt.rx_time = header ? header->last_heard : 0; + pkt.which_payload_variant = meshtastic_MeshPacket_decoded_tag; + pkt.decoded.portnum = meshtastic_PortNum_TELEMETRY_APP; + meshtastic_Telemetry fullTel = meshtastic_Telemetry_init_default; + fullTel.time = pkt.rx_time; + fullTel.which_variant = meshtastic_Telemetry_device_metrics_tag; + fullTel.variant.device_metrics = metrics; + size_t len = + pb_encode_to_bytes(pkt.decoded.payload.bytes, sizeof(pkt.decoded.payload.bytes), &meshtastic_Telemetry_msg, &fullTel); + pkt.decoded.payload.size = (pb_size_t)len; + return pkt; +} + +void PhoneAPI::beginReplayPositions() +{ +#if MESHTASTIC_EXCLUDE_POSITIONDB + // Build excluded entirely - leave the order list empty so the state arm + // immediately drains and advances. + replayPositionOrder.clear(); + replayPositionIndex = 0; +#else + if (!clientWantsGradientSync()) { + replayPositionOrder.clear(); + replayPositionIndex = 0; + return; + } + // Snapshot the keyset at phase start so concurrent inserts/erases on the + // map don't invalidate iteration. Skip our own node - the phone already + // got our position bundled in STATE_SEND_OWN_NODEINFO. + replayPositionOrder = nodeDB->snapshotPositionNodeNums(nodeDB->getNodeNum()); + replayPositionIndex = 0; + LOG_INFO("Begin position replay: %u entries millis=%u", (unsigned)replayPositionOrder.size(), millis()); +#endif +} + +void PhoneAPI::prefetchReplayPositions() +{ +#if MESHTASTIC_EXCLUDE_POSITIONDB + return; +#else + if (!clientWantsGradientSync()) + return; + bool added = false; + bool wasEmpty = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + wasEmpty = replayQueue.empty(); + while (replayQueue.size() < kReplayPrefetchDepth && replayPositionIndex < replayPositionOrder.size()) { + NodeNum num = replayPositionOrder[replayPositionIndex++]; + meshtastic_PositionLite pos; + if (!nodeDB->copyNodePosition(num, pos)) + continue; // entry was evicted between snapshot and now + replayQueue.push_back(makeReplayPositionPacket(num, pos)); + added = true; + } + } + if (added && wasEmpty) + onNowHasData(0); +#endif +} + +void PhoneAPI::beginReplayTelemetry() +{ +#if MESHTASTIC_EXCLUDE_TELEMETRYDB + replayTelemetryOrder.clear(); + replayTelemetryIndex = 0; +#else + if (!clientWantsGradientSync()) { + replayTelemetryOrder.clear(); + replayTelemetryIndex = 0; + return; + } + replayTelemetryOrder = nodeDB->snapshotTelemetryNodeNums(nodeDB->getNodeNum()); + replayTelemetryIndex = 0; + LOG_INFO("Begin telemetry replay: %u entries millis=%u", (unsigned)replayTelemetryOrder.size(), millis()); +#endif +} + +void PhoneAPI::prefetchReplayTelemetry() +{ +#if MESHTASTIC_EXCLUDE_TELEMETRYDB + return; +#else + if (!clientWantsGradientSync()) + return; + bool added = false; + bool wasEmpty = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + wasEmpty = replayQueue.empty(); + while (replayQueue.size() < kReplayPrefetchDepth && replayTelemetryIndex < replayTelemetryOrder.size()) { + NodeNum num = replayTelemetryOrder[replayTelemetryIndex++]; + meshtastic_DeviceMetrics dm; + if (!nodeDB->copyNodeTelemetry(num, dm)) + continue; + replayQueue.push_back(makeReplayTelemetryPacket(num, dm)); + added = true; + } + } + if (added && wasEmpty) + onNowHasData(0); +#endif +} + +meshtastic_MeshPacket PhoneAPI::makeReplayEnvironmentPacket(uint32_t num, const meshtastic_EnvironmentMetrics &env) +{ + meshtastic_MeshPacket pkt = meshtastic_MeshPacket_init_default; + pkt.from = num; + pkt.to = nodeDB->getNodeNum(); + pkt.id = generatePacketId(); + const meshtastic_NodeInfoLite *header = nodeDB->getMeshNode(num); + pkt.rx_time = header ? header->last_heard : 0; + pkt.which_payload_variant = meshtastic_MeshPacket_decoded_tag; + pkt.decoded.portnum = meshtastic_PortNum_TELEMETRY_APP; + meshtastic_Telemetry fullTel = meshtastic_Telemetry_init_default; + fullTel.time = pkt.rx_time; + fullTel.which_variant = meshtastic_Telemetry_environment_metrics_tag; + fullTel.variant.environment_metrics = env; + size_t len = + pb_encode_to_bytes(pkt.decoded.payload.bytes, sizeof(pkt.decoded.payload.bytes), &meshtastic_Telemetry_msg, &fullTel); + pkt.decoded.payload.size = (pb_size_t)len; + return pkt; +} + +void PhoneAPI::beginReplayEnvironment() +{ +#if MESHTASTIC_EXCLUDE_ENVIRONMENTDB + replayEnvironmentOrder.clear(); + replayEnvironmentIndex = 0; +#else + if (!clientWantsGradientSync()) { + replayEnvironmentOrder.clear(); + replayEnvironmentIndex = 0; + return; + } + replayEnvironmentOrder = nodeDB->snapshotEnvironmentNodeNums(nodeDB->getNodeNum()); + replayEnvironmentIndex = 0; + LOG_INFO("Begin environment replay: %u entries millis=%u", (unsigned)replayEnvironmentOrder.size(), millis()); +#endif +} + +void PhoneAPI::prefetchReplayEnvironment() +{ +#if MESHTASTIC_EXCLUDE_ENVIRONMENTDB + return; +#else + if (!clientWantsGradientSync()) + return; + bool added = false; + bool wasEmpty = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + wasEmpty = replayQueue.empty(); + while (replayQueue.size() < kReplayPrefetchDepth && replayEnvironmentIndex < replayEnvironmentOrder.size()) { + NodeNum num = replayEnvironmentOrder[replayEnvironmentIndex++]; + meshtastic_EnvironmentMetrics env; + if (!nodeDB->copyNodeEnvironment(num, env)) + continue; + replayQueue.push_back(makeReplayEnvironmentPacket(num, env)); + added = true; + } + } + if (added && wasEmpty) + onNowHasData(0); +#endif +} + +meshtastic_MeshPacket PhoneAPI::makeReplayStatusPacket(uint32_t num, const meshtastic_StatusMessage &status) +{ + meshtastic_MeshPacket pkt = meshtastic_MeshPacket_init_default; + pkt.from = num; + pkt.to = nodeDB->getNodeNum(); + pkt.id = generatePacketId(); + // StatusMessage has no native timestamp; use last_heard. + const meshtastic_NodeInfoLite *header = nodeDB->getMeshNode(num); + pkt.rx_time = header ? header->last_heard : 0; + pkt.which_payload_variant = meshtastic_MeshPacket_decoded_tag; + pkt.decoded.portnum = meshtastic_PortNum_NODE_STATUS_APP; + size_t len = + pb_encode_to_bytes(pkt.decoded.payload.bytes, sizeof(pkt.decoded.payload.bytes), &meshtastic_StatusMessage_msg, &status); + pkt.decoded.payload.size = (pb_size_t)len; + return pkt; +} + +void PhoneAPI::beginReplayStatus() +{ +#if MESHTASTIC_EXCLUDE_STATUSDB + replayStatusOrder.clear(); + replayStatusIndex = 0; +#else + if (!clientWantsGradientSync()) { + replayStatusOrder.clear(); + replayStatusIndex = 0; + return; + } + replayStatusOrder = nodeDB->snapshotStatusNodeNums(nodeDB->getNodeNum()); + replayStatusIndex = 0; + LOG_INFO("Begin status replay: %u entries millis=%u", (unsigned)replayStatusOrder.size(), millis()); +#endif +} + +void PhoneAPI::prefetchReplayStatus() +{ +#if MESHTASTIC_EXCLUDE_STATUSDB + return; +#else + if (!clientWantsGradientSync()) + return; + bool added = false; + bool wasEmpty = false; + { + concurrency::LockGuard guard(&nodeInfoMutex); + wasEmpty = replayQueue.empty(); + while (replayQueue.size() < kReplayPrefetchDepth && replayStatusIndex < replayStatusOrder.size()) { + NodeNum num = replayStatusOrder[replayStatusIndex++]; + meshtastic_StatusMessage status; + if (!nodeDB->copyNodeStatus(num, status) || status.status[0] == '\0') + continue; + replayQueue.push_back(makeReplayStatusPacket(num, status)); + added = true; + } + } + if (added && wasEmpty) + onNowHasData(0); +#endif +} + void PhoneAPI::releaseMqttClientProxyPhonePacket() { if (mqttClientProxyMessageForPhone) { @@ -728,6 +1115,31 @@ bool PhoneAPI::available() PREFETCH_NODEINFO: prefetchNodeInfos(); return true; + case STATE_REPLAY_POSITIONS: { + // Prime the iterator if we haven't yet, then top up the queue. + if (replayPositionOrder.empty() && replayPositionIndex == 0) + beginReplayPositions(); + prefetchReplayPositions(); + return true; // Always advance state machine; arm itself transitions when drained + } + case STATE_REPLAY_TELEMETRY: { + if (replayTelemetryOrder.empty() && replayTelemetryIndex == 0) + beginReplayTelemetry(); + prefetchReplayTelemetry(); + return true; + } + case STATE_REPLAY_ENVIRONMENT: { + if (replayEnvironmentOrder.empty() && replayEnvironmentIndex == 0) + beginReplayEnvironment(); + prefetchReplayEnvironment(); + return true; + } + case STATE_REPLAY_STATUS: { + if (replayStatusOrder.empty() && replayStatusIndex == 0) + beginReplayStatus(); + prefetchReplayStatus(); + return true; + } case STATE_SEND_PACKETS: { if (!queueStatusPacketForPhone) queueStatusPacketForPhone = service->getQueueStatusForPhone(); diff --git a/src/mesh/PhoneAPI.h b/src/mesh/PhoneAPI.h index 7f79b5792..642fdd7e0 100644 --- a/src/mesh/PhoneAPI.h +++ b/src/mesh/PhoneAPI.h @@ -4,12 +4,16 @@ #include "concurrency/Lock.h" #include "mesh-pb-constants.h" #include "meshtastic/portnums.pb.h" +#include #include #include #include #include #include +// NodeNum stored as raw uint32_t below; including MeshTypes.h here breaks the +// LOG_WARN include order via Arduino.h/MemoryPool.h. + // Make sure that we never let our packets grow too large for one BLE packet #define MAX_TO_FROM_RADIO_SIZE 512 @@ -22,6 +26,9 @@ #define SPECIAL_NONCE_ONLY_CONFIG 69420 #define SPECIAL_NONCE_ONLY_NODES 69421 // ( ͡° ͜ʖ ͡°) +// Gradient sync: phone sends one of these to opt into thin-header + replay. +#define SPECIAL_NONCE_GRADIENT_SYNC 69422 +#define SPECIAL_NONCE_GRADIENT_ONLY_NODES 69423 /** * Provides our protobuf based API which phone/PC clients can use to talk to our device @@ -45,7 +52,13 @@ class PhoneAPI STATE_SEND_CONFIG, // Replacement for the old Radioconfig STATE_SEND_MODULECONFIG, // Send Module specific config STATE_SEND_OTHER_NODEINFOS, // states progress in this order as the device sends to to the client - STATE_SEND_FILEMANIFEST, // Send file manifest + // Drain satellite DBs as synthetic POSITION_APP / TELEMETRY_APP / + // NODE_STATUS_APP packets when the phone opted into gradient sync. + STATE_REPLAY_POSITIONS, + STATE_REPLAY_TELEMETRY, + STATE_REPLAY_ENVIRONMENT, + STATE_REPLAY_STATUS, + STATE_SEND_FILEMANIFEST, // Send file manifest STATE_SEND_COMPLETE_ID, STATE_SEND_PACKETS // send packets or debug strings }; @@ -88,6 +101,20 @@ class PhoneAPI // Protect nodeInfoForPhone + nodeInfoQueue because NimBLE callbacks run in a separate FreeRTOS task. concurrency::Lock nodeInfoMutex; + // Synthetic-packet replay queue (paced via prefetch). + std::deque replayQueue; + static constexpr size_t kReplayPrefetchDepth = 4; + // NodeNum snapshots taken at each replay phase start so concurrent + // satellite-map mutations don't invalidate iteration. + std::vector replayPositionOrder; + std::vector replayTelemetryOrder; + std::vector replayEnvironmentOrder; + std::vector replayStatusOrder; + size_t replayPositionIndex = 0; + size_t replayTelemetryIndex = 0; + size_t replayEnvironmentIndex = 0; + size_t replayStatusIndex = 0; + meshtastic_ToRadio toRadioScratch = { 0}; // this is a static scratch object, any data must be copied elsewhere before returning @@ -137,6 +164,10 @@ class PhoneAPI bool isConnected() { return state != STATE_SEND_NOTHING; } bool isSendingPackets() { return state == STATE_SEND_PACKETS; } + bool clientWantsGradientSync() const + { + return config_nonce == SPECIAL_NONCE_GRADIENT_SYNC || config_nonce == SPECIAL_NONCE_GRADIENT_ONLY_NODES; + } protected: /// Our fromradio packet while it is being assembled @@ -185,6 +216,18 @@ class PhoneAPI void releaseQueueStatusPhonePacket(); void prefetchNodeInfos(); + void beginReplayPositions(); + void prefetchReplayPositions(); + void beginReplayTelemetry(); + void prefetchReplayTelemetry(); + void beginReplayEnvironment(); + void prefetchReplayEnvironment(); + void beginReplayStatus(); + void prefetchReplayStatus(); + meshtastic_MeshPacket makeReplayPositionPacket(uint32_t num, const meshtastic_PositionLite &pos); + meshtastic_MeshPacket makeReplayTelemetryPacket(uint32_t num, const meshtastic_DeviceMetrics &metrics); + meshtastic_MeshPacket makeReplayEnvironmentPacket(uint32_t num, const meshtastic_EnvironmentMetrics &env); + meshtastic_MeshPacket makeReplayStatusPacket(uint32_t num, const meshtastic_StatusMessage &status); void releaseMqttClientProxyPhonePacket(); diff --git a/src/mesh/ProtobufModule.h b/src/mesh/ProtobufModule.h index 27e653efe..fa2417c46 100644 --- a/src/mesh/ProtobufModule.h +++ b/src/mesh/ProtobufModule.h @@ -58,7 +58,7 @@ template class ProtobufModule : protected SinglePortModule const char *getSenderShortName(const meshtastic_MeshPacket &mp) { auto node = nodeDB->getMeshNode(getFrom(&mp)); - const char *sender = (node) ? node->user.short_name : "???"; + const char *sender = (node && nodeInfoLiteHasUser(node) && node->short_name[0]) ? node->short_name : "???"; return sender; } @@ -121,4 +121,4 @@ template class ProtobufModule : protected SinglePortModule return alterReceivedProtobuf(mp, decoded); } } -}; \ No newline at end of file +}; diff --git a/src/mesh/RF95Interface.cpp b/src/mesh/RF95Interface.cpp index 32c92de93..43149ef8b 100644 --- a/src/mesh/RF95Interface.cpp +++ b/src/mesh/RF95Interface.cpp @@ -252,7 +252,7 @@ bool RF95Interface::reconfigure() startReceive(); // restart receiving - return RADIOLIB_ERR_NONE; + return true; } /** diff --git a/src/mesh/RadioInterface.cpp b/src/mesh/RadioInterface.cpp index b7b672d8b..f4f25f80c 100644 --- a/src/mesh/RadioInterface.cpp +++ b/src/mesh/RadioInterface.cpp @@ -5,6 +5,7 @@ #include "LR1110Interface.h" #include "LR1120Interface.h" #include "LR1121Interface.h" +#include "LR2021Interface.h" #include "MeshRadio.h" #include "MeshService.h" #include "NodeDB.h" @@ -478,6 +479,20 @@ std::unique_ptr initLoRa() } #endif +#if defined(USE_LR2021) && RADIOLIB_EXCLUDE_LR2021 != 1 + if (!rIf) { + rIf = std::unique_ptr( + new LR2021Interface(loraHal, LR2021_SPI_NSS_PIN, LR2021_IRQ_PIN, LR2021_NRESET_PIN, LR2021_BUSY_PIN)); + if (!rIf->init()) { + LOG_WARN("No LR2021 radio"); + rIf = nullptr; + } else { + LOG_INFO("LR2021 init success"); + radioType = LR2021_RADIO; + } + } +#endif + #if defined(USE_SX1280) && RADIOLIB_EXCLUDE_SX128X != 1 if (!rIf) { rIf = std::unique_ptr(new SX1280Interface(loraHal, SX128X_CS, SX128X_DIO1, SX128X_RESET, SX128X_BUSY)); diff --git a/src/mesh/ReliableRouter.cpp b/src/mesh/ReliableRouter.cpp index 42c24c783..a1cfb1200 100644 --- a/src/mesh/ReliableRouter.cpp +++ b/src/mesh/ReliableRouter.cpp @@ -115,7 +115,7 @@ void ReliableRouter::sniffReceived(const meshtastic_MeshPacket *p, const meshtas sendAckNak(meshtastic_Routing_Error_NONE, getFrom(p), p->id, p->channel, 0); } } else if (p->which_payload_variant == meshtastic_MeshPacket_encrypted_tag && p->channel == 0 && - (nodeDB->getMeshNode(p->from) == nullptr || nodeDB->getMeshNode(p->from)->user.public_key.size == 0)) { + (nodeDB->getMeshNode(p->from) == nullptr || nodeDB->getMeshNode(p->from)->public_key.size == 0)) { LOG_INFO("PKI packet from unknown node, send PKI_UNKNOWN_PUBKEY"); sendAckNak(meshtastic_Routing_Error_PKI_UNKNOWN_PUBKEY, getFrom(p), p->id, channels.getPrimaryIndex(), routingModule->getHopLimitForResponse(*p)); diff --git a/src/mesh/Router.cpp b/src/mesh/Router.cpp index ffeb7c539..980ceb6df 100644 --- a/src/mesh/Router.cpp +++ b/src/mesh/Router.cpp @@ -120,17 +120,17 @@ bool Router::shouldDecrementHopLimit(const meshtastic_MeshPacket *p) if (!node) continue; - // Check 1: is_favorite (cheapest - single bool) - if (!node->is_favorite) + // Check 1: is_favorite (cheapest - single bit test) + if (!nodeInfoLiteIsFavorite(node)) continue; - // Check 2: has_user (cheap - single bool) - if (!node->has_user) + // Check 2: has_user (cheap - single bit test) + if (!nodeInfoLiteHasUser(node)) continue; // Check 3: role check (moderate cost - multiple comparisons) - if (!IS_ONE_OF(node->user.role, meshtastic_Config_DeviceConfig_Role_ROUTER, - meshtastic_Config_DeviceConfig_Role_ROUTER_LATE, meshtastic_Config_DeviceConfig_Role_CLIENT_BASE)) { + if (!IS_ONE_OF(node->role, meshtastic_Config_DeviceConfig_Role_ROUTER, meshtastic_Config_DeviceConfig_Role_ROUTER_LATE, + meshtastic_Config_DeviceConfig_Role_CLIENT_BASE)) { continue; } @@ -329,7 +329,7 @@ ErrorCode Router::send(meshtastic_MeshPacket *p) cn->reply_id = p->id; cn->level = meshtastic_LogRecord_Level_WARNING; cn->time = getValidTime(RTCQualityFromNet); - sprintf(cn->message, "Duty cycle limit exceeded. You can send again in %d mins", silentMinutes); + snprintf(cn->message, sizeof(cn->message), "Duty cycle limit exceeded. You can send again in %d mins", silentMinutes); service->sendClientNotification(cn); meshtastic_Routing_Error err = meshtastic_Routing_Error_DUTY_CYCLE_LIMIT; @@ -433,7 +433,7 @@ DecodeState perhapsDecode(meshtastic_MeshPacket *p) concurrency::LockGuard g(cryptLock); if (config.device.rebroadcast_mode == meshtastic_Config_DeviceConfig_RebroadcastMode_KNOWN_ONLY && - (nodeDB->getMeshNode(p->from) == NULL || !nodeDB->getMeshNode(p->from)->has_user)) { + !nodeInfoLiteHasUser(nodeDB->getMeshNode(p->from))) { LOG_DEBUG("Node 0x%x not in nodeDB-> Rebroadcast mode KNOWN_ONLY will ignore packet", p->from); return DecodeState::DECODE_FAILURE; } @@ -451,11 +451,11 @@ DecodeState perhapsDecode(meshtastic_MeshPacket *p) #if !(MESHTASTIC_EXCLUDE_PKI) // Attempt PKI decryption first if (p->channel == 0 && isToUs(p) && p->to > 0 && !isBroadcast(p->to) && nodeDB->getMeshNode(p->from) != nullptr && - nodeDB->getMeshNode(p->from)->user.public_key.size > 0 && nodeDB->getMeshNode(p->to)->user.public_key.size > 0 && - rawSize > MESHTASTIC_PKC_OVERHEAD) { + nodeDB->getMeshNode(p->from)->public_key.size > 0 && nodeDB->getMeshNode(p->to) != nullptr && + nodeDB->getMeshNode(p->to)->public_key.size > 0 && rawSize > MESHTASTIC_PKC_OVERHEAD) { LOG_DEBUG("Attempt PKI decryption"); - if (crypto->decryptCurve25519(p->from, nodeDB->getMeshNode(p->from)->user.public_key, p->id, rawSize, p->encrypted.bytes, + if (crypto->decryptCurve25519(p->from, nodeDB->getMeshNode(p->from)->public_key, p->id, rawSize, p->encrypted.bytes, bytes)) { LOG_INFO("PKI Decryption worked!"); @@ -467,7 +467,7 @@ DecodeState perhapsDecode(meshtastic_MeshPacket *p) decrypted = true; LOG_INFO("Packet decrypted using PKI!"); p->pki_encrypted = true; - memcpy(&p->public_key.bytes, nodeDB->getMeshNode(p->from)->user.public_key.bytes, 32); + memcpy(&p->public_key.bytes, nodeDB->getMeshNode(p->from)->public_key.bytes, 32); p->public_key.size = 32; p->decoded = decodedtmp; p->which_payload_variant = meshtastic_MeshPacket_decoded_tag; // change type to decoded @@ -665,18 +665,18 @@ meshtastic_Routing_Error perhapsEncode(meshtastic_MeshPacket *p) if (numbytes + MESHTASTIC_HEADER_LENGTH + MESHTASTIC_PKC_OVERHEAD > MAX_LORA_PAYLOAD_LEN) return meshtastic_Routing_Error_TOO_LARGE; // Check for a known public key for the destination - if (node == nullptr || node->user.public_key.size != 32) { + if (node == nullptr || node->public_key.size != 32) { LOG_WARN("Unknown public key for destination node 0x%08x (portnum %d), refusing to send legacy DM", p->to, p->decoded.portnum); return meshtastic_Routing_Error_PKI_SEND_FAIL_PUBLIC_KEY; } if (p->pki_encrypted && !memfll(p->public_key.bytes, 0, 32) && - memcmp(p->public_key.bytes, node->user.public_key.bytes, 32) != 0) { + memcmp(p->public_key.bytes, node->public_key.bytes, 32) != 0) { LOG_WARN("Client public key differs from requested: 0x%02x, stored key begins 0x%02x", *p->public_key.bytes, - *node->user.public_key.bytes); + *node->public_key.bytes); return meshtastic_Routing_Error_PKI_FAILED; } - crypto->encryptCurve25519(p->to, getFrom(p), node->user.public_key, p->id, numbytes, bytes, p->encrypted.bytes); + crypto->encryptCurve25519(p->to, getFrom(p), node->public_key, p->id, numbytes, bytes, p->encrypted.bytes); numbytes += MESHTASTIC_PKC_OVERHEAD; p->channel = 0; p->pki_encrypted = true; @@ -860,7 +860,7 @@ void Router::perhapsHandleReceived(meshtastic_MeshPacket *p) } meshtastic_NodeInfoLite const *node = nodeDB->getMeshNode(p->from); - if (node != NULL && node->is_ignored) { + if (nodeInfoLiteIsIgnored(node)) { LOG_DEBUG("Ignore msg, 0x%x is ignored", p->from); packetPool.release(p); return; diff --git a/src/mesh/SX126xInterface.cpp b/src/mesh/SX126xInterface.cpp index 44c4a805a..3513bbba3 100644 --- a/src/mesh/SX126xInterface.cpp +++ b/src/mesh/SX126xInterface.cpp @@ -262,7 +262,7 @@ template bool SX126xInterface::reconfigure() startReceive(); // restart receiving - return RADIOLIB_ERR_NONE; + return true; } template void SX126xInterface::disableInterrupt() diff --git a/src/mesh/SX128xInterface.cpp b/src/mesh/SX128xInterface.cpp index cb21c0770..64d71921a 100644 --- a/src/mesh/SX128xInterface.cpp +++ b/src/mesh/SX128xInterface.cpp @@ -153,7 +153,7 @@ template bool SX128xInterface::reconfigure() startReceive(); // restart receiving - return RADIOLIB_ERR_NONE; + return true; } template void SX128xInterface::disableInterrupt() diff --git a/src/mesh/TypeConversions.cpp b/src/mesh/TypeConversions.cpp index 3798daf28..254af6132 100644 --- a/src/mesh/TypeConversions.cpp +++ b/src/mesh/TypeConversions.cpp @@ -3,7 +3,9 @@ #include "mesh/generated/meshtastic/mesh.pb.h" #include "meshUtils.h" -meshtastic_NodeInfo TypeConversions::ConvertToNodeInfo(const meshtastic_NodeInfoLite *lite) +meshtastic_NodeInfo TypeConversions::ConvertToNodeInfo(const meshtastic_NodeInfoLite *lite, + const meshtastic_PositionLite *position, + const meshtastic_DeviceMetrics *deviceMetrics) { meshtastic_NodeInfo info = meshtastic_NodeInfo_init_default; @@ -11,42 +13,56 @@ meshtastic_NodeInfo TypeConversions::ConvertToNodeInfo(const meshtastic_NodeInfo info.snr = lite->snr; info.last_heard = lite->last_heard; info.channel = lite->channel; - info.via_mqtt = lite->via_mqtt; - info.is_favorite = lite->is_favorite; - info.is_ignored = lite->is_ignored; - info.is_key_manually_verified = lite->bitfield & NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; - info.is_muted = lite->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK; + info.via_mqtt = nodeInfoLiteViaMqtt(lite); + info.is_favorite = nodeInfoLiteIsFavorite(lite); + info.is_ignored = nodeInfoLiteIsIgnored(lite); + info.is_key_manually_verified = nodeInfoLiteIsKeyManuallyVerified(lite); + info.is_muted = nodeInfoLiteIsMuted(lite); if (lite->has_hops_away) { info.has_hops_away = true; info.hops_away = lite->hops_away; } - if (lite->has_position) { + if (position) { info.has_position = true; - if (lite->position.latitude_i != 0) + if (position->latitude_i != 0) info.position.has_latitude_i = true; - info.position.latitude_i = lite->position.latitude_i; - if (lite->position.longitude_i != 0) + info.position.latitude_i = position->latitude_i; + if (position->longitude_i != 0) info.position.has_longitude_i = true; - info.position.longitude_i = lite->position.longitude_i; - if (lite->position.altitude != 0) + info.position.longitude_i = position->longitude_i; + if (position->altitude != 0) info.position.has_altitude = true; - info.position.altitude = lite->position.altitude; - info.position.location_source = lite->position.location_source; - info.position.time = lite->position.time; + info.position.altitude = position->altitude; + info.position.location_source = position->location_source; + info.position.time = position->time; } - if (lite->has_user) { + if (nodeInfoLiteHasUser(lite)) { info.has_user = true; - info.user = ConvertToUser(lite->num, lite->user); + info.user = ConvertToUser(lite); } - if (lite->has_device_metrics) { + if (deviceMetrics) { info.has_device_metrics = true; - info.device_metrics = lite->device_metrics; + info.device_metrics = *deviceMetrics; } return info; } +meshtastic_NodeInfo TypeConversions::ConvertToNodeInfo(const meshtastic_NodeInfoLite *lite) +{ + meshtastic_PositionLite posScratch; + meshtastic_DeviceMetrics dmScratch; + const meshtastic_PositionLite *pos = (nodeDB && nodeDB->copyNodePosition(lite->num, posScratch)) ? &posScratch : nullptr; + const meshtastic_DeviceMetrics *dm = (nodeDB && nodeDB->copyNodeTelemetry(lite->num, dmScratch)) ? &dmScratch : nullptr; + return ConvertToNodeInfo(lite, pos, dm); +} + +meshtastic_NodeInfo TypeConversions::ConvertToNodeInfoThin(const meshtastic_NodeInfoLite *lite) +{ + return ConvertToNodeInfo(lite, nullptr, nullptr); +} + meshtastic_PositionLite TypeConversions::ConvertToPositionLite(meshtastic_Position position) { meshtastic_PositionLite lite = meshtastic_PositionLite_init_default; @@ -77,46 +93,50 @@ meshtastic_Position TypeConversions::ConvertToPosition(meshtastic_PositionLite l return position; } -meshtastic_UserLite TypeConversions::ConvertToUserLite(meshtastic_User user) +void TypeConversions::CopyUserToNodeInfoLite(meshtastic_NodeInfoLite *lite, const meshtastic_User &user) { - meshtastic_UserLite lite = meshtastic_UserLite_init_default; + if (!lite) + return; - strncpy(lite.long_name, user.long_name, sizeof(lite.long_name)); - lite.long_name[sizeof(lite.long_name) - 1] = '\0'; - sanitizeUtf8(lite.long_name, sizeof(lite.long_name)); - strncpy(lite.short_name, user.short_name, sizeof(lite.short_name)); - lite.short_name[sizeof(lite.short_name) - 1] = '\0'; - sanitizeUtf8(lite.short_name, sizeof(lite.short_name)); - lite.hw_model = user.hw_model; - lite.role = user.role; - lite.is_licensed = user.is_licensed; - memcpy(lite.macaddr, user.macaddr, sizeof(lite.macaddr)); - memcpy(lite.public_key.bytes, user.public_key.bytes, sizeof(lite.public_key.bytes)); - lite.public_key.size = user.public_key.size; - lite.has_is_unmessagable = user.has_is_unmessagable; - lite.is_unmessagable = user.is_unmessagable; - return lite; + strncpy(lite->long_name, user.long_name, sizeof(lite->long_name)); + lite->long_name[sizeof(lite->long_name) - 1] = '\0'; + sanitizeUtf8(lite->long_name, sizeof(lite->long_name)); + strncpy(lite->short_name, user.short_name, sizeof(lite->short_name)); + lite->short_name[sizeof(lite->short_name) - 1] = '\0'; + sanitizeUtf8(lite->short_name, sizeof(lite->short_name)); + lite->hw_model = user.hw_model; + lite->role = user.role; + memcpy(lite->public_key.bytes, user.public_key.bytes, sizeof(lite->public_key.bytes)); + lite->public_key.size = user.public_key.size; + + nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_IS_LICENSED_MASK, user.is_licensed); + nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK, user.has_is_unmessagable); + nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK, user.has_is_unmessagable && user.is_unmessagable); + nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_HAS_USER_MASK, true); } -meshtastic_User TypeConversions::ConvertToUser(uint32_t nodeNum, meshtastic_UserLite lite) +meshtastic_User TypeConversions::ConvertToUser(const meshtastic_NodeInfoLite *lite) { meshtastic_User user = meshtastic_User_init_default; + if (!lite) + return user; - snprintf(user.id, sizeof(user.id), "!%08x", nodeNum); - strncpy(user.long_name, lite.long_name, sizeof(user.long_name)); + snprintf(user.id, sizeof(user.id), "!%08x", lite->num); + strncpy(user.long_name, lite->long_name, sizeof(user.long_name)); user.long_name[sizeof(user.long_name) - 1] = '\0'; sanitizeUtf8(user.long_name, sizeof(user.long_name)); - strncpy(user.short_name, lite.short_name, sizeof(user.short_name)); + strncpy(user.short_name, lite->short_name, sizeof(user.short_name)); user.short_name[sizeof(user.short_name) - 1] = '\0'; sanitizeUtf8(user.short_name, sizeof(user.short_name)); - user.hw_model = lite.hw_model; - user.role = lite.role; - user.is_licensed = lite.is_licensed; - memcpy(user.macaddr, lite.macaddr, sizeof(user.macaddr)); - memcpy(user.public_key.bytes, lite.public_key.bytes, sizeof(user.public_key.bytes)); - user.public_key.size = lite.public_key.size; - user.has_is_unmessagable = lite.has_is_unmessagable; - user.is_unmessagable = lite.is_unmessagable; + user.hw_model = lite->hw_model; + user.role = lite->role; + user.is_licensed = nodeInfoLiteIsLicensed(lite); + memcpy(user.public_key.bytes, lite->public_key.bytes, sizeof(user.public_key.bytes)); + user.public_key.size = lite->public_key.size; + user.has_is_unmessagable = nodeInfoLiteHasIsUnmessagable(lite); + user.is_unmessagable = nodeInfoLiteIsUnmessagable(lite); + // macaddr is gone from the slim header; zero-fill for old clients that read it. + memset(user.macaddr, 0, sizeof(user.macaddr)); return user; -} \ No newline at end of file +} diff --git a/src/mesh/TypeConversions.h b/src/mesh/TypeConversions.h index 19e471f98..f9792b7c3 100644 --- a/src/mesh/TypeConversions.h +++ b/src/mesh/TypeConversions.h @@ -7,9 +7,16 @@ class TypeConversions { public: + // Either pointer may be null; the corresponding has_* fields stay unset. + static meshtastic_NodeInfo ConvertToNodeInfo(const meshtastic_NodeInfoLite *lite, const meshtastic_PositionLite *position, + const meshtastic_DeviceMetrics *deviceMetrics); static meshtastic_NodeInfo ConvertToNodeInfo(const meshtastic_NodeInfoLite *lite); + // Identity + link-state only; satellite payloads are replayed afterward. + static meshtastic_NodeInfo ConvertToNodeInfoThin(const meshtastic_NodeInfoLite *lite); + static meshtastic_PositionLite ConvertToPositionLite(meshtastic_Position position); static meshtastic_Position ConvertToPosition(meshtastic_PositionLite lite); - static meshtastic_UserLite ConvertToUserLite(meshtastic_User user); - static meshtastic_User ConvertToUser(uint32_t nodeNum, meshtastic_UserLite lite); + + static void CopyUserToNodeInfoLite(meshtastic_NodeInfoLite *lite, const meshtastic_User &user); + static meshtastic_User ConvertToUser(const meshtastic_NodeInfoLite *lite); }; diff --git a/src/mesh/generated/meshtastic/deviceonly.pb.cpp b/src/mesh/generated/meshtastic/deviceonly.pb.cpp index 5a9695702..558086637 100644 --- a/src/mesh/generated/meshtastic/deviceonly.pb.cpp +++ b/src/mesh/generated/meshtastic/deviceonly.pb.cpp @@ -18,6 +18,18 @@ PB_BIND(meshtastic_NodeInfoLite, meshtastic_NodeInfoLite, AUTO) PB_BIND(meshtastic_DeviceState, meshtastic_DeviceState, 2) +PB_BIND(meshtastic_NodePositionEntry, meshtastic_NodePositionEntry, AUTO) + + +PB_BIND(meshtastic_NodeTelemetryEntry, meshtastic_NodeTelemetryEntry, AUTO) + + +PB_BIND(meshtastic_NodeEnvironmentEntry, meshtastic_NodeEnvironmentEntry, AUTO) + + +PB_BIND(meshtastic_NodeStatusEntry, meshtastic_NodeStatusEntry, AUTO) + + PB_BIND(meshtastic_NodeDatabase, meshtastic_NodeDatabase, AUTO) diff --git a/src/mesh/generated/meshtastic/deviceonly.pb.h b/src/mesh/generated/meshtastic/deviceonly.pb.h index 6d03dc643..5e0b844f1 100644 --- a/src/mesh/generated/meshtastic/deviceonly.pb.h +++ b/src/mesh/generated/meshtastic/deviceonly.pb.h @@ -63,43 +63,35 @@ typedef struct _meshtastic_UserLite { bool is_unmessagable; } meshtastic_UserLite; +typedef PB_BYTES_ARRAY_T(32) meshtastic_NodeInfoLite_public_key_t; typedef struct _meshtastic_NodeInfoLite { /* The node number */ uint32_t num; - /* The user info for this node */ - bool has_user; - meshtastic_UserLite user; - /* This position data. Note: before 1.2.14 we would also store the last time we've heard from this node in position.time, that is no longer true. - Position.time now indicates the last time we received a POSITION from that node. */ - bool has_position; - meshtastic_PositionLite position; /* Returns the Signal-to-noise ratio (SNR) of the last received message, as measured by the receiver. Return SNR of the last received message in dB */ float snr; /* Set to indicate the last time we received a packet from this node */ uint32_t last_heard; - /* The latest device metrics for the node. */ - bool has_device_metrics; - meshtastic_DeviceMetrics device_metrics; /* local channel index we heard that node on. Only populated if its not the default channel. */ uint8_t channel; - /* True if we witnessed the node over MQTT instead of LoRA transport */ - bool via_mqtt; /* Number of hops away from us this node is (0 if direct neighbor) */ bool has_hops_away; uint8_t hops_away; - /* True if node is in our favorites list - Persists between NodeDB internal clean ups */ - bool is_favorite; - /* True if node is in our ignored list - Persists between NodeDB internal clean ups */ - bool is_ignored; /* Last byte of the node number of the node that should be used as the next hop to reach this node. */ uint8_t next_hop; - /* Bitfield for storing booleans. - LSB 0 is_key_manually_verified - LSB 1 is_muted */ + /* Bitfield for storing booleans. See NODEINFO_BITFIELD_* in src/mesh/NodeDB.h. */ uint32_t bitfield; + /* A full name for this user, i.e. "Kevin Hester". */ + char long_name[25]; + /* A VERY short name, ideally two characters or an emoji. + Suitable for a tiny OLED screen. */ + char short_name[5]; + /* Hardware model the user's device is running. */ + meshtastic_HardwareModel hw_model; + /* The user's role in the mesh. */ + meshtastic_Config_DeviceConfig_Role role; + /* The public key of the user's device, for PKI-based encrypted DMs. */ + meshtastic_NodeInfoLite_public_key_t public_key; } meshtastic_NodeInfoLite; /* This message is never sent over the wire, but it is used for serializing DB @@ -143,6 +135,30 @@ typedef struct _meshtastic_DeviceState { meshtastic_NodeRemoteHardwarePin node_remote_hardware_pins[12]; } meshtastic_DeviceState; +typedef struct _meshtastic_NodePositionEntry { + uint32_t num; + bool has_position; + meshtastic_PositionLite position; +} meshtastic_NodePositionEntry; + +typedef struct _meshtastic_NodeTelemetryEntry { + uint32_t num; + bool has_device_metrics; + meshtastic_DeviceMetrics device_metrics; +} meshtastic_NodeTelemetryEntry; + +typedef struct _meshtastic_NodeEnvironmentEntry { + uint32_t num; + bool has_environment_metrics; + meshtastic_EnvironmentMetrics environment_metrics; +} meshtastic_NodeEnvironmentEntry; + +typedef struct _meshtastic_NodeStatusEntry { + uint32_t num; + bool has_status; + meshtastic_StatusMessage status; +} meshtastic_NodeStatusEntry; + typedef struct _meshtastic_NodeDatabase { /* A version integer used to invalidate old save files when we make incompatible changes This integer is set at build time and is private to @@ -150,6 +166,12 @@ typedef struct _meshtastic_NodeDatabase { uint32_t version; /* New lite version of NodeDB to decrease memory footprint */ std::vector nodes; + /* Per-NodeNum satellite arrays. Constrained platforms (e.g. STM32WL) omit + these via MESHTASTIC_EXCLUDE_*DB build flags. */ + std::vector positions; + std::vector telemetry; + std::vector status; + std::vector environment; } meshtastic_NodeDatabase; /* The on-disk saved channels */ @@ -191,16 +213,24 @@ extern "C" { /* Initializer values for message structs */ #define meshtastic_PositionLite_init_default {0, 0, 0, 0, _meshtastic_Position_LocSource_MIN} #define meshtastic_UserLite_init_default {{0}, "", "", _meshtastic_HardwareModel_MIN, 0, _meshtastic_Config_DeviceConfig_Role_MIN, {0, {0}}, false, 0} -#define meshtastic_NodeInfoLite_init_default {0, false, meshtastic_UserLite_init_default, false, meshtastic_PositionLite_init_default, 0, 0, false, meshtastic_DeviceMetrics_init_default, 0, 0, false, 0, 0, 0, 0, 0} +#define meshtastic_NodeInfoLite_init_default {0, 0, 0, 0, false, 0, 0, 0, "", "", _meshtastic_HardwareModel_MIN, _meshtastic_Config_DeviceConfig_Role_MIN, {0, {0}}} #define meshtastic_DeviceState_init_default {false, meshtastic_MyNodeInfo_init_default, false, meshtastic_User_init_default, 0, {meshtastic_MeshPacket_init_default}, false, meshtastic_MeshPacket_init_default, 0, 0, 0, false, meshtastic_MeshPacket_init_default, 0, {meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default, meshtastic_NodeRemoteHardwarePin_init_default}} -#define meshtastic_NodeDatabase_init_default {0, {0}} +#define meshtastic_NodePositionEntry_init_default {0, false, meshtastic_PositionLite_init_default} +#define meshtastic_NodeTelemetryEntry_init_default {0, false, meshtastic_DeviceMetrics_init_default} +#define meshtastic_NodeEnvironmentEntry_init_default {0, false, meshtastic_EnvironmentMetrics_init_default} +#define meshtastic_NodeStatusEntry_init_default {0, false, meshtastic_StatusMessage_init_default} +#define meshtastic_NodeDatabase_init_default {0, {0}, {0}, {0}, {0}, {0}} #define meshtastic_ChannelFile_init_default {0, {meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default, meshtastic_Channel_init_default}, 0} #define meshtastic_BackupPreferences_init_default {0, 0, false, meshtastic_LocalConfig_init_default, false, meshtastic_LocalModuleConfig_init_default, false, meshtastic_ChannelFile_init_default, false, meshtastic_User_init_default} #define meshtastic_PositionLite_init_zero {0, 0, 0, 0, _meshtastic_Position_LocSource_MIN} #define meshtastic_UserLite_init_zero {{0}, "", "", _meshtastic_HardwareModel_MIN, 0, _meshtastic_Config_DeviceConfig_Role_MIN, {0, {0}}, false, 0} -#define meshtastic_NodeInfoLite_init_zero {0, false, meshtastic_UserLite_init_zero, false, meshtastic_PositionLite_init_zero, 0, 0, false, meshtastic_DeviceMetrics_init_zero, 0, 0, false, 0, 0, 0, 0, 0} +#define meshtastic_NodeInfoLite_init_zero {0, 0, 0, 0, false, 0, 0, 0, "", "", _meshtastic_HardwareModel_MIN, _meshtastic_Config_DeviceConfig_Role_MIN, {0, {0}}} #define meshtastic_DeviceState_init_zero {false, meshtastic_MyNodeInfo_init_zero, false, meshtastic_User_init_zero, 0, {meshtastic_MeshPacket_init_zero}, false, meshtastic_MeshPacket_init_zero, 0, 0, 0, false, meshtastic_MeshPacket_init_zero, 0, {meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero, meshtastic_NodeRemoteHardwarePin_init_zero}} -#define meshtastic_NodeDatabase_init_zero {0, {0}} +#define meshtastic_NodePositionEntry_init_zero {0, false, meshtastic_PositionLite_init_zero} +#define meshtastic_NodeTelemetryEntry_init_zero {0, false, meshtastic_DeviceMetrics_init_zero} +#define meshtastic_NodeEnvironmentEntry_init_zero {0, false, meshtastic_EnvironmentMetrics_init_zero} +#define meshtastic_NodeStatusEntry_init_zero {0, false, meshtastic_StatusMessage_init_zero} +#define meshtastic_NodeDatabase_init_zero {0, {0}, {0}, {0}, {0}, {0}} #define meshtastic_ChannelFile_init_zero {0, {meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero, meshtastic_Channel_init_zero}, 0} #define meshtastic_BackupPreferences_init_zero {0, 0, false, meshtastic_LocalConfig_init_zero, false, meshtastic_LocalModuleConfig_init_zero, false, meshtastic_ChannelFile_init_zero, false, meshtastic_User_init_zero} @@ -219,18 +249,17 @@ extern "C" { #define meshtastic_UserLite_public_key_tag 7 #define meshtastic_UserLite_is_unmessagable_tag 9 #define meshtastic_NodeInfoLite_num_tag 1 -#define meshtastic_NodeInfoLite_user_tag 2 -#define meshtastic_NodeInfoLite_position_tag 3 #define meshtastic_NodeInfoLite_snr_tag 4 #define meshtastic_NodeInfoLite_last_heard_tag 5 -#define meshtastic_NodeInfoLite_device_metrics_tag 6 #define meshtastic_NodeInfoLite_channel_tag 7 -#define meshtastic_NodeInfoLite_via_mqtt_tag 8 #define meshtastic_NodeInfoLite_hops_away_tag 9 -#define meshtastic_NodeInfoLite_is_favorite_tag 10 -#define meshtastic_NodeInfoLite_is_ignored_tag 11 #define meshtastic_NodeInfoLite_next_hop_tag 12 #define meshtastic_NodeInfoLite_bitfield_tag 13 +#define meshtastic_NodeInfoLite_long_name_tag 14 +#define meshtastic_NodeInfoLite_short_name_tag 15 +#define meshtastic_NodeInfoLite_hw_model_tag 16 +#define meshtastic_NodeInfoLite_role_tag 17 +#define meshtastic_NodeInfoLite_public_key_tag 18 #define meshtastic_DeviceState_my_node_tag 2 #define meshtastic_DeviceState_owner_tag 3 #define meshtastic_DeviceState_receive_queue_tag 5 @@ -240,8 +269,20 @@ extern "C" { #define meshtastic_DeviceState_did_gps_reset_tag 11 #define meshtastic_DeviceState_rx_waypoint_tag 12 #define meshtastic_DeviceState_node_remote_hardware_pins_tag 13 +#define meshtastic_NodePositionEntry_num_tag 1 +#define meshtastic_NodePositionEntry_position_tag 2 +#define meshtastic_NodeTelemetryEntry_num_tag 1 +#define meshtastic_NodeTelemetryEntry_device_metrics_tag 2 +#define meshtastic_NodeEnvironmentEntry_num_tag 1 +#define meshtastic_NodeEnvironmentEntry_environment_metrics_tag 2 +#define meshtastic_NodeStatusEntry_num_tag 1 +#define meshtastic_NodeStatusEntry_status_tag 2 #define meshtastic_NodeDatabase_version_tag 1 #define meshtastic_NodeDatabase_nodes_tag 2 +#define meshtastic_NodeDatabase_positions_tag 3 +#define meshtastic_NodeDatabase_telemetry_tag 4 +#define meshtastic_NodeDatabase_status_tag 5 +#define meshtastic_NodeDatabase_environment_tag 6 #define meshtastic_ChannelFile_channels_tag 1 #define meshtastic_ChannelFile_version_tag 2 #define meshtastic_BackupPreferences_version_tag 1 @@ -275,23 +316,19 @@ X(a, STATIC, OPTIONAL, BOOL, is_unmessagable, 9) #define meshtastic_NodeInfoLite_FIELDLIST(X, a) \ X(a, STATIC, SINGULAR, UINT32, num, 1) \ -X(a, STATIC, OPTIONAL, MESSAGE, user, 2) \ -X(a, STATIC, OPTIONAL, MESSAGE, position, 3) \ X(a, STATIC, SINGULAR, FLOAT, snr, 4) \ X(a, STATIC, SINGULAR, FIXED32, last_heard, 5) \ -X(a, STATIC, OPTIONAL, MESSAGE, device_metrics, 6) \ X(a, STATIC, SINGULAR, UINT32, channel, 7) \ -X(a, STATIC, SINGULAR, BOOL, via_mqtt, 8) \ X(a, STATIC, OPTIONAL, UINT32, hops_away, 9) \ -X(a, STATIC, SINGULAR, BOOL, is_favorite, 10) \ -X(a, STATIC, SINGULAR, BOOL, is_ignored, 11) \ X(a, STATIC, SINGULAR, UINT32, next_hop, 12) \ -X(a, STATIC, SINGULAR, UINT32, bitfield, 13) +X(a, STATIC, SINGULAR, UINT32, bitfield, 13) \ +X(a, STATIC, SINGULAR, STRING, long_name, 14) \ +X(a, STATIC, SINGULAR, STRING, short_name, 15) \ +X(a, STATIC, SINGULAR, UENUM, hw_model, 16) \ +X(a, STATIC, SINGULAR, UENUM, role, 17) \ +X(a, STATIC, SINGULAR, BYTES, public_key, 18) #define meshtastic_NodeInfoLite_CALLBACK NULL #define meshtastic_NodeInfoLite_DEFAULT NULL -#define meshtastic_NodeInfoLite_user_MSGTYPE meshtastic_UserLite -#define meshtastic_NodeInfoLite_position_MSGTYPE meshtastic_PositionLite -#define meshtastic_NodeInfoLite_device_metrics_MSGTYPE meshtastic_DeviceMetrics #define meshtastic_DeviceState_FIELDLIST(X, a) \ X(a, STATIC, OPTIONAL, MESSAGE, my_node, 2) \ @@ -312,13 +349,49 @@ X(a, STATIC, REPEATED, MESSAGE, node_remote_hardware_pins, 13) #define meshtastic_DeviceState_rx_waypoint_MSGTYPE meshtastic_MeshPacket #define meshtastic_DeviceState_node_remote_hardware_pins_MSGTYPE meshtastic_NodeRemoteHardwarePin +#define meshtastic_NodePositionEntry_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, num, 1) \ +X(a, STATIC, OPTIONAL, MESSAGE, position, 2) +#define meshtastic_NodePositionEntry_CALLBACK NULL +#define meshtastic_NodePositionEntry_DEFAULT NULL +#define meshtastic_NodePositionEntry_position_MSGTYPE meshtastic_PositionLite + +#define meshtastic_NodeTelemetryEntry_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, num, 1) \ +X(a, STATIC, OPTIONAL, MESSAGE, device_metrics, 2) +#define meshtastic_NodeTelemetryEntry_CALLBACK NULL +#define meshtastic_NodeTelemetryEntry_DEFAULT NULL +#define meshtastic_NodeTelemetryEntry_device_metrics_MSGTYPE meshtastic_DeviceMetrics + +#define meshtastic_NodeEnvironmentEntry_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, num, 1) \ +X(a, STATIC, OPTIONAL, MESSAGE, environment_metrics, 2) +#define meshtastic_NodeEnvironmentEntry_CALLBACK NULL +#define meshtastic_NodeEnvironmentEntry_DEFAULT NULL +#define meshtastic_NodeEnvironmentEntry_environment_metrics_MSGTYPE meshtastic_EnvironmentMetrics + +#define meshtastic_NodeStatusEntry_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, num, 1) \ +X(a, STATIC, OPTIONAL, MESSAGE, status, 2) +#define meshtastic_NodeStatusEntry_CALLBACK NULL +#define meshtastic_NodeStatusEntry_DEFAULT NULL +#define meshtastic_NodeStatusEntry_status_MSGTYPE meshtastic_StatusMessage + #define meshtastic_NodeDatabase_FIELDLIST(X, a) \ X(a, STATIC, SINGULAR, UINT32, version, 1) \ -X(a, CALLBACK, REPEATED, MESSAGE, nodes, 2) +X(a, CALLBACK, REPEATED, MESSAGE, nodes, 2) \ +X(a, CALLBACK, REPEATED, MESSAGE, positions, 3) \ +X(a, CALLBACK, REPEATED, MESSAGE, telemetry, 4) \ +X(a, CALLBACK, REPEATED, MESSAGE, status, 5) \ +X(a, CALLBACK, REPEATED, MESSAGE, environment, 6) extern bool meshtastic_NodeDatabase_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_t *field); #define meshtastic_NodeDatabase_CALLBACK meshtastic_NodeDatabase_callback #define meshtastic_NodeDatabase_DEFAULT NULL #define meshtastic_NodeDatabase_nodes_MSGTYPE meshtastic_NodeInfoLite +#define meshtastic_NodeDatabase_positions_MSGTYPE meshtastic_NodePositionEntry +#define meshtastic_NodeDatabase_telemetry_MSGTYPE meshtastic_NodeTelemetryEntry +#define meshtastic_NodeDatabase_status_MSGTYPE meshtastic_NodeStatusEntry +#define meshtastic_NodeDatabase_environment_MSGTYPE meshtastic_NodeEnvironmentEntry #define meshtastic_ChannelFile_FIELDLIST(X, a) \ X(a, STATIC, REPEATED, MESSAGE, channels, 1) \ @@ -345,6 +418,10 @@ extern const pb_msgdesc_t meshtastic_PositionLite_msg; extern const pb_msgdesc_t meshtastic_UserLite_msg; extern const pb_msgdesc_t meshtastic_NodeInfoLite_msg; extern const pb_msgdesc_t meshtastic_DeviceState_msg; +extern const pb_msgdesc_t meshtastic_NodePositionEntry_msg; +extern const pb_msgdesc_t meshtastic_NodeTelemetryEntry_msg; +extern const pb_msgdesc_t meshtastic_NodeEnvironmentEntry_msg; +extern const pb_msgdesc_t meshtastic_NodeStatusEntry_msg; extern const pb_msgdesc_t meshtastic_NodeDatabase_msg; extern const pb_msgdesc_t meshtastic_ChannelFile_msg; extern const pb_msgdesc_t meshtastic_BackupPreferences_msg; @@ -354,6 +431,10 @@ extern const pb_msgdesc_t meshtastic_BackupPreferences_msg; #define meshtastic_UserLite_fields &meshtastic_UserLite_msg #define meshtastic_NodeInfoLite_fields &meshtastic_NodeInfoLite_msg #define meshtastic_DeviceState_fields &meshtastic_DeviceState_msg +#define meshtastic_NodePositionEntry_fields &meshtastic_NodePositionEntry_msg +#define meshtastic_NodeTelemetryEntry_fields &meshtastic_NodeTelemetryEntry_msg +#define meshtastic_NodeEnvironmentEntry_fields &meshtastic_NodeEnvironmentEntry_msg +#define meshtastic_NodeStatusEntry_fields &meshtastic_NodeStatusEntry_msg #define meshtastic_NodeDatabase_fields &meshtastic_NodeDatabase_msg #define meshtastic_ChannelFile_fields &meshtastic_ChannelFile_msg #define meshtastic_BackupPreferences_fields &meshtastic_BackupPreferences_msg @@ -364,7 +445,11 @@ extern const pb_msgdesc_t meshtastic_BackupPreferences_msg; #define meshtastic_BackupPreferences_size 2432 #define meshtastic_ChannelFile_size 718 #define meshtastic_DeviceState_size 1737 -#define meshtastic_NodeInfoLite_size 196 +#define meshtastic_NodeEnvironmentEntry_size 170 +#define meshtastic_NodeInfoLite_size 105 +#define meshtastic_NodePositionEntry_size 36 +#define meshtastic_NodeStatusEntry_size 89 +#define meshtastic_NodeTelemetryEntry_size 35 #define meshtastic_PositionLite_size 28 #define meshtastic_UserLite_size 98 diff --git a/src/mesh/generated/meshtastic/deviceonly_legacy.pb.cpp b/src/mesh/generated/meshtastic/deviceonly_legacy.pb.cpp new file mode 100644 index 000000000..04c3a98d9 --- /dev/null +++ b/src/mesh/generated/meshtastic/deviceonly_legacy.pb.cpp @@ -0,0 +1,15 @@ +/* Automatically generated nanopb constant definitions */ +/* Generated by nanopb-0.4.9.1 */ + +#include "meshtastic/deviceonly_legacy.pb.h" +#if PB_PROTO_HEADER_VERSION != 40 +#error Regenerate this file with the current version of nanopb generator. +#endif + +PB_BIND(meshtastic_NodeInfoLite_Legacy, meshtastic_NodeInfoLite_Legacy, AUTO) + + +PB_BIND(meshtastic_NodeDatabase_Legacy, meshtastic_NodeDatabase_Legacy, AUTO) + + + diff --git a/src/mesh/generated/meshtastic/deviceonly_legacy.pb.h b/src/mesh/generated/meshtastic/deviceonly_legacy.pb.h new file mode 100644 index 000000000..916951419 --- /dev/null +++ b/src/mesh/generated/meshtastic/deviceonly_legacy.pb.h @@ -0,0 +1,124 @@ +/* Automatically generated nanopb header */ +/* Generated by nanopb-0.4.9.1 */ + +#ifndef PB_MESHTASTIC_MESHTASTIC_DEVICEONLY_LEGACY_PB_H_INCLUDED +#define PB_MESHTASTIC_MESHTASTIC_DEVICEONLY_LEGACY_PB_H_INCLUDED +#include +#include +#include "meshtastic/deviceonly.pb.h" +#include "meshtastic/telemetry.pb.h" + +#if PB_PROTO_HEADER_VERSION != 40 +#error Regenerate this file with the current version of nanopb generator. +#endif + +/* Struct definitions */ +/* Legacy NodeInfoLite descriptor used only to decode pre-split + /prefs/nodes.proto saves during the v24 -> v25 migration boot. + This preserves the original NodeInfoLite-compatible field numbers needed + to parse old wire bytes cleanly, including user (2), position (3), + device_metrics (6), and the legacy-only compatibility fields via_mqtt (8), + is_favorite (10), and is_ignored (11). Steady-state code does not use + this struct; it is dropped after migration completes. This file should be + removed once DEVICESTATE_MIN_VER advances past 24. */ +typedef struct _meshtastic_NodeInfoLite_Legacy { + uint32_t num; + bool has_user; + meshtastic_UserLite user; + bool has_position; + meshtastic_PositionLite position; + float snr; + uint32_t last_heard; + bool has_device_metrics; + meshtastic_DeviceMetrics device_metrics; + uint8_t channel; + bool via_mqtt; + bool has_hops_away; + uint8_t hops_away; + bool is_favorite; + bool is_ignored; + uint8_t next_hop; + uint32_t bitfield; +} meshtastic_NodeInfoLite_Legacy; + +/* Legacy NodeDatabase shape: one repeated array of fat NodeInfoLite_Legacy + with no satellite position/telemetry arrays. */ +typedef struct _meshtastic_NodeDatabase_Legacy { + uint32_t version; + std::vector nodes; +} meshtastic_NodeDatabase_Legacy; + + +#ifdef __cplusplus +extern "C" { +#endif + +/* Initializer values for message structs */ +#define meshtastic_NodeInfoLite_Legacy_init_default {0, false, meshtastic_UserLite_init_default, false, meshtastic_PositionLite_init_default, 0, 0, false, meshtastic_DeviceMetrics_init_default, 0, 0, false, 0, 0, 0, 0, 0} +#define meshtastic_NodeDatabase_Legacy_init_default {0, {0}} +#define meshtastic_NodeInfoLite_Legacy_init_zero {0, false, meshtastic_UserLite_init_zero, false, meshtastic_PositionLite_init_zero, 0, 0, false, meshtastic_DeviceMetrics_init_zero, 0, 0, false, 0, 0, 0, 0, 0} +#define meshtastic_NodeDatabase_Legacy_init_zero {0, {0}} + +/* Field tags (for use in manual encoding/decoding) */ +#define meshtastic_NodeInfoLite_Legacy_num_tag 1 +#define meshtastic_NodeInfoLite_Legacy_user_tag 2 +#define meshtastic_NodeInfoLite_Legacy_position_tag 3 +#define meshtastic_NodeInfoLite_Legacy_snr_tag 4 +#define meshtastic_NodeInfoLite_Legacy_last_heard_tag 5 +#define meshtastic_NodeInfoLite_Legacy_device_metrics_tag 6 +#define meshtastic_NodeInfoLite_Legacy_channel_tag 7 +#define meshtastic_NodeInfoLite_Legacy_via_mqtt_tag 8 +#define meshtastic_NodeInfoLite_Legacy_hops_away_tag 9 +#define meshtastic_NodeInfoLite_Legacy_is_favorite_tag 10 +#define meshtastic_NodeInfoLite_Legacy_is_ignored_tag 11 +#define meshtastic_NodeInfoLite_Legacy_next_hop_tag 12 +#define meshtastic_NodeInfoLite_Legacy_bitfield_tag 13 +#define meshtastic_NodeDatabase_Legacy_version_tag 1 +#define meshtastic_NodeDatabase_Legacy_nodes_tag 2 + +/* Struct field encoding specification for nanopb */ +#define meshtastic_NodeInfoLite_Legacy_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, num, 1) \ +X(a, STATIC, OPTIONAL, MESSAGE, user, 2) \ +X(a, STATIC, OPTIONAL, MESSAGE, position, 3) \ +X(a, STATIC, SINGULAR, FLOAT, snr, 4) \ +X(a, STATIC, SINGULAR, FIXED32, last_heard, 5) \ +X(a, STATIC, OPTIONAL, MESSAGE, device_metrics, 6) \ +X(a, STATIC, SINGULAR, UINT32, channel, 7) \ +X(a, STATIC, SINGULAR, BOOL, via_mqtt, 8) \ +X(a, STATIC, OPTIONAL, UINT32, hops_away, 9) \ +X(a, STATIC, SINGULAR, BOOL, is_favorite, 10) \ +X(a, STATIC, SINGULAR, BOOL, is_ignored, 11) \ +X(a, STATIC, SINGULAR, UINT32, next_hop, 12) \ +X(a, STATIC, SINGULAR, UINT32, bitfield, 13) +#define meshtastic_NodeInfoLite_Legacy_CALLBACK NULL +#define meshtastic_NodeInfoLite_Legacy_DEFAULT NULL +#define meshtastic_NodeInfoLite_Legacy_user_MSGTYPE meshtastic_UserLite +#define meshtastic_NodeInfoLite_Legacy_position_MSGTYPE meshtastic_PositionLite +#define meshtastic_NodeInfoLite_Legacy_device_metrics_MSGTYPE meshtastic_DeviceMetrics + +#define meshtastic_NodeDatabase_Legacy_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, UINT32, version, 1) \ +X(a, CALLBACK, REPEATED, MESSAGE, nodes, 2) +extern bool meshtastic_NodeDatabase_Legacy_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_t *field); +#define meshtastic_NodeDatabase_Legacy_CALLBACK meshtastic_NodeDatabase_Legacy_callback +#define meshtastic_NodeDatabase_Legacy_DEFAULT NULL +#define meshtastic_NodeDatabase_Legacy_nodes_MSGTYPE meshtastic_NodeInfoLite_Legacy + +extern const pb_msgdesc_t meshtastic_NodeInfoLite_Legacy_msg; +extern const pb_msgdesc_t meshtastic_NodeDatabase_Legacy_msg; + +/* Defines for backwards compatibility with code written before nanopb-0.4.0 */ +#define meshtastic_NodeInfoLite_Legacy_fields &meshtastic_NodeInfoLite_Legacy_msg +#define meshtastic_NodeDatabase_Legacy_fields &meshtastic_NodeDatabase_Legacy_msg + +/* Maximum encoded size of messages (where known) */ +/* meshtastic_NodeDatabase_Legacy_size depends on runtime parameters */ +#define MESHTASTIC_MESHTASTIC_DEVICEONLY_LEGACY_PB_H_MAX_SIZE meshtastic_NodeInfoLite_Legacy_size +#define meshtastic_NodeInfoLite_Legacy_size 196 + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif diff --git a/src/mesh/generated/meshtastic/mesh.pb.h b/src/mesh/generated/meshtastic/mesh.pb.h index f22825030..facf00a7f 100644 --- a/src/mesh/generated/meshtastic/mesh.pb.h +++ b/src/mesh/generated/meshtastic/mesh.pb.h @@ -317,6 +317,8 @@ typedef enum _meshtastic_HardwareModel { meshtastic_HardwareModel_THINKNODE_M9 = 131, /* The Heltec-V4-R8 uses an ESP32S3R8 chip, plus an SX1262. */ meshtastic_HardwareModel_HELTEC_V4_R8 = 132, + /* The HELTEC_MESH_NODE_T1 uses an NRF52840 chip, plus an SX1262. */ + meshtastic_HardwareModel_HELTEC_MESH_NODE_T1 = 133, /* ------------------------------------------------------------------------------------------------------------------------------------------ Reserved ID For developing private Ports. These will show up in live traffic sparsely, so we can use a high number. Keep it within 8 bits. ------------------------------------------------------------------------------------------------------------------------------------------ */ diff --git a/src/mesh/generated/meshtastic/telemetry.pb.h b/src/mesh/generated/meshtastic/telemetry.pb.h index 8c0fdd563..e8d337049 100644 --- a/src/mesh/generated/meshtastic/telemetry.pb.h +++ b/src/mesh/generated/meshtastic/telemetry.pb.h @@ -115,7 +115,11 @@ typedef enum _meshtastic_TelemetrySensorType { /* SHT family of sensors for temperature and humidity */ meshtastic_TelemetrySensorType_SHTXX = 50, /* DS248X Bridge for one-wire temperature sensors */ - meshtastic_TelemetrySensorType_DS248X = 51 + meshtastic_TelemetrySensorType_DS248X = 51, + /* MMC5983MA 3-Axis Digital Magnetic Sensor */ + meshtastic_TelemetrySensorType_MMC5983MA = 52, + /* ICM-42607-P 6‑Axis IMU */ + meshtastic_TelemetrySensorType_ICM42607P = 53 } meshtastic_TelemetrySensorType; /* Struct definitions */ @@ -496,8 +500,8 @@ extern "C" { /* Helper constants for enums */ #define _meshtastic_TelemetrySensorType_MIN meshtastic_TelemetrySensorType_SENSOR_UNSET -#define _meshtastic_TelemetrySensorType_MAX meshtastic_TelemetrySensorType_DS248X -#define _meshtastic_TelemetrySensorType_ARRAYSIZE ((meshtastic_TelemetrySensorType)(meshtastic_TelemetrySensorType_DS248X+1)) +#define _meshtastic_TelemetrySensorType_MAX meshtastic_TelemetrySensorType_ICM42607P +#define _meshtastic_TelemetrySensorType_ARRAYSIZE ((meshtastic_TelemetrySensorType)(meshtastic_TelemetrySensorType_ICM42607P+1)) diff --git a/src/mesh/http/ContentHandler.cpp b/src/mesh/http/ContentHandler.cpp index 6b3aa4859..57bbfe61b 100644 --- a/src/mesh/http/ContentHandler.cpp +++ b/src/mesh/http/ContentHandler.cpp @@ -729,7 +729,7 @@ void handleNodes(HTTPRequest *req, HTTPResponse *res) uint32_t readIndex = 0; const meshtastic_NodeInfoLite *tempNodeInfo = nodeDB->readNextMeshNode(readIndex); while (tempNodeInfo != NULL) { - if (tempNodeInfo->has_user) { + if (nodeInfoLiteHasUser(tempNodeInfo)) { JSONObject node; char id[16]; @@ -737,26 +737,26 @@ void handleNodes(HTTPRequest *req, HTTPResponse *res) node["id"] = new JSONValue(id); node["snr"] = new JSONValue(tempNodeInfo->snr); - node["via_mqtt"] = new JSONValue(BoolToString(tempNodeInfo->via_mqtt)); + node["via_mqtt"] = new JSONValue(BoolToString(nodeInfoLiteViaMqtt(tempNodeInfo))); node["last_heard"] = new JSONValue((int)tempNodeInfo->last_heard); node["position"] = new JSONValue(); if (nodeDB->hasValidPosition(tempNodeInfo)) { - JSONObject position; - position["latitude"] = new JSONValue((float)tempNodeInfo->position.latitude_i * 1e-7); - position["longitude"] = new JSONValue((float)tempNodeInfo->position.longitude_i * 1e-7); - position["altitude"] = new JSONValue((int)tempNodeInfo->position.altitude); - node["position"] = new JSONValue(position); + meshtastic_PositionLite posLite; + if (nodeDB->copyNodePosition(tempNodeInfo->num, posLite)) { + JSONObject position; + position["latitude"] = new JSONValue((float)posLite.latitude_i * 1e-7); + position["longitude"] = new JSONValue((float)posLite.longitude_i * 1e-7); + position["altitude"] = new JSONValue((int)posLite.altitude); + node["position"] = new JSONValue(position); + } } - node["long_name"] = new JSONValue(tempNodeInfo->user.long_name); - node["short_name"] = new JSONValue(tempNodeInfo->user.short_name); - char macStr[18]; - snprintf(macStr, sizeof(macStr), "%02X:%02X:%02X:%02X:%02X:%02X", tempNodeInfo->user.macaddr[0], - tempNodeInfo->user.macaddr[1], tempNodeInfo->user.macaddr[2], tempNodeInfo->user.macaddr[3], - tempNodeInfo->user.macaddr[4], tempNodeInfo->user.macaddr[5]); - node["mac_address"] = new JSONValue(macStr); - node["hw_model"] = new JSONValue(tempNodeInfo->user.hw_model); + node["long_name"] = new JSONValue(tempNodeInfo->long_name); + node["short_name"] = new JSONValue(tempNodeInfo->short_name); + // mac_address dropped from NodeInfoLite as part of the slim refactor; emit zeros. + node["mac_address"] = new JSONValue("00:00:00:00:00:00"); + node["hw_model"] = new JSONValue(tempNodeInfo->hw_model); nodesArray.push_back(new JSONValue(node)); } @@ -930,4 +930,4 @@ void handleScanNetworks(HTTPRequest *req, HTTPResponse *res) res->print(jsonString.c_str()); delete value; } -#endif \ No newline at end of file +#endif diff --git a/src/mesh/mesh-pb-constants.h b/src/mesh/mesh-pb-constants.h index eea7d4efc..2fbb355b4 100644 --- a/src/mesh/mesh-pb-constants.h +++ b/src/mesh/mesh-pb-constants.h @@ -37,9 +37,43 @@ #define MAX_RX_NOTIFICATION_TOPHONE 2 #endif -/// Verify baseline assumption of node size. If it increases, we need to reevaluate -/// the impact of its memory footprint, notably on MAX_NUM_NODES. -static_assert(sizeof(meshtastic_NodeInfoLite) <= 200, "NodeInfoLite size increased. Reconsider impact on MAX_NUM_NODES."); +/// Tighten this when the slim header shrinks; loosen only with deliberate +/// awareness of MAX_NUM_NODES impact per platform. +static_assert(sizeof(meshtastic_NodeInfoLite) <= 130, "NodeInfoLite size increased. Reconsider impact on MAX_NUM_NODES."); + +// Compile satellite NodeDBs out on STM32WL (and the status DB also follows +// MESHTASTIC_EXCLUDE_STATUS). +#ifndef MESHTASTIC_EXCLUDE_POSITIONDB +#if defined(ARCH_STM32WL) +#define MESHTASTIC_EXCLUDE_POSITIONDB 1 +#else +#define MESHTASTIC_EXCLUDE_POSITIONDB 0 +#endif +#endif + +#ifndef MESHTASTIC_EXCLUDE_TELEMETRYDB +#if defined(ARCH_STM32WL) +#define MESHTASTIC_EXCLUDE_TELEMETRYDB 1 +#else +#define MESHTASTIC_EXCLUDE_TELEMETRYDB 0 +#endif +#endif + +#ifndef MESHTASTIC_EXCLUDE_ENVIRONMENTDB +#if defined(ARCH_STM32WL) +#define MESHTASTIC_EXCLUDE_ENVIRONMENTDB 1 +#else +#define MESHTASTIC_EXCLUDE_ENVIRONMENTDB 0 +#endif +#endif + +#ifndef MESHTASTIC_EXCLUDE_STATUSDB +#if defined(ARCH_STM32WL) || defined(MESHTASTIC_EXCLUDE_STATUS) +#define MESHTASTIC_EXCLUDE_STATUSDB 1 +#else +#define MESHTASTIC_EXCLUDE_STATUSDB 0 +#endif +#endif /// max number of nodes allowed in the nodeDB #ifndef MAX_NUM_NODES diff --git a/src/modules/AdminModule.cpp b/src/modules/AdminModule.cpp index 7b847e9a1..ba1cfdb97 100644 --- a/src/modules/AdminModule.cpp +++ b/src/modules/AdminModule.cpp @@ -107,14 +107,14 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta // Automatically favorite the node that is using the admin key auto remoteNode = nodeDB->getMeshNode(mp.from); - if (remoteNode && !remoteNode->is_favorite) { + if (remoteNode && !nodeInfoLiteIsFavorite(remoteNode)) { if (config.device.role == meshtastic_Config_DeviceConfig_Role_CLIENT_BASE) { // Special case for CLIENT_BASE: is_favorite has special meaning, and we don't want to automatically set it // without the user doing so deliberately. LOG_INFO("PKC admin valid, but not auto-favoriting node %x because role==CLIENT_BASE", mp.from); } else { LOG_INFO("PKC admin valid. Auto-favoriting node %x", mp.from); - remoteNode->is_favorite = true; + nodeInfoLiteSetBit(remoteNode, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); } } } else { @@ -393,7 +393,7 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received set_favorite_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->set_favorite_node); if (node != NULL) { - node->is_favorite = true; + nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); saveChanges(SEGMENT_NODEDATABASE, false); if (screen) screen->setFrames(graphics::Screen::FOCUS_PRESERVE); // <-- Rebuild screens @@ -404,7 +404,7 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received remove_favorite_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->remove_favorite_node); if (node != NULL) { - node->is_favorite = false; + nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_FAVORITE_MASK, false); saveChanges(SEGMENT_NODEDATABASE, false); if (screen) screen->setFrames(graphics::Screen::FOCUS_PRESERVE); // <-- Rebuild screens @@ -415,11 +415,10 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received set_ignored_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->set_ignored_node); if (node != NULL) { - node->is_ignored = true; - node->has_device_metrics = false; - node->has_position = false; - node->user.public_key.size = 0; - memset(node->user.public_key.bytes, 0, sizeof(node->user.public_key.bytes)); + nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); + nodeDB->eraseNodeSatellites(node->num); + node->public_key.size = 0; + memset(node->public_key.bytes, 0, sizeof(node->public_key.bytes)); saveChanges(SEGMENT_NODEDATABASE, false); } break; @@ -428,7 +427,7 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received remove_ignored_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->remove_ignored_node); if (node != NULL) { - node->is_ignored = false; + nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_IGNORED_MASK, false); saveChanges(SEGMENT_NODEDATABASE, false); } break; @@ -437,7 +436,7 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received toggle_muted_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->toggle_muted_node); if (node != NULL) { - node->bitfield ^= (1 << NODEINFO_BITFIELD_IS_MUTED_SHIFT); + nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_MUTED_MASK, !nodeInfoLiteIsMuted(node)); saveChanges(SEGMENT_NODEDATABASE, false); } break; @@ -445,9 +444,11 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta case meshtastic_AdminMessage_set_fixed_position_tag: { LOG_INFO("Client received set_fixed_position command"); - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); - node->has_position = true; - node->position = TypeConversions::ConvertToPositionLite(r->set_fixed_position); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); + // Route the fixed position through updatePosition so it lands in the + // satellite map (or, on builds with PositionDB excluded, just sets + // localPosition for the local broadcast path). + nodeDB->updatePosition(node->num, r->set_fixed_position, RX_SRC_LOCAL); nodeDB->setLocalPosition(r->set_fixed_position); config.position.fixed_position = true; saveChanges(SEGMENT_NODEDATABASE | SEGMENT_CONFIG, false); diff --git a/src/modules/CannedMessageModule.cpp b/src/modules/CannedMessageModule.cpp index 65e903134..a29b9fa58 100644 --- a/src/modules/CannedMessageModule.cpp +++ b/src/modules/CannedMessageModule.cpp @@ -141,7 +141,7 @@ void CannedMessageModule::LaunchFreetextWithDestination(NodeNum newDest, uint8_t static bool returnToCannedList = false; bool hasKeyForNode(const meshtastic_NodeInfoLite *node) { - return node && node->has_user && node->user.public_key.size > 0; + return nodeInfoLiteHasUser(node) && node->public_key.size > 0; } /** * @brief Items in array this->messages will be set to be pointing on the right @@ -262,10 +262,10 @@ void CannedMessageModule::updateDestinationSelectionList() for (size_t i = 0; i < numMeshNodes; ++i) { meshtastic_NodeInfoLite *node = nodeDB->getMeshNodeByIndex(i); - if (!node || node->num == myNodeNum || !node->has_user || node->user.public_key.size != 32) + if (!node || node->num == myNodeNum || !nodeInfoLiteHasUser(node) || node->public_key.size != 32) continue; - const String &nodeName = node->user.long_name; + const String &nodeName = node->long_name; if (searchQuery.length() == 0) { this->filteredNodes.push_back({node, sinceLastSeen(node)}); @@ -1054,7 +1054,7 @@ void CannedMessageModule::sendText(NodeNum dest, ChannelIndex channel, const cha } NodeNum myNodeNum = nodeDB->getNodeNum(); - if (node && node->num != myNodeNum && node->has_user && node->user.public_key.size == 32) { + if (node && node->num != myNodeNum && nodeInfoLiteHasUser(node) && node->public_key.size == 32) { p->pki_encrypted = true; p->channel = 0; // force PKI } @@ -1415,8 +1415,8 @@ const char *CannedMessageModule::getNodeName(NodeNum node) return "Broadcast"; meshtastic_NodeInfoLite *info = nodeDB->getMeshNode(node); - if (info && info->has_user && strlen(info->user.long_name) > 0) { - return info->user.long_name; + if (nodeInfoLiteHasUser(info) && strlen(info->long_name) > 0) { + return info->long_name; } static char fallback[12]; @@ -1723,17 +1723,17 @@ void CannedMessageModule::drawDestinationSelectionScreen(OLEDDisplay *display, O meshtastic_NodeInfoLite *node = this->filteredNodes[nodeIndex].node; if (node) { if (display->getWidth() <= 64) { - entryText = node->user.short_name; - } else if (node->user.long_name[0]) { - entryText = node->user.long_name; + entryText = node->short_name; + } else if (node->long_name[0]) { + entryText = node->long_name; } else { - entryText = node->user.short_name; + entryText = node->short_name; } } int availWidth = display->getWidth() - ((graphics::currentResolution == graphics::ScreenResolution::High) ? 40 : 20) - - ((node && node->is_favorite) ? 10 : 0); + ((nodeInfoLiteIsFavorite(node)) ? 10 : 0); if (availWidth < 0) availWidth = 0; char truncatedEntry[96]; @@ -1742,7 +1742,7 @@ void CannedMessageModule::drawDestinationSelectionScreen(OLEDDisplay *display, O entryText = truncatedEntry; // Prepend "* " if this is a favorite - if (node && node->is_favorite) { + if (nodeInfoLiteIsFavorite(node)) { entryText = "* " + entryText; } graphics::UIRenderer::truncateStringWithEmotes(display, entryText.c_str(), truncatedEntry, sizeof(truncatedEntry), diff --git a/src/modules/ExternalNotificationModule.cpp b/src/modules/ExternalNotificationModule.cpp index 16ccdd744..1a0b69930 100644 --- a/src/modules/ExternalNotificationModule.cpp +++ b/src/modules/ExternalNotificationModule.cpp @@ -372,7 +372,7 @@ ProcessMessage ExternalNotificationModule::handleReceived(const meshtastic_MeshP // If we receive a direct message and the receipent is us, apply DM mute setting // Else we just handle it as not muted. const bool isDmToUs = !isBroadcast(mp.to) && isToUs(&mp); - bool is_muted = isDmToUs ? (sender && ((sender->bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0)) + bool is_muted = isDmToUs ? nodeInfoLiteIsMuted(sender) : (ch.settings.has_module_settings && ch.settings.module_settings.is_muted); const bool buzzerModeIsDirectOnly = diff --git a/src/modules/KeyVerificationModule.cpp b/src/modules/KeyVerificationModule.cpp index 6d0255d53..67317ac9b 100644 --- a/src/modules/KeyVerificationModule.cpp +++ b/src/modules/KeyVerificationModule.cpp @@ -10,6 +10,18 @@ KeyVerificationModule *keyVerificationModule; +namespace +{ +void copyNodeLongNameOrUnknown(char *dest, size_t destSize, const meshtastic_NodeInfoLite *node) +{ + if (!dest || destSize == 0) + return; + const char *name = (node && nodeInfoLiteHasUser(node) && node->long_name[0]) ? node->long_name : "Unknown"; + strncpy(dest, name, destSize - 1); + dest[destSize - 1] = '\0'; +} +} // namespace + KeyVerificationModule::KeyVerificationModule() : ProtobufModule("KeyVerification", meshtastic_PortNum_KEY_VERIFICATION_APP, &meshtastic_KeyVerification_msg) { @@ -37,7 +49,8 @@ AdminMessageHandleResult KeyVerificationModule::handleAdminMessageForModule(cons } else if (request->key_verification.message_type == meshtastic_KeyVerificationAdmin_MessageType_DO_VERIFY && request->key_verification.nonce == currentNonce) { auto remoteNodePtr = nodeDB->getMeshNode(currentRemoteNode); - remoteNodePtr->bitfield |= NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; + if (remoteNodePtr) + remoteNodePtr->bitfield |= NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; resetToIdle(); } else if (request->key_verification.message_type == meshtastic_KeyVerificationAdmin_MessageType_DO_NOT_VERIFY) { resetToIdle(); @@ -72,9 +85,9 @@ bool KeyVerificationModule::handleReceivedProtobuf(const meshtastic_MeshPacket & sprintf(cn->message, "Enter Security Number for Key Verification"); cn->which_payload_variant = meshtastic_ClientNotification_key_verification_number_request_tag; cn->payload_variant.key_verification_number_request.nonce = currentNonce; - strncpy(cn->payload_variant.key_verification_number_request.remote_longname, // should really check for nulls, etc - nodeDB->getMeshNode(currentRemoteNode)->user.long_name, - sizeof(cn->payload_variant.key_verification_number_request.remote_longname)); + copyNodeLongNameOrUnknown(cn->payload_variant.key_verification_number_request.remote_longname, + sizeof(cn->payload_variant.key_verification_number_request.remote_longname), + nodeDB->getMeshNode(currentRemoteNode)); service->sendClientNotification(cn); LOG_INFO("Received hash2"); currentState = KEY_VERIFICATION_SENDER_AWAITING_NUMBER; @@ -95,7 +108,8 @@ bool KeyVerificationModule::handleReceivedProtobuf(const meshtastic_MeshPacket & [=](int selected) { if (selected == 1) { auto remoteNodePtr = nodeDB->getMeshNode(currentRemoteNode); - remoteNodePtr->bitfield |= NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; + if (remoteNodePtr) + remoteNodePtr->bitfield |= NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK; } }; screen->showOverlayBanner(options);) @@ -104,9 +118,9 @@ bool KeyVerificationModule::handleReceivedProtobuf(const meshtastic_MeshPacket & sprintf(cn->message, "Final confirmation for incoming manual key verification %s", message); cn->which_payload_variant = meshtastic_ClientNotification_key_verification_final_tag; cn->payload_variant.key_verification_final.nonce = currentNonce; - strncpy(cn->payload_variant.key_verification_final.remote_longname, // should really check for nulls, etc - nodeDB->getMeshNode(currentRemoteNode)->user.long_name, - sizeof(cn->payload_variant.key_verification_final.remote_longname)); + copyNodeLongNameOrUnknown(cn->payload_variant.key_verification_final.remote_longname, + sizeof(cn->payload_variant.key_verification_final.remote_longname), + nodeDB->getMeshNode(currentRemoteNode)); cn->payload_variant.key_verification_final.isSender = false; service->sendClientNotification(cn); @@ -202,9 +216,9 @@ meshtastic_MeshPacket *KeyVerificationModule::allocReply() currentSecurityNumber % 1000); cn->which_payload_variant = meshtastic_ClientNotification_key_verification_number_inform_tag; cn->payload_variant.key_verification_number_inform.nonce = currentNonce; - strncpy(cn->payload_variant.key_verification_number_inform.remote_longname, // should really check for nulls, etc - nodeDB->getMeshNode(currentRemoteNode)->user.long_name, - sizeof(cn->payload_variant.key_verification_number_inform.remote_longname)); + copyNodeLongNameOrUnknown(cn->payload_variant.key_verification_number_inform.remote_longname, + sizeof(cn->payload_variant.key_verification_number_inform.remote_longname), + nodeDB->getMeshNode(currentRemoteNode)); cn->payload_variant.key_verification_number_inform.security_number = currentSecurityNumber; service->sendClientNotification(cn); LOG_WARN("Security Number %04u, nonce %llu", currentSecurityNumber, currentNonce); @@ -219,7 +233,7 @@ void KeyVerificationModule::processSecurityNumber(uint32_t incomingNumber) LOG_WARN("received security number: %u", incomingNumber); meshtastic_NodeInfoLite *remoteNodePtr = nullptr; remoteNodePtr = nodeDB->getMeshNode(currentRemoteNode); - if (remoteNodePtr == nullptr || !remoteNodePtr->has_user || remoteNodePtr->user.public_key.size != 32) { + if (!remoteNodePtr || !nodeInfoLiteHasUser(remoteNodePtr) || remoteNodePtr->public_key.size != 32) { currentState = KEY_VERIFICATION_IDLE; return; // should we throw an error here? } @@ -232,7 +246,7 @@ void KeyVerificationModule::processSecurityNumber(uint32_t incomingNumber) hash.update(¤tRemoteNode, sizeof(currentRemoteNode)); hash.update(owner.public_key.bytes, owner.public_key.size); - hash.update(remoteNodePtr->user.public_key.bytes, remoteNodePtr->user.public_key.size); + hash.update(remoteNodePtr->public_key.bytes, remoteNodePtr->public_key.size); hash.finalize(hash1, 32); hash.reset(); @@ -265,9 +279,9 @@ void KeyVerificationModule::processSecurityNumber(uint32_t incomingNumber) sprintf(cn->message, "Final confirmation for outgoing manual key verification %s", message); cn->which_payload_variant = meshtastic_ClientNotification_key_verification_final_tag; cn->payload_variant.key_verification_final.nonce = currentNonce; - strncpy(cn->payload_variant.key_verification_final.remote_longname, // should really check for nulls, etc - nodeDB->getMeshNode(currentRemoteNode)->user.long_name, - sizeof(cn->payload_variant.key_verification_final.remote_longname)); + copyNodeLongNameOrUnknown(cn->payload_variant.key_verification_final.remote_longname, + sizeof(cn->payload_variant.key_verification_final.remote_longname), + nodeDB->getMeshNode(currentRemoteNode)); cn->payload_variant.key_verification_final.isSender = true; service->sendClientNotification(cn); LOG_INFO(message); @@ -310,4 +324,4 @@ void KeyVerificationModule::generateVerificationCode(char *readableCode) readableCode[i] = (hash1[i] >> 2) + 48; // not a standardized base64, but workable and avoids having a dictionary. } } -#endif \ No newline at end of file +#endif diff --git a/src/modules/PositionModule.cpp b/src/modules/PositionModule.cpp index ac81e9c57..c13904c31 100644 --- a/src/modules/PositionModule.cpp +++ b/src/modules/PositionModule.cpp @@ -183,8 +183,7 @@ meshtastic_MeshPacket *PositionModule::allocPositionPacket() return nullptr; } - meshtastic_NodeInfoLite *node = service->refreshLocalMeshNode(); // should guarantee there is now a position - assert(node->has_position); + const meshtastic_NodeInfoLite *node = service->refreshLocalMeshNode(); // should guarantee there is now a position // configuration of POSITION packet // consider making this a function argument? @@ -194,7 +193,9 @@ meshtastic_MeshPacket *PositionModule::allocPositionPacket() meshtastic_Position p = meshtastic_Position_init_default; // Start with an empty structure // if localPosition is totally empty, put our last saved position (lite) in there if (localPosition.latitude_i == 0 && localPosition.longitude_i == 0) { - nodeDB->setLocalPosition(TypeConversions::ConvertToPosition(node->position)); + meshtastic_PositionLite cachedSelf; + if (nodeDB->copyNodePosition(node->num, cachedSelf)) + nodeDB->setLocalPosition(TypeConversions::ConvertToPosition(cachedSelf)); } localPosition.seq_number++; @@ -420,7 +421,7 @@ int32_t PositionModule::runOnce() doDeepSleep(nightyNightMs, false, false); } - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); if (node == nullptr) return RUNONCE_INTERVAL; @@ -445,8 +446,11 @@ int32_t PositionModule::runOnce() } else if (nodeDB->hasValidPosition(node)) { lastGpsSend = now; - lastGpsLatitude = node->position.latitude_i; - lastGpsLongitude = node->position.longitude_i; + meshtastic_PositionLite selfPos; + if (nodeDB->copyNodePosition(node->num, selfPos)) { + lastGpsLatitude = selfPos.latitude_i; + lastGpsLongitude = selfPos.longitude_i; + } if (transmitHistory) transmitHistory->setLastSentToMesh(meshtastic_PortNum_POSITION_APP); @@ -461,7 +465,10 @@ int32_t PositionModule::runOnce() if (nodeDB->hasValidPosition(node2)) { // The minimum time (in seconds) that would pass before we are able to send a new position packet. - auto smartPosition = getDistanceTraveledSinceLastSend(node->position); + meshtastic_PositionLite selfPos; + if (!nodeDB->copyNodePosition(node->num, selfPos)) + return RUNONCE_INTERVAL; // Defensive: hasValidPosition should imply this is non-null + auto smartPosition = getDistanceTraveledSinceLastSend(selfPos); msSinceLastSend = now - lastGpsSend; if (smartPosition.hasTraveledOverThreshold && @@ -479,8 +486,8 @@ int32_t PositionModule::runOnce() msSinceLastSend, minimumTimeThreshold); // Set the current coords as our last ones, after we've compared distance with current and decided to send - lastGpsLatitude = node->position.latitude_i; - lastGpsLongitude = node->position.longitude_i; + lastGpsLatitude = selfPos.latitude_i; + lastGpsLongitude = selfPos.longitude_i; } } } @@ -563,11 +570,14 @@ struct SmartPosition PositionModule::getDistanceTraveledSinceLastSend(meshtastic void PositionModule::handleNewPosition() { - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(nodeDB->getNodeNum()); const meshtastic_NodeInfoLite *node2 = service->refreshLocalMeshNode(); // should guarantee there is now a position // We limit our GPS broadcasts to a max rate if (nodeDB->hasValidPosition(node2)) { - auto smartPosition = getDistanceTraveledSinceLastSend(node->position); + meshtastic_PositionLite selfPos; + if (!nodeDB->copyNodePosition(node->num, selfPos)) + return; + auto smartPosition = getDistanceTraveledSinceLastSend(selfPos); uint32_t msSinceLastSend = millis() - lastGpsSend; if (smartPosition.hasTraveledOverThreshold && Throttle::execute( @@ -583,8 +593,8 @@ void PositionModule::handleNewPosition() minimumTimeThreshold); // Set the current coords as our last ones, after we've compared distance with current and decided to send - lastGpsLatitude = node->position.latitude_i; - lastGpsLongitude = node->position.longitude_i; + lastGpsLatitude = selfPos.latitude_i; + lastGpsLongitude = selfPos.longitude_i; } } } diff --git a/src/modules/RangeTestModule.cpp b/src/modules/RangeTestModule.cpp index 026b3028d..9d3c3ffba 100644 --- a/src/modules/RangeTestModule.cpp +++ b/src/modules/RangeTestModule.cpp @@ -268,26 +268,36 @@ bool RangeTestModuleRadio::appendFile(const meshtastic_MeshPacket &mp) fileToAppend.printf("??:??:??,"); // Time } - fileToAppend.printf("%d,", getFrom(&mp)); // From - fileToAppend.printf("%s,", n->user.long_name); // Long Name - fileToAppend.printf("%f,", n->position.latitude_i * 1e-7); // Sender Lat - fileToAppend.printf("%f,", n->position.longitude_i * 1e-7); // Sender Long + fileToAppend.printf("%d,", getFrom(&mp)); // From + fileToAppend.printf("%s,", n ? n->long_name : ""); // Long Name + meshtastic_PositionLite senderPos; + const bool haveSenderPos = nodeDB->copyNodePosition(getFrom(&mp), senderPos); + if (haveSenderPos) { + fileToAppend.printf("%f,", senderPos.latitude_i * 1e-7); // Sender Lat + fileToAppend.printf("%f,", senderPos.longitude_i * 1e-7); // Sender Long + } else { + fileToAppend.printf("0.0,0.0,"); + } if (gpsStatus->getIsConnected() || config.position.fixed_position) { fileToAppend.printf("%f,", gpsStatus->getLatitude() * 1e-7); // RX Lat fileToAppend.printf("%f,", gpsStatus->getLongitude() * 1e-7); // RX Long fileToAppend.printf("%d,", gpsStatus->getAltitude()); // RX Altitude } else { // When the phone API is in use, the node info will be updated with position - meshtastic_NodeInfoLite *us = nodeDB->getMeshNode(nodeDB->getNodeNum()); - fileToAppend.printf("%f,", us->position.latitude_i * 1e-7); // RX Lat - fileToAppend.printf("%f,", us->position.longitude_i * 1e-7); // RX Long - fileToAppend.printf("%d,", us->position.altitude); // RX Altitude + meshtastic_PositionLite usPos; + if (nodeDB->copyNodePosition(nodeDB->getNodeNum(), usPos)) { + fileToAppend.printf("%f,", usPos.latitude_i * 1e-7); // RX Lat + fileToAppend.printf("%f,", usPos.longitude_i * 1e-7); // RX Long + fileToAppend.printf("%d,", usPos.altitude); // RX Altitude + } else { + fileToAppend.printf("0.0,0.0,0,"); + } } fileToAppend.printf("%f,", mp.rx_snr); // RX SNR - if (n->position.latitude_i && n->position.longitude_i && gpsStatus->getLatitude() && gpsStatus->getLongitude()) { - float distance = GeoCoord::latLongToMeter(n->position.latitude_i * 1e-7, n->position.longitude_i * 1e-7, + if (haveSenderPos && senderPos.latitude_i && senderPos.longitude_i && gpsStatus->getLatitude() && gpsStatus->getLongitude()) { + float distance = GeoCoord::latLongToMeter(senderPos.latitude_i * 1e-7, senderPos.longitude_i * 1e-7, gpsStatus->getLatitude() * 1e-7, gpsStatus->getLongitude() * 1e-7); fileToAppend.printf("%f,", distance); // Distance in meters } else { @@ -340,4 +350,4 @@ bool RangeTestModuleRadio::removeFile() return 0; #endif -} \ No newline at end of file +} diff --git a/src/modules/RoutingModule.cpp b/src/modules/RoutingModule.cpp index 85e7f8c06..a46323ac6 100644 --- a/src/modules/RoutingModule.cpp +++ b/src/modules/RoutingModule.cpp @@ -17,8 +17,7 @@ bool RoutingModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, mesh config.device.rebroadcast_mode == meshtastic_Config_DeviceConfig_RebroadcastMode_KNOWN_ONLY)) { if (!maybePKI) return false; - if ((nodeDB->getMeshNode(mp.from) == NULL || !nodeDB->getMeshNode(mp.from)->has_user) && - (nodeDB->getMeshNode(mp.to) == NULL || !nodeDB->getMeshNode(mp.to)->has_user)) + if (!nodeInfoLiteHasUser(nodeDB->getMeshNode(mp.from)) && !nodeInfoLiteHasUser(nodeDB->getMeshNode(mp.to))) return false; } else if (owner.is_licensed && ((nodeDB->getLicenseStatus(mp.from) == UserLicenseStatus::NotLicensed) || (nodeDB->getLicenseStatus(mp.to) == UserLicenseStatus::NotLicensed))) { diff --git a/src/modules/SerialModule.cpp b/src/modules/SerialModule.cpp index 20d4d7d8c..0266bcec5 100644 --- a/src/modules/SerialModule.cpp +++ b/src/modules/SerialModule.cpp @@ -250,9 +250,12 @@ int32_t SerialModule::runOnce() uint32_t readIndex = 0; const meshtastic_NodeInfoLite *tempNodeInfo = nodeDB->readNextMeshNode(readIndex); while (tempNodeInfo != NULL) { - if (tempNodeInfo->has_user && nodeDB->hasValidPosition(tempNodeInfo)) { - printWPL(outbuf, sizeof(outbuf), tempNodeInfo->position, tempNodeInfo->user.long_name, true); - serialPrint->printf("%s", outbuf); + if (nodeInfoLiteHasUser(tempNodeInfo) && nodeDB->hasValidPosition(tempNodeInfo)) { + meshtastic_PositionLite pos; + if (nodeDB->copyNodePosition(tempNodeInfo->num, pos)) { + printWPL(outbuf, sizeof(outbuf), pos, tempNodeInfo->long_name, true); + serialPrint->printf("%s", outbuf); + } } tempNodeInfo = nodeDB->readNextMeshNode(readIndex); } @@ -401,7 +404,7 @@ ProcessMessage SerialModuleRadio::handleReceived(const meshtastic_MeshPacket &mp serialPrint->write(p.payload.bytes, p.payload.size); } else if (moduleConfig.serial.mode == meshtastic_ModuleConfig_SerialConfig_Serial_Mode_TEXTMSG) { meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(getFrom(&mp)); - const char *sender = (node && node->has_user) ? node->user.short_name : "???"; + const char *sender = nodeInfoLiteHasUser(node) ? node->short_name : "???"; serialPrint->println(); serialPrint->printf("%s: %s", sender, p.payload.bytes); serialPrint->println(); @@ -417,9 +420,13 @@ ProcessMessage SerialModuleRadio::handleReceived(const meshtastic_MeshPacket &mp decoded = &scratch; } // send position packet as WPL to the serial port - printWPL(outbuf, sizeof(outbuf), *decoded, nodeDB->getMeshNode(getFrom(&mp))->user.long_name, - moduleConfig.serial.mode == meshtastic_ModuleConfig_SerialConfig_Serial_Mode_CALTOPO); - serialPrint->printf("%s", outbuf); + { + meshtastic_NodeInfoLite *senderNode = nodeDB->getMeshNode(getFrom(&mp)); + const char *senderName = senderNode ? senderNode->long_name : ""; + printWPL(outbuf, sizeof(outbuf), *decoded, senderName, + moduleConfig.serial.mode == meshtastic_ModuleConfig_SerialConfig_Serial_Mode_CALTOPO); + serialPrint->printf("%s", outbuf); + } } } } @@ -714,4 +721,4 @@ void SerialModule::processWXSerial() #endif return; } -#endif \ No newline at end of file +#endif diff --git a/src/modules/StatusLEDModule.cpp b/src/modules/StatusLEDModule.cpp index f828f4a16..4ea34fb52 100644 --- a/src/modules/StatusLEDModule.cpp +++ b/src/modules/StatusLEDModule.cpp @@ -95,8 +95,23 @@ int32_t StatusLEDModule::runOnce() } } } - - if (power_state != charging && power_state != charged && !doing_fast_blink) { +// If we want a LED to be dedicated to the simple hearbeat, we can use that instead of the charge LED +#if defined(LED_HEARTBEAT) + if (power_state != charging && power_state != charged && !doing_fast_blink && !config.device.led_heartbeat_disabled) { + if (HEARTBEAT_LED_state == LED_STATE_ON) { + HEARTBEAT_LED_state = LED_STATE_OFF; + my_interval = 999; + } else { + HEARTBEAT_LED_state = LED_STATE_ON; + my_interval = 1; + } + digitalWrite(LED_HEARTBEAT, HEARTBEAT_LED_state); + } else { + HEARTBEAT_LED_state = LED_STATE_OFF; + digitalWrite(LED_HEARTBEAT, HEARTBEAT_LED_state); + } +#else + if (power_state != charging && power_state != charged && !doing_fast_blink && !config.device.led_heartbeat_disabled) { if (CHARGE_LED_state == LED_STATE_ON) { CHARGE_LED_state = LED_STATE_OFF; my_interval = 999; @@ -105,7 +120,7 @@ int32_t StatusLEDModule::runOnce() my_interval = 1; } } - +#endif if (!config.bluetooth.enabled || PAIRING_LED_starttime + 30 * 1000 < millis() || doing_fast_blink) { PAIRING_LED_state = LED_STATE_OFF; } else if (ble_state == unpaired) { diff --git a/src/modules/StatusLEDModule.h b/src/modules/StatusLEDModule.h index 972e26737..f66a536f6 100644 --- a/src/modules/StatusLEDModule.h +++ b/src/modules/StatusLEDModule.h @@ -43,6 +43,9 @@ class StatusLEDModule : private concurrency::OSThread private: bool CHARGE_LED_state = LED_STATE_OFF; bool PAIRING_LED_state = LED_STATE_OFF; +#if defined(LED_HEARTBEAT) + bool HEARTBEAT_LED_state = LED_STATE_OFF; +#endif uint32_t PAIRING_LED_starttime = 0; uint32_t lastUserbuttonTime = 0; diff --git a/src/modules/StatusMessageModule.cpp b/src/modules/StatusMessageModule.cpp index 0707a4f7d..2f05517f3 100644 --- a/src/modules/StatusMessageModule.cpp +++ b/src/modules/StatusMessageModule.cpp @@ -2,6 +2,7 @@ #include "StatusMessageModule.h" #include "MeshService.h" +#include "NodeDB.h" #include "ProtobufModule.h" StatusMessageModule *statusMessageModule; @@ -36,16 +37,8 @@ ProcessMessage StatusMessageModule::handleReceived(const meshtastic_MeshPacket & LOG_INFO("Received a NodeStatus message %s", incomingMessage.status); - RecentStatus entry; - entry.fromNodeId = mp.from; - entry.statusText = incomingMessage.status; - - recentReceived.push_back(std::move(entry)); - - // Keep only last MAX_RECENT_STATUSMESSAGES - if (recentReceived.size() > MAX_RECENT_STATUSMESSAGES) { - recentReceived.erase(recentReceived.begin()); // drop oldest - } + if (nodeDB) + nodeDB->setNodeStatus(mp.from, incomingMessage); } } return ProcessMessage::CONTINUE; diff --git a/src/modules/StatusMessageModule.h b/src/modules/StatusMessageModule.h index 5090066e6..2238a8437 100644 --- a/src/modules/StatusMessageModule.h +++ b/src/modules/StatusMessageModule.h @@ -2,8 +2,6 @@ #if !MESHTASTIC_EXCLUDE_STATUS #include "SinglePortModule.h" #include "configuration.h" -#include -#include class StatusMessageModule : public SinglePortModule, private concurrency::OSThread { @@ -20,29 +18,16 @@ class StatusMessageModule : public SinglePortModule, private concurrency::OSThre this->setInterval(1000 * 12 * 60 * 60); } // TODO: If we have a string, set the initial delay (15 minutes maybe) - - // Keep vector from reallocating as we fill up to MAX_RECENT_STATUSMESSAGES - recentReceived.reserve(MAX_RECENT_STATUSMESSAGES); } virtual int32_t runOnce() override; - struct RecentStatus { - uint32_t fromNodeId; // mp.from - std::string statusText; // incomingMessage.status - }; - - const std::vector &getRecentReceived() const { return recentReceived; } - protected: - /** Called to handle a particular incoming message + /** Called to handle a particular incoming message. The cached most-recent + * status for each node lives on NodeDB; use nodeDB->copyNodeStatus(num, ...). */ virtual ProcessMessage handleReceived(const meshtastic_MeshPacket &mp) override; - - private: - static constexpr size_t MAX_RECENT_STATUSMESSAGES = 5; - std::vector recentReceived; }; extern StatusMessageModule *statusMessageModule; -#endif \ No newline at end of file +#endif diff --git a/src/modules/StoreForwardModule.cpp b/src/modules/StoreForwardModule.cpp index 6c2efe83f..413006d6e 100644 --- a/src/modules/StoreForwardModule.cpp +++ b/src/modules/StoreForwardModule.cpp @@ -80,9 +80,9 @@ void StoreForwardModule::populatePSRAM() (this->records ? this->records : (((memGet.getFreePsram() / 4) * 3) / sizeof(PacketHistoryStruct))); this->records = numberOfPackets; #if defined(ARCH_ESP32) - this->packetHistory = static_cast(ps_calloc(numberOfPackets, sizeof(PacketHistoryStruct))); + this->packetHistory.reset(static_cast(ps_calloc(numberOfPackets, sizeof(PacketHistoryStruct)))); #elif defined(ARCH_PORTDUINO) - this->packetHistory = static_cast(calloc(numberOfPackets, sizeof(PacketHistoryStruct))); + this->packetHistory.reset(static_cast(calloc(numberOfPackets, sizeof(PacketHistoryStruct)))); #endif diff --git a/src/modules/StoreForwardModule.h b/src/modules/StoreForwardModule.h index 148568e1b..77565b22c 100644 --- a/src/modules/StoreForwardModule.h +++ b/src/modules/StoreForwardModule.h @@ -7,6 +7,7 @@ #include "configuration.h" #include #include +#include #include struct PacketHistoryStruct { @@ -29,11 +30,17 @@ struct PacketHistoryStruct { class StoreForwardModule : private concurrency::OSThread, public ProtobufModule { + // packetHistory is allocated with ps_calloc / calloc, so it must be released with free(), + // not delete[]. + struct CFreeDeleter { + void operator()(PacketHistoryStruct *p) const noexcept { free(p); } + }; + bool busy = 0; uint32_t busyTo = 0; char routerMessage[meshtastic_Constants_DATA_PAYLOAD_LEN] = {0}; - PacketHistoryStruct *packetHistory = 0; + std::unique_ptr packetHistory; uint32_t packetHistoryTotalCount = 0; uint32_t last_time = 0; uint32_t requestCount = 0; diff --git a/src/modules/Telemetry/EnvironmentTelemetry.cpp b/src/modules/Telemetry/EnvironmentTelemetry.cpp index 684d408a1..04a5370de 100644 --- a/src/modules/Telemetry/EnvironmentTelemetry.cpp +++ b/src/modules/Telemetry/EnvironmentTelemetry.cpp @@ -515,6 +515,10 @@ bool EnvironmentTelemetryModule::handleReceivedProtobuf(const meshtastic_MeshPac packetPool.release(lastMeasurementPacket); lastMeasurementPacket = packetPool.allocCopy(mp); + + // Cache the latest env metrics per node on NodeDB so the phone can + // pull last-known values across reboots and replays. + nodeDB->updateTelemetry(getFrom(&mp), *t, RX_SRC_RADIO); } return false; // Let others look at this message also if they want diff --git a/src/modules/TraceRouteModule.cpp b/src/modules/TraceRouteModule.cpp index 3371c405d..b6cb5d041 100644 --- a/src/modules/TraceRouteModule.cpp +++ b/src/modules/TraceRouteModule.cpp @@ -492,12 +492,12 @@ TraceRouteModule::TraceRouteModule() const char *TraceRouteModule::getNodeName(NodeNum node) { meshtastic_NodeInfoLite *info = nodeDB->getMeshNode(node); - if (info && info->has_user) { - if (strlen(info->user.short_name) > 0) { - return info->user.short_name; + if (nodeInfoLiteHasUser(info)) { + if (strlen(info->short_name) > 0) { + return info->short_name; } - if (strlen(info->user.long_name) > 0) { - return info->user.long_name; + if (strlen(info->long_name) > 0) { + return info->long_name; } } diff --git a/src/modules/TrafficManagementModule.cpp b/src/modules/TrafficManagementModule.cpp index 1ecb68c4b..4c73c2947 100644 --- a/src/modules/TrafficManagementModule.cpp +++ b/src/modules/TrafficManagementModule.cpp @@ -1226,9 +1226,9 @@ bool TrafficManagementModule::shouldRespondToNodeInfo(const meshtastic_MeshPacke // Fallback only when PSRAM cache is unavailable on this target. // In this mode we use the node-wide table maintained by NodeInfoModule. const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(p->to); - if (!node || !node->has_user) + if (!nodeInfoLiteHasUser(node)) return false; - cachedUser = TypeConversions::ConvertToUser(node->num, node->user); + cachedUser = TypeConversions::ConvertToUser(node); } if (!sendResponse) diff --git a/src/modules/WaypointModule.cpp b/src/modules/WaypointModule.cpp index 632727b92..ca96eaa7a 100644 --- a/src/modules/WaypointModule.cpp +++ b/src/modules/WaypointModule.cpp @@ -100,7 +100,7 @@ void WaypointModule::drawFrame(OLEDDisplay *display, OLEDDisplayUiState *state, char distStr[20] = ""; // Get our node, to use our own position - meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); + const meshtastic_NodeInfoLite *ourNode = nodeDB->getMeshNode(nodeDB->getNodeNum()); // Match compass sizing/placement to favorite node screen logic. const int w = display->getWidth(); @@ -147,8 +147,10 @@ void WaypointModule::drawFrame(OLEDDisplay *display, OLEDDisplayUiState *state, const char *statusLine2 = nullptr; // Distance only needs our own position fix; compass/bearing additionally needs heading. - if (hasOwnPositionFix) { - const meshtastic_PositionLite &op = ourNode->position; + meshtastic_PositionLite ownPos; + const bool haveOwnPos = ourNode && nodeDB->copyNodePosition(ourNode->num, ownPos); + if (hasOwnPositionFix && haveOwnPos) { + const meshtastic_PositionLite &op = ownPos; const float d = GeoCoord::latLongToMeter(DegD(wp.latitude_i), DegD(wp.longitude_i), DegD(op.latitude_i), DegD(op.longitude_i)); diff --git a/src/mqtt/MQTT.cpp b/src/mqtt/MQTT.cpp index e47c9ba56..a6ae79614 100644 --- a/src/mqtt/MQTT.cpp +++ b/src/mqtt/MQTT.cpp @@ -144,7 +144,7 @@ inline void onReceiveProto(char *topic, byte *payload, size_t length) const meshtastic_NodeInfoLite *rx = nodeDB->getMeshNode(p->to); // Only accept PKI messages to us, or if we have both the sender and receiver in our nodeDB, as then it's // likely they discovered each other via a channel we have downlink enabled for - if (isToUs(p.get()) || (tx && tx->has_user && rx && rx->has_user)) + if (isToUs(p.get()) || (nodeInfoLiteHasUser(tx) && nodeInfoLiteHasUser(rx))) router->enqueueReceivedMessage(p.release()); } else if (router && perhapsDecode(p.get()) == DecodeState::DECODE_SUCCESS) // ignore messages if we don't have the channel key @@ -415,6 +415,9 @@ void MQTT::onReceive(char *topic, byte *payload, size_t length) void mqttInit() { + if (!moduleConfig.mqtt.enabled) + return; + new MQTT(); } diff --git a/src/platform/portduino/PortduinoGlue.cpp b/src/platform/portduino/PortduinoGlue.cpp index 6e9f7fca2..59d9f7526 100644 --- a/src/platform/portduino/PortduinoGlue.cpp +++ b/src/platform/portduino/PortduinoGlue.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -301,10 +302,9 @@ void portduinoSetup() // Try CH341 try { std::cout << "autoconf: Looking for CH341 device..." << std::endl; - ch341Hal = new Ch341Hal(0, portduino_config.lora_usb_serial_num, portduino_config.lora_usb_vid, - portduino_config.lora_usb_pid); - ch341Hal->getProductString(autoconf_product, 95); - delete ch341Hal; + auto probe = std::unique_ptr(new Ch341Hal(0, portduino_config.lora_usb_serial_num, + portduino_config.lora_usb_vid, portduino_config.lora_usb_pid)); + probe->getProductString(autoconf_product, 95); std::cout << "autoconf: Found CH341 device " << autoconf_product << std::endl; found_ch341 = true; diff --git a/src/serialization/MeshPacketSerializer.cpp b/src/serialization/MeshPacketSerializer.cpp index 819ba3da5..36ed4b507 100644 --- a/src/serialization/MeshPacketSerializer.cpp +++ b/src/serialization/MeshPacketSerializer.cpp @@ -319,10 +319,14 @@ std::string MeshPacketSerializer::JsonSerialize(const meshtastic_MeshPacket *mp, // Lambda function for adding a long name to the route auto addToRoute = [](JSONArray *route, NodeNum num) { char long_name[40] = "Unknown"; - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(num); - bool name_known = node ? node->has_user : false; - if (name_known) - memcpy(long_name, node->user.long_name, sizeof(long_name)); + const meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(num); + bool name_known = nodeInfoLiteHasUser(node); + if (name_known) { + const size_t copy_len = + (sizeof(node->long_name) < sizeof(long_name)) ? sizeof(node->long_name) : sizeof(long_name) - 1; + memcpy(long_name, node->long_name, copy_len); + long_name[copy_len] = '\0'; + } route->push_back(new JSONValue(long_name)); }; addToRoute(&route, mp->to); // Started at the original transmitter (destination of response) @@ -473,4 +477,4 @@ std::string MeshPacketSerializer::JsonSerializeEncrypted(const meshtastic_MeshPa delete value; return jsonStr; } -#endif \ No newline at end of file +#endif diff --git a/src/serialization/MeshPacketSerializer_nRF52.cpp b/src/serialization/MeshPacketSerializer_nRF52.cpp index bd0a29c51..c79b3d269 100644 --- a/src/serialization/MeshPacketSerializer_nRF52.cpp +++ b/src/serialization/MeshPacketSerializer_nRF52.cpp @@ -292,9 +292,13 @@ std::string MeshPacketSerializer::JsonSerialize(const meshtastic_MeshPacket *mp, auto addToRoute = [](JsonArray *route, NodeNum num) { char long_name[40] = "Unknown"; meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(num); - bool name_known = node ? node->has_user : false; - if (name_known) - memcpy(long_name, node->user.long_name, sizeof(long_name)); + bool name_known = nodeInfoLiteHasUser(node); + if (name_known) { + const size_t copy_len = + (sizeof(node->long_name) < sizeof(long_name)) ? sizeof(node->long_name) : sizeof(long_name) - 1; + memcpy(long_name, node->long_name, copy_len); + long_name[copy_len] = '\0'; + } route->add(long_name); }; @@ -418,4 +422,4 @@ std::string MeshPacketSerializer::JsonSerializeEncrypted(const meshtastic_MeshPa return jsonStr; } -#endif \ No newline at end of file +#endif diff --git a/test/test_crypto/test_main.cpp b/test/test_crypto/test_main.cpp index 36dc37b9d..15f372398 100644 --- a/test/test_crypto/test_main.cpp +++ b/test/test_crypto/test_main.cpp @@ -113,7 +113,7 @@ void test_DH25519(void) void test_PKC(void) { uint8_t private_key[32]; - meshtastic_UserLite_public_key_t public_key; + meshtastic_NodeInfoLite_public_key_t public_key; uint8_t expected_shared[32]; uint8_t expected_decrypted[32]; uint8_t radioBytes[128] __attribute__((__aligned__)); diff --git a/test/test_mqtt/MQTT.cpp b/test/test_mqtt/MQTT.cpp index edf9a3983..523e1d4ac 100644 --- a/test/test_mqtt/MQTT.cpp +++ b/test/test_mqtt/MQTT.cpp @@ -800,6 +800,17 @@ void test_disabled(void) TEST_ASSERT_FALSE(mqtt->isEnabled()); } +void test_mqttInitSkipsAllocationWhenDisabled(void) +{ + delete unitTest; + mqtt = unitTest = NULL; + + moduleConfig.mqtt.enabled = false; + mqttInit(); + + TEST_ASSERT_NULL(mqtt); +} + // Subscriptions contain the moduleConfig.mqtt.root prefix. void test_customMqttRoot(void) { @@ -912,6 +923,7 @@ void setup() RUN_TEST(test_usingCustomServer); RUN_TEST(test_enabled); RUN_TEST(test_disabled); + RUN_TEST(test_mqttInitSkipsAllocationWhenDisabled); RUN_TEST(test_customMqttRoot); RUN_TEST(test_configEmptyIsValid); RUN_TEST(test_configEnabledEmptyIsValid); diff --git a/test/test_packet_history/test_main.cpp b/test/test_packet_history/test_main.cpp index 2453956c5..4f4782ab6 100644 --- a/test/test_packet_history/test_main.cpp +++ b/test/test_packet_history/test_main.cpp @@ -641,30 +641,12 @@ void test_sender_zero_substituted(void) TEST_ASSERT_TRUE(ph->wasSeenRecently(&p2, false)); } -void test_uninitialized_wasSeenRecently(void) -{ - // Simulate uninitialized state — create a PacketHistory that looks uninitialized - // We can't easily make allocation fail, but we can test the initOk guard with a destructed one - PacketHistory h(4); - TEST_ASSERT_TRUE(h.initOk()); // sanity check - h.~PacketHistory(); - - auto p = makePacket(0x1111, 100); - TEST_ASSERT_FALSE(h.wasSeenRecently(&p)); - - // Reconstruct in place to allow proper destruction - new (&h) PacketHistory(4); -} - -void test_uninitialized_wasRelayer(void) -{ - PacketHistory h(4); - h.~PacketHistory(); - - TEST_ASSERT_FALSE(h.wasRelayer(0xAA, 100, 0x1111)); - - new (&h) PacketHistory(4); -} +// NOTE: Tests for the !initOk() short-circuit in wasSeenRecently / wasRelayer +// were removed: the only way to drive that branch from a test is to destruct +// the object and then call methods on it, which is UB and trips ASAN under +// `pio test -e coverage`. In practice the guard can only fire if `new[]` throws +// (in which case the constructor doesn't return), so the lost coverage is +// purely defensive. void test_multiple_instances_independent(void) { @@ -819,8 +801,6 @@ void setup() // Group 10 — Edge Cases RUN_TEST(test_packet_id_zero_not_stored); RUN_TEST(test_sender_zero_substituted); - RUN_TEST(test_uninitialized_wasSeenRecently); - RUN_TEST(test_uninitialized_wasRelayer); RUN_TEST(test_multiple_instances_independent); // Group 11 — Hash Table Stress diff --git a/test/test_traffic_management/test_main.cpp b/test/test_traffic_management/test_main.cpp index ec54f2312..734b4a970 100644 --- a/test/test_traffic_management/test_main.cpp +++ b/test/test_traffic_management/test_main.cpp @@ -50,7 +50,7 @@ class MockNodeDB : public NodeDB hasCachedNode = true; cachedNodeNum = n; cachedNode.num = n; - cachedNode.has_user = true; + cachedNode.bitfield |= NODEINFO_BITFIELD_HAS_USER_MASK; } private: @@ -494,9 +494,9 @@ static void test_tm_nodeinfo_directResponse_learnsRequestorNodeInfo(void) TEST_ASSERT_EQUAL_INT(static_cast(ProcessMessage::STOP), static_cast(result)); TEST_ASSERT_NOT_NULL(requestor); - TEST_ASSERT_TRUE(requestor->has_user); - TEST_ASSERT_EQUAL_STRING("requester-long", requestor->user.long_name); - TEST_ASSERT_EQUAL_STRING("rq", requestor->user.short_name); + TEST_ASSERT_TRUE((requestor->bitfield & NODEINFO_BITFIELD_HAS_USER_MASK) != 0); + TEST_ASSERT_EQUAL_STRING("requester-long", requestor->long_name); + TEST_ASSERT_EQUAL_STRING("rq", requestor->short_name); TEST_ASSERT_EQUAL_UINT8(request.channel, requestor->channel); } diff --git a/test/test_type_conversions/test_main.cpp b/test/test_type_conversions/test_main.cpp new file mode 100644 index 000000000..a02d9bdf8 --- /dev/null +++ b/test/test_type_conversions/test_main.cpp @@ -0,0 +1,406 @@ +// Tests for src/mesh/TypeConversions.cpp covering: +// - bitfield bit collapse on store + extraction round-trip +// - long_name / short_name truncation at the new max_size:25 / 5 boundaries +// - public_key / hw_model / role pass-through +// - thin vs bundled NodeInfo emission +// +// All exercised via the explicit-args overload of ConvertToNodeInfo so we don't +// touch the global nodeDB pointer (which isn't initialized in this test env). + +#include "NodeDB.h" +#include "TestUtil.h" +#include "TypeConversions.h" +#include +#include +#include + +void setUp(void) {} +void tearDown(void) {} + +// ---------- helpers ----------------------------------------------------------- + +static meshtastic_User makeUser(const char *longName, const char *shortName) +{ + meshtastic_User u = meshtastic_User_init_zero; + if (longName) + strncpy(u.long_name, longName, sizeof(u.long_name) - 1); + if (shortName) + strncpy(u.short_name, shortName, sizeof(u.short_name) - 1); + u.hw_model = meshtastic_HardwareModel_TBEAM; + u.role = meshtastic_Config_DeviceConfig_Role_CLIENT; + return u; +} + +// ---------- has_user / id / macaddr ------------------------------------------ + +void test_copy_user_sets_has_user_bit(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser("Kevin Hester", "kh"); + TEST_ASSERT_FALSE((lite.bitfield & NODEINFO_BITFIELD_HAS_USER_MASK) != 0); + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_HAS_USER_MASK) != 0); +} + +void test_convert_to_user_id_derived_from_nodenum(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0x12345678; + meshtastic_User u = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_EQUAL_STRING("!12345678", u.id); +} + +void test_convert_to_user_zero_fills_macaddr(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xCAFEBABE; + meshtastic_User u = TypeConversions::ConvertToUser(&lite); + uint8_t zeros[sizeof(u.macaddr)] = {0}; + TEST_ASSERT_EQUAL_MEMORY(zeros, u.macaddr, sizeof(u.macaddr)); +} + +// ---------- long_name truncation --------------------------------------------- + +void test_long_name_short_passes_through(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser("Kevin Hester", "kh"); // 12 chars, fits + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_EQUAL_STRING("Kevin Hester", lite.long_name); +} + +void test_long_name_exact_24_fits(void) +{ + // 24 chars -> stored as 24 chars + NUL inside char[25]. + const char *exact24 = "abcdefghijklmnopqrstuvwx"; + TEST_ASSERT_EQUAL_INT(24, (int)strlen(exact24)); + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser(exact24, "ex"); + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_EQUAL_STRING(exact24, lite.long_name); + TEST_ASSERT_EQUAL_INT(24, (int)strlen(lite.long_name)); +} + +void test_long_name_truncates_when_too_long(void) +{ + // 33 chars in, must fit in 24 + NUL. + const char *tooLong = "North-County Search & Rescue Base"; + TEST_ASSERT_EQUAL_INT(33, (int)strlen(tooLong)); + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser(tooLong, "nc"); + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_EQUAL_INT(24, (int)strlen(lite.long_name)); + TEST_ASSERT_EQUAL_STRING_LEN(tooLong, lite.long_name, 24); + TEST_ASSERT_EQUAL_INT('\0', lite.long_name[24]); +} + +void test_long_name_round_trip_to_wire(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("Mountain Repeater Site E", "mr"); // exactly 24 + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_EQUAL_STRING(in.long_name, out.long_name); +} + +void test_long_name_truncated_utf8_boundary_sanitized(void) +{ + // Suffix the 24th byte with the start of a 4-byte emoji; truncation should + // leave the dangling bytes for sanitizeUtf8 to replace with '?'. + char input[40] = {0}; + memset(input, 'a', 22); // 22 ASCII + input[22] = static_cast(0xF0); // emoji lead byte at position 22 + input[23] = static_cast(0x9F); // continuation + input[24] = static_cast(0xA4); // continuation - past the cap + input[25] = static_cast(0x96); // continuation - past the cap + input[26] = '\0'; + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser(input, "u8"); + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + // 24 bytes survive plus NUL. sanitizeUtf8 should have turned the dangling + // multi-byte head into '?' since its continuation bytes were chopped. + TEST_ASSERT_EQUAL_INT(24, (int)strlen(lite.long_name)); + for (int i = 0; i < 22; ++i) + TEST_ASSERT_EQUAL_INT('a', lite.long_name[i]); + // The 4-byte sequence got truncated mid-codepoint; sanitizeUtf8 replaces + // any invalid lead/continuation byte with '?'. + TEST_ASSERT_EQUAL_INT('?', lite.long_name[22]); + TEST_ASSERT_EQUAL_INT('?', lite.long_name[23]); +} + +// ---------- short_name truncation -------------------------------------------- + +void test_short_name_passes_through(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser("Test", "abcd"); // 4 chars, fits short_name[5] + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_EQUAL_STRING("abcd", lite.short_name); +} + +void test_short_name_truncates_when_too_long(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User u = makeUser("Test", "abcdefgh"); + TypeConversions::CopyUserToNodeInfoLite(&lite, u); + TEST_ASSERT_EQUAL_INT(4, (int)strlen(lite.short_name)); + TEST_ASSERT_EQUAL_INT('\0', lite.short_name[4]); +} + +// ---------- bitfield collapse + round-trip per bool -------------------------- + +void test_bitfield_is_licensed_round_trip(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.is_licensed = true; + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_LICENSED_MASK) != 0); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_TRUE(out.is_licensed); + + in.is_licensed = false; + meshtastic_NodeInfoLite lite2 = meshtastic_NodeInfoLite_init_default; + TypeConversions::CopyUserToNodeInfoLite(&lite2, in); + TEST_ASSERT_FALSE((lite2.bitfield & NODEINFO_BITFIELD_IS_LICENSED_MASK) != 0); + meshtastic_User out2 = TypeConversions::ConvertToUser(&lite2); + TEST_ASSERT_FALSE(out2.is_licensed); +} + +void test_bitfield_unmessagable_present_and_true(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.has_is_unmessagable = true; + in.is_unmessagable = true; + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK) != 0); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK) != 0); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_TRUE(out.has_is_unmessagable); + TEST_ASSERT_TRUE(out.is_unmessagable); +} + +void test_bitfield_unmessagable_present_but_false(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.has_is_unmessagable = true; + in.is_unmessagable = false; + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK) != 0); + TEST_ASSERT_FALSE((lite.bitfield & NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK) != 0); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_TRUE(out.has_is_unmessagable); + TEST_ASSERT_FALSE(out.is_unmessagable); +} + +void test_bitfield_unmessagable_absent(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.has_is_unmessagable = false; + in.is_unmessagable = true; // explicitly true to make sure absence still wins + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_FALSE((lite.bitfield & NODEINFO_BITFIELD_HAS_IS_UNMESSAGABLE_MASK) != 0); + TEST_ASSERT_FALSE((lite.bitfield & NODEINFO_BITFIELD_IS_UNMESSAGABLE_MASK) != 0); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_FALSE(out.has_is_unmessagable); + TEST_ASSERT_FALSE(out.is_unmessagable); +} + +void test_copy_user_preserves_unrelated_bits(void) +{ + // Pre-set is_muted and is_key_manually_verified - CopyUserToNodeInfoLite + // must not stomp them. + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.bitfield = NODEINFO_BITFIELD_IS_MUTED_MASK | NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK | + NODEINFO_BITFIELD_VIA_MQTT_MASK | NODEINFO_BITFIELD_IS_FAVORITE_MASK | NODEINFO_BITFIELD_IS_IGNORED_MASK; + meshtastic_User in = makeUser("a", "a"); + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_MUTED_MASK) != 0); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK) != 0); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_VIA_MQTT_MASK) != 0); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_FAVORITE_MASK) != 0); + TEST_ASSERT_TRUE((lite.bitfield & NODEINFO_BITFIELD_IS_IGNORED_MASK) != 0); +} + +void test_bitfield_bits_are_independent(void) +{ + // Set just is_licensed, verify only that bit (and HAS_USER) is set. + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.is_licensed = true; + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + const uint32_t expected = NODEINFO_BITFIELD_HAS_USER_MASK | NODEINFO_BITFIELD_IS_LICENSED_MASK; + TEST_ASSERT_EQUAL_HEX32(expected, lite.bitfield); +} + +// ---------- public_key / hw_model / role pass-through ------------------------ + +void test_public_key_round_trip(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.public_key.size = 32; + for (int i = 0; i < 32; ++i) + in.public_key.bytes[i] = (uint8_t)(i ^ 0xA5); + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_EQUAL_INT(32, lite.public_key.size); + TEST_ASSERT_EQUAL_MEMORY(in.public_key.bytes, lite.public_key.bytes, 32); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_EQUAL_INT(32, out.public_key.size); + TEST_ASSERT_EQUAL_MEMORY(in.public_key.bytes, out.public_key.bytes, 32); +} + +void test_hw_model_and_role_round_trip(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + meshtastic_User in = makeUser("a", "a"); + in.hw_model = meshtastic_HardwareModel_HELTEC_V3; + in.role = meshtastic_Config_DeviceConfig_Role_ROUTER; + TypeConversions::CopyUserToNodeInfoLite(&lite, in); + TEST_ASSERT_EQUAL_INT(meshtastic_HardwareModel_HELTEC_V3, lite.hw_model); + TEST_ASSERT_EQUAL_INT(meshtastic_Config_DeviceConfig_Role_ROUTER, lite.role); + meshtastic_User out = TypeConversions::ConvertToUser(&lite); + TEST_ASSERT_EQUAL_INT(meshtastic_HardwareModel_HELTEC_V3, out.hw_model); + TEST_ASSERT_EQUAL_INT(meshtastic_Config_DeviceConfig_Role_ROUTER, out.role); +} + +// ---------- ConvertToNodeInfo (3-arg) ---------------------------------------- + +void test_convert_to_node_info_thin_omits_position_and_metrics(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xAA; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfoThin(&lite); + TEST_ASSERT_FALSE(info.has_position); + TEST_ASSERT_FALSE(info.has_device_metrics); +} + +void test_convert_to_node_info_3arg_with_position(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xAA; + meshtastic_PositionLite pos = meshtastic_PositionLite_init_default; + pos.latitude_i = 374200000; + pos.longitude_i = -1221000000; + pos.altitude = 30; + pos.time = 12345; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, &pos, nullptr); + TEST_ASSERT_TRUE(info.has_position); + TEST_ASSERT_FALSE(info.has_device_metrics); + TEST_ASSERT_EQUAL_INT32(374200000, info.position.latitude_i); + TEST_ASSERT_EQUAL_INT32(-1221000000, info.position.longitude_i); + TEST_ASSERT_EQUAL_INT32(30, info.position.altitude); + TEST_ASSERT_EQUAL_UINT32(12345, info.position.time); +} + +void test_convert_to_node_info_3arg_with_metrics(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xAA; + meshtastic_DeviceMetrics dm = meshtastic_DeviceMetrics_init_default; + dm.battery_level = 88; + dm.has_battery_level = true; + dm.voltage = 3.71f; + dm.has_voltage = true; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, nullptr, &dm); + TEST_ASSERT_FALSE(info.has_position); + TEST_ASSERT_TRUE(info.has_device_metrics); + TEST_ASSERT_EQUAL_INT(88, info.device_metrics.battery_level); + TEST_ASSERT_TRUE(info.device_metrics.has_battery_level); + TEST_ASSERT_EQUAL_FLOAT(3.71f, info.device_metrics.voltage); +} + +void test_convert_to_node_info_3arg_null_inputs(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xAA; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, nullptr, nullptr); + TEST_ASSERT_FALSE(info.has_position); + TEST_ASSERT_FALSE(info.has_device_metrics); +} + +void test_convert_to_node_info_extracts_bitfield_bools(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xBB; + lite.bitfield = NODEINFO_BITFIELD_VIA_MQTT_MASK | NODEINFO_BITFIELD_IS_FAVORITE_MASK | NODEINFO_BITFIELD_IS_IGNORED_MASK | + NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK | NODEINFO_BITFIELD_IS_MUTED_MASK; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, nullptr, nullptr); + TEST_ASSERT_TRUE(info.via_mqtt); + TEST_ASSERT_TRUE(info.is_favorite); + TEST_ASSERT_TRUE(info.is_ignored); + TEST_ASSERT_TRUE(info.is_key_manually_verified); + TEST_ASSERT_TRUE(info.is_muted); +} + +void test_convert_to_node_info_extracts_bitfield_bools_none_set(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0xBB; + lite.bitfield = 0; + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, nullptr, nullptr); + TEST_ASSERT_FALSE(info.via_mqtt); + TEST_ASSERT_FALSE(info.is_favorite); + TEST_ASSERT_FALSE(info.is_ignored); + TEST_ASSERT_FALSE(info.is_key_manually_verified); + TEST_ASSERT_FALSE(info.is_muted); +} + +void test_convert_to_node_info_user_only_when_has_user_bit_set(void) +{ + meshtastic_NodeInfoLite lite = meshtastic_NodeInfoLite_init_default; + lite.num = 0x01; + strcpy(lite.long_name, "Alpha"); + strcpy(lite.short_name, "A"); + // No HAS_USER bit -> user fields ignored on emit. + meshtastic_NodeInfo info = TypeConversions::ConvertToNodeInfo(&lite, nullptr, nullptr); + TEST_ASSERT_FALSE(info.has_user); + + lite.bitfield |= NODEINFO_BITFIELD_HAS_USER_MASK; + meshtastic_NodeInfo info2 = TypeConversions::ConvertToNodeInfo(&lite, nullptr, nullptr); + TEST_ASSERT_TRUE(info2.has_user); + TEST_ASSERT_EQUAL_STRING("Alpha", info2.user.long_name); + TEST_ASSERT_EQUAL_STRING("A", info2.user.short_name); + TEST_ASSERT_EQUAL_STRING("!00000001", info2.user.id); +} + +// ---------- entry point ------------------------------------------------------- + +void setup() +{ + delay(10); + delay(2000); + initializeTestEnvironment(); + UNITY_BEGIN(); + RUN_TEST(test_copy_user_sets_has_user_bit); + RUN_TEST(test_convert_to_user_id_derived_from_nodenum); + RUN_TEST(test_convert_to_user_zero_fills_macaddr); + RUN_TEST(test_long_name_short_passes_through); + RUN_TEST(test_long_name_exact_24_fits); + RUN_TEST(test_long_name_truncates_when_too_long); + RUN_TEST(test_long_name_round_trip_to_wire); + RUN_TEST(test_long_name_truncated_utf8_boundary_sanitized); + RUN_TEST(test_short_name_passes_through); + RUN_TEST(test_short_name_truncates_when_too_long); + RUN_TEST(test_bitfield_is_licensed_round_trip); + RUN_TEST(test_bitfield_unmessagable_present_and_true); + RUN_TEST(test_bitfield_unmessagable_present_but_false); + RUN_TEST(test_bitfield_unmessagable_absent); + RUN_TEST(test_copy_user_preserves_unrelated_bits); + RUN_TEST(test_bitfield_bits_are_independent); + RUN_TEST(test_public_key_round_trip); + RUN_TEST(test_hw_model_and_role_round_trip); + RUN_TEST(test_convert_to_node_info_thin_omits_position_and_metrics); + RUN_TEST(test_convert_to_node_info_3arg_with_position); + RUN_TEST(test_convert_to_node_info_3arg_with_metrics); + RUN_TEST(test_convert_to_node_info_3arg_null_inputs); + RUN_TEST(test_convert_to_node_info_extracts_bitfield_bools); + RUN_TEST(test_convert_to_node_info_extracts_bitfield_bools_none_set); + RUN_TEST(test_convert_to_node_info_user_only_when_has_user_bit_set); + exit(UNITY_END()); +} + +void loop() {} diff --git a/variants/esp32/chatter2/platformio.ini b/variants/esp32/chatter2/platformio.ini index a14e407a1..b0adeee4b 100644 --- a/variants/esp32/chatter2/platformio.ini +++ b/variants/esp32/chatter2/platformio.ini @@ -12,4 +12,4 @@ build_flags = lib_deps = ${esp32_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 diff --git a/variants/esp32/esp32-common.ini b/variants/esp32/esp32-common.ini index b9d6f5c50..cd5a8d593 100644 --- a/variants/esp32/esp32-common.ini +++ b/variants/esp32/esp32-common.ini @@ -71,7 +71,7 @@ lib_deps = # renovate: datasource=custom.pio depName=NimBLE-Arduino packageName=h2zero/library/NimBLE-Arduino h2zero/NimBLE-Arduino@1.4.3 # renovate: datasource=git-refs depName=libpax packageName=https://github.com/dbinfrago/libpax gitBranch=master - https://github.com/dbinfrago/libpax/archive/3cdc0371c375676a97967547f4065607d4c53fd1.zip + https://github.com/dbinfrago/libpax/archive/df424747f9acb86ab07c5a206ded1e8e3650726a.zip # renovate: datasource=custom.pio depName=XPowersLib packageName=lewisxhe/library/XPowersLib lewisxhe/XPowersLib@0.3.3 # renovate: datasource=custom.pio depName=rweather/Crypto packageName=rweather/library/Crypto diff --git a/variants/esp32/m5stack_core/platformio.ini b/variants/esp32/m5stack_core/platformio.ini index 8fbbae895..4f0b556ac 100644 --- a/variants/esp32/m5stack_core/platformio.ini +++ b/variants/esp32/m5stack_core/platformio.ini @@ -35,4 +35,4 @@ lib_ignore = lib_deps = ${esp32_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 diff --git a/variants/esp32/wiphone/platformio.ini b/variants/esp32/wiphone/platformio.ini index fbd77be75..b1c1f8bf7 100644 --- a/variants/esp32/wiphone/platformio.ini +++ b/variants/esp32/wiphone/platformio.ini @@ -11,7 +11,7 @@ build_flags = lib_deps = ${esp32_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=custom.pio depName=SX1509 IO Expander packageName=sparkfun/library/SX1509 IO Expander sparkfun/SX1509 IO Expander@3.0.6 # renovate: datasource=custom.pio depName=APA102 packageName=pololu/library/APA102 diff --git a/variants/esp32s3/heltec_v4/platformio.ini b/variants/esp32s3/heltec_v4/platformio.ini index ca81ab435..f42cd10b5 100644 --- a/variants/esp32s3/heltec_v4/platformio.ini +++ b/variants/esp32s3/heltec_v4/platformio.ini @@ -133,6 +133,6 @@ build_flags = lib_deps = ${heltec_v4_base.lib_deps} ${device-ui_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=git-refs depName=Quency-D_chsc6x packageName=https://github.com/Quency-D/chsc6x gitBranch=master https://github.com/Quency-D/chsc6x/archive/5cbead829d6b432a8d621ed1aafd4eb474fd4f27.zip diff --git a/variants/esp32s3/heltec_v4_r8/platformio.ini b/variants/esp32s3/heltec_v4_r8/platformio.ini index 747cc8c49..7799acf43 100644 --- a/variants/esp32s3/heltec_v4_r8/platformio.ini +++ b/variants/esp32s3/heltec_v4_r8/platformio.ini @@ -140,6 +140,6 @@ build_flags = lib_deps = ${heltec_v4_r8_base.lib_deps} ${device-ui_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=git-refs depName=Quency-D_chsc6x packageName=https://github.com/Quency-D/chsc6x gitBranch=master https://github.com/Quency-D/chsc6x/archive/3b2b6cebf3177b3e2c33d06e07909b0b10159516.zip \ No newline at end of file diff --git a/variants/esp32s3/heltec_wireless_tracker/platformio.ini b/variants/esp32s3/heltec_wireless_tracker/platformio.ini index 33643c541..1450bb45c 100644 --- a/variants/esp32s3/heltec_wireless_tracker/platformio.ini +++ b/variants/esp32s3/heltec_wireless_tracker/platformio.ini @@ -24,4 +24,4 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 diff --git a/variants/esp32s3/heltec_wireless_tracker_V1_0/platformio.ini b/variants/esp32s3/heltec_wireless_tracker_V1_0/platformio.ini index ab6592afb..ffcc1fe95 100644 --- a/variants/esp32s3/heltec_wireless_tracker_V1_0/platformio.ini +++ b/variants/esp32s3/heltec_wireless_tracker_V1_0/platformio.ini @@ -22,4 +22,4 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 diff --git a/variants/esp32s3/heltec_wireless_tracker_v2/platformio.ini b/variants/esp32s3/heltec_wireless_tracker_v2/platformio.ini index ebf0118bb..53f698c9e 100644 --- a/variants/esp32s3/heltec_wireless_tracker_v2/platformio.ini +++ b/variants/esp32s3/heltec_wireless_tracker_v2/platformio.ini @@ -21,4 +21,4 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 diff --git a/variants/esp32s3/mesh-tab/platformio.ini b/variants/esp32s3/mesh-tab/platformio.ini index a153ba9fb..fcb58d36a 100644 --- a/variants/esp32s3/mesh-tab/platformio.ini +++ b/variants/esp32s3/mesh-tab/platformio.ini @@ -55,7 +55,7 @@ lib_deps = ${esp32s3_base.lib_deps} ${device-ui_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 [mesh_tab_xpt2046] extends = mesh_tab_base diff --git a/variants/esp32s3/picomputer-s3/platformio.ini b/variants/esp32s3/picomputer-s3/platformio.ini index 6f218a126..b5a4ff178 100644 --- a/variants/esp32s3/picomputer-s3/platformio.ini +++ b/variants/esp32s3/picomputer-s3/platformio.ini @@ -25,7 +25,7 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 build_src_filter = ${esp32s3_base.build_src_filter} diff --git a/variants/esp32s3/rak_wismesh_tap_v2/platformio.ini b/variants/esp32s3/rak_wismesh_tap_v2/platformio.ini index 7847410ae..1d3314aab 100644 --- a/variants/esp32s3/rak_wismesh_tap_v2/platformio.ini +++ b/variants/esp32s3/rak_wismesh_tap_v2/platformio.ini @@ -37,7 +37,7 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 [env:rak_wismesh_tap_v2-tft] extends = env:rak_wismesh_tap_v2 diff --git a/variants/esp32s3/t-deck/platformio.ini b/variants/esp32s3/t-deck/platformio.ini index 1b3599464..a7701549f 100644 --- a/variants/esp32s3/t-deck/platformio.ini +++ b/variants/esp32s3/t-deck/platformio.ini @@ -29,7 +29,7 @@ build_flags = ${esp32s3_base.build_flags} lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=git-refs depName=ESP8266Audio packageName=https://github.com/meshtastic/ESP8266Audio gitBranch=meshtastic-2.0.0-dacfix https://github.com/meshtastic/ESP8266Audio/archive/343024632ee78d6216907b2353fc943a62422d80.zip # renovate: datasource=custom.pio depName=ESP8266SAM packageName=earlephilhower/library/ESP8266SAM diff --git a/variants/esp32s3/t-watch-s3/platformio.ini b/variants/esp32s3/t-watch-s3/platformio.ini index 352396818..6e7db8630 100644 --- a/variants/esp32s3/t-watch-s3/platformio.ini +++ b/variants/esp32s3/t-watch-s3/platformio.ini @@ -22,7 +22,7 @@ build_flags = ${esp32s3_base.build_flags} lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=custom.pio depName=SensorLib packageName=lewisxhe/library/SensorLib lewisxhe/SensorLib@0.3.4 # renovate: datasource=custom.pio depName=Adafruit DRV2605 packageName=adafruit/library/Adafruit DRV2605 Library diff --git a/variants/esp32s3/tlora-pager/platformio.ini b/variants/esp32s3/tlora-pager/platformio.ini index 832f9d7d7..15abfadf3 100644 --- a/variants/esp32s3/tlora-pager/platformio.ini +++ b/variants/esp32s3/tlora-pager/platformio.ini @@ -33,7 +33,7 @@ build_flags = ${esp32s3_base.build_flags} lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # renovate: datasource=git-refs depName=ESP8266Audio packageName=https://github.com/meshtastic/ESP8266Audio gitBranch=meshtastic-2.0.0-dacfix https://github.com/meshtastic/ESP8266Audio/archive/343024632ee78d6216907b2353fc943a62422d80.zip # renovate: datasource=custom.pio depName=ESP8266SAM packageName=earlephilhower/library/ESP8266SAM diff --git a/variants/esp32s3/tracksenger/platformio.ini b/variants/esp32s3/tracksenger/platformio.ini index c006cf835..44d07d9e8 100644 --- a/variants/esp32s3/tracksenger/platformio.ini +++ b/variants/esp32s3/tracksenger/platformio.ini @@ -22,7 +22,7 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 [env:tracksenger-lcd] custom_meshtastic_hw_model = 48 @@ -48,7 +48,7 @@ build_flags = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 [env:tracksenger-oled] custom_meshtastic_hw_model = 48 diff --git a/variants/esp32s3/unphone/platformio.ini b/variants/esp32s3/unphone/platformio.ini index 3c342e2ac..56838b1fc 100644 --- a/variants/esp32s3/unphone/platformio.ini +++ b/variants/esp32s3/unphone/platformio.ini @@ -37,7 +37,7 @@ build_src_filter = lib_deps = ${esp32s3_base.lib_deps} # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 # TODO renovate https://gitlab.com/hamishcunningham/unphonelibrary#meshtastic@9.0.0 https://gitlab.com/hamishcunningham/unphonelibrary/-/archive/meshtastic/unphonelibrary-meshtastic.zip diff --git a/variants/native/portduino.ini b/variants/native/portduino.ini index 6ff3d0686..9ab45d1ab 100644 --- a/variants/native/portduino.ini +++ b/variants/native/portduino.ini @@ -27,7 +27,7 @@ lib_deps = # renovate: datasource=custom.pio depName=rweather/Crypto packageName=rweather/library/Crypto rweather/Crypto@0.4.0 # renovate: datasource=custom.pio depName=LovyanGFX packageName=lovyan03/library/LovyanGFX - lovyan03/LovyanGFX@1.2.19 + lovyan03/LovyanGFX@1.2.21 ; # renovate: datasource=git-refs depName=libch341-spi-userspace packageName=https://github.com/pine64/libch341-spi-userspace gitBranch=main https://github.com/pine64/libch341-spi-userspace/archive/23c42319a69cffcb65868e3c72e6bed83974a393.zip # renovate: datasource=custom.pio depName=adafruit/Adafruit seesaw Library packageName=adafruit/library/Adafruit seesaw Library diff --git a/variants/nrf52840/ELECROW-ThinkNode-M4/variant.cpp b/variants/nrf52840/ELECROW-ThinkNode-M4/variant.cpp index 999f326db..da257a39b 100644 --- a/variants/nrf52840/ELECROW-ThinkNode-M4/variant.cpp +++ b/variants/nrf52840/ELECROW-ThinkNode-M4/variant.cpp @@ -35,6 +35,9 @@ void initVariant() pinMode(LED_PAIRING, OUTPUT); ledOff(LED_PAIRING); + pinMode(LED_HEARTBEAT, OUTPUT); + ledOff(LED_HEARTBEAT); + pinMode(Battery_LED_1, OUTPUT); ledOff(Battery_LED_1); pinMode(Battery_LED_2, OUTPUT); diff --git a/variants/nrf52840/ELECROW-ThinkNode-M4/variant.h b/variants/nrf52840/ELECROW-ThinkNode-M4/variant.h index 2164bcedc..052ee77c0 100644 --- a/variants/nrf52840/ELECROW-ThinkNode-M4/variant.h +++ b/variants/nrf52840/ELECROW-ThinkNode-M4/variant.h @@ -41,7 +41,8 @@ extern "C" { // LEDs #define LED_BLUE -1 -#define LED_NOTIFICATION (32 + 9) +// #define LED_NOTIFICATION (32 + 9) +#define LED_HEARTBEAT (32 + 9) #define LED_PAIRING (13) #define Battery_LED_1 (15) diff --git a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/platformio.ini b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/platformio.ini index 82a4e1953..d111b011f 100644 --- a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/platformio.ini +++ b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/platformio.ini @@ -15,6 +15,7 @@ board = promicro-nrf52840 build_flags = ${nrf52840_base.build_flags} -I variants/nrf52840/diy/nrf52_promicro_diy_tcxo -D NRF52_PROMICRO_DIY + ; -D RADIOLIB_GODMODE=1 ; needed for some LR2021 items, but not enabled by default. build_src_filter = ${nrf52_base.build_src_filter} +<../variants/nrf52840/diy/nrf52_promicro_diy_tcxo> debug_tool = jlink diff --git a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/rfswitch.h b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/rfswitch.h index ac7ef57c4..714c58c24 100644 --- a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/rfswitch.h +++ b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/rfswitch.h @@ -1,5 +1,11 @@ +#pragma once #include "RadioLib.h" +// Keep LR20x0 naming while RadioLib exposes LR2021 symbols. +#ifndef LR20x0 +#define LR20x0 LR2021 +#endif + // This is rewritten to match the requirements of the E80-900M2213S // The E80 does not conform to the reference Semtech switches(!) and therefore needs a custom matrix. // See footnote #3 in "https://www.cdebyte.com/products/E80-900M2213S/2#Pin" @@ -13,14 +19,35 @@ static const uint32_t rfswitch_dio_pins[] = {RADIOLIB_LR11X0_DIO5, RADIOLIB_LR11 static const Module::RfSwitchMode_t rfswitch_table[] = { // clang-format off - // mode DIO5 DIO6 DIO7 - {LR11x0::MODE_STBY, {LOW, LOW, LOW}}, - {LR11x0::MODE_RX, {LOW, HIGH, LOW}}, - {LR11x0::MODE_TX, {HIGH, HIGH, LOW}}, - {LR11x0::MODE_TX_HP, {HIGH, LOW, LOW}}, - {LR11x0::MODE_TX_HF, {LOW, LOW, LOW}}, - {LR11x0::MODE_GNSS, {LOW, LOW, HIGH}}, - {LR11x0::MODE_WIFI, {LOW, LOW, LOW}}, + // mode DIO5 DIO6 DIO7 + {LR11x0::MODE_STBY, {LOW, LOW, LOW}}, + {LR11x0::MODE_RX, {LOW, HIGH, LOW}}, + {LR11x0::MODE_TX, {HIGH, HIGH, LOW}}, + {LR11x0::MODE_TX_HP, {HIGH, LOW, LOW}}, + {LR11x0::MODE_TX_HF, {LOW, LOW, LOW}}, + {LR11x0::MODE_GNSS, {LOW, LOW, HIGH}}, + {LR11x0::MODE_WIFI, {LOW, LOW, LOW}}, + END_OF_MODE_TABLE, + // clang-format on +}; + +// LR2021 RF switch matrix following the standard Semtech / Seeed T1000-E reference topology. +// DIO5 -> antenna path select (HIGH = sub-GHz LF) +// DIO6 -> TX enable / HP PA select +// DIO7 -> not connected (no GNSS on LR2021) +// DIO8 -> RF front-end power enable + +static const uint32_t lr20x0_rfswitch_dio_pins[] = {RADIOLIB_LR2021_DIO5, RADIOLIB_LR2021_DIO6, RADIOLIB_LR2021_DIO7, + RADIOLIB_LR2021_DIO8, RADIOLIB_NC}; + +static const Module::RfSwitchMode_t lr20x0_rfswitch_table[] = { + // clang-format off + // mode DIO5 DIO6 DIO7 DIO8 + {LR20x0::MODE_STBY, {LOW, LOW, LOW, LOW}}, + {LR20x0::MODE_RX, {HIGH, LOW, LOW, HIGH}}, + {LR20x0::MODE_TX, {HIGH, HIGH, LOW, HIGH}}, + {LR20x0::MODE_RX_HF, {LOW, LOW, LOW, LOW}}, + {LR20x0::MODE_TX_HF, {LOW, LOW, LOW, LOW}}, END_OF_MODE_TABLE, // clang-format on }; diff --git a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/variant.h b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/variant.h index 8e10141f5..82178d321 100644 --- a/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/variant.h +++ b/variants/nrf52840/diy/nrf52_promicro_diy_tcxo/variant.h @@ -155,6 +155,16 @@ NRF52 PRO MICRO PIN ASSIGNMENT #define LR11X0_DIO_AS_RF_SWITCH #endif +// LR2021 +#define USE_LR2021 +#define LR2021_IRQ_PIN (0 + 10) // P0.10 IRQ +#define LR2021_NRESET_PIN LORA_RESET // P0.09 NRST +#define LR2021_BUSY_PIN (0 + 29) // P0.29 BUSY +#define LR2021_SPI_NSS_PIN LORA_CS // P1.13 +#define LR2021_DIO3_TCXO_VOLTAGE 1.8 +#define LR2021_DIO_AS_RF_SWITCH +#define LR2021_IRQ_DIO_NUM 9 // DIO9 → P0.10 + // #define SX126X_MAX_POWER 8 set this if using a high-power board! /* diff --git a/version.properties b/version.properties index 4ee342bb8..56ea39317 100644 --- a/version.properties +++ b/version.properties @@ -1,4 +1,4 @@ [VERSION] major = 2 minor = 7 -build = 23 +build = 24