ElixirForge/bom-local-service

Fork 0

mirror of https://github.com/alexhopeoconnor/bom-local-service.git synced 2026-10-04 03:28:11 +10:00

Files

T

alex 0b6d266e79 Replace large image with smaller icon in README

2025-12-24 02:15:35 +10:00

46 KiB

Raw Blame History

BOM Local Service

A local caching service for Australian Bureau of Meteorology (BOM) radar data, designed to provide a reliable API for Home Assistant integrations and other local services.

Background

The Australian Bureau of Meteorology's radar API endpoint (https://api.weather.bom.gov.au/v1/radar/capabilities) stopped working in December 2024, returning errors. This broke integrations like the popular bom-radar-card for Home Assistant, which can no longer render due to the Cross-Origin Request Blocked error.

The issue was reported in GitHub issue #40, where multiple users confirmed the card completely fails to render.

This project was created to bridge that gap by:

Caching radar data from the BOM website
Storing it locally in a structured format
Providing a simple REST API for local services to consume
Automatically managing cache updates and cleanup

Features

🌧️ Automatic Cache Management: Background service automatically updates radar data for all cached locations
📊 Historical Data: Access historical radar frames across multiple time periods
🎯 Location-Based: Support for any Australian suburb/state combination
🖼️ Image API: Direct access to individual radar frame images
🔄 Auto-Refresh: Configurable cache expiration and refresh intervals
🧹 Automatic Cleanup: Old cache data is automatically purged based on retention settings
🎨 Demo SPA: Built-in web interface for testing and demonstration
🐳 Easy Docker Deployment: Quick setup with Docker

Architecture

Built on ASP.NET Core 9.0, the service uses a service-oriented architecture with clear separation of concerns:

Core Services

BomRadarService: Main orchestrator that coordinates cache operations, browser automation, and data retrieval
CacheService: Manages file-based storage of radar screenshots and metadata in organized directory structures
BrowserService: Handles Playwright browser automation for headless browser sessions
ScrapingService: Coordinates web scraping workflows (simplified orchestrator)
SelectorService: Finds page elements using configurable CSS selectors with fallback support
TimeParsingService: Parses and converts time formats from BOM data
DebugService: Provides debug functionality for troubleshooting

Scraping Architecture

The scraping system uses a workflow-based architecture with configurable steps:

Workflows: Define fixed sequences of steps for different data types (e.g., RadarScrapingWorkflow, TemperatureMapWorkflow). Each workflow specifies its response type via generics (IWorkflow<TResponse>)
Steps: Individual, testable units that perform specific actions (navigation, search, map interaction, capture). Steps declare prerequisites and validate page state before execution
Step Registry: Manages and discovers available scraping steps
Workflow Factory: Creates typed workflow instances based on configuration

Configuration-Driven Design:

Selectors: All CSS selectors are configurable via appsettings.json with fallback options, allowing adaptation to website changes without code modifications
JavaScript Templates: JavaScript code for page evaluation is externalized in configuration, making it easy to update logic as the website evolves
Text Patterns: Regex patterns for parsing page content are configurable, enabling quick adjustments to parsing logic
Workflow Steps: Individual steps within workflows can be enabled/disabled via configuration, providing flexibility for testing and troubleshooting

Background Services

CacheManagementService: Periodically checks cache validity for all cached locations and triggers updates when data expires
CacheCleanupService: Removes cache files older than the configured retention period

API Layer

RadarController: REST endpoints for accessing radar data (/api/radar/{suburb}/{state})
CacheController: REST endpoints for cache management operations (/api/cache/{suburb}/{state})
RadarTestController: MVC controller serving the demo SPA at /radar/{suburb}/{state}

Installation

Docker Image

Pre-built Docker images are available on GitHub Container Registry:

Image: ghcr.io/alexhopeoconnor/bom-local-service

Multi-architecture support: Images are built for both linux/amd64 and linux/arm64 platforms

Pull the latest version:

docker pull ghcr.io/alexhopeoconnor/bom-local-service:latest

Pull a specific version:

docker pull ghcr.io/alexhopeoconnor/bom-local-service:v0.0.1

See all available versions on the releases page.

Prerequisites

Docker Engine 20.10+ or Docker Desktop
Docker Compose v2.0+ (optional, for easier management)

Quick Start

Option 1: Using Pre-built Docker Image (Recommended)

Pull the latest image from GitHub Container Registry:

docker pull ghcr.io/alexhopeoconnor/bom-local-service:latest

Then run the container:

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Available tags:

latest - Latest release
v0.0.1 - Specific version (see releases for all versions)

Option 2: Build from Source

Clone the repository:

git clone https://github.com/alexhopeoconnor/bom-local-service.git
cd bom-local-service

Build the Docker image:
```
docker build -t bom-local-service .
```
The build process uses a multi-stage Dockerfile:
- Build stage: Compiles the .NET application
- Runtime stage: Uses the official Playwright .NET image with browsers pre-installed
- Sets up a virtual display (Xvfb) for headless browser operation

Run the container:

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  --shm-size=1gb \
  --ipc=host \
  bom-local-service

Note: The --shm-size=1gb and --ipc=host flags are important for Playwright to function correctly in Docker.

Using Docker Compose

The included docker-compose.yml provides a convenient way to run the service with all configuration options.

Using pre-built image (recommended):

Update docker-compose.yml to use the GitHub Container Registry image:

services:
  bom-local-service:
    image: ghcr.io/alexhopeoconnor/bom-local-service:latest
    # Remove or comment out the 'build:' section

Then run:

docker-compose up -d

Building from source:

If you want to build locally, keep the build: section in docker-compose.yml and run:

docker-compose up -d

Mounting custom appsettings.json:

To use a custom configuration file, add it to the volumes section in docker-compose.yml:

services:
  bom-local-service:
    volumes:
      - ./cache:/app/cache
      - ./appsettings.json:/app/appsettings.json:ro  # Custom config

This will:

Pull/build the image as configured
Start the service on port 8082 (configurable via HOST_PORT)
Mount the ./cache directory for persistent storage
Mount custom appsettings.json if specified
Apply all environment variable configurations (which override appsettings.json)

To view logs:

docker-compose logs -f

To stop the service:

docker-compose down

Configuration

All configuration can be done via environment variables, which override the default values in appsettings.json. The service uses ASP.NET Core's configuration system, which supports nested configuration via double underscores (__).

Environment Variables

ASP.NET Core Configuration

Variable	Description	Default
`ASPNETCORE_ENVIRONMENT`	Runtime environment (Development/Production)	`Production`
`ASPNETCORE_URLS`	URLs the service listens on	`http://+:8080`
`ENABLEHTTPSREDIRECTION`	Enable HTTPS redirection	`false`

Application Configuration

Variable	Description	Default	Example
`CACHEDIRECTORY`	Directory path for cache storage	`/app/cache`	`/data/bom-cache`
`CACHERETENTIONHOURS`	Hours to retain cached data before cleanup (can be any positive integer)	`24`	`48`, `72`, `168` (1 week)
`CACHEEXPIRATIONMINUTES`	Minutes before cache is considered expired	`12.5`	`15`
`TIMEZONE`	Timezone for time parsing (IANA format)	`Australia/Brisbane`	`Australia/Sydney`

Cache Management

Variable	Description	Default	Example
`CACHEMANAGEMENT__CHECKINTERVALMINUTES`	Interval between cache validity checks	`5`	`10`
`CACHEMANAGEMENT__INITIALDELAYSECONDS`	Delay before first cache check on startup	`10`	`30`
`CACHEMANAGEMENT__LOCATIONSTAGGERSECONDS`	Delay between processing different locations (used for both initial and periodic updates)	`1`	`2`

Cache Cleanup

Variable	Description	Default	Example
`CACHECLEANUP__INTERVALHOURS`	Interval between cleanup runs	`1`	`2`

Screenshot Configuration

Variable	Description	Default	Example
`SCREENSHOT__DYNAMICCONTENTWAITMS`	Milliseconds to wait for dynamic content to load	`1500`	`2000`
`SCREENSHOT__TILERENDERWAITMS`	Milliseconds to wait for map tiles to render	`3000`	`5000`
`SCREENSHOT__CROP__X`	X offset in pixels for screenshot cropping	`250`	`300`
`SCREENSHOT__CROP__Y`	Y offset in pixels for screenshot cropping	`0`	`50`
`SCREENSHOT__CROP__RIGHTOFFSET`	Right offset in pixels for screenshot cropping	`250`	`300`
`SCREENSHOT__CROP__HEIGHT`	Height in pixels for screenshot cropping (null = full height)	`null`	`800`

Scraping Configuration

The scraping system is highly configurable through appsettings.json. Most scraping settings (selectors, JavaScript templates, text patterns, workflow steps) are configured in appsettings.json, but can be overridden in Docker deployments.

Option 1: Mount Custom appsettings.json (Recommended for Docker)

Mount a custom appsettings.json file as a volume:

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -v $(pwd)/appsettings.json:/app/appsettings.json:ro \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Or in docker-compose.yml:

services:
  bom-local-service:
    volumes:
      - ./cache:/app/cache
      - ./appsettings.json:/app/appsettings.json:ro  # Add this line

Option 2: Environment Variables (Simple Overrides)

For quick overrides of commonly needed settings:

Variable	Description	Default	Example
`SCRAPING__BASEURL`	Base URL for the BOM website	`https://www.bom.gov.au/`	`https://www.bom.gov.au/`

Note: Complex configurations (selectors, JavaScript templates, text patterns, workflow steps) are best managed via a mounted appsettings.json file. See the Configuration File section below for the complete structure.

Debug Configuration

Variable	Description	Default	Example
`DEBUG__ENABLED`	Enable debug mode (saves debug screenshots)	`false`	`true`
`DEBUG__WAITMS`	Additional wait time in debug mode	`2000`	`5000`

Time Series Configuration

Variable	Description	Default	Example
`TIMESERIES__WARNINGFOLDERCOUNT`	Number of cache folders that triggers a warning log when processing time series requests	`200`	`300`
`TIMESERIES__MAXTIMERANGEHOURS`	Maximum time range allowed for time series queries (null = use CacheRetentionHours)	`null` (uses CacheRetentionHours)	`72`

Docker Compose Port Configuration

Variable	Description	Default
`HOST_PORT`	Port on host machine to map to container	`8082`
`CONTAINER_PORT`	Port inside container (usually 8080)	`8080`

Configuration Examples

Basic Configuration

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -e CACHERETENTIONHOURS=48 \
  -e CACHEEXPIRATIONMINUTES=15 \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Custom Cache Directory

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v /data/bom-cache:/app/cache \
  -e CACHEDIRECTORY=/app/cache \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Development Mode with Debug

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -e ASPNETCORE_ENVIRONMENT=Development \
  -e DEBUG__ENABLED=true \
  -e DEBUG__WAITMS=5000 \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Using docker-compose with Custom Settings

Create a .env file in the project root:

HOST_PORT=8082
CACHERETENTIONHOURS=48
CACHEEXPIRATIONMINUTES=15
CACHEMANAGEMENT__CHECKINTERVALMINUTES=10
TIMEZONE=Australia/Sydney

Then run:

docker-compose up -d

Custom appsettings.json (For Selector/Scraping Configuration)

If you need to customize selectors, JavaScript templates, or workflow steps:

Copy the default appsettings.json from the repository
Edit the sections you need (e.g., Scraping:Selectors)
Mount it as a volume:

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -v $(pwd)/appsettings.json:/app/appsettings.json:ro \
  --shm-size=1gb \
  --ipc=host \
  ghcr.io/alexhopeoconnor/bom-local-service:latest

Or in docker-compose.yml:

services:
  bom-local-service:
    volumes:
      - ./cache:/app/cache
      - ./appsettings.json:/app/appsettings.json:ro

Note: Environment variables still override values in the mounted appsettings.json, so you can use env vars for simple overrides and the mounted file for complex configurations.

Configuration File

For advanced scraping configuration, edit appsettings.json directly. When using Docker, you can mount a custom appsettings.json file as a volume (see Scraping Configuration above).

The scraping system supports extensive configuration:

Scraping Selectors

All CSS selectors used to find page elements are configurable with fallback options:

{
  "Scraping": {
    "Selectors": {
      "SearchButton": {
        "Name": "Search Button",
        "Selectors": [
          "button[data-testid='searchLabel']",
          "button[aria-label='Search for a location']",
          "button.search-location__trigger-button"
        ],
        "TimeoutMs": 10000,
        "Required": true,
        "ErrorMessage": "Could not find search button"
      }
    }
  }
}

JavaScript Templates

JavaScript code used for page evaluation is externalized and configurable:

{
  "Scraping": {
    "JavaScriptTemplates": {
      "WaitForSearchResults": "() => { /* template code */ }",
      "ExtractSearchResults": "() => { /* template code */ }"
    }
  }
}

Text Patterns

Regex patterns for parsing page content are configurable:

{
  "Scraping": {
    "TextPatterns": {
      "ResultsCountPattern": "(\\d+)\\s+of\\s+(\\d+)",
      "TimestampPattern": "(?:[A-Za-z]+\\s+)?\\d{1,2}\\s+[A-Za-z]{3},?\\s+\\d{1,2}:\\d{2}\\s+(?:am|pm)",
      "TimestampPattern": "(?:[A-Za-z]+\\s+)?\\d{1,2}\\s+[A-Za-z]{3},?\\s+\\d{1,2}:\\d{2}\\s+(?:am|pm)"
    }
  }
}

Workflow Steps

Individual workflow steps can be enabled/disabled and configured:

{
  "Scraping": {
    "Workflows": {
      "RadarScraping": {
        "Description": "Scrapes radar images for a location",
        "Steps": {
          "NavigateHomepage": { "Enabled": true },
          "ClickSearchButton": { "Enabled": true },
          "CaptureFrames": {
            "Enabled": true,
            "Parameters": {
              "FrameCount": 7,
              "WaitBetweenFrames": 5000
            }
          }
        }
      }
    }
  }
}

Note: Step order is fixed within workflows due to dependencies. Steps can be disabled but not reordered. See appsettings.json for the complete configuration structure.

API Documentation

The service provides RESTful API endpoints for accessing radar data and managing the cache.

Base URL

When running locally with default settings: http://localhost:8082

Endpoints

Get Radar Data

Get the latest radar frames for a location.

GET /api/radar/{suburb}/{state}

Parameters:

suburb (path): Suburb name (e.g., "Brisbane")
state (path): State abbreviation (e.g., "QLD")

Response:

{
  "frames": [
    {
      "frameIndex": 0,
      "imageUrl": "/api/radar/Brisbane/QLD/frame/0",
      "absoluteObservationTime": "2025-01-15T10:00:00Z"
    }
  ],
  "observationTime": "2025-01-15T10:00:00Z",
  "forecastTime": "2025-01-15T10:00:00Z",
  "weatherStation": "Brisbane",
  "distance": "5.2 km",
  "cacheIsValid": true,
  "cacheExpiresAt": "2025-01-15T10:12:30Z",
  "isUpdating": false,
  "nextUpdateTime": "2025-01-15T10:12:30Z"
}

Response Fields:

frames: Array of radar frame objects with image URLs. Each frame contains absoluteObservationTime (UTC timestamp). Client should calculate "minutes ago" dynamically from this timestamp.
observationTime: UTC timestamp when the observation was made
forecastTime: UTC timestamp for the forecast
weatherStation: Name of the weather station
distance: Distance from location to weather station
cacheIsValid: Whether the cache is still valid (not expired)
cacheExpiresAt: UTC timestamp when the cache expires
isUpdating: Whether a cache update is currently in progress
nextUpdateTime: Estimated UTC timestamp for when the cache will be updated or when an in-progress update will complete. This value is calculated using:
- Metrics-based estimation (preferred): When historical data is available, uses median durations from previous cache updates to provide hardware-adaptive estimates
- Calculated estimation (fallback): When no metrics are available yet (e.g., first update), calculates based on configured wait times and frame count
- Progress-aware: During active updates, estimates improve as progress is tracked through phases (Initializing → CapturingFrames → Saving)

Status Codes:

200 OK: Radar data available

404 Not Found: Cache is being generated (check response for retry information)

{
  "errorCode": "CACHE_NOT_FOUND",
  "errorType": "CacheError",
  "message": "No cached data found for this location (fresh start). Cache update has been triggered in background.",
  "details": {
    "location": { "suburb": "Brisbane", "state": "QLD" },
    "cacheExists": false,
    "cacheIsValid": false,
    "updateTriggered": true,
    "nextUpdateTime": "2025-01-15T10:12:30Z"
  },
  "suggestions": {
    "action": "retry_after_seconds",
    "retryAfter": 30,
    "refreshEndpoint": "/api/cache/Brisbane/QLD/refresh"
  },
  "note": "The retryAfter value is dynamically calculated based on the estimated cache update duration. On first startup with no cache, it uses a calculated estimate. After metrics are collected from completed updates, it uses hardware-adaptive estimates based on actual performance."
  "timestamp": "2025-01-15T10:00:00Z"
}

400 Bad Request: Invalid location parameters

{
  "errorCode": "VALIDATION_ERROR",
  "errorType": "ValidationError",
  "message": "Invalid state abbreviation. Use: NSW, VIC, QLD, SA, WA, TAS, NT, ACT",
  "details": {
    "field": "state"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

Get Frame Image

Get a specific radar frame image.

GET /api/radar/{suburb}/{state}/frame/{frameIndex}

Parameters:

suburb (path): Suburb name
state (path): State abbreviation
frameIndex (path): Frame index (0-6 for default 7 frames)
cacheFolder (query, optional): Specific cache folder name for historical data

Response:

200 OK: PNG image

404 Not Found: Frame not found

{
  "errorCode": "NOT_FOUND",
  "errorType": "NotFoundError",
  "message": "Frame 3 not found for Brisbane, QLD",
  "details": {
    "resourceType": "Frame",
    "identifier": "Frame 3 for Brisbane, QLD",
    "frameIndex": 3,
    "location": { "suburb": "Brisbane", "state": "QLD" }
  },
  "suggestions": {
    "suggestion": "The frame may not exist yet. Try refreshing the cache or checking if cache update is in progress."
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

Get Metadata

Get metadata about cached radar data.

GET /api/radar/{suburb}/{state}/metadata

Response:

{
  "lastUpdated": "2025-01-15T10:00:00Z",
  "observationTime": "2025-01-15T10:00:00Z",
  "forecastTime": "2025-01-15T10:00:00Z",
  "weatherStation": "Brisbane",
  "distance": "5.2 km"
}

Get Time Series

Get historical radar data across multiple cache folders.

GET /api/radar/{suburb}/{state}/timeseries?startTime={iso8601}&endTime={iso8601}

Parameters:

suburb (path): Suburb name
state (path): State abbreviation
startTime (query, optional): ISO 8601 start time (e.g., 2025-01-15T00:00:00Z)
endTime (query, optional): ISO 8601 end time (defaults to now)

Response (200 OK):

{
  "cacheFolders": [
    {
      "cacheFolderName": "Brisbane_QLD_20250115_100000",
      "cacheTimestamp": "2025-01-15T10:00:00Z",
      "observationTime": "2025-01-15T10:00:00Z",
      "frames": [
        {
          "frameIndex": 0,
          "imageUrl": "/api/radar/Brisbane/QLD/frame/0?cacheFolder=Brisbane_QLD_20250115_100000",
          "absoluteObservationTime": "2025-01-15T10:00:00Z"
        }
      ]
    }
  ],
  "startTime": "2025-01-15T07:00:00Z",
  "endTime": "2025-01-15T10:00:00Z",
  "totalFrames": 7
}

Status Codes:

200 OK: Historical data available

400 Bad Request: Invalid request (e.g., time range exceeds maximum allowed duration, invalid time format, startTime after endTime)

Invalid time format:

{
  "errorCode": "VALIDATION_ERROR",
  "errorType": "ValidationError",
  "message": "Invalid startTime format. Use ISO 8601 format (e.g., 2025-12-07T00:00:00Z)",
  "details": {
    "field": "startTime"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

Time range exceeds maximum:

{
  "errorCode": "TIME_RANGE_ERROR",
  "errorType": "ValidationError",
  "message": "Time range exceeds maximum allowed duration of 24 hours (based on cache retention: 24 hours). Please specify a smaller range.",
  "details": {
    "requestedRange": {
      "start": "2025-01-15T00:00:00Z",
      "end": "2025-01-15T25:00:00Z",
      "requestedHours": 25.0
    }
  },
  "suggestions": {
    "action": "adjust_time_range"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

404 Not Found:

Location not cached: No cache exists for this location. Cache update is triggered in background:

{
  "errorCode": "CACHE_NOT_FOUND",
  "errorType": "CacheError",
  "message": "No cached data found for this location. Cache update has been triggered in background.",
  "details": {
    "location": { "suburb": "Brisbane", "state": "QLD" },
    "cacheExists": false,
    "cacheIsValid": false,
    "updateTriggered": true,
    "cacheExpiresAt": null,
    "nextUpdateTime": "2025-01-15T10:12:30Z",
    "statusMessage": "No cache exists, update triggered"
  },
  "suggestions": {
    "action": "retry_after_seconds",
    "retryAfter": 30,
    "refreshEndpoint": "/api/cache/Brisbane/QLD/refresh",
    "statusEndpoint": "/api/cache/Brisbane/QLD/range"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

No data in range: Cache exists but no data in the requested time range:

{
  "errorCode": "TIME_RANGE_ERROR",
  "errorType": "ValidationError",
  "message": "No historical data found for the specified time range.",
  "details": {
    "availableRange": {
      "oldest": "2025-01-15T08:00:00Z",
      "newest": "2025-01-15T10:00:00Z",
      "totalCacheFolders": 10,
      "timeSpanMinutes": 120
    },
    "requestedRange": {
      "start": "2025-01-15T00:00:00Z",
      "end": "2025-01-15T10:00:00Z"
    }
  },
  "suggestions": {
    "action": "adjust_time_range",
    "suggestedRange": {
      "start": "2025-01-15T08:00:00Z",
      "end": "2025-01-15T10:00:00Z"
    },
    "suggestion": "Try querying data between 2025-01-15T08:00:00Z and 2025-01-15T10:00:00Z"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

Time Range Limits:

Maximum time range is configurable via TimeSeries:MaxTimeRangeHours (defaults to CacheRetentionHours or minimum 24 hours)
If TimeSeries:MaxTimeRangeHours is not set, the limit automatically matches your CacheRetentionHours setting
This ensures you can always query all available cached data (e.g., if retention is 72 hours, you can query up to 72 hours)

Error Response Format

All API endpoints return standardized error responses using the ApiErrorResponse format:

{
  "errorCode": "CACHE_NOT_FOUND",
  "errorType": "CacheError",
  "message": "Human-readable error message",
  "details": {
    "location": { "suburb": "Brisbane", "state": "QLD" },
    "cacheExists": false,
    "cacheIsValid": false
  },
  "suggestions": {
    "action": "retry_after_seconds",
    "retryAfter": 30,
    "refreshEndpoint": "/api/cache/Brisbane/QLD/refresh"
  },
  "timestamp": "2025-01-15T10:00:00Z"
}

Error Response Fields:

errorCode: Machine-readable error code (e.g., CACHE_NOT_FOUND, VALIDATION_ERROR, TIME_RANGE_ERROR)
errorType: Error category (CacheError, ValidationError, ServiceError, NotFoundError)
message: Human-readable error description
details: Additional context (varies by error type)
suggestions: Actionable guidance (retry times, endpoints, etc.)
timestamp: UTC timestamp when error occurred

Common Error Codes:

CACHE_NOT_FOUND: No cached data exists for the location (fresh start scenario)
VALIDATION_ERROR: Invalid request parameters
TIME_RANGE_ERROR: Time range validation failed or no data in range
NOT_FOUND: Specific resource not found (e.g., frame, metadata)
CACHE_UPDATE_FAILED: Cache update operation failed
INTERNAL_ERROR: Server-side error occurred
If no time range is specified, returns all available historical data
Note: CacheRetentionHours can be set to any positive integer value (24, 48, 72, 168, etc.)

Get Cache Range

Get information about available historical cache data.

GET /api/cache/{suburb}/{state}/range

Response:

{
  "oldestCache": {
    "cacheFolderName": "Brisbane_QLD_20250115_000000",
    "cacheTimestamp": "2025-01-15T00:00:00Z"
  },
  "newestCache": {
    "cacheFolderName": "Brisbane_QLD_20250115_100000",
    "cacheTimestamp": "2025-01-15T10:00:00Z"
  },
  "totalCacheFolders": 10,
  "timeSpanMinutes": 600
}

Refresh Cache

Manually trigger a cache update for a location.

POST /api/cache/{suburb}/{state}/refresh

Response:

{
  "updateTriggered": true,
  "cacheIsValid": false,
  "cacheExpiresAt": null,
  "nextUpdateTime": "2025-01-15T10:12:30Z",
  "message": "Cache update triggered"
}

Note on nextUpdateTime: The estimated completion time is calculated using:

Metrics-based estimation: Uses historical median durations from previous cache updates (more accurate, hardware-adaptive)
Calculated estimation: Falls back to calculated estimates based on configuration when no metrics are available yet
Estimates improve in real-time as the update progresses through phases (Initializing → CapturingFrames → Saving)

Delete Cache

Delete cached data for a location.

DELETE /api/cache/{suburb}/{state}

Response:

{
  "message": "Cache deleted for Brisbane, QLD"
}

Cache Update Estimation

The service uses a metrics-based estimation system to provide accurate estimates of cache update completion times. This ensures clients receive meaningful nextUpdateTime values that adapt to the actual hardware performance.

How It Works

Progress Tracking: During cache updates, the service tracks progress through three phases:
- Initializing: Browser setup, navigation, and page loading (~0-20% of total time)
- CapturingFrames: Frame capture loop (~20-95% of total time)
- Saving: Metadata and cleanup operations (~95-100% of total time)
Metrics Collection: After each successful cache update, the service records:
- Total duration of the update
- Duration of each phase (Initializing, CapturingFrames, Saving)
- Duration of each individual scraping step (NavigateHomepage, ClickSearchButton, etc.)
- Frame-level progress during capture
Estimation Strategy:
- Metrics-based (preferred): Uses median durations from the last 20 completed updates to provide hardware-adaptive estimates
- Progress-aware: During active updates, estimates improve in real-time based on current phase and frame progress
- Calculated fallback: When no metrics are available (e.g., first update), falls back to calculated estimates based on configuration values
Benefits:
- Hardware-adaptive: Estimates automatically adjust to slower/faster hardware
- Improves over time: More accurate estimates as more updates complete
- Real-time refinement: Estimates become more precise as updates progress
- Works from clean start: Provides reasonable estimates even on first run

Example Scenarios

First Update (No Metrics):

Uses calculated estimate based on Screenshot:DynamicContentWaitMs, Screenshot:TileRenderWaitMs, and frame count
Example: ~100 seconds for 7 frames with default settings (optimized wait times)

Subsequent Updates (With Metrics):

Uses median duration from historical data
Example: If previous updates averaged 95 seconds, estimates will use ~95 seconds (with buffer)

In-Progress Update:

If capturing frame 3 of 7, estimates remaining time based on:
- Average frame duration from historical data
- Remaining frames (4 frames × avg frame duration)
- Plus estimated time for saving phase

Demo SPA

The service includes a built-in Single Page Application (SPA) for testing and demonstration purposes. This provides a visual interface to:

View radar frames in a slideshow
Test API endpoints
Configure playback settings
View historical data across extended time periods
Monitor cache status and update information

Accessing the Demo

Navigate to:

http://localhost:8082/radar/{suburb}/{state}

Example:

http://localhost:8082/radar/Brisbane/QLD

Features

Slideshow Playback: Play, pause, and navigate through radar frames
Frame Navigation: Use slider, buttons, or keyboard shortcuts (arrow keys, spacebar)
Extended Timespans: View historical data with configurable time ranges (based on cache retention settings)
Custom Time Ranges: Select specific start and end times for historical viewing
Auto-Refresh: Automatically checks for new data at configurable intervals (minimum 5 seconds, no maximum)
Cache Status: Real-time display of cache validity, expiration, and update status
Settings Panel: Configure frame intervals (minimum 0.1 seconds, no maximum), refresh rates, and playback options

Keyboard Shortcuts

← / →: Navigate to previous/next frame
Shift + ← / Shift + →: Jump back/forward 10 frames
Home / End: Jump to first/last frame
Space: Play/pause slideshow

Using the Demo for Integration Development

The demo SPA serves as a reference implementation demonstrating best practices for consuming the API. Key implementation patterns:

Basic Radar Data Fetching: The simplest pattern - fetch data and handle the case where cache is being generated:

async function getRadarData(suburb, state) {
    const response = await fetch(`http://localhost:8082/api/radar/${suburb}/${state}`);
    
    if (response.status === 404) {
        // Cache is being generated - trigger refresh and show message
        const error = await response.json();
        
        // Use standardized error response format
        if (error.errorCode === 'CACHE_NOT_FOUND' && error.suggestions?.refreshEndpoint) {
            // Trigger cache update in background
            fetch(error.suggestions.refreshEndpoint, { method: 'POST' }).catch(() => {});
        }
        
        const retryAfter = error.suggestions?.retryAfter || 30;
        return { 
            frames: [], 
            message: error.message || `Cache being generated. Retry in ${retryAfter} seconds.` 
        };
    }
    
    if (!response.ok) {
        const error = await response.json().catch(() => ({ message: `HTTP ${response.status}` }));
        throw new Error(error.message || `HTTP ${response.status}`);
    }
    
    return await response.json();
}

Displaying Frames: Frame objects include ready-to-use imageUrl properties:

const radarData = await getRadarData('Brisbane', 'QLD');
if (radarData.frames && radarData.frames.length > 0) {
    // Display first frame
    document.getElementById('radar-image').src = radarData.frames[0].imageUrl;
    
    // Or loop through all frames for animation
    radarData.frames.forEach((frame, index) => {
        const minutesAgo = frame.absoluteObservationTime 
            ? Math.round((Date.now() - new Date(frame.absoluteObservationTime).getTime()) / 60000)
            : null;
        console.log(`Frame ${index}: ${frame.imageUrl}${minutesAgo !== null ? ` (${minutesAgo} min ago)` : ''}`);
    });
}

Historical Data: Fetch extended time series by specifying a time range:

async function getHistoricalRadar(suburb, state, hoursBack = 3) {
    const endTime = new Date();
    const startTime = new Date(endTime.getTime() - (hoursBack * 60 * 60 * 1000));
    
    const response = await fetch(
        `/api/radar/${suburb}/${state}/timeseries?startTime=${startTime.toISOString()}&endTime=${endTime.toISOString()}`
    );
    
    if (response.status === 400) {
        const error = await response.json();
        // Use standardized error format
        throw new Error(error.message || 'Invalid time range request');
    }
    
    if (response.status === 404) {
        const error = await response.json();
        
        // Check error code to determine type
        if (error.errorCode === 'CACHE_NOT_FOUND') {
            // Location doesn't exist - trigger cache update if endpoint provided
            if (error.suggestions?.refreshEndpoint) {
                fetch(error.suggestions.refreshEndpoint, { method: 'POST' }).catch(() => {});
            }
            throw new Error(error.message || 'Cache update triggered, please retry in a few moments.');
        }
        
        // Cache exists but no data in range
        if (error.errorCode === 'TIME_RANGE_ERROR' && error.details?.availableRange) {
            const range = error.details.availableRange;
            const rangeMsg = range.oldest && range.newest 
                ? ` Available data: ${new Date(range.oldest).toLocaleString()} to ${new Date(range.newest).toLocaleString()}.`
                : '';
            throw new Error(error.message + rangeMsg);
        }
        
        throw new Error(error.message || 'No historical data found');
    }
    
    if (!response.ok) {
        throw new Error(`HTTP ${response.status}`);
    }
    
    const data = await response.json();
    // Flatten frames from all cache folders
    const allFrames = data.cacheFolders.flatMap(folder => folder.frames);
    return allFrames;
}

Auto-Refresh with Error Handling: Implement periodic updates that handle API downtime:

let refreshInterval;

function startAutoRefresh(suburb, state, intervalSeconds = 30) {
    refreshInterval = setInterval(async () => {
        try {
            const data = await getRadarData(suburb, state);
            if (data.frames && data.frames.length > 0) {
                updateDisplay(data);
            }
        } catch (error) {
            console.error('Failed to refresh:', error);
            // Optionally stop auto-refresh on persistent errors
        }
    }, intervalSeconds * 1000);
}

// Start auto-refresh
startAutoRefresh('Brisbane', 'QLD', 30);

// Stop auto-refresh
if (refreshInterval) {
    clearInterval(refreshInterval);
}

The complete implementation with all patterns is available in Views/RadarTest/Index.cshtml for reference.

Usage Examples

Home Assistant Integration

To use this service with Home Assistant, you'll need to create a custom integration or use a REST sensor. Here's a basic example:

# configuration.yaml
rest:
  - sensor:
      name: "BOM Radar Brisbane"
      resource: "http://localhost:8082/api/radar/Brisbane/QLD"
      value_template: "{{ value_json.frames[0].imageUrl }}"
      scan_interval: 300

cURL Examples

Get latest radar data:

curl http://localhost:8082/api/radar/Brisbane/QLD

Get specific frame image:

curl http://localhost:8082/api/radar/Brisbane/QLD/frame/0 -o frame.png

Trigger cache refresh:

curl -X POST http://localhost:8082/api/cache/Brisbane/QLD/refresh

Get historical data (last 3 hours):

curl "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z&endTime=2025-01-15T10:00:00Z"

Get historical data with error handling:

# Check response status
response=$(curl -s -w "\n%{http_code}" "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z&endTime=2025-01-15T10:00:00Z")
http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | sed '$d')

if [ "$http_code" = "200" ]; then
    echo "Success: $body"
elif [ "$http_code" = "400" ]; then
    echo "Bad Request: $body"
elif [ "$http_code" = "404" ]; then
    echo "Not Found: $body"
    # Check if cache update was triggered
    if echo "$body" | grep -q "updateTriggered"; then
        echo "Cache update triggered, retry in a few moments"
    fi
else
    echo "Error ($http_code): $body"
fi

Troubleshooting

Service Won't Start

Check Docker logs: docker logs bom-local-service
Verify port availability: Ensure port 8082 (or your configured port) is not in use
Shared memory configuration: The docker-compose.yml includes shm_size: '1gb' and ipc: host which are required for Playwright. If running with docker run, ensure you include:
```
docker run --shm-size=1gb --ipc=host ...
```
Browser launch failures: If Playwright browsers fail to launch, check for seccomp security restrictions. You may need to add --security-opt seccomp:unconfined (though this is generally not needed with the included configuration)

No Radar Data Available

Wait for initial cache: First request triggers cache generation, which takes 30-60 seconds
Check location format: Ensure suburb and state are correctly formatted (e.g., "Brisbane", "QLD")
Verify cache directory: Check that the cache volume is mounted and writable
Check logs: Look for errors in browser automation or data capture

Images Not Loading

Verify frame URLs: Check that frame image URLs are correctly formatted
Check cache status: Use /api/cache/{suburb}/{state}/range to verify cache exists
Browser automation issues: Check logs for Playwright errors

Scraping Failures

If scraping fails (e.g., "Could not find element"), the BOM website structure may have changed:

Check debug screenshots: Enable DEBUG__ENABLED=true to see what the browser sees at each step
Update selectors:
- Docker: Mount a custom appsettings.json with updated selectors (see Scraping Configuration)
- Local: Edit appsettings.json under Scraping:Selectors to add new CSS selectors as fallbacks
Check step logs: Each step logs its execution - look for which step failed
Selector fallbacks: The system tries multiple selectors in order, so add new selectors to the existing arrays
Workflow steps: Individual steps can be disabled via Scraping:Workflows:RadarScraping:Steps:{StepName}:Enabled: false if needed temporarily

Quick Fix for Docker Users:

Copy the default appsettings.json from the repository
Edit the selectors that are failing
Mount it as a volume: -v $(pwd)/appsettings.json:/app/appsettings.json:ro
Restart the container

Playwright Resource Usage

Playwright browsers (Chromium) can consume significant CPU and memory:

Limit CPU Cores: Add CPU limits to your docker-compose.yml:

services:
  bom-local-service:
    deploy:
      resources:
        limits:
          cpus: '1.0'  # Limit to 1 CPU core
        reservations:
          cpus: '0.5'  # Reserve 0.5 cores

Or with docker run:

docker run --cpus="1.0" ...

Common Playwright + Docker Issues:

Browser crashes with "out of memory": Ensure shm_size: '1gb' is set. Chromium uses /dev/shm for shared memory, and the default 64MB is insufficient.
High CPU usage during idle: This is normal - Playwright browsers can consume CPU even when idle. Consider reducing CACHEMANAGEMENT__CHECKINTERVALMINUTES to check less frequently.
Browser processes not terminating: Check logs for stuck browser processes. The service includes cleanup logic, but you may need to restart the container if processes hang.
"Protocol error" or connection failures: Usually indicates insufficient shared memory or IPC namespace issues. Verify ipc: host is set in docker-compose.yml.

Optimize Cache Settings:

Reduce retention: Lower CACHERETENTIONHOURS to keep less data on disk
Increase cleanup frequency: Lower CACHECLEANUP__INTERVALHOURS to clean up more often
Limit locations: The service automatically manages all cached locations; reduce the number of locations being cached to lower resource usage

Performance Monitoring

The service logs detailed performance metrics for each scraping workflow:

Step-Level Timing:

Each step logs its duration and compares it to historical averages
Example: Step WaitForMapReady completed in 45.30s (avg: 43.76s)
Steps that are >50% slower than average trigger warnings: ⚠️ Step WaitForMapReady took significantly longer than average: 75.45s (avg: 50.30s, +25.15s, +50.0% slower)

Workflow-Level Timing:

Complete workflow duration is logged with step breakdown
Example: Workflow RadarScraping completed in 144.40s. Step breakdown: NavigateHomepage=4.96s, ClickSearchButton=2.70s, ...
Workflows that are >30% slower than average trigger warnings: ⚠️ Workflow RadarScraping took significantly longer than average: 189.45s (avg: 145.67s, +43.78s, +30.0% slower)

Metrics Storage:

Step and phase durations are stored in memory (last 20 samples)
Used for performance estimation and identifying bottlenecks
Metrics improve over time as more updates complete

Development

Building Locally

dotnet restore
dotnet build
dotnet run

Debug Mode

Enable debug mode to save intermediate screenshots during data capture:

docker run -e DEBUG__ENABLED=true -e DEBUG__WAITMS=5000 ...

Debug screenshots are saved in {CACHEDIRECTORY}/debug/. Each scraping step saves a screenshot, HTML snapshot, and logs, making it easy to diagnose issues.

Extending the Scraping System

The workflow-based architecture makes it easy to extend the scraping system:

Adding a New Workflow:

Create a new workflow class in Services/Scraping/Workflows/ implementing IWorkflow<TResponse> where TResponse is your response type
Define the step sequence (can reuse existing steps)
Register the workflow in WorkflowFactory
Add workflow configuration to appsettings.json

Adding a New Step:

Create a step class inheriting from BaseScrapingStep
Implement Name, Prerequisites, CanExecute, and ExecuteAsync
The step will be auto-registered on startup
Add the step to a workflow's StepNames array

Updating Selectors:

Edit appsettings.json under Scraping:Selectors
Add new CSS selectors to the Selectors array (tried in order)
Adjust TimeoutMs if needed
No code changes required

Updating JavaScript Templates:

Edit appsettings.json under Scraping:JavaScriptTemplates
Update the template code as needed
No code changes required

Updating Text Patterns:

Edit appsettings.json under Scraping:TextPatterns
Update regex patterns as needed (e.g., TimestampPattern)
The TimestampPattern supports parsing timestamps like "Wednesday 17 Dec, 11:05 pm" when the BOM website changes format
No code changes required

License

MIT License - see LICENSE file for details.

Contributing

This is a hobby project, but contributions are welcome! Feel free to open issues or submit pull requests.

Acknowledgments

Inspired by the bom-radar-card project
Built with Playwright for browser automation
Uses ASP.NET Core for the web framework

Disclaimer

This project is intended for local, personal use only. The radar data and images cached by this service are the property of the Australian Bureau of Meteorology (BOM) and are subject to copyright.

Important Notes:

This service is designed to run on your local network for personal use
Do not redistribute or republish BOM radar data or images
Respect BOM's terms of service and data usage policies
The service caches publicly available data for local consumption only
This project is not affiliated with or endorsed by the Australian Bureau of Meteorology

For official BOM data and services, visit bom.gov.au.

46 KiB Raw Blame History Unescape Escape

BOM Local Service

Background

Features

Architecture

Core Services

Scraping Architecture

Background Services

API Layer

Installation

Docker Image

Prerequisites

Quick Start

Option 1: Using Pre-built Docker Image (Recommended)

Option 2: Build from Source

Using Docker Compose

Configuration

Environment Variables

ASP.NET Core Configuration

Application Configuration

Cache Management

Cache Cleanup

Screenshot Configuration

Scraping Configuration

Debug Configuration

Time Series Configuration

Docker Compose Port Configuration

Configuration Examples

Basic Configuration

Custom Cache Directory

Development Mode with Debug

Using docker-compose with Custom Settings

Custom appsettings.json (For Selector/Scraping Configuration)

Configuration File

Scraping Selectors

JavaScript Templates

Text Patterns

Workflow Steps

API Documentation

Base URL

Endpoints

Get Radar Data

Get Frame Image

Get Metadata

Get Time Series

Error Response Format

Get Cache Range

Refresh Cache

Delete Cache

Cache Update Estimation

How It Works

Example Scenarios

Demo SPA

Accessing the Demo

Features

Keyboard Shortcuts

Using the Demo for Integration Development

Usage Examples

Home Assistant Integration

cURL Examples

Troubleshooting

Service Won't Start

No Radar Data Available

Images Not Loading

Scraping Failures

Playwright Resource Usage

Performance Monitoring

Development

Building Locally

Debug Mode

Extending the Scraping System

License

Contributing

Acknowledgments

Disclaimer

46 KiB

Raw Blame History