diff --git a/.github/workflows/docker-build-release.yml b/.github/workflows/docker-build-release.yml new file mode 100644 index 0000000..510e540 --- /dev/null +++ b/.github/workflows/docker-build-release.yml @@ -0,0 +1,52 @@ +name: Publish Docker image + +on: + release: + types: [published] + +env: + REGISTRY: ghcr.io + IMAGE_NAME: ${{ github.repository }} + +jobs: + build-and-push-image: + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + attestations: write + id-token: write + + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Log in to the Container registry + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Extract metadata (tags, labels) for Docker + id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + + - name: Build and push Docker image + id: push + uses: docker/build-push-action@v6 + with: + context: . + file: ./Dockerfile + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + + - name: Generate artifact attestation + uses: actions/attest-build-provenance@v3 + with: + subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + subject-digest: ${{ steps.push.outputs.digest }} + push-to-registry: true diff --git a/Dockerfile b/Dockerfile index 8248c26..37df569 100644 --- a/Dockerfile +++ b/Dockerfile @@ -41,6 +41,11 @@ export DISPLAY=:99\n\ exec dotnet BomLocalService.dll' > /app/start.sh && \ chmod +x /app/start.sh +# OCI labels for GitHub Container Registry metadata +LABEL org.opencontainers.image.source="https://github.com/alexhopeoconnor/bom-local-service" +LABEL org.opencontainers.image.description="BOM Local Service - Australian Bureau of Meteorology radar data caching service for Home Assistant and local services" +LABEL org.opencontainers.image.licenses=MIT + EXPOSE 8080 ENTRYPOINT ["/app/start.sh"] diff --git a/README.md b/README.md new file mode 100644 index 0000000..e1aaaa4 --- /dev/null +++ b/README.md @@ -0,0 +1,694 @@ +# BOM Local Service + +A local caching service for Australian Bureau of Meteorology (BOM) radar data, designed to provide a reliable API for Home Assistant integrations and other local services. + +## Background + +The Australian Bureau of Meteorology's radar API endpoint (`https://api.weather.bom.gov.au/v1/radar/capabilities`) stopped working in December 2024, returning errors. This broke integrations like the popular [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) for Home Assistant, which can no longer render due to the Cross-Origin Request Blocked error. + +The issue was reported in [GitHub issue #40](https://github.com/Makin-Things/bom-radar-card/issues/40), where multiple users confirmed the card completely fails to render. + +This project was created to bridge that gap by: +- Caching radar data from the BOM website +- Storing it locally in a structured format +- Providing a simple REST API for local services to consume +- Automatically managing cache updates and cleanup + +## Features + +- ๐ŸŒง๏ธ **Automatic Cache Management**: Background service automatically updates radar data for all cached locations +- ๐Ÿ“Š **Historical Data**: Access historical radar frames across multiple time periods +- ๐ŸŽฏ **Location-Based**: Support for any Australian suburb/state combination +- ๐Ÿ–ผ๏ธ **Image API**: Direct access to individual radar frame images +- ๐Ÿ”„ **Auto-Refresh**: Configurable cache expiration and refresh intervals +- ๐Ÿงน **Automatic Cleanup**: Old cache data is automatically purged based on retention settings +- ๐ŸŽจ **Demo SPA**: Built-in web interface for testing and demonstration +- ๐Ÿณ **Easy Docker Deployment**: Quick setup with Docker + +## Architecture + +Built on ASP.NET Core 9.0, the service uses a service-oriented architecture with clear separation of concerns: + +### Core Services +- **BomRadarService**: Main orchestrator that coordinates cache operations, browser automation, and data retrieval +- **CacheService**: Manages file-based storage of radar screenshots and metadata in organized directory structures +- **BrowserService**: Handles Playwright browser automation for headless browser sessions +- **ScrapingService**: Performs web scraping operations using the browser service to navigate and capture radar data from the BOM website +- **TimeParsingService**: Parses and converts time formats from BOM data +- **DebugService**: Provides debug functionality for troubleshooting + +### Background Services +- **CacheManagementService**: Periodically checks cache validity for all cached locations and triggers updates when data expires +- **CacheCleanupService**: Removes cache files older than the configured retention period + +### API Layer +- **RadarController**: REST endpoints for accessing radar data (`/api/radar/{suburb}/{state}`) +- **CacheController**: REST endpoints for cache management operations (`/api/cache/{suburb}/{state}`) +- **RadarTestController**: MVC controller serving the demo SPA at `/radar/{suburb}/{state}` + +## Building with Docker + +### Prerequisites + +- Docker Engine 20.10+ or Docker Desktop +- Docker Compose v2.0+ (optional, for easier management) + +### Quick Start + +1. **Clone the repository**: + ```bash + git clone + cd bom-local-service + ``` + +2. **Build the Docker image**: + ```bash + docker build -t bom-local-service . + ``` + + The build process uses a multi-stage Dockerfile: + - **Build stage**: Compiles the .NET application + - **Runtime stage**: Uses the official Playwright .NET image with browsers pre-installed + - Sets up a virtual display (Xvfb) for headless browser operation + +3. **Run the container**: + ```bash + docker run -d \ + --name bom-local-service \ + -p 8082:8080 \ + -v $(pwd)/cache:/app/cache \ + --shm-size=1gb \ + --ipc=host \ + bom-local-service + ``` + + **Note**: The `--shm-size=1gb` and `--ipc=host` flags are important for Playwright to function correctly in Docker. + +### Using Docker Compose + +The included `docker-compose.yml` provides a convenient way to run the service with all configuration options: + +```bash +docker-compose up -d +``` + +This will: +- Build the image if it doesn't exist +- Start the service on port 8082 (configurable via `HOST_PORT`) +- Mount the `./cache` directory for persistent storage +- Apply all environment variable configurations + +To view logs: +```bash +docker-compose logs -f +``` + +To stop the service: +```bash +docker-compose down +``` + +## Configuration + +All configuration can be done via environment variables, which override the default values in `appsettings.json`. The service uses ASP.NET Core's configuration system, which supports nested configuration via double underscores (`__`). + +### Environment Variables + +#### ASP.NET Core Configuration + +| Variable | Description | Default | +|----------|-------------|---------| +| `ASPNETCORE_ENVIRONMENT` | Runtime environment (Development/Production) | `Production` | +| `ASPNETCORE_URLS` | URLs the service listens on | `http://+:8080` | +| `ENABLE_HTTPS_REDIRECTION` | Enable HTTPS redirection | `false` | + +#### Application Configuration + +| Variable | Description | Default | Example | +|----------|-------------|---------|---------| +| `CACHEDIRECTORY` | Directory path for cache storage | `/app/cache` | `/data/bom-cache` | +| `CACHERETENTIONHOURS` | Hours to retain cached data before cleanup | `24` | `48` | +| `CACHEEXPIRATIONMINUTES` | Minutes before cache is considered expired | `12.5` | `15` | +| `TIMEZONE` | Timezone for time parsing (IANA format) | `Australia/Brisbane` | `Australia/Sydney` | + +#### Cache Management + +| Variable | Description | Default | Example | +|----------|-------------|---------|---------| +| `CACHEMANAGEMENT__CHECKINTERVALMINUTES` | Interval between cache validity checks | `5` | `10` | +| `CACHEMANAGEMENT__INITIALDELAYSECONDS` | Delay before first cache check on startup | `10` | `30` | +| `CACHEMANAGEMENT__UPDATESTAGGERSECONDS` | Delay between triggering cache updates | `2` | `5` | +| `CACHEMANAGEMENT__LOCATIONSTAGGERSECONDS` | Delay between processing different locations | `1` | `2` | + +#### Cache Cleanup + +| Variable | Description | Default | Example | +|----------|-------------|---------|---------| +| `CACHECLEANUP__INTERVALHOURS` | Interval between cleanup runs | `1` | `2` | + +#### Screenshot Configuration + +| Variable | Description | Default | Example | +|----------|-------------|---------|---------| +| `SCREENSHOT__DYNAMICCONTENTWAITMS` | Milliseconds to wait for dynamic content to load | `2000` | `3000` | +| `SCREENSHOT__TILERENDERWAITMS` | Milliseconds to wait for map tiles to render | `5000` | `7000` | + +#### Debug Configuration + +| Variable | Description | Default | Example | +|----------|-------------|---------|---------| +| `DEBUG__ENABLED` | Enable debug mode (saves debug screenshots) | `false` | `true` | +| `DEBUG__WAITMS` | Additional wait time in debug mode | `2000` | `5000` | + +#### Docker Compose Port Configuration + +| Variable | Description | Default | +|----------|-------------|---------| +| `HOST_PORT` | Port on host machine to map to container | `8082` | +| `CONTAINER_PORT` | Port inside container (usually 8080) | `8080` | + +### Configuration Examples + +#### Basic Configuration + +```bash +docker run -d \ + --name bom-local-service \ + -p 8082:8080 \ + -v $(pwd)/cache:/app/cache \ + -e CACHERETENTIONHOURS=48 \ + -e CACHEEXPIRATIONMINUTES=15 \ + --shm-size=1gb \ + --ipc=host \ + bom-local-service +``` + +#### Custom Cache Directory + +```bash +docker run -d \ + --name bom-local-service \ + -p 8082:8080 \ + -v /data/bom-cache:/app/cache \ + -e CACHEDIRECTORY=/app/cache \ + --shm-size=1gb \ + --ipc=host \ + bom-local-service +``` + +#### Development Mode with Debug + +```bash +docker run -d \ + --name bom-local-service \ + -p 8082:8080 \ + -v $(pwd)/cache:/app/cache \ + -e ASPNETCORE_ENVIRONMENT=Development \ + -e DEBUG__ENABLED=true \ + -e DEBUG__WAITMS=5000 \ + --shm-size=1gb \ + --ipc=host \ + bom-local-service +``` + +#### Using docker-compose with Custom Settings + +Create a `.env` file in the project root: + +```env +HOST_PORT=8082 +CACHERETENTIONHOURS=48 +CACHEEXPIRATIONMINUTES=15 +CACHEMANAGEMENT__CHECKINTERVALMINUTES=10 +TIMEZONE=Australia/Sydney +``` + +Then run: +```bash +docker-compose up -d +``` + +## API Documentation + +The service provides RESTful API endpoints for accessing radar data and managing the cache. + +### Base URL + +When running locally with default settings: `http://localhost:8082` + +### Endpoints + +#### Get Radar Data + +Get the latest radar frames for a location. + +```http +GET /api/radar/{suburb}/{state} +``` + +**Parameters:** +- `suburb` (path): Suburb name (e.g., "Brisbane") +- `state` (path): State abbreviation (e.g., "QLD") + +**Response:** +```json +{ + "frames": [ + { + "frameIndex": 0, + "imageUrl": "/api/radar/Brisbane/QLD/frame/0", + "minutesAgo": 0, + "observationTime": "2025-01-15T10:00:00Z" + } + ], + "observationTime": "2025-01-15T10:00:00Z", + "forecastTime": "2025-01-15T10:00:00Z", + "weatherStation": "Brisbane", + "distance": "5.2 km", + "cacheIsValid": true, + "cacheExpiresAt": "2025-01-15T10:12:30Z", + "isUpdating": false, + "nextUpdateTime": "2025-01-15T10:12:30Z" +} +``` + +**Status Codes:** +- `200 OK`: Radar data available +- `404 Not Found`: Cache is being generated (check response for retry information) +- `400 Bad Request`: Invalid location parameters + +#### Get Frame Image + +Get a specific radar frame image. + +```http +GET /api/radar/{suburb}/{state}/frame/{frameIndex} +``` + +**Parameters:** +- `suburb` (path): Suburb name +- `state` (path): State abbreviation +- `frameIndex` (path): Frame index (0-6 for default 7 frames) +- `cacheFolder` (query, optional): Specific cache folder name for historical data + +**Response:** +- `200 OK`: PNG image +- `404 Not Found`: Frame not found + +#### Get Metadata + +Get metadata about cached radar data. + +```http +GET /api/radar/{suburb}/{state}/metadata +``` + +**Response:** +```json +{ + "lastUpdated": "2025-01-15T10:00:00Z", + "observationTime": "2025-01-15T10:00:00Z", + "forecastTime": "2025-01-15T10:00:00Z", + "weatherStation": "Brisbane", + "distance": "5.2 km" +} +``` + +#### Get Time Series + +Get historical radar data across multiple cache folders. + +```http +GET /api/radar/{suburb}/{state}/timeseries?startTime={iso8601}&endTime={iso8601} +``` + +**Parameters:** +- `suburb` (path): Suburb name +- `state` (path): State abbreviation +- `startTime` (query, optional): ISO 8601 start time (e.g., `2025-01-15T00:00:00Z`) +- `endTime` (query, optional): ISO 8601 end time (defaults to now) + +**Response:** +```json +{ + "cacheFolders": [ + { + "cacheFolderName": "Brisbane_QLD_20250115_100000", + "cacheTimestamp": "2025-01-15T10:00:00Z", + "observationTime": "2025-01-15T10:00:00Z", + "frames": [ + { + "frameIndex": 0, + "imageUrl": "/api/radar/Brisbane/QLD/frame/0?cacheFolder=Brisbane_QLD_20250115_100000", + "absoluteObservationTime": "2025-01-15T10:00:00Z", + "minutesAgo": 0 + } + ] + } + ], + "totalFrames": 7, + "timeSpanMinutes": 60 +} +``` + +#### Get Cache Range + +Get information about available historical cache data. + +```http +GET /api/cache/{suburb}/{state}/range +``` + +**Response:** +```json +{ + "oldestCache": { + "cacheFolderName": "Brisbane_QLD_20250115_000000", + "cacheTimestamp": "2025-01-15T00:00:00Z" + }, + "newestCache": { + "cacheFolderName": "Brisbane_QLD_20250115_100000", + "cacheTimestamp": "2025-01-15T10:00:00Z" + }, + "totalCacheFolders": 10, + "timeSpanMinutes": 600 +} +``` + +#### Refresh Cache + +Manually trigger a cache update for a location. + +```http +POST /api/cache/{suburb}/{state}/refresh +``` + +**Response:** +```json +{ + "updateTriggered": true, + "cacheIsValid": false, + "cacheExpiresAt": null, + "nextUpdateTime": "2025-01-15T10:12:30Z", + "message": "Cache update triggered" +} +``` + +#### Delete Cache + +Delete cached data for a location. + +```http +DELETE /api/cache/{suburb}/{state} +``` + +**Response:** +```json +{ + "message": "Cache deleted for Brisbane, QLD" +} +``` + +### OpenAPI Documentation + +When running in Development mode, OpenAPI documentation is available at: +``` +http://localhost:8082/openapi/v1.json +``` + +## Demo SPA + +The service includes a built-in Single Page Application (SPA) for testing and demonstration purposes. This provides a visual interface to: + +- View radar frames in a slideshow +- Test API endpoints +- Configure playback settings +- View historical data across extended time periods +- Monitor cache status and update information + +### Accessing the Demo + +Navigate to: +``` +http://localhost:8082/radar/{suburb}/{state} +``` + +**Example:** +``` +http://localhost:8082/radar/Brisbane/QLD +``` + +### Features + +- **Slideshow Playback**: Play, pause, and navigate through radar frames +- **Frame Navigation**: Use slider, buttons, or keyboard shortcuts (arrow keys, spacebar) +- **Extended Timespans**: View historical data from 1 hour to 24 hours +- **Custom Time Ranges**: Select specific start and end times for historical viewing +- **Auto-Refresh**: Automatically checks for new data at configurable intervals +- **Cache Status**: Real-time display of cache validity, expiration, and update status +- **Settings Panel**: Configure frame intervals, refresh rates, and playback options + +### Keyboard Shortcuts + +- `โ†` / `โ†’`: Navigate to previous/next frame +- `Shift + โ†` / `Shift + โ†’`: Jump back/forward 10 frames +- `Home` / `End`: Jump to first/last frame +- `Space`: Play/pause slideshow + +### Using the Demo for Integration Development + +The demo SPA serves as a reference implementation demonstrating best practices for consuming the API. Key implementation patterns: + +**Basic Radar Data Fetching**: The simplest pattern - fetch data and handle the case where cache is being generated: + +```javascript +async function getRadarData(suburb, state) { + const response = await fetch(`http://localhost:8082/api/radar/${suburb}/${state}`); + + if (response.status === 404) { + // Cache is being generated - trigger refresh and show message + const error = await response.json(); + if (error.refreshEndpoint) { + // Trigger cache update in background + fetch(error.refreshEndpoint, { method: 'POST' }).catch(() => {}); + } + return { frames: [], message: `Cache being generated. Retry in ${error.retryAfter} seconds.` }; + } + + if (!response.ok) { + throw new Error(`HTTP ${response.status}`); + } + + return await response.json(); +} +``` + +**Displaying Frames**: Frame objects include ready-to-use `imageUrl` properties: + +```javascript +const radarData = await getRadarData('Brisbane', 'QLD'); +if (radarData.frames && radarData.frames.length > 0) { + // Display first frame + document.getElementById('radar-image').src = radarData.frames[0].imageUrl; + + // Or loop through all frames for animation + radarData.frames.forEach((frame, index) => { + console.log(`Frame ${index}: ${frame.imageUrl} (${frame.minutesAgo} min ago)`); + }); +} +``` + +**Historical Data**: Fetch extended time series by specifying a time range: + +```javascript +async function getHistoricalRadar(suburb, state, hoursBack = 3) { + const endTime = new Date(); + const startTime = new Date(endTime.getTime() - (hoursBack * 60 * 60 * 1000)); + + const response = await fetch( + `/api/radar/${suburb}/${state}/timeseries?startTime=${startTime.toISOString()}&endTime=${endTime.toISOString()}` + ); + + if (!response.ok) { + throw new Error(`HTTP ${response.status}`); + } + + const data = await response.json(); + // Flatten frames from all cache folders + const allFrames = data.cacheFolders.flatMap(folder => folder.frames); + return allFrames; +} +``` + +**Auto-Refresh with Error Handling**: Implement periodic updates that handle API downtime: + +```javascript +let refreshInterval; + +function startAutoRefresh(suburb, state, intervalSeconds = 30) { + refreshInterval = setInterval(async () => { + try { + const data = await getRadarData(suburb, state); + if (data.frames && data.frames.length > 0) { + updateDisplay(data); + } + } catch (error) { + console.error('Failed to refresh:', error); + // Optionally stop auto-refresh on persistent errors + } + }, intervalSeconds * 1000); +} + +// Start auto-refresh +startAutoRefresh('Brisbane', 'QLD', 30); + +// Stop auto-refresh +if (refreshInterval) { + clearInterval(refreshInterval); +} +``` + +The complete implementation with all patterns is available in `Views/RadarTest/Index.cshtml` for reference. + +## Usage Examples + +### Home Assistant Integration + +To use this service with Home Assistant, you'll need to create a custom integration or use a REST sensor. Here's a basic example: + +```yaml +# configuration.yaml +rest: + - sensor: + name: "BOM Radar Brisbane" + resource: "http://localhost:8082/api/radar/Brisbane/QLD" + value_template: "{{ value_json.frames[0].imageUrl }}" + scan_interval: 300 +``` + +### cURL Examples + +**Get latest radar data:** +```bash +curl http://localhost:8082/api/radar/Brisbane/QLD +``` + +**Get specific frame image:** +```bash +curl http://localhost:8082/api/radar/Brisbane/QLD/frame/0 -o frame.png +``` + +**Trigger cache refresh:** +```bash +curl -X POST http://localhost:8082/api/cache/Brisbane/QLD/refresh +``` + +**Get historical data (last 3 hours):** +```bash +curl "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z" +``` + +## Troubleshooting + +### Service Won't Start + +- **Check Docker logs**: `docker logs bom-local-service` +- **Verify port availability**: Ensure port 8082 (or your configured port) is not in use +- **Shared memory configuration**: The docker-compose.yml includes `shm_size: '1gb'` and `ipc: host` which are required for Playwright. If running with `docker run`, ensure you include: + ```bash + docker run --shm-size=1gb --ipc=host ... + ``` +- **Browser launch failures**: If Playwright browsers fail to launch, check for seccomp security restrictions. You may need to add `--security-opt seccomp:unconfined` (though this is generally not needed with the included configuration) + +### No Radar Data Available + +- **Wait for initial cache**: First request triggers cache generation, which takes 30-60 seconds +- **Check location format**: Ensure suburb and state are correctly formatted (e.g., "Brisbane", "QLD") +- **Verify cache directory**: Check that the cache volume is mounted and writable +- **Check logs**: Look for errors in browser automation or data capture + +### Images Not Loading + +- **Verify frame URLs**: Check that frame image URLs are correctly formatted +- **Check cache status**: Use `/api/cache/{suburb}/{state}/range` to verify cache exists +- **Browser automation issues**: Check logs for Playwright errors + +### Playwright Resource Usage + +Playwright browsers (Chromium) can consume significant CPU and memory: + +**Limit CPU Cores**: Add CPU limits to your docker-compose.yml: +```yaml +services: + bom-local-service: + deploy: + resources: + limits: + cpus: '1.0' # Limit to 1 CPU core + reservations: + cpus: '0.5' # Reserve 0.5 cores +``` + +Or with `docker run`: +```bash +docker run --cpus="1.0" ... +``` + +**Common Playwright + Docker Issues**: + +- **Browser crashes with "out of memory"**: Ensure `shm_size: '1gb'` is set. Chromium uses `/dev/shm` for shared memory, and the default 64MB is insufficient. +- **High CPU usage during idle**: This is normal - Playwright browsers can consume CPU even when idle. Consider reducing `CACHEMANAGEMENT__CHECKINTERVALMINUTES` to check less frequently. +- **Browser processes not terminating**: Check logs for stuck browser processes. The service includes cleanup logic, but you may need to restart the container if processes hang. +- **"Protocol error" or connection failures**: Usually indicates insufficient shared memory or IPC namespace issues. Verify `ipc: host` is set in docker-compose.yml. + +**Optimize Cache Settings**: +- **Reduce retention**: Lower `CACHERETENTIONHOURS` to keep less data on disk +- **Increase cleanup frequency**: Lower `CACHECLEANUP__INTERVALHOURS` to clean up more often +- **Limit locations**: The service automatically manages all cached locations; reduce the number of locations being cached to lower resource usage + +## Development + +### Building Locally + +```bash +dotnet restore +dotnet build +dotnet run +``` + +### Debug Mode + +Enable debug mode to save intermediate screenshots during data capture: + +```bash +docker run -e DEBUG__ENABLED=true -e DEBUG__WAITMS=5000 ... +``` + +Debug screenshots are saved in `{CACHEDIRECTORY}/debug/`. + +## License + +MIT License - see [LICENSE](LICENSE) file for details. + +## Contributing + +This is a hobby project, but contributions are welcome! Feel free to open issues or submit pull requests. + +## Acknowledgments + +- Inspired by the [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) project +- Built with [Playwright](https://playwright.dev/) for browser automation +- Uses [ASP.NET Core](https://dotnet.microsoft.com/) for the web framework + +## Disclaimer + +This project is intended for **local, personal use only**. The radar data and images cached by this service are the property of the Australian Bureau of Meteorology (BOM) and are subject to copyright. + +**Important Notes:** +- This service is designed to run on your local network for personal use +- Do not redistribute or republish BOM radar data or images +- Respect BOM's terms of service and data usage policies +- The service caches publicly available data for local consumption only +- This project is not affiliated with or endorsed by the Australian Bureau of Meteorology + +For official BOM data and services, visit [bom.gov.au](https://www.bom.gov.au/).