Add GitHub Actions workflow for Docker image publishing on releases

2026-10-04 03:28:11 +10:00 · 2025-12-09 02:33:01 +10:00
parent 2d9d45d430
commit ec81fc1da6
3 changed files with 751 additions and 0 deletions
@@ -0,0 +1,694 @@
+# BOM Local Service
+
+A local caching service for Australian Bureau of Meteorology (BOM) radar data, designed to provide a reliable API for Home Assistant integrations and other local services.
+
+## Background
+
+The Australian Bureau of Meteorology's radar API endpoint (`https://api.weather.bom.gov.au/v1/radar/capabilities`) stopped working in December 2024, returning errors. This broke integrations like the popular [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) for Home Assistant, which can no longer render due to the Cross-Origin Request Blocked error. 
+
+The issue was reported in [GitHub issue #40](https://github.com/Makin-Things/bom-radar-card/issues/40), where multiple users confirmed the card completely fails to render.
+
+This project was created to bridge that gap by:
+- Caching radar data from the BOM website
+- Storing it locally in a structured format
+- Providing a simple REST API for local services to consume
+- Automatically managing cache updates and cleanup
+
+## Features
+
+- 🌧️ **Automatic Cache Management**: Background service automatically updates radar data for all cached locations
+- 📊 **Historical Data**: Access historical radar frames across multiple time periods
+- 🎯 **Location-Based**: Support for any Australian suburb/state combination
+- 🖼️ **Image API**: Direct access to individual radar frame images
+- 🔄 **Auto-Refresh**: Configurable cache expiration and refresh intervals
+- 🧹 **Automatic Cleanup**: Old cache data is automatically purged based on retention settings
+- 🎨 **Demo SPA**: Built-in web interface for testing and demonstration
+- 🐳 **Easy Docker Deployment**: Quick setup with Docker
+
+## Architecture
+
+Built on ASP.NET Core 9.0, the service uses a service-oriented architecture with clear separation of concerns:
+
+### Core Services
+- **BomRadarService**: Main orchestrator that coordinates cache operations, browser automation, and data retrieval
+- **CacheService**: Manages file-based storage of radar screenshots and metadata in organized directory structures
+- **BrowserService**: Handles Playwright browser automation for headless browser sessions
+- **ScrapingService**: Performs web scraping operations using the browser service to navigate and capture radar data from the BOM website
+- **TimeParsingService**: Parses and converts time formats from BOM data
+- **DebugService**: Provides debug functionality for troubleshooting
+
+### Background Services
+- **CacheManagementService**: Periodically checks cache validity for all cached locations and triggers updates when data expires
+- **CacheCleanupService**: Removes cache files older than the configured retention period
+
+### API Layer
+- **RadarController**: REST endpoints for accessing radar data (`/api/radar/{suburb}/{state}`)
+- **CacheController**: REST endpoints for cache management operations (`/api/cache/{suburb}/{state}`)
+- **RadarTestController**: MVC controller serving the demo SPA at `/radar/{suburb}/{state}`
+
+## Building with Docker
+
+### Prerequisites
+
+- Docker Engine 20.10+ or Docker Desktop
+- Docker Compose v2.0+ (optional, for easier management)
+
+### Quick Start
+
+1. **Clone the repository**:
+   ```bash
+   git clone <repository-url>
+   cd bom-local-service
+   ```
+
+2. **Build the Docker image**:
+   ```bash
+   docker build -t bom-local-service .
+   ```
+
+   The build process uses a multi-stage Dockerfile:
+   - **Build stage**: Compiles the .NET application
+   - **Runtime stage**: Uses the official Playwright .NET image with browsers pre-installed
+   - Sets up a virtual display (Xvfb) for headless browser operation
+
+3. **Run the container**:
+   ```bash
+   docker run -d \
+     --name bom-local-service \
+     -p 8082:8080 \
+     -v $(pwd)/cache:/app/cache \
+     --shm-size=1gb \
+     --ipc=host \
+     bom-local-service
+   ```
+
+   **Note**: The `--shm-size=1gb` and `--ipc=host` flags are important for Playwright to function correctly in Docker.
+
+### Using Docker Compose
+
+The included `docker-compose.yml` provides a convenient way to run the service with all configuration options:
+
+```bash
+docker-compose up -d
+```
+
+This will:
+- Build the image if it doesn't exist
+- Start the service on port 8082 (configurable via `HOST_PORT`)
+- Mount the `./cache` directory for persistent storage
+- Apply all environment variable configurations
+
+To view logs:
+```bash
+docker-compose logs -f
+```
+
+To stop the service:
+```bash
+docker-compose down
+```
+
+## Configuration
+
+All configuration can be done via environment variables, which override the default values in `appsettings.json`. The service uses ASP.NET Core's configuration system, which supports nested configuration via double underscores (`__`).
+
+### Environment Variables
+
+#### ASP.NET Core Configuration
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `ASPNETCORE_ENVIRONMENT` | Runtime environment (Development/Production) | `Production` |
+| `ASPNETCORE_URLS` | URLs the service listens on | `http://+:8080` |
+| `ENABLE_HTTPS_REDIRECTION` | Enable HTTPS redirection | `false` |
+
+#### Application Configuration
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `CACHEDIRECTORY` | Directory path for cache storage | `/app/cache` | `/data/bom-cache` |
+| `CACHERETENTIONHOURS` | Hours to retain cached data before cleanup | `24` | `48` |
+| `CACHEEXPIRATIONMINUTES` | Minutes before cache is considered expired | `12.5` | `15` |
+| `TIMEZONE` | Timezone for time parsing (IANA format) | `Australia/Brisbane` | `Australia/Sydney` |
+
+#### Cache Management
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `CACHEMANAGEMENT__CHECKINTERVALMINUTES` | Interval between cache validity checks | `5` | `10` |
+| `CACHEMANAGEMENT__INITIALDELAYSECONDS` | Delay before first cache check on startup | `10` | `30` |
+| `CACHEMANAGEMENT__UPDATESTAGGERSECONDS` | Delay between triggering cache updates | `2` | `5` |
+| `CACHEMANAGEMENT__LOCATIONSTAGGERSECONDS` | Delay between processing different locations | `1` | `2` |
+
+#### Cache Cleanup
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `CACHECLEANUP__INTERVALHOURS` | Interval between cleanup runs | `1` | `2` |
+
+#### Screenshot Configuration
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `SCREENSHOT__DYNAMICCONTENTWAITMS` | Milliseconds to wait for dynamic content to load | `2000` | `3000` |
+| `SCREENSHOT__TILERENDERWAITMS` | Milliseconds to wait for map tiles to render | `5000` | `7000` |
+
+#### Debug Configuration
+
+| Variable | Description | Default | Example |
+|----------|-------------|---------|---------|
+| `DEBUG__ENABLED` | Enable debug mode (saves debug screenshots) | `false` | `true` |
+| `DEBUG__WAITMS` | Additional wait time in debug mode | `2000` | `5000` |
+
+#### Docker Compose Port Configuration
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `HOST_PORT` | Port on host machine to map to container | `8082` |
+| `CONTAINER_PORT` | Port inside container (usually 8080) | `8080` |
+
+### Configuration Examples
+
+#### Basic Configuration
+
+```bash
+docker run -d \
+  --name bom-local-service \
+  -p 8082:8080 \
+  -v $(pwd)/cache:/app/cache \
+  -e CACHERETENTIONHOURS=48 \
+  -e CACHEEXPIRATIONMINUTES=15 \
+  --shm-size=1gb \
+  --ipc=host \
+  bom-local-service
+```
+
+#### Custom Cache Directory
+
+```bash
+docker run -d \
+  --name bom-local-service \
+  -p 8082:8080 \
+  -v /data/bom-cache:/app/cache \
+  -e CACHEDIRECTORY=/app/cache \
+  --shm-size=1gb \
+  --ipc=host \
+  bom-local-service
+```
+
+#### Development Mode with Debug
+
+```bash
+docker run -d \
+  --name bom-local-service \
+  -p 8082:8080 \
+  -v $(pwd)/cache:/app/cache \
+  -e ASPNETCORE_ENVIRONMENT=Development \
+  -e DEBUG__ENABLED=true \
+  -e DEBUG__WAITMS=5000 \
+  --shm-size=1gb \
+  --ipc=host \
+  bom-local-service
+```
+
+#### Using docker-compose with Custom Settings
+
+Create a `.env` file in the project root:
+
+```env
+HOST_PORT=8082
+CACHERETENTIONHOURS=48
+CACHEEXPIRATIONMINUTES=15
+CACHEMANAGEMENT__CHECKINTERVALMINUTES=10
+TIMEZONE=Australia/Sydney
+```
+
+Then run:
+```bash
+docker-compose up -d
+```
+
+## API Documentation
+
+The service provides RESTful API endpoints for accessing radar data and managing the cache.
+
+### Base URL
+
+When running locally with default settings: `http://localhost:8082`
+
+### Endpoints
+
+#### Get Radar Data
+
+Get the latest radar frames for a location.
+
+```http
+GET /api/radar/{suburb}/{state}
+```
+
+**Parameters:**
+- `suburb` (path): Suburb name (e.g., "Brisbane")
+- `state` (path): State abbreviation (e.g., "QLD")
+
+**Response:**
+```json
+{
+  "frames": [
+    {
+      "frameIndex": 0,
+      "imageUrl": "/api/radar/Brisbane/QLD/frame/0",
+      "minutesAgo": 0,
+      "observationTime": "2025-01-15T10:00:00Z"
+    }
+  ],
+  "observationTime": "2025-01-15T10:00:00Z",
+  "forecastTime": "2025-01-15T10:00:00Z",
+  "weatherStation": "Brisbane",
+  "distance": "5.2 km",
+  "cacheIsValid": true,
+  "cacheExpiresAt": "2025-01-15T10:12:30Z",
+  "isUpdating": false,
+  "nextUpdateTime": "2025-01-15T10:12:30Z"
+}
+```
+
+**Status Codes:**
+- `200 OK`: Radar data available
+- `404 Not Found`: Cache is being generated (check response for retry information)
+- `400 Bad Request`: Invalid location parameters
+
+#### Get Frame Image
+
+Get a specific radar frame image.
+
+```http
+GET /api/radar/{suburb}/{state}/frame/{frameIndex}
+```
+
+**Parameters:**
+- `suburb` (path): Suburb name
+- `state` (path): State abbreviation
+- `frameIndex` (path): Frame index (0-6 for default 7 frames)
+- `cacheFolder` (query, optional): Specific cache folder name for historical data
+
+**Response:**
+- `200 OK`: PNG image
+- `404 Not Found`: Frame not found
+
+#### Get Metadata
+
+Get metadata about cached radar data.
+
+```http
+GET /api/radar/{suburb}/{state}/metadata
+```
+
+**Response:**
+```json
+{
+  "lastUpdated": "2025-01-15T10:00:00Z",
+  "observationTime": "2025-01-15T10:00:00Z",
+  "forecastTime": "2025-01-15T10:00:00Z",
+  "weatherStation": "Brisbane",
+  "distance": "5.2 km"
+}
+```
+
+#### Get Time Series
+
+Get historical radar data across multiple cache folders.
+
+```http
+GET /api/radar/{suburb}/{state}/timeseries?startTime={iso8601}&endTime={iso8601}
+```
+
+**Parameters:**
+- `suburb` (path): Suburb name
+- `state` (path): State abbreviation
+- `startTime` (query, optional): ISO 8601 start time (e.g., `2025-01-15T00:00:00Z`)
+- `endTime` (query, optional): ISO 8601 end time (defaults to now)
+
+**Response:**
+```json
+{
+  "cacheFolders": [
+    {
+      "cacheFolderName": "Brisbane_QLD_20250115_100000",
+      "cacheTimestamp": "2025-01-15T10:00:00Z",
+      "observationTime": "2025-01-15T10:00:00Z",
+      "frames": [
+        {
+          "frameIndex": 0,
+          "imageUrl": "/api/radar/Brisbane/QLD/frame/0?cacheFolder=Brisbane_QLD_20250115_100000",
+          "absoluteObservationTime": "2025-01-15T10:00:00Z",
+          "minutesAgo": 0
+        }
+      ]
+    }
+  ],
+  "totalFrames": 7,
+  "timeSpanMinutes": 60
+}
+```
+
+#### Get Cache Range
+
+Get information about available historical cache data.
+
+```http
+GET /api/cache/{suburb}/{state}/range
+```
+
+**Response:**
+```json
+{
+  "oldestCache": {
+    "cacheFolderName": "Brisbane_QLD_20250115_000000",
+    "cacheTimestamp": "2025-01-15T00:00:00Z"
+  },
+  "newestCache": {
+    "cacheFolderName": "Brisbane_QLD_20250115_100000",
+    "cacheTimestamp": "2025-01-15T10:00:00Z"
+  },
+  "totalCacheFolders": 10,
+  "timeSpanMinutes": 600
+}
+```
+
+#### Refresh Cache
+
+Manually trigger a cache update for a location.
+
+```http
+POST /api/cache/{suburb}/{state}/refresh
+```
+
+**Response:**
+```json
+{
+  "updateTriggered": true,
+  "cacheIsValid": false,
+  "cacheExpiresAt": null,
+  "nextUpdateTime": "2025-01-15T10:12:30Z",
+  "message": "Cache update triggered"
+}
+```
+
+#### Delete Cache
+
+Delete cached data for a location.
+
+```http
+DELETE /api/cache/{suburb}/{state}
+```
+
+**Response:**
+```json
+{
+  "message": "Cache deleted for Brisbane, QLD"
+}
+```
+
+### OpenAPI Documentation
+
+When running in Development mode, OpenAPI documentation is available at:
+```
+http://localhost:8082/openapi/v1.json
+```
+
+## Demo SPA
+
+The service includes a built-in Single Page Application (SPA) for testing and demonstration purposes. This provides a visual interface to:
+
+- View radar frames in a slideshow
+- Test API endpoints
+- Configure playback settings
+- View historical data across extended time periods
+- Monitor cache status and update information
+
+### Accessing the Demo
+
+Navigate to:
+```
+http://localhost:8082/radar/{suburb}/{state}
+```
+
+**Example:**
+```
+http://localhost:8082/radar/Brisbane/QLD
+```
+
+### Features
+
+- **Slideshow Playback**: Play, pause, and navigate through radar frames
+- **Frame Navigation**: Use slider, buttons, or keyboard shortcuts (arrow keys, spacebar)
+- **Extended Timespans**: View historical data from 1 hour to 24 hours
+- **Custom Time Ranges**: Select specific start and end times for historical viewing
+- **Auto-Refresh**: Automatically checks for new data at configurable intervals
+- **Cache Status**: Real-time display of cache validity, expiration, and update status
+- **Settings Panel**: Configure frame intervals, refresh rates, and playback options
+
+### Keyboard Shortcuts
+
+- `←` / `→`: Navigate to previous/next frame
+- `Shift + ←` / `Shift + →`: Jump back/forward 10 frames
+- `Home` / `End`: Jump to first/last frame
+- `Space`: Play/pause slideshow
+
+### Using the Demo for Integration Development
+
+The demo SPA serves as a reference implementation demonstrating best practices for consuming the API. Key implementation patterns:
+
+**Basic Radar Data Fetching**: The simplest pattern - fetch data and handle the case where cache is being generated:
+
+```javascript
+async function getRadarData(suburb, state) {
+    const response = await fetch(`http://localhost:8082/api/radar/${suburb}/${state}`);
+    
+    if (response.status === 404) {
+        // Cache is being generated - trigger refresh and show message
+        const error = await response.json();
+        if (error.refreshEndpoint) {
+            // Trigger cache update in background
+            fetch(error.refreshEndpoint, { method: 'POST' }).catch(() => {});
+        }
+        return { frames: [], message: `Cache being generated. Retry in ${error.retryAfter} seconds.` };
+    }
+    
+    if (!response.ok) {
+        throw new Error(`HTTP ${response.status}`);
+    }
+    
+    return await response.json();
+}
+```
+
+**Displaying Frames**: Frame objects include ready-to-use `imageUrl` properties:
+
+```javascript
+const radarData = await getRadarData('Brisbane', 'QLD');
+if (radarData.frames && radarData.frames.length > 0) {
+    // Display first frame
+    document.getElementById('radar-image').src = radarData.frames[0].imageUrl;
+    
+    // Or loop through all frames for animation
+    radarData.frames.forEach((frame, index) => {
+        console.log(`Frame ${index}: ${frame.imageUrl} (${frame.minutesAgo} min ago)`);
+    });
+}
+```
+
+**Historical Data**: Fetch extended time series by specifying a time range:
+
+```javascript
+async function getHistoricalRadar(suburb, state, hoursBack = 3) {
+    const endTime = new Date();
+    const startTime = new Date(endTime.getTime() - (hoursBack * 60 * 60 * 1000));
+    
+    const response = await fetch(
+        `/api/radar/${suburb}/${state}/timeseries?startTime=${startTime.toISOString()}&endTime=${endTime.toISOString()}`
+    );
+    
+    if (!response.ok) {
+        throw new Error(`HTTP ${response.status}`);
+    }
+    
+    const data = await response.json();
+    // Flatten frames from all cache folders
+    const allFrames = data.cacheFolders.flatMap(folder => folder.frames);
+    return allFrames;
+}
+```
+
+**Auto-Refresh with Error Handling**: Implement periodic updates that handle API downtime:
+
+```javascript
+let refreshInterval;
+
+function startAutoRefresh(suburb, state, intervalSeconds = 30) {
+    refreshInterval = setInterval(async () => {
+        try {
+            const data = await getRadarData(suburb, state);
+            if (data.frames && data.frames.length > 0) {
+                updateDisplay(data);
+            }
+        } catch (error) {
+            console.error('Failed to refresh:', error);
+            // Optionally stop auto-refresh on persistent errors
+        }
+    }, intervalSeconds * 1000);
+}
+
+// Start auto-refresh
+startAutoRefresh('Brisbane', 'QLD', 30);
+
+// Stop auto-refresh
+if (refreshInterval) {
+    clearInterval(refreshInterval);
+}
+```
+
+The complete implementation with all patterns is available in `Views/RadarTest/Index.cshtml` for reference.
+
+## Usage Examples
+
+### Home Assistant Integration
+
+To use this service with Home Assistant, you'll need to create a custom integration or use a REST sensor. Here's a basic example:
+
+```yaml
+# configuration.yaml
+rest:
+  - sensor:
+      name: "BOM Radar Brisbane"
+      resource: "http://localhost:8082/api/radar/Brisbane/QLD"
+      value_template: "{{ value_json.frames[0].imageUrl }}"
+      scan_interval: 300
+```
+
+### cURL Examples
+
+**Get latest radar data:**
+```bash
+curl http://localhost:8082/api/radar/Brisbane/QLD
+```
+
+**Get specific frame image:**
+```bash
+curl http://localhost:8082/api/radar/Brisbane/QLD/frame/0 -o frame.png
+```
+
+**Trigger cache refresh:**
+```bash
+curl -X POST http://localhost:8082/api/cache/Brisbane/QLD/refresh
+```
+
+**Get historical data (last 3 hours):**
+```bash
+curl "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z"
+```
+
+## Troubleshooting
+
+### Service Won't Start
+
+- **Check Docker logs**: `docker logs bom-local-service`
+- **Verify port availability**: Ensure port 8082 (or your configured port) is not in use
+- **Shared memory configuration**: The docker-compose.yml includes `shm_size: '1gb'` and `ipc: host` which are required for Playwright. If running with `docker run`, ensure you include:
+  ```bash
+  docker run --shm-size=1gb --ipc=host ...
+  ```
+- **Browser launch failures**: If Playwright browsers fail to launch, check for seccomp security restrictions. You may need to add `--security-opt seccomp:unconfined` (though this is generally not needed with the included configuration)
+
+### No Radar Data Available
+
+- **Wait for initial cache**: First request triggers cache generation, which takes 30-60 seconds
+- **Check location format**: Ensure suburb and state are correctly formatted (e.g., "Brisbane", "QLD")
+- **Verify cache directory**: Check that the cache volume is mounted and writable
+- **Check logs**: Look for errors in browser automation or data capture
+
+### Images Not Loading
+
+- **Verify frame URLs**: Check that frame image URLs are correctly formatted
+- **Check cache status**: Use `/api/cache/{suburb}/{state}/range` to verify cache exists
+- **Browser automation issues**: Check logs for Playwright errors
+
+### Playwright Resource Usage
+
+Playwright browsers (Chromium) can consume significant CPU and memory:
+
+**Limit CPU Cores**: Add CPU limits to your docker-compose.yml:
+```yaml
+services:
+  bom-local-service:
+    deploy:
+      resources:
+        limits:
+          cpus: '1.0'  # Limit to 1 CPU core
+        reservations:
+          cpus: '0.5'  # Reserve 0.5 cores
+```
+
+Or with `docker run`:
+```bash
+docker run --cpus="1.0" ...
+```
+
+**Common Playwright + Docker Issues**:
+
+- **Browser crashes with "out of memory"**: Ensure `shm_size: '1gb'` is set. Chromium uses `/dev/shm` for shared memory, and the default 64MB is insufficient.
+- **High CPU usage during idle**: This is normal - Playwright browsers can consume CPU even when idle. Consider reducing `CACHEMANAGEMENT__CHECKINTERVALMINUTES` to check less frequently.
+- **Browser processes not terminating**: Check logs for stuck browser processes. The service includes cleanup logic, but you may need to restart the container if processes hang.
+- **"Protocol error" or connection failures**: Usually indicates insufficient shared memory or IPC namespace issues. Verify `ipc: host` is set in docker-compose.yml.
+
+**Optimize Cache Settings**:
+- **Reduce retention**: Lower `CACHERETENTIONHOURS` to keep less data on disk
+- **Increase cleanup frequency**: Lower `CACHECLEANUP__INTERVALHOURS` to clean up more often
+- **Limit locations**: The service automatically manages all cached locations; reduce the number of locations being cached to lower resource usage
+
+## Development
+
+### Building Locally
+
+```bash
+dotnet restore
+dotnet build
+dotnet run
+```
+
+### Debug Mode
+
+Enable debug mode to save intermediate screenshots during data capture:
+
+```bash
+docker run -e DEBUG__ENABLED=true -e DEBUG__WAITMS=5000 ...
+```
+
+Debug screenshots are saved in `{CACHEDIRECTORY}/debug/`.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Contributing
+
+This is a hobby project, but contributions are welcome! Feel free to open issues or submit pull requests.
+
+## Acknowledgments
+
+- Inspired by the [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) project
+- Built with [Playwright](https://playwright.dev/) for browser automation
+- Uses [ASP.NET Core](https://dotnet.microsoft.com/) for the web framework
+
+## Disclaimer
+
+This project is intended for **local, personal use only**. The radar data and images cached by this service are the property of the Australian Bureau of Meteorology (BOM) and are subject to copyright. 
+
+**Important Notes:**
+- This service is designed to run on your local network for personal use
+- Do not redistribute or republish BOM radar data or images
+- Respect BOM's terms of service and data usage policies
+- The service caches publicly available data for local consumption only
+- This project is not affiliated with or endorsed by the Australian Bureau of Meteorology
+
+For official BOM data and services, visit [bom.gov.au](https://www.bom.gov.au/).