Add GitHub Actions workflow for Docker image publishing on releases

This commit is contained in:
2025-12-09 02:33:01 +10:00
parent 2d9d45d430
commit ec81fc1da6
3 changed files with 751 additions and 0 deletions
@@ -0,0 +1,52 @@
name: Publish Docker image
on:
release:
types: [published]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
attestations: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v5
- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
- name: Build and push Docker image
id: push
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
- name: Generate artifact attestation
uses: actions/attest-build-provenance@v3
with:
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true
+5
View File
@@ -41,6 +41,11 @@ export DISPLAY=:99\n\
exec dotnet BomLocalService.dll' > /app/start.sh && \ exec dotnet BomLocalService.dll' > /app/start.sh && \
chmod +x /app/start.sh chmod +x /app/start.sh
# OCI labels for GitHub Container Registry metadata
LABEL org.opencontainers.image.source="https://github.com/alexhopeoconnor/bom-local-service"
LABEL org.opencontainers.image.description="BOM Local Service - Australian Bureau of Meteorology radar data caching service for Home Assistant and local services"
LABEL org.opencontainers.image.licenses=MIT
EXPOSE 8080 EXPOSE 8080
ENTRYPOINT ["/app/start.sh"] ENTRYPOINT ["/app/start.sh"]
+694
View File
@@ -0,0 +1,694 @@
# BOM Local Service
A local caching service for Australian Bureau of Meteorology (BOM) radar data, designed to provide a reliable API for Home Assistant integrations and other local services.
## Background
The Australian Bureau of Meteorology's radar API endpoint (`https://api.weather.bom.gov.au/v1/radar/capabilities`) stopped working in December 2024, returning errors. This broke integrations like the popular [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) for Home Assistant, which can no longer render due to the Cross-Origin Request Blocked error.
The issue was reported in [GitHub issue #40](https://github.com/Makin-Things/bom-radar-card/issues/40), where multiple users confirmed the card completely fails to render.
This project was created to bridge that gap by:
- Caching radar data from the BOM website
- Storing it locally in a structured format
- Providing a simple REST API for local services to consume
- Automatically managing cache updates and cleanup
## Features
- 🌧️ **Automatic Cache Management**: Background service automatically updates radar data for all cached locations
- 📊 **Historical Data**: Access historical radar frames across multiple time periods
- 🎯 **Location-Based**: Support for any Australian suburb/state combination
- 🖼️ **Image API**: Direct access to individual radar frame images
- 🔄 **Auto-Refresh**: Configurable cache expiration and refresh intervals
- 🧹 **Automatic Cleanup**: Old cache data is automatically purged based on retention settings
- 🎨 **Demo SPA**: Built-in web interface for testing and demonstration
- 🐳 **Easy Docker Deployment**: Quick setup with Docker
## Architecture
Built on ASP.NET Core 9.0, the service uses a service-oriented architecture with clear separation of concerns:
### Core Services
- **BomRadarService**: Main orchestrator that coordinates cache operations, browser automation, and data retrieval
- **CacheService**: Manages file-based storage of radar screenshots and metadata in organized directory structures
- **BrowserService**: Handles Playwright browser automation for headless browser sessions
- **ScrapingService**: Performs web scraping operations using the browser service to navigate and capture radar data from the BOM website
- **TimeParsingService**: Parses and converts time formats from BOM data
- **DebugService**: Provides debug functionality for troubleshooting
### Background Services
- **CacheManagementService**: Periodically checks cache validity for all cached locations and triggers updates when data expires
- **CacheCleanupService**: Removes cache files older than the configured retention period
### API Layer
- **RadarController**: REST endpoints for accessing radar data (`/api/radar/{suburb}/{state}`)
- **CacheController**: REST endpoints for cache management operations (`/api/cache/{suburb}/{state}`)
- **RadarTestController**: MVC controller serving the demo SPA at `/radar/{suburb}/{state}`
## Building with Docker
### Prerequisites
- Docker Engine 20.10+ or Docker Desktop
- Docker Compose v2.0+ (optional, for easier management)
### Quick Start
1. **Clone the repository**:
```bash
git clone <repository-url>
cd bom-local-service
```
2. **Build the Docker image**:
```bash
docker build -t bom-local-service .
```
The build process uses a multi-stage Dockerfile:
- **Build stage**: Compiles the .NET application
- **Runtime stage**: Uses the official Playwright .NET image with browsers pre-installed
- Sets up a virtual display (Xvfb) for headless browser operation
3. **Run the container**:
```bash
docker run -d \
--name bom-local-service \
-p 8082:8080 \
-v $(pwd)/cache:/app/cache \
--shm-size=1gb \
--ipc=host \
bom-local-service
```
**Note**: The `--shm-size=1gb` and `--ipc=host` flags are important for Playwright to function correctly in Docker.
### Using Docker Compose
The included `docker-compose.yml` provides a convenient way to run the service with all configuration options:
```bash
docker-compose up -d
```
This will:
- Build the image if it doesn't exist
- Start the service on port 8082 (configurable via `HOST_PORT`)
- Mount the `./cache` directory for persistent storage
- Apply all environment variable configurations
To view logs:
```bash
docker-compose logs -f
```
To stop the service:
```bash
docker-compose down
```
## Configuration
All configuration can be done via environment variables, which override the default values in `appsettings.json`. The service uses ASP.NET Core's configuration system, which supports nested configuration via double underscores (`__`).
### Environment Variables
#### ASP.NET Core Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `ASPNETCORE_ENVIRONMENT` | Runtime environment (Development/Production) | `Production` |
| `ASPNETCORE_URLS` | URLs the service listens on | `http://+:8080` |
| `ENABLE_HTTPS_REDIRECTION` | Enable HTTPS redirection | `false` |
#### Application Configuration
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `CACHEDIRECTORY` | Directory path for cache storage | `/app/cache` | `/data/bom-cache` |
| `CACHERETENTIONHOURS` | Hours to retain cached data before cleanup | `24` | `48` |
| `CACHEEXPIRATIONMINUTES` | Minutes before cache is considered expired | `12.5` | `15` |
| `TIMEZONE` | Timezone for time parsing (IANA format) | `Australia/Brisbane` | `Australia/Sydney` |
#### Cache Management
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `CACHEMANAGEMENT__CHECKINTERVALMINUTES` | Interval between cache validity checks | `5` | `10` |
| `CACHEMANAGEMENT__INITIALDELAYSECONDS` | Delay before first cache check on startup | `10` | `30` |
| `CACHEMANAGEMENT__UPDATESTAGGERSECONDS` | Delay between triggering cache updates | `2` | `5` |
| `CACHEMANAGEMENT__LOCATIONSTAGGERSECONDS` | Delay between processing different locations | `1` | `2` |
#### Cache Cleanup
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `CACHECLEANUP__INTERVALHOURS` | Interval between cleanup runs | `1` | `2` |
#### Screenshot Configuration
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `SCREENSHOT__DYNAMICCONTENTWAITMS` | Milliseconds to wait for dynamic content to load | `2000` | `3000` |
| `SCREENSHOT__TILERENDERWAITMS` | Milliseconds to wait for map tiles to render | `5000` | `7000` |
#### Debug Configuration
| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `DEBUG__ENABLED` | Enable debug mode (saves debug screenshots) | `false` | `true` |
| `DEBUG__WAITMS` | Additional wait time in debug mode | `2000` | `5000` |
#### Docker Compose Port Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `HOST_PORT` | Port on host machine to map to container | `8082` |
| `CONTAINER_PORT` | Port inside container (usually 8080) | `8080` |
### Configuration Examples
#### Basic Configuration
```bash
docker run -d \
--name bom-local-service \
-p 8082:8080 \
-v $(pwd)/cache:/app/cache \
-e CACHERETENTIONHOURS=48 \
-e CACHEEXPIRATIONMINUTES=15 \
--shm-size=1gb \
--ipc=host \
bom-local-service
```
#### Custom Cache Directory
```bash
docker run -d \
--name bom-local-service \
-p 8082:8080 \
-v /data/bom-cache:/app/cache \
-e CACHEDIRECTORY=/app/cache \
--shm-size=1gb \
--ipc=host \
bom-local-service
```
#### Development Mode with Debug
```bash
docker run -d \
--name bom-local-service \
-p 8082:8080 \
-v $(pwd)/cache:/app/cache \
-e ASPNETCORE_ENVIRONMENT=Development \
-e DEBUG__ENABLED=true \
-e DEBUG__WAITMS=5000 \
--shm-size=1gb \
--ipc=host \
bom-local-service
```
#### Using docker-compose with Custom Settings
Create a `.env` file in the project root:
```env
HOST_PORT=8082
CACHERETENTIONHOURS=48
CACHEEXPIRATIONMINUTES=15
CACHEMANAGEMENT__CHECKINTERVALMINUTES=10
TIMEZONE=Australia/Sydney
```
Then run:
```bash
docker-compose up -d
```
## API Documentation
The service provides RESTful API endpoints for accessing radar data and managing the cache.
### Base URL
When running locally with default settings: `http://localhost:8082`
### Endpoints
#### Get Radar Data
Get the latest radar frames for a location.
```http
GET /api/radar/{suburb}/{state}
```
**Parameters:**
- `suburb` (path): Suburb name (e.g., "Brisbane")
- `state` (path): State abbreviation (e.g., "QLD")
**Response:**
```json
{
"frames": [
{
"frameIndex": 0,
"imageUrl": "/api/radar/Brisbane/QLD/frame/0",
"minutesAgo": 0,
"observationTime": "2025-01-15T10:00:00Z"
}
],
"observationTime": "2025-01-15T10:00:00Z",
"forecastTime": "2025-01-15T10:00:00Z",
"weatherStation": "Brisbane",
"distance": "5.2 km",
"cacheIsValid": true,
"cacheExpiresAt": "2025-01-15T10:12:30Z",
"isUpdating": false,
"nextUpdateTime": "2025-01-15T10:12:30Z"
}
```
**Status Codes:**
- `200 OK`: Radar data available
- `404 Not Found`: Cache is being generated (check response for retry information)
- `400 Bad Request`: Invalid location parameters
#### Get Frame Image
Get a specific radar frame image.
```http
GET /api/radar/{suburb}/{state}/frame/{frameIndex}
```
**Parameters:**
- `suburb` (path): Suburb name
- `state` (path): State abbreviation
- `frameIndex` (path): Frame index (0-6 for default 7 frames)
- `cacheFolder` (query, optional): Specific cache folder name for historical data
**Response:**
- `200 OK`: PNG image
- `404 Not Found`: Frame not found
#### Get Metadata
Get metadata about cached radar data.
```http
GET /api/radar/{suburb}/{state}/metadata
```
**Response:**
```json
{
"lastUpdated": "2025-01-15T10:00:00Z",
"observationTime": "2025-01-15T10:00:00Z",
"forecastTime": "2025-01-15T10:00:00Z",
"weatherStation": "Brisbane",
"distance": "5.2 km"
}
```
#### Get Time Series
Get historical radar data across multiple cache folders.
```http
GET /api/radar/{suburb}/{state}/timeseries?startTime={iso8601}&endTime={iso8601}
```
**Parameters:**
- `suburb` (path): Suburb name
- `state` (path): State abbreviation
- `startTime` (query, optional): ISO 8601 start time (e.g., `2025-01-15T00:00:00Z`)
- `endTime` (query, optional): ISO 8601 end time (defaults to now)
**Response:**
```json
{
"cacheFolders": [
{
"cacheFolderName": "Brisbane_QLD_20250115_100000",
"cacheTimestamp": "2025-01-15T10:00:00Z",
"observationTime": "2025-01-15T10:00:00Z",
"frames": [
{
"frameIndex": 0,
"imageUrl": "/api/radar/Brisbane/QLD/frame/0?cacheFolder=Brisbane_QLD_20250115_100000",
"absoluteObservationTime": "2025-01-15T10:00:00Z",
"minutesAgo": 0
}
]
}
],
"totalFrames": 7,
"timeSpanMinutes": 60
}
```
#### Get Cache Range
Get information about available historical cache data.
```http
GET /api/cache/{suburb}/{state}/range
```
**Response:**
```json
{
"oldestCache": {
"cacheFolderName": "Brisbane_QLD_20250115_000000",
"cacheTimestamp": "2025-01-15T00:00:00Z"
},
"newestCache": {
"cacheFolderName": "Brisbane_QLD_20250115_100000",
"cacheTimestamp": "2025-01-15T10:00:00Z"
},
"totalCacheFolders": 10,
"timeSpanMinutes": 600
}
```
#### Refresh Cache
Manually trigger a cache update for a location.
```http
POST /api/cache/{suburb}/{state}/refresh
```
**Response:**
```json
{
"updateTriggered": true,
"cacheIsValid": false,
"cacheExpiresAt": null,
"nextUpdateTime": "2025-01-15T10:12:30Z",
"message": "Cache update triggered"
}
```
#### Delete Cache
Delete cached data for a location.
```http
DELETE /api/cache/{suburb}/{state}
```
**Response:**
```json
{
"message": "Cache deleted for Brisbane, QLD"
}
```
### OpenAPI Documentation
When running in Development mode, OpenAPI documentation is available at:
```
http://localhost:8082/openapi/v1.json
```
## Demo SPA
The service includes a built-in Single Page Application (SPA) for testing and demonstration purposes. This provides a visual interface to:
- View radar frames in a slideshow
- Test API endpoints
- Configure playback settings
- View historical data across extended time periods
- Monitor cache status and update information
### Accessing the Demo
Navigate to:
```
http://localhost:8082/radar/{suburb}/{state}
```
**Example:**
```
http://localhost:8082/radar/Brisbane/QLD
```
### Features
- **Slideshow Playback**: Play, pause, and navigate through radar frames
- **Frame Navigation**: Use slider, buttons, or keyboard shortcuts (arrow keys, spacebar)
- **Extended Timespans**: View historical data from 1 hour to 24 hours
- **Custom Time Ranges**: Select specific start and end times for historical viewing
- **Auto-Refresh**: Automatically checks for new data at configurable intervals
- **Cache Status**: Real-time display of cache validity, expiration, and update status
- **Settings Panel**: Configure frame intervals, refresh rates, and playback options
### Keyboard Shortcuts
- `←` / `→`: Navigate to previous/next frame
- `Shift + ←` / `Shift + →`: Jump back/forward 10 frames
- `Home` / `End`: Jump to first/last frame
- `Space`: Play/pause slideshow
### Using the Demo for Integration Development
The demo SPA serves as a reference implementation demonstrating best practices for consuming the API. Key implementation patterns:
**Basic Radar Data Fetching**: The simplest pattern - fetch data and handle the case where cache is being generated:
```javascript
async function getRadarData(suburb, state) {
const response = await fetch(`http://localhost:8082/api/radar/${suburb}/${state}`);
if (response.status === 404) {
// Cache is being generated - trigger refresh and show message
const error = await response.json();
if (error.refreshEndpoint) {
// Trigger cache update in background
fetch(error.refreshEndpoint, { method: 'POST' }).catch(() => {});
}
return { frames: [], message: `Cache being generated. Retry in ${error.retryAfter} seconds.` };
}
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
return await response.json();
}
```
**Displaying Frames**: Frame objects include ready-to-use `imageUrl` properties:
```javascript
const radarData = await getRadarData('Brisbane', 'QLD');
if (radarData.frames && radarData.frames.length > 0) {
// Display first frame
document.getElementById('radar-image').src = radarData.frames[0].imageUrl;
// Or loop through all frames for animation
radarData.frames.forEach((frame, index) => {
console.log(`Frame ${index}: ${frame.imageUrl} (${frame.minutesAgo} min ago)`);
});
}
```
**Historical Data**: Fetch extended time series by specifying a time range:
```javascript
async function getHistoricalRadar(suburb, state, hoursBack = 3) {
const endTime = new Date();
const startTime = new Date(endTime.getTime() - (hoursBack * 60 * 60 * 1000));
const response = await fetch(
`/api/radar/${suburb}/${state}/timeseries?startTime=${startTime.toISOString()}&endTime=${endTime.toISOString()}`
);
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const data = await response.json();
// Flatten frames from all cache folders
const allFrames = data.cacheFolders.flatMap(folder => folder.frames);
return allFrames;
}
```
**Auto-Refresh with Error Handling**: Implement periodic updates that handle API downtime:
```javascript
let refreshInterval;
function startAutoRefresh(suburb, state, intervalSeconds = 30) {
refreshInterval = setInterval(async () => {
try {
const data = await getRadarData(suburb, state);
if (data.frames && data.frames.length > 0) {
updateDisplay(data);
}
} catch (error) {
console.error('Failed to refresh:', error);
// Optionally stop auto-refresh on persistent errors
}
}, intervalSeconds * 1000);
}
// Start auto-refresh
startAutoRefresh('Brisbane', 'QLD', 30);
// Stop auto-refresh
if (refreshInterval) {
clearInterval(refreshInterval);
}
```
The complete implementation with all patterns is available in `Views/RadarTest/Index.cshtml` for reference.
## Usage Examples
### Home Assistant Integration
To use this service with Home Assistant, you'll need to create a custom integration or use a REST sensor. Here's a basic example:
```yaml
# configuration.yaml
rest:
- sensor:
name: "BOM Radar Brisbane"
resource: "http://localhost:8082/api/radar/Brisbane/QLD"
value_template: "{{ value_json.frames[0].imageUrl }}"
scan_interval: 300
```
### cURL Examples
**Get latest radar data:**
```bash
curl http://localhost:8082/api/radar/Brisbane/QLD
```
**Get specific frame image:**
```bash
curl http://localhost:8082/api/radar/Brisbane/QLD/frame/0 -o frame.png
```
**Trigger cache refresh:**
```bash
curl -X POST http://localhost:8082/api/cache/Brisbane/QLD/refresh
```
**Get historical data (last 3 hours):**
```bash
curl "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z"
```
## Troubleshooting
### Service Won't Start
- **Check Docker logs**: `docker logs bom-local-service`
- **Verify port availability**: Ensure port 8082 (or your configured port) is not in use
- **Shared memory configuration**: The docker-compose.yml includes `shm_size: '1gb'` and `ipc: host` which are required for Playwright. If running with `docker run`, ensure you include:
```bash
docker run --shm-size=1gb --ipc=host ...
```
- **Browser launch failures**: If Playwright browsers fail to launch, check for seccomp security restrictions. You may need to add `--security-opt seccomp:unconfined` (though this is generally not needed with the included configuration)
### No Radar Data Available
- **Wait for initial cache**: First request triggers cache generation, which takes 30-60 seconds
- **Check location format**: Ensure suburb and state are correctly formatted (e.g., "Brisbane", "QLD")
- **Verify cache directory**: Check that the cache volume is mounted and writable
- **Check logs**: Look for errors in browser automation or data capture
### Images Not Loading
- **Verify frame URLs**: Check that frame image URLs are correctly formatted
- **Check cache status**: Use `/api/cache/{suburb}/{state}/range` to verify cache exists
- **Browser automation issues**: Check logs for Playwright errors
### Playwright Resource Usage
Playwright browsers (Chromium) can consume significant CPU and memory:
**Limit CPU Cores**: Add CPU limits to your docker-compose.yml:
```yaml
services:
bom-local-service:
deploy:
resources:
limits:
cpus: '1.0' # Limit to 1 CPU core
reservations:
cpus: '0.5' # Reserve 0.5 cores
```
Or with `docker run`:
```bash
docker run --cpus="1.0" ...
```
**Common Playwright + Docker Issues**:
- **Browser crashes with "out of memory"**: Ensure `shm_size: '1gb'` is set. Chromium uses `/dev/shm` for shared memory, and the default 64MB is insufficient.
- **High CPU usage during idle**: This is normal - Playwright browsers can consume CPU even when idle. Consider reducing `CACHEMANAGEMENT__CHECKINTERVALMINUTES` to check less frequently.
- **Browser processes not terminating**: Check logs for stuck browser processes. The service includes cleanup logic, but you may need to restart the container if processes hang.
- **"Protocol error" or connection failures**: Usually indicates insufficient shared memory or IPC namespace issues. Verify `ipc: host` is set in docker-compose.yml.
**Optimize Cache Settings**:
- **Reduce retention**: Lower `CACHERETENTIONHOURS` to keep less data on disk
- **Increase cleanup frequency**: Lower `CACHECLEANUP__INTERVALHOURS` to clean up more often
- **Limit locations**: The service automatically manages all cached locations; reduce the number of locations being cached to lower resource usage
## Development
### Building Locally
```bash
dotnet restore
dotnet build
dotnet run
```
### Debug Mode
Enable debug mode to save intermediate screenshots during data capture:
```bash
docker run -e DEBUG__ENABLED=true -e DEBUG__WAITMS=5000 ...
```
Debug screenshots are saved in `{CACHEDIRECTORY}/debug/`.
## License
MIT License - see [LICENSE](LICENSE) file for details.
## Contributing
This is a hobby project, but contributions are welcome! Feel free to open issues or submit pull requests.
## Acknowledgments
- Inspired by the [bom-radar-card](https://github.com/Makin-Things/bom-radar-card) project
- Built with [Playwright](https://playwright.dev/) for browser automation
- Uses [ASP.NET Core](https://dotnet.microsoft.com/) for the web framework
## Disclaimer
This project is intended for **local, personal use only**. The radar data and images cached by this service are the property of the Australian Bureau of Meteorology (BOM) and are subject to copyright.
**Important Notes:**
- This service is designed to run on your local network for personal use
- Do not redistribute or republish BOM radar data or images
- Respect BOM's terms of service and data usage policies
- The service caches publicly available data for local consumption only
- This project is not affiliated with or endorsed by the Australian Bureau of Meteorology
For official BOM data and services, visit [bom.gov.au](https://www.bom.gov.au/).