BOM Local Service

A local caching service for Australian Bureau of Meteorology (BOM) radar data, designed to provide a reliable API for Home Assistant integrations and other local services.

Background

The Australian Bureau of Meteorology's radar API endpoint (https://api.weather.bom.gov.au/v1/radar/capabilities) stopped working in December 2024, returning errors. This broke integrations like the popular bom-radar-card for Home Assistant, which can no longer render due to the Cross-Origin Request Blocked error.

The issue was reported in GitHub issue #40, where multiple users confirmed the card completely fails to render.

This project was created to bridge that gap by:

• Caching radar data from the BOM website
• Storing it locally in a structured format
• Providing a simple REST API for local services to consume
• Automatically managing cache updates and cleanup

Features

• 🌧️ Automatic Cache Management: Background service automatically updates radar data for all cached locations
• 📊 Historical Data: Access historical radar frames across multiple time periods
• 🎯 Location-Based: Support for any Australian suburb/state combination
• 🖼️ Image API: Direct access to individual radar frame images
• 🔄 Auto-Refresh: Configurable cache expiration and refresh intervals
• 🧹 Automatic Cleanup: Old cache data is automatically purged based on retention settings
• 🎨 Demo SPA: Built-in web interface for testing and demonstration
• 🐳 Easy Docker Deployment: Quick setup with Docker

Architecture

Built on ASP.NET Core 9.0, the service uses a service-oriented architecture with clear separation of concerns:

Core Services

• BomRadarService: Main orchestrator that coordinates cache operations, browser automation, and data retrieval
• CacheService: Manages file-based storage of radar screenshots and metadata in organized directory structures
• BrowserService: Handles Playwright browser automation for headless browser sessions
• ScrapingService: Performs web scraping operations using the browser service to navigate and capture radar data from the BOM website
• TimeParsingService: Parses and converts time formats from BOM data
• DebugService: Provides debug functionality for troubleshooting

Background Services

• CacheManagementService: Periodically checks cache validity for all cached locations and triggers updates when data expires
• CacheCleanupService: Removes cache files older than the configured retention period

API Layer

• RadarController: REST endpoints for accessing radar data (/api/radar/{suburb}/{state})
• CacheController: REST endpoints for cache management operations (/api/cache/{suburb}/{state})
• RadarTestController: MVC controller serving the demo SPA at /radar/{suburb}/{state}

Building with Docker

Prerequisites

• Docker Engine 20.10+ or Docker Desktop
• Docker Compose v2.0+ (optional, for easier management)

Quick Start

1. Clone the repository:

   git clone <repository-url>
   cd bom-local-service
   

2. Build the Docker image:

   docker build -t bom-local-service .
   

   The build process uses a multi-stage Dockerfile:

   • Build stage: Compiles the .NET application
   • Runtime stage: Uses the official Playwright .NET image with browsers pre-installed
   • Sets up a virtual display (Xvfb) for headless browser operation

3. Run the container:

   docker run -d \
     --name bom-local-service \
     -p 8082:8080 \
     -v $(pwd)/cache:/app/cache \
     --shm-size=1gb \
     --ipc=host \
     bom-local-service
   

   Note: The --shm-size=1gb and --ipc=host flags are important for Playwright to function correctly in Docker.

Using Docker Compose

The included docker-compose.yml provides a convenient way to run the service with all configuration options:

docker-compose up -d

This will:

• Build the image if it doesn't exist
• Start the service on port 8082 (configurable via HOST_PORT)
• Mount the ./cache directory for persistent storage
• Apply all environment variable configurations

To view logs:

docker-compose logs -f

To stop the service:

docker-compose down

Configuration

All configuration can be done via environment variables, which override the default values in appsettings.json. The service uses ASP.NET Core's configuration system, which supports nested configuration via double underscores (__).

Environment Variables

ASP.NET Core Configuration

Variable	Description	Default
ASPNETCORE_ENVIRONMENT	Runtime environment (Development/Production)	Production
ASPNETCORE_URLS	URLs the service listens on	http://+:8080
ENABLE_HTTPS_REDIRECTION	Enable HTTPS redirection	false

Application Configuration

Variable	Description	Default	Example
CACHEDIRECTORY	Directory path for cache storage	/app/cache	/data/bom-cache
CACHERETENTIONHOURS	Hours to retain cached data before cleanup	24	48
CACHEEXPIRATIONMINUTES	Minutes before cache is considered expired	12.5	15
TIMEZONE	Timezone for time parsing (IANA format)	Australia/Brisbane	Australia/Sydney

Cache Management

Variable	Description	Default	Example
CACHEMANAGEMENT__CHECKINTERVALMINUTES	Interval between cache validity checks	5	10
CACHEMANAGEMENT__INITIALDELAYSECONDS	Delay before first cache check on startup	10	30
CACHEMANAGEMENT__UPDATESTAGGERSECONDS	Delay between triggering cache updates	2	5
CACHEMANAGEMENT__LOCATIONSTAGGERSECONDS	Delay between processing different locations	1	2

Cache Cleanup

Variable	Description	Default	Example
CACHECLEANUP__INTERVALHOURS	Interval between cleanup runs	1	2

Screenshot Configuration

Variable	Description	Default	Example
SCREENSHOT__DYNAMICCONTENTWAITMS	Milliseconds to wait for dynamic content to load	2000	3000
SCREENSHOT__TILERENDERWAITMS	Milliseconds to wait for map tiles to render	5000	7000

Debug Configuration

Variable	Description	Default	Example
DEBUG__ENABLED	Enable debug mode (saves debug screenshots)	false	true
DEBUG__WAITMS	Additional wait time in debug mode	2000	5000

Docker Compose Port Configuration

Variable	Description	Default
HOST_PORT	Port on host machine to map to container	8082
CONTAINER_PORT	Port inside container (usually 8080)	8080

Configuration Examples

Basic Configuration

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -e CACHERETENTIONHOURS=48 \
  -e CACHEEXPIRATIONMINUTES=15 \
  --shm-size=1gb \
  --ipc=host \
  bom-local-service

Custom Cache Directory

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v /data/bom-cache:/app/cache \
  -e CACHEDIRECTORY=/app/cache \
  --shm-size=1gb \
  --ipc=host \
  bom-local-service

Development Mode with Debug

docker run -d \
  --name bom-local-service \
  -p 8082:8080 \
  -v $(pwd)/cache:/app/cache \
  -e ASPNETCORE_ENVIRONMENT=Development \
  -e DEBUG__ENABLED=true \
  -e DEBUG__WAITMS=5000 \
  --shm-size=1gb \
  --ipc=host \
  bom-local-service

Using docker-compose with Custom Settings

Create a .env file in the project root:

HOST_PORT=8082
CACHERETENTIONHOURS=48
CACHEEXPIRATIONMINUTES=15
CACHEMANAGEMENT__CHECKINTERVALMINUTES=10
TIMEZONE=Australia/Sydney

Then run:

docker-compose up -d

API Documentation

The service provides RESTful API endpoints for accessing radar data and managing the cache.

Base URL

When running locally with default settings: http://localhost:8082

Endpoints

Get Radar Data

Get the latest radar frames for a location.

GET /api/radar/{suburb}/{state}

Parameters:

• suburb (path): Suburb name (e.g., "Brisbane")
• state (path): State abbreviation (e.g., "QLD")

Response:

{
  "frames": [
    {
      "frameIndex": 0,
      "imageUrl": "/api/radar/Brisbane/QLD/frame/0",
      "minutesAgo": 0,
      "observationTime": "2025-01-15T10:00:00Z"
    }
  ],
  "observationTime": "2025-01-15T10:00:00Z",
  "forecastTime": "2025-01-15T10:00:00Z",
  "weatherStation": "Brisbane",
  "distance": "5.2 km",
  "cacheIsValid": true,
  "cacheExpiresAt": "2025-01-15T10:12:30Z",
  "isUpdating": false,
  "nextUpdateTime": "2025-01-15T10:12:30Z"
}

Status Codes:

• 200 OK: Radar data available
• 404 Not Found: Cache is being generated (check response for retry information)
• 400 Bad Request: Invalid location parameters

Get Frame Image

Get a specific radar frame image.

GET /api/radar/{suburb}/{state}/frame/{frameIndex}

Parameters:

• suburb (path): Suburb name
• state (path): State abbreviation
• frameIndex (path): Frame index (0-6 for default 7 frames)
• cacheFolder (query, optional): Specific cache folder name for historical data

Response:

• 200 OK: PNG image
• 404 Not Found: Frame not found

Get Metadata

Get metadata about cached radar data.

GET /api/radar/{suburb}/{state}/metadata

Response:

{
  "lastUpdated": "2025-01-15T10:00:00Z",
  "observationTime": "2025-01-15T10:00:00Z",
  "forecastTime": "2025-01-15T10:00:00Z",
  "weatherStation": "Brisbane",
  "distance": "5.2 km"
}

Get Time Series

Get historical radar data across multiple cache folders.

GET /api/radar/{suburb}/{state}/timeseries?startTime={iso8601}&endTime={iso8601}

Parameters:

• suburb (path): Suburb name
• state (path): State abbreviation
• startTime (query, optional): ISO 8601 start time (e.g., 2025-01-15T00:00:00Z)
• endTime (query, optional): ISO 8601 end time (defaults to now)

Response:

{
  "cacheFolders": [
    {
      "cacheFolderName": "Brisbane_QLD_20250115_100000",
      "cacheTimestamp": "2025-01-15T10:00:00Z",
      "observationTime": "2025-01-15T10:00:00Z",
      "frames": [
        {
          "frameIndex": 0,
          "imageUrl": "/api/radar/Brisbane/QLD/frame/0?cacheFolder=Brisbane_QLD_20250115_100000",
          "absoluteObservationTime": "2025-01-15T10:00:00Z",
          "minutesAgo": 0
        }
      ]
    }
  ],
  "totalFrames": 7,
  "timeSpanMinutes": 60
}

Get Cache Range

Get information about available historical cache data.

GET /api/cache/{suburb}/{state}/range

Response:

{
  "oldestCache": {
    "cacheFolderName": "Brisbane_QLD_20250115_000000",
    "cacheTimestamp": "2025-01-15T00:00:00Z"
  },
  "newestCache": {
    "cacheFolderName": "Brisbane_QLD_20250115_100000",
    "cacheTimestamp": "2025-01-15T10:00:00Z"
  },
  "totalCacheFolders": 10,
  "timeSpanMinutes": 600
}

Refresh Cache

Manually trigger a cache update for a location.

POST /api/cache/{suburb}/{state}/refresh

Response:

{
  "updateTriggered": true,
  "cacheIsValid": false,
  "cacheExpiresAt": null,
  "nextUpdateTime": "2025-01-15T10:12:30Z",
  "message": "Cache update triggered"
}

Delete Cache

Delete cached data for a location.

DELETE /api/cache/{suburb}/{state}

Response:

{
  "message": "Cache deleted for Brisbane, QLD"
}

OpenAPI Documentation

When running in Development mode, OpenAPI documentation is available at:

http://localhost:8082/openapi/v1.json

Demo SPA

The service includes a built-in Single Page Application (SPA) for testing and demonstration purposes. This provides a visual interface to:

• View radar frames in a slideshow
• Test API endpoints
• Configure playback settings
• View historical data across extended time periods
• Monitor cache status and update information

Accessing the Demo

Navigate to:

http://localhost:8082/radar/{suburb}/{state}

Example:

http://localhost:8082/radar/Brisbane/QLD

Features

• Slideshow Playback: Play, pause, and navigate through radar frames
• Frame Navigation: Use slider, buttons, or keyboard shortcuts (arrow keys, spacebar)
• Extended Timespans: View historical data from 1 hour to 24 hours
• Custom Time Ranges: Select specific start and end times for historical viewing
• Auto-Refresh: Automatically checks for new data at configurable intervals
• Cache Status: Real-time display of cache validity, expiration, and update status
• Settings Panel: Configure frame intervals, refresh rates, and playback options

Keyboard Shortcuts

• ← / →: Navigate to previous/next frame
• Shift + ← / Shift + →: Jump back/forward 10 frames
• Home / End: Jump to first/last frame
• Space: Play/pause slideshow

Using the Demo for Integration Development

The demo SPA serves as a reference implementation demonstrating best practices for consuming the API. Key implementation patterns:

Basic Radar Data Fetching: The simplest pattern - fetch data and handle the case where cache is being generated:

async function getRadarData(suburb, state) {
    const response = await fetch(`http://localhost:8082/api/radar/${suburb}/${state}`);
    
    if (response.status === 404) {
        // Cache is being generated - trigger refresh and show message
        const error = await response.json();
        if (error.refreshEndpoint) {
            // Trigger cache update in background
            fetch(error.refreshEndpoint, { method: 'POST' }).catch(() => {});
        }
        return { frames: [], message: `Cache being generated. Retry in ${error.retryAfter} seconds.` };
    }
    
    if (!response.ok) {
        throw new Error(`HTTP ${response.status}`);
    }
    
    return await response.json();
}

Displaying Frames: Frame objects include ready-to-use imageUrl properties:

const radarData = await getRadarData('Brisbane', 'QLD');
if (radarData.frames && radarData.frames.length > 0) {
    // Display first frame
    document.getElementById('radar-image').src = radarData.frames[0].imageUrl;
    
    // Or loop through all frames for animation
    radarData.frames.forEach((frame, index) => {
        console.log(`Frame ${index}: ${frame.imageUrl} (${frame.minutesAgo} min ago)`);
    });
}

Historical Data: Fetch extended time series by specifying a time range:

async function getHistoricalRadar(suburb, state, hoursBack = 3) {
    const endTime = new Date();
    const startTime = new Date(endTime.getTime() - (hoursBack * 60 * 60 * 1000));
    
    const response = await fetch(
        `/api/radar/${suburb}/${state}/timeseries?startTime=${startTime.toISOString()}&endTime=${endTime.toISOString()}`
    );
    
    if (!response.ok) {
        throw new Error(`HTTP ${response.status}`);
    }
    
    const data = await response.json();
    // Flatten frames from all cache folders
    const allFrames = data.cacheFolders.flatMap(folder => folder.frames);
    return allFrames;
}

Auto-Refresh with Error Handling: Implement periodic updates that handle API downtime:

let refreshInterval;

function startAutoRefresh(suburb, state, intervalSeconds = 30) {
    refreshInterval = setInterval(async () => {
        try {
            const data = await getRadarData(suburb, state);
            if (data.frames && data.frames.length > 0) {
                updateDisplay(data);
            }
        } catch (error) {
            console.error('Failed to refresh:', error);
            // Optionally stop auto-refresh on persistent errors
        }
    }, intervalSeconds * 1000);
}

// Start auto-refresh
startAutoRefresh('Brisbane', 'QLD', 30);

// Stop auto-refresh
if (refreshInterval) {
    clearInterval(refreshInterval);
}

The complete implementation with all patterns is available in Views/RadarTest/Index.cshtml for reference.

Usage Examples

Home Assistant Integration

To use this service with Home Assistant, you'll need to create a custom integration or use a REST sensor. Here's a basic example:

# configuration.yaml
rest:
  - sensor:
      name: "BOM Radar Brisbane"
      resource: "http://localhost:8082/api/radar/Brisbane/QLD"
      value_template: "{{ value_json.frames[0].imageUrl }}"
      scan_interval: 300

cURL Examples

Get latest radar data:

curl http://localhost:8082/api/radar/Brisbane/QLD

Get specific frame image:

curl http://localhost:8082/api/radar/Brisbane/QLD/frame/0 -o frame.png

Trigger cache refresh:

curl -X POST http://localhost:8082/api/cache/Brisbane/QLD/refresh

Get historical data (last 3 hours):

curl "http://localhost:8082/api/radar/Brisbane/QLD/timeseries?startTime=2025-01-15T07:00:00Z"

Troubleshooting

Service Won't Start

• Check Docker logs: docker logs bom-local-service
• Verify port availability: Ensure port 8082 (or your configured port) is not in use
• Shared memory configuration: The docker-compose.yml includes shm_size: '1gb' and ipc: host which are required for Playwright. If running with docker run, ensure you include:

  docker run --shm-size=1gb --ipc=host ...
  

• Browser launch failures: If Playwright browsers fail to launch, check for seccomp security restrictions. You may need to add --security-opt seccomp:unconfined (though this is generally not needed with the included configuration)

No Radar Data Available

• Wait for initial cache: First request triggers cache generation, which takes 30-60 seconds
• Check location format: Ensure suburb and state are correctly formatted (e.g., "Brisbane", "QLD")
• Verify cache directory: Check that the cache volume is mounted and writable
• Check logs: Look for errors in browser automation or data capture

Images Not Loading

• Verify frame URLs: Check that frame image URLs are correctly formatted
• Check cache status: Use /api/cache/{suburb}/{state}/range to verify cache exists
• Browser automation issues: Check logs for Playwright errors

Playwright Resource Usage

Playwright browsers (Chromium) can consume significant CPU and memory:

Limit CPU Cores: Add CPU limits to your docker-compose.yml:

services:
  bom-local-service:
    deploy:
      resources:
        limits:
          cpus: '1.0'  # Limit to 1 CPU core
        reservations:
          cpus: '0.5'  # Reserve 0.5 cores

Or with docker run:

docker run --cpus="1.0" ...

Common Playwright + Docker Issues:

• Browser crashes with "out of memory": Ensure shm_size: '1gb' is set. Chromium uses /dev/shm for shared memory, and the default 64MB is insufficient.
• High CPU usage during idle: This is normal - Playwright browsers can consume CPU even when idle. Consider reducing CACHEMANAGEMENT__CHECKINTERVALMINUTES to check less frequently.
• Browser processes not terminating: Check logs for stuck browser processes. The service includes cleanup logic, but you may need to restart the container if processes hang.
• "Protocol error" or connection failures: Usually indicates insufficient shared memory or IPC namespace issues. Verify ipc: host is set in docker-compose.yml.

Optimize Cache Settings:

• Reduce retention: Lower CACHERETENTIONHOURS to keep less data on disk
• Increase cleanup frequency: Lower CACHECLEANUP__INTERVALHOURS to clean up more often
• Limit locations: The service automatically manages all cached locations; reduce the number of locations being cached to lower resource usage

Development

Building Locally

dotnet restore
dotnet build
dotnet run

Debug Mode

Enable debug mode to save intermediate screenshots during data capture:

docker run -e DEBUG__ENABLED=true -e DEBUG__WAITMS=5000 ...

Debug screenshots are saved in {CACHEDIRECTORY}/debug/.

License

MIT License - see LICENSE file for details.

Contributing

This is a hobby project, but contributions are welcome! Feel free to open issues or submit pull requests.

Acknowledgments

• Inspired by the bom-radar-card project
• Built with Playwright for browser automation
• Uses ASP.NET Core for the web framework

Disclaimer

This project is intended for local, personal use only. The radar data and images cached by this service are the property of the Australian Bureau of Meteorology (BOM) and are subject to copyright.

Important Notes:

• This service is designed to run on your local network for personal use
• Do not redistribute or republish BOM radar data or images
• Respect BOM's terms of service and data usage policies
• The service caches publicly available data for local consumption only
• This project is not affiliated with or endorsed by the Australian Bureau of Meteorology

For official BOM data and services, visit bom.gov.au.