Major architectural changes:
- Replace monolithic ScrapingService with workflow-based system
- Extract scraping logic into discrete, testable steps
- Add configuration-driven selectors, JavaScript templates, and text patterns
- Implement step registry and workflow factory for extensibility
- Add PageState tracking and step dependencies
- Make IWorkflow generic to support different response types
New components:
- Scraping steps organized by category (Navigation, Search, Map, Metadata, Capture)
- BaseScrapingStep base class with common dependencies
- ScrapingContext for shared state between steps
- RadarScrapingWorkflow orchestrating step execution
- SelectorService for configurable element selection
- Configuration models (SelectorConfig, TextPatternsConfig, JavaScriptTemplatesConfig)
Performance and monitoring:
- Add step-level timing metrics with historical tracking
- Log step durations and compare to averages
- Warn on slow steps/workflows (50%+ slower than average)
- Integrate metrics with CacheService
Configuration improvements:
- Externalize all selectors to appsettings.json
- Externalize JavaScript templates to appsettings.json
- Externalize text patterns to appsettings.json
- Support Docker configuration via mounted appsettings.json
- Remove hardcoded waits, use configurable timeouts
Bug fixes:
- Fix DebugService collection modification exception
- Fix ResetToFirstFrameStep click interception with JavaScript fallback
- Remove minutes ago fallback, use timestamp parsing only
- Update timestamp parsing for new BOM website format
Documentation:
- Update README with new architecture details
- Add configuration section with Docker guidance
- Add development guidelines for extending scraping system
- Add performance monitoring documentation
- Add active cache folder exclusion to GetCachedFrameAsync
- Prevents TaskCanceledException when reading metadata from folders being written to
- Matches pattern used in GetCachedRadarAsync for consistency
- Add MVC test page at /radar/{suburb}/{state} demonstrating API usage
- Implement auto-play on page load with cookie-based settings
- Add connection monitoring with automatic retry on API downtime
- Show update status, cache state, and next check times
- Fix NextUpdateTime calculation to account for background service intervals
- Add graceful error handling and visual status indicators
- Move cache folder completeness check to CacheHelper utility
- Replace Width with RightOffset in screenshot crop config
- Add startup cleanup to remove incomplete cache folders
- Validate cache folders before use in BomRadarService
- HTTPS redirection now configurable via ENABLE_HTTPS_REDIRECTION env var
- Removed UseAuthorization() for Docker flexibility
- Improved regex group validation in ScrapingService
alex