- Refactor TimeParsingService to parse time strings to UTC only
- Remove minutesAgo from RadarFrame and FrameMetadata models
- Update CaptureFramesStep to extract UTC timestamps directly
- Update CacheService to store/load ObservationTime instead of MinutesAgo
- Update BomRadarService to use AbsoluteObservationTime directly
- Remove all backward compatibility code for old cache folders
- Update documentation and test SPA to calculate minutes ago on client
- Fix step registration to register concrete types for DI resolution
- Fix hosted service registration to use reflection correctly
- Remove OpenAPI/Swagger support and all related code
- Standardize configuration: remove hardcoded defaults, all defaults from appsettings.json
- Configuration values throw exceptions if missing (except Debug:Enabled defaults to false)
- Refactor CacheManagementService: extract ProcessLocationsAsync helper, remove duplicate code
- Combine UpdateStaggerSeconds and LocationStaggerSeconds into single LocationStaggerSeconds config
- Move config reads to constructors for better performance
- Update README and docker-compose.yml to reflect configuration changes
Major architectural changes:
- Replace monolithic ScrapingService with workflow-based system
- Extract scraping logic into discrete, testable steps
- Add configuration-driven selectors, JavaScript templates, and text patterns
- Implement step registry and workflow factory for extensibility
- Add PageState tracking and step dependencies
- Make IWorkflow generic to support different response types
New components:
- Scraping steps organized by category (Navigation, Search, Map, Metadata, Capture)
- BaseScrapingStep base class with common dependencies
- ScrapingContext for shared state between steps
- RadarScrapingWorkflow orchestrating step execution
- SelectorService for configurable element selection
- Configuration models (SelectorConfig, TextPatternsConfig, JavaScriptTemplatesConfig)
Performance and monitoring:
- Add step-level timing metrics with historical tracking
- Log step durations and compare to averages
- Warn on slow steps/workflows (50%+ slower than average)
- Integrate metrics with CacheService
Configuration improvements:
- Externalize all selectors to appsettings.json
- Externalize JavaScript templates to appsettings.json
- Externalize text patterns to appsettings.json
- Support Docker configuration via mounted appsettings.json
- Remove hardcoded waits, use configurable timeouts
Bug fixes:
- Fix DebugService collection modification exception
- Fix ResetToFirstFrameStep click interception with JavaScript fallback
- Remove minutes ago fallback, use timestamp parsing only
- Update timestamp parsing for new BOM website format
Documentation:
- Update README with new architecture details
- Add configuration section with Docker guidance
- Add development guidelines for extending scraping system
- Add performance monitoring documentation
- Fix TimeRangeError to properly extract properties from anonymous objects
- Add suggestions field to timeseries error response examples in README
- Enhance demo app to use suggested ranges from error responses
Active Folder Exclusion Fixes:
- Exclude active cache folders in GetCachedFrameAsync, GetCachedFramesAsync, and GetAllCacheFoldersAsync
- Prevents TaskCanceledException when reading metadata from folders being written to
- Matches pattern used in GetCachedRadarAsync for consistency
Cache Configuration Fixes:
- Standardize default CacheExpirationMinutes to 12.5 (matches appsettings.json)
- Add cache configuration to ScrapingService (removes hardcoded values)
- Fix check interval calculation bug for intervals that don't divide 60 evenly
- Fix NextUpdateTime calculation to account for background check intervals
- Add configuration validation to all service constructors
All fixes verified - service running stably with 0 exceptions in recent logs
- Add active cache folder exclusion to GetCachedFrameAsync
- Prevents TaskCanceledException when reading metadata from folders being written to
- Matches pattern used in GetCachedRadarAsync for consistency
- Fix scraping to wait for display label to change after step forward, preventing duplicate minutesAgo values within same folder
- Add hashset-based deduplication by AbsoluteObservationTime across cache folders
- Filter frames that are too close together (< 4 minutes apart)
- Preserve original folder structure while pruning filtered frames
- Sort folders by earliest frame time to maintain chronological order
- Add validation to detect folder overlaps that could cause out-of-order frames
- Move cache folder completeness check to CacheHelper utility
- Replace Width with RightOffset in screenshot crop config
- Add startup cleanup to remove incomplete cache folders
- Validate cache folders before use in BomRadarService
- Track active cache folders being written to prevent returning incomplete data
- Add cache state fields to RadarResponse (CacheIsValid, CacheExpiresAt, IsUpdating, NextUpdateTime)
- Detect active updates before acquiring semaphore for better performance
- Prevent duplicate cache updates when one is already in progress
- Clean up empty cache folders created but not used
- Capture all 7 radar frames (0-6) instead of single screenshot
- Store frames in folder-based cache structure with metadata
- Extract and store minutesAgo for each frame from page display
- Add configurable screenshot cropping (default: 200px left/right)
- Fix step numbering to use whole numbers (1-22)
- Add scrubber position verification before frame capture
- Handle modal overlay blocking step forward button clicks
- Only create debug folders when scraping is actually needed
- Update API endpoints to return JSON with frame URLs
- Add separate endpoints for individual frame images
alex