ElixirForge/bom-local-service
1
0
Fork 0
mirror of https://github.com/alexhopeoconnor/bom-local-service.git synced 2026-10-03 21:21:59 +10:00
Code Releases 7 Activity

7 Releases 7 Tags

RSS Feed
  • v0.1.0 12dd0eedad
    Compare

    v0.1.0 - Workflow-Based Scraping Architecture Stable

    elixir released this 2025-12-18 00:54:57 +10:00 | 8 commits to main since this release

    Major Architectural Refactoring

    This release introduces a complete refactoring of the scraping system to improve maintainability, configurability, and extensibility.

    Key Changes

    New Architecture:

    • Workflow-based scraping system replacing monolithic service
    • Discrete, testable scraping steps organized by category
    • Configuration-driven selectors, JavaScript templates, and text patterns
    • Step registry and workflow factory for extensibility
    • PageState tracking and step dependencies
    • Generic IWorkflow interface supporting different response types

    New Components:

    • Scraping steps: Navigation, Search, Map, Metadata, Capture
    • BaseScrapingStep base class with common dependencies
    • ScrapingContext for shared state between steps
    • RadarScrapingWorkflow orchestrating step execution
    • SelectorService for configurable element selection
    • Configuration models for externalized settings

    Performance & Monitoring:

    • Step-level timing metrics with historical tracking
    • Automatic warnings for slow steps/workflows (50%+ slower than average)
    • Integrated metrics with CacheService

    Configuration:

    • All selectors externalized to appsettings.json
    • JavaScript templates externalized to appsettings.json
    • Text patterns externalized to appsettings.json
    • Docker configuration support via mounted appsettings.json
    • Reduced hardcoded waits, using configurable timeouts

    Bug Fixes:

    • Fixed DebugService collection modification exception
    • Fixed ResetToFirstFrameStep click interception with JavaScript fallback
    • Removed deprecated minutes ago fallback, using timestamp parsing only
    • Updated timestamp parsing for new BOM website format

    Documentation:

    • Updated README with architecture details
    • Added configuration section with Docker guidance
    • Added development guidelines for extending scraping system
    • Added performance monitoring documentation
    Downloads