Refactor scraping system to workflow-based architecture

Major architectural changes:
- Replace monolithic ScrapingService with workflow-based system
- Extract scraping logic into discrete, testable steps
- Add configuration-driven selectors, JavaScript templates, and text patterns
- Implement step registry and workflow factory for extensibility
- Add PageState tracking and step dependencies
- Make IWorkflow generic to support different response types

New components:
- Scraping steps organized by category (Navigation, Search, Map, Metadata, Capture)
- BaseScrapingStep base class with common dependencies
- ScrapingContext for shared state between steps
- RadarScrapingWorkflow orchestrating step execution
- SelectorService for configurable element selection
- Configuration models (SelectorConfig, TextPatternsConfig, JavaScriptTemplatesConfig)

Performance and monitoring:
- Add step-level timing metrics with historical tracking
- Log step durations and compare to averages
- Warn on slow steps/workflows (50%+ slower than average)
- Integrate metrics with CacheService

Configuration improvements:
- Externalize all selectors to appsettings.json
- Externalize JavaScript templates to appsettings.json
- Externalize text patterns to appsettings.json
- Support Docker configuration via mounted appsettings.json
- Remove hardcoded waits, use configurable timeouts

Bug fixes:
- Fix DebugService collection modification exception
- Fix ResetToFirstFrameStep click interception with JavaScript fallback
- Remove minutes ago fallback, use timestamp parsing only
- Update timestamp parsing for new BOM website format

Documentation:
- Update README with new architecture details
- Add configuration section with Docker guidance
- Add development guidelines for extending scraping system
- Add performance monitoring documentation
This commit is contained in:
2025-12-18 00:54:57 +10:00
parent 5e31ed33c6
commit 12dd0eedad
40 changed files with 2868 additions and 1020 deletions
+33
View File
@@ -0,0 +1,33 @@
namespace BomLocalService.Services.Scraping;
public class ScrapingStepRegistry : IScrapingStepRegistry
{
private readonly Dictionary<string, IScrapingStep> _steps = new();
private readonly ILogger<ScrapingStepRegistry> _logger;
public ScrapingStepRegistry(ILogger<ScrapingStepRegistry> logger)
{
_logger = logger;
}
public void RegisterStep(IScrapingStep step)
{
if (_steps.ContainsKey(step.Name))
{
_logger.LogWarning("Step {Name} is already registered, overwriting", step.Name);
}
_steps[step.Name] = step;
_logger.LogDebug("Registered step: {Name}", step.Name);
}
public IScrapingStep? GetStep(string name)
{
return _steps.TryGetValue(name, out var step) ? step : null;
}
public IEnumerable<IScrapingStep> GetAllSteps()
{
return _steps.Values;
}
}