feat: Python supervisor + feature flags
Supervisor (replaces supervisor.sh): - supervisor.py — cron-based pipeline orchestration, reads workflows.toml on every tick, runs due extractors in topological waves with parallel execution, then SQLMesh transform + serving export - workflows.toml — workflow registry: overpass (monthly), eurostat (monthly), playtomic_tenants (weekly), playtomic_availability (daily), playtomic_recheck (hourly 6–23) - padelnomics-supervisor.service — updated ExecStart to Python supervisor Extraction enhancements: - proxy.py — optional round-robin/sticky proxy rotation via PROXY_URLS env - playtomic_availability.py — parallel fetch (EXTRACT_WORKERS), recheck mode (main_recheck) re-queries imminent slots for accurate occupancy measurement - _shared.py — realistic browser User-Agent on all extractor sessions - stg_playtomic_availability.sql — reads morning + recheck snapshots, tags each - fct_daily_availability.sql — prefers recheck over morning for same slot Feature flags (replaces WAITLIST_MODE env var): - migration 0019 — feature_flags table, 5 initial flags: markets (on), payments/planner_export/supplier_signup/lead_unlock (off) - core.py — is_flag_enabled() + feature_gate() decorator - routes — payments, markets, planner_export, supplier_signup, lead_unlock gated - admin flags UI — /admin/flags toggle page + nav link - app.py — flag() injected as Jinja2 global Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
33
infra/supervisor/workflows.toml
Normal file
33
infra/supervisor/workflows.toml
Normal file
@@ -0,0 +1,33 @@
|
||||
# Workflow registry — the supervisor reads this file on every tick.
|
||||
# To add a new extractor: add a [section] here and create the Python module.
|
||||
#
|
||||
# Fields:
|
||||
# module — Python module path (must have a main() function)
|
||||
# schedule — named preset ("hourly", "daily", "weekly", "monthly")
|
||||
# or raw cron expression (e.g. "0 6-23 * * *")
|
||||
# entry — optional: function name if not "main" (default: "main")
|
||||
# depends_on — optional: list of workflow names that must run first
|
||||
# proxy_mode — optional: "round-robin" (default) or "sticky"
|
||||
|
||||
[overpass]
|
||||
module = "padelnomics_extract.overpass"
|
||||
schedule = "monthly"
|
||||
|
||||
[eurostat]
|
||||
module = "padelnomics_extract.eurostat"
|
||||
schedule = "monthly"
|
||||
|
||||
[playtomic_tenants]
|
||||
module = "padelnomics_extract.playtomic_tenants"
|
||||
schedule = "weekly"
|
||||
|
||||
[playtomic_availability]
|
||||
module = "padelnomics_extract.playtomic_availability"
|
||||
schedule = "daily"
|
||||
depends_on = ["playtomic_tenants"]
|
||||
|
||||
[playtomic_recheck]
|
||||
module = "padelnomics_extract.playtomic_availability"
|
||||
entry = "main_recheck"
|
||||
schedule = "0 6-23 * * *"
|
||||
depends_on = ["playtomic_availability"]
|
||||
Reference in New Issue
Block a user