5d7d53a260
Port padelnomics' schedule-aware Python supervisor to materia: - src/materia/supervisor.py — croniter scheduling, topological wave execution (parallel independent workflows), tag-based git pull + deploy, status CLI subcommand - infra/supervisor/workflows.toml — workflow registry (psd daily, cot weekly, prices daily, ice daily, weather daily) - infra/supervisor/materia-supervisor.service — updated ExecStart to Python supervisor, added SUPERVISOR_GIT_PULL=1 Adaptations from padelnomics: - Uses extract_core.state.open_state_db (not padelnomics_extract.utils) - uv run sqlmesh -p transform/sqlmesh_materia run - uv run materia pipeline run export_serving - web/deploy.sh path (materia's deploy.sh is under web/) - Removed proxy_mode (not used in materia) Also: add croniter dependency to src/materia, delete old supervisor.sh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
35 lines
968 B
TOML
35 lines
968 B
TOML
# Workflow registry — the supervisor reads this file on every tick.
|
|
# To add a new extractor: add a [section] here and create the Python module.
|
|
#
|
|
# Fields:
|
|
# module — Python module path (must expose an entry function)
|
|
# entry — function name in the module (default: "main")
|
|
# schedule — named preset ("hourly", "daily", "weekly", "monthly")
|
|
# or raw cron expression (e.g. "0 6 * * 1-5")
|
|
# depends_on — optional: list of workflow names that must complete first
|
|
|
|
[extract_psd]
|
|
module = "psdonline.execute"
|
|
entry = "extract_psd_dataset"
|
|
schedule = "daily"
|
|
|
|
[extract_cot]
|
|
module = "cftc_cot.execute"
|
|
entry = "extract_cot_dataset"
|
|
schedule = "weekly"
|
|
|
|
[extract_prices]
|
|
module = "coffee_prices.execute"
|
|
entry = "extract_coffee_prices"
|
|
schedule = "daily"
|
|
|
|
[extract_ice]
|
|
module = "ice_stocks.execute"
|
|
entry = "extract_ice_all"
|
|
schedule = "daily"
|
|
|
|
[extract_weather]
|
|
module = "openmeteo.execute"
|
|
entry = "extract_weather"
|
|
schedule = "daily"
|