feat(supervisor): port Python supervisor from padelnomics + workflows.toml

Port padelnomics' schedule-aware Python supervisor to materia:
- src/materia/supervisor.py — croniter scheduling, topological wave
  execution (parallel independent workflows), tag-based git pull + deploy,
  status CLI subcommand
- infra/supervisor/workflows.toml — workflow registry (psd daily, cot
  weekly, prices daily, ice daily, weather daily)
- infra/supervisor/materia-supervisor.service — updated ExecStart to Python
  supervisor, added SUPERVISOR_GIT_PULL=1

Adaptations from padelnomics:
- Uses extract_core.state.open_state_db (not padelnomics_extract.utils)
- uv run sqlmesh -p transform/sqlmesh_materia run
- uv run materia pipeline run export_serving
- web/deploy.sh path (materia's deploy.sh is under web/)
- Removed proxy_mode (not used in materia)

Also: add croniter dependency to src/materia, delete old supervisor.sh.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-26 11:59:55 +01:00
parent 520da2c920
commit 5d7d53a260
6 changed files with 503 additions and 86 deletions

View File

@@ -7,13 +7,14 @@ Wants=network-online.target
Type=simple
User=root
WorkingDirectory=/opt/materia
ExecStart=/opt/materia/infra/supervisor/supervisor.sh
ExecStart=/bin/sh -c 'exec uv run python src/materia/supervisor.py'
Restart=always
RestartSec=10
EnvironmentFile=/opt/materia/.env
Environment=LANDING_DIR=/data/materia/landing
Environment=DUCKDB_PATH=/data/materia/lakehouse.duckdb
Environment=SERVING_DUCKDB_PATH=/data/materia/analytics.duckdb
Environment=SUPERVISOR_GIT_PULL=1
# Resource limits
LimitNOFILE=65536