Files

Deeman 5d7d53a260 feat(supervisor): port Python supervisor from padelnomics + workflows.toml

Port padelnomics' schedule-aware Python supervisor to materia:
- src/materia/supervisor.py — croniter scheduling, topological wave
  execution (parallel independent workflows), tag-based git pull + deploy,
  status CLI subcommand
- infra/supervisor/workflows.toml — workflow registry (psd daily, cot
  weekly, prices daily, ice daily, weather daily)
- infra/supervisor/materia-supervisor.service — updated ExecStart to Python
  supervisor, added SUPERVISOR_GIT_PULL=1

Adaptations from padelnomics:
- Uses extract_core.state.open_state_db (not padelnomics_extract.utils)
- uv run sqlmesh -p transform/sqlmesh_materia run
- uv run materia pipeline run export_serving
- web/deploy.sh path (materia's deploy.sh is under web/)
- Removed proxy_mode (not used in materia)

Also: add croniter dependency to src/materia, delete old supervisor.sh.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-26 11:59:55 +01:00

backup

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

supervisor

feat(supervisor): port Python supervisor from padelnomics + workflows.toml

2026-02-26 11:59:55 +01:00

bootstrap_supervisor.sh

fix: export_serving — Arrow-based copy, rename to analytics.duckdb

2026-02-22 12:54:39 +01:00

future_enhancements.md

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.dev.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.prod.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

readme.md

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

setup_server.sh

infra: fix CRLF line endings in setup_server.sh

2026-02-22 15:24:22 +01:00

readme.md

Materia Infrastructure

Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.

Architecture

Hetzner Server (NVMe)
├── /opt/materia/              # Git repo, code, uv environment
├── /data/materia/landing/     # Extracted USDA data (year/month subdirs)
├── /data/materia/lakehouse.duckdb  # SQLMesh output database
└── systemd services:
    ├── materia-supervisor     # Pulls git, runs extract + transform daily
    └── materia-backup.timer   # Syncs landing/ to R2 every 6 hours

Data Flow

Extract: USDA API → /data/materia/landing/psd/{year}/{month}/{etag}.csv.gzip
Transform: SQLMesh reads landing CSVs → writes to /data/materia/lakehouse.duckdb
Backup: rclone syncs /data/materia/landing/ → R2 materia-raw/landing/
Web: Reads lakehouse.duckdb (read-only)

Setup

Prerequisites

Hetzner server with NVMe storage
Pulumi ESC configured (beanflows/prod environment)
GITLAB_READ_TOKEN and PULUMI_ACCESS_TOKEN set

Bootstrap

# From local machine or CI:
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh

This installs dependencies, clones the repo, creates data directories, and starts the supervisor service.

R2 Backup

Install rclone: apt install rclone
Copy and configure: cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
Fill in R2 credentials from Pulumi ESC
Install systemd units:

cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer

Pulumi IaC

Still manages Cloudflare R2 buckets and can provision Hetzner instances:

cd infra
pulumi login
pulumi stack select prod
pulumi up

Monitoring

# Supervisor status and logs
systemctl status materia-supervisor
journalctl -u materia-supervisor -f

# Backup timer status
systemctl list-timers materia-backup.timer
journalctl -u materia-backup -f

Cost

Resource	Type	Cost
Hetzner Server	CCX22 (4 vCPU, 16GB)	~€24/mo
R2 Storage	Backup (~10 GB)	$0.15/mo
R2 Egress	Zero	$0.00
Total		~~€24/mo (~~$26)