Files

Deeman c1d00dcdc4 Refactor to local-first architecture on Hetzner NVMe

Remove distributed R2/Iceberg/SSH pipeline architecture in favor of
local subprocess execution with NVMe storage. Landing data backed up
to R2 via rclone timer.

- Strip Iceberg catalog, httpfs, boto3, paramiko, prefect, pyarrow
- Pipelines run via subprocess.run() with bounded timeouts
- Extract writes to {LANDING_DIR}/psd/{year}/{month}/{etag}.csv.gzip
- SQLMesh reads LANDING_DIR variable, writes to DUCKDB_PATH
- Delete unused provider stubs (ovh, scaleway, oracle)
- Add rclone systemd timer for R2 backup every 6h
- Update supervisor to run pipelines with env vars

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-18 19:50:19 +01:00

backup

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

supervisor

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

bootstrap_supervisor.sh

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

future_enhancements.md

cleanup and prefect service setup

2026-02-04 22:24:55 +01:00

pulumi.dev.yaml

cleanup and prefect service setup

2026-02-04 22:24:55 +01:00

pulumi.prod.yaml

cleanup and prefect service setup

2026-02-04 22:24:55 +01:00

pulumi.yaml

cleanup and prefect service setup

2026-02-04 22:24:55 +01:00

readme.md

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

readme.md

Materia Infrastructure

Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.

Architecture

Hetzner Server (NVMe)
├── /opt/materia/              # Git repo, code, uv environment
├── /data/materia/landing/     # Extracted USDA data (year/month subdirs)
├── /data/materia/lakehouse.duckdb  # SQLMesh output database
└── systemd services:
    ├── materia-supervisor     # Pulls git, runs extract + transform daily
    └── materia-backup.timer   # Syncs landing/ to R2 every 6 hours

Data Flow

Extract: USDA API → /data/materia/landing/psd/{year}/{month}/{etag}.csv.gzip
Transform: SQLMesh reads landing CSVs → writes to /data/materia/lakehouse.duckdb
Backup: rclone syncs /data/materia/landing/ → R2 materia-raw/landing/
Web: Reads lakehouse.duckdb (read-only)

Setup

Prerequisites

Hetzner server with NVMe storage
Pulumi ESC configured (beanflows/prod environment)
GITLAB_READ_TOKEN and PULUMI_ACCESS_TOKEN set

Bootstrap

# From local machine or CI:
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh

This installs dependencies, clones the repo, creates data directories, and starts the supervisor service.

R2 Backup

Install rclone: apt install rclone
Copy and configure: cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
Fill in R2 credentials from Pulumi ESC
Install systemd units:

cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer

Pulumi IaC

Still manages Cloudflare R2 buckets and can provision Hetzner instances:

cd infra
pulumi login
pulumi stack select prod
pulumi up

Monitoring

# Supervisor status and logs
systemctl status materia-supervisor
journalctl -u materia-supervisor -f

# Backup timer status
systemctl list-timers materia-backup.timer
journalctl -u materia-backup -f

Cost

Resource	Type	Cost
Hetzner Server	CCX22 (4 vCPU, 16GB)	~€24/mo
R2 Storage	Backup (~10 GB)	$0.15/mo
R2 Egress	Zero	$0.00
Total		~~€24/mo (~~$26)