Files

Deeman 95f881827e feat(infra): replace Pulumi ESC with SOPS in bootstrap + setup scripts

- bootstrap_supervisor.sh: remove esc CLI + PULUMI_ACCESS_TOKEN; install
  sops+age; check age keypair exists; decrypt .env.prod.sops → .env;
  checkout latest release tag; use uv sync --all-packages
- setup_server.sh: add age keypair generation at /opt/materia/age-key.txt;
  install age binary; print public key with .sops.yaml instructions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-26 12:03:11 +01:00

backup

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

supervisor

feat(supervisor): port Python supervisor from padelnomics + workflows.toml

2026-02-26 11:59:55 +01:00

bootstrap_supervisor.sh

feat(infra): replace Pulumi ESC with SOPS in bootstrap + setup scripts

2026-02-26 12:03:11 +01:00

future_enhancements.md

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.dev.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.prod.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

readme.md

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

setup_server.sh

feat(infra): replace Pulumi ESC with SOPS in bootstrap + setup scripts

2026-02-26 12:03:11 +01:00

readme.md

Materia Infrastructure

Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.

Architecture

Hetzner Server (NVMe)
├── /opt/materia/              # Git repo, code, uv environment
├── /data/materia/landing/     # Extracted USDA data (year/month subdirs)
├── /data/materia/lakehouse.duckdb  # SQLMesh output database
└── systemd services:
    ├── materia-supervisor     # Pulls git, runs extract + transform daily
    └── materia-backup.timer   # Syncs landing/ to R2 every 6 hours

Data Flow

Extract: USDA API → /data/materia/landing/psd/{year}/{month}/{etag}.csv.gzip
Transform: SQLMesh reads landing CSVs → writes to /data/materia/lakehouse.duckdb
Backup: rclone syncs /data/materia/landing/ → R2 materia-raw/landing/
Web: Reads lakehouse.duckdb (read-only)

Setup

Prerequisites

Hetzner server with NVMe storage
Pulumi ESC configured (beanflows/prod environment)
GITLAB_READ_TOKEN and PULUMI_ACCESS_TOKEN set

Bootstrap

# From local machine or CI:
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh

This installs dependencies, clones the repo, creates data directories, and starts the supervisor service.

R2 Backup

Install rclone: apt install rclone
Copy and configure: cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
Fill in R2 credentials from Pulumi ESC
Install systemd units:

cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer

Pulumi IaC

Still manages Cloudflare R2 buckets and can provision Hetzner instances:

cd infra
pulumi login
pulumi stack select prod
pulumi up

Monitoring

# Supervisor status and logs
systemctl status materia-supervisor
journalctl -u materia-supervisor -f

# Backup timer status
systemctl list-timers materia-backup.timer
journalctl -u materia-backup -f

Cost

Resource	Type	Cost
Hetzner Server	CCX22 (4 vCPU, 16GB)	~€24/mo
R2 Storage	Backup (~10 GB)	$0.15/mo
R2 Egress	Zero	$0.00
Total		~~€24/mo (~~$26)