beanflows/infra/readme.md

Materia Infrastructure

Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.

Architecture

Hetzner Server (NVMe)
├── /opt/materia/                    # Git repo (checked out at latest release tag)
├── /opt/materia/age-key.txt         # Server age keypair (chmod 600, gitignored)
├── /opt/materia/.env                # Decrypted from .env.prod.sops at deploy time
├── /data/materia/landing/           # Extracted raw data (immutable, content-addressed)
├── /data/materia/lakehouse.duckdb   # SQLMesh exclusive write
├── /data/materia/analytics.duckdb  # Read-only serving copy for web app
└── systemd services:
    ├── materia-supervisor           # Python supervisor: extract → transform → export → deploy
    └── materia-backup.timer         # rclone: syncs landing/ to R2 every 6 hours

Data Flow

1. Extract — Supervisor runs due extractors per infra/supervisor/workflows.toml
2. Transform — SQLMesh reads landing → writes lakehouse.duckdb
3. Export — export_serving copies serving.* → analytics.duckdb (atomic rename)
4. Backup — rclone syncs /data/materia/landing/ → R2 materia-raw/landing/
5. Web — Web app reads analytics.duckdb read-only (per-thread connections)

Setup (new server)

1. Run setup_server.sh

bash infra/setup_server.sh

This creates data directories, installs age, and generates the server age keypair at /opt/materia/age-key.txt. It prints the server's age public key.

2. Add the server key to SOPS

On your workstation:

# Add the server public key to .sops.yaml
# Then re-encrypt prod secrets to include the server key:
sops updatekeys .env.prod.sops
git add .sops.yaml .env.prod.sops
git commit -m "chore: add server age key"
git push

3. Bootstrap the supervisor

# Requires GITLAB_READ_TOKEN (GitLab project access token, read-only)
export GITLAB_READ_TOKEN=<token>
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh

This installs uv + sops + age, clones the repo, decrypts secrets, installs Python dependencies, and starts the supervisor service.

4. Set up R2 backup

apt install rclone
cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
# Fill in R2 credentials from .env.prod.sops (ACCESS_KEY_ID, SECRET_ACCESS_KEY, bucket endpoint)
cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer

Secrets management

Secrets are stored as SOPS-encrypted dotenv files in the repo root:

File	Purpose
.env.dev.sops	Dev defaults (safe values, local paths)
.env.prod.sops	Production secrets
.sops.yaml	Maps file patterns to age public keys

# Decrypt for local dev
make secrets-decrypt-dev

# Edit prod secrets
make secrets-edit-prod

bootstrap_supervisor.sh decrypts .env.prod.sops → /opt/materia/.env during setup. web/deploy.sh re-decrypts on every deploy (so secret rotations take effect automatically).

Deploy model (pull-based)

No SSH keys or deploy credentials in CI.

1. CI runs tests (test:cli, test:sqlmesh, test:web)
2. On master, CI creates tag v${CI_PIPELINE_IID} using built-in CI_JOB_TOKEN
3. Supervisor polls for new tags every 60s
4. When a new tag appears: git checkout --detach <tag> + uv sync --all-packages
5. If web/ files changed: ./web/deploy.sh (Docker blue/green + health check)

Monitoring

# Supervisor status and logs
systemctl status materia-supervisor
journalctl -u materia-supervisor -f

# Workflow status table
cd /opt/materia && uv run python src/materia/supervisor.py status

# Backup timer status
systemctl list-timers materia-backup.timer
journalctl -u materia-backup -f

# Extraction state DB
sqlite3 /data/materia/landing/.state.sqlite \
  "SELECT extractor, status, finished_at FROM extraction_runs ORDER BY run_id DESC LIMIT 20"

Pulumi IaC

Still manages Cloudflare R2 buckets:

cd infra
pulumi login
pulumi stack select prod
pulumi up

Cost

Resource	Type	Cost
Hetzner Server	CCX22 (4 vCPU, 16GB)	~€24/mo
R2 Storage	Backup (~10 GB)	$0.15/mo
R2 Egress	Zero	$0.00
Total		€24/mo ($26)