Files

Deeman 8d6d79345c infra: add setup_server.sh for one-time server provisioning

Creates the beanflows system user, /opt/beanflows directory, and an
ed25519 GitLab deploy key. Prints the public key to add as a read-only
deploy key on the repo.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-02-22 15:12:21 +01:00

backup

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

supervisor

fix: export_serving — Arrow-based copy, rename to analytics.duckdb

2026-02-22 12:54:39 +01:00

bootstrap_supervisor.sh

fix: export_serving — Arrow-based copy, rename to analytics.duckdb

2026-02-22 12:54:39 +01:00

future_enhancements.md

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.dev.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.prod.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

pulumi.yaml

cleanup and prefect service setup

2026-02-05 20:01:50 +01:00

readme.md

Refactor to local-first architecture on Hetzner NVMe

2026-02-18 19:50:19 +01:00

setup_server.sh

infra: add setup_server.sh for one-time server provisioning

2026-02-22 15:12:21 +01:00

readme.md

Materia Infrastructure

Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.

Architecture

Hetzner Server (NVMe)
├── /opt/materia/              # Git repo, code, uv environment
├── /data/materia/landing/     # Extracted USDA data (year/month subdirs)
├── /data/materia/lakehouse.duckdb  # SQLMesh output database
└── systemd services:
    ├── materia-supervisor     # Pulls git, runs extract + transform daily
    └── materia-backup.timer   # Syncs landing/ to R2 every 6 hours

Data Flow

Extract: USDA API → /data/materia/landing/psd/{year}/{month}/{etag}.csv.gzip
Transform: SQLMesh reads landing CSVs → writes to /data/materia/lakehouse.duckdb
Backup: rclone syncs /data/materia/landing/ → R2 materia-raw/landing/
Web: Reads lakehouse.duckdb (read-only)

Setup

Prerequisites

Hetzner server with NVMe storage
Pulumi ESC configured (beanflows/prod environment)
GITLAB_READ_TOKEN and PULUMI_ACCESS_TOKEN set

Bootstrap

# From local machine or CI:
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh

This installs dependencies, clones the repo, creates data directories, and starts the supervisor service.

R2 Backup

Install rclone: apt install rclone
Copy and configure: cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
Fill in R2 credentials from Pulumi ESC
Install systemd units:

cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer

Pulumi IaC

Still manages Cloudflare R2 buckets and can provision Hetzner instances:

cd infra
pulumi login
pulumi stack select prod
pulumi up

Monitoring

# Supervisor status and logs
systemctl status materia-supervisor
journalctl -u materia-supervisor -f

# Backup timer status
systemctl list-timers materia-backup.timer
journalctl -u materia-backup -f

Cost

Resource	Type	Cost
Hetzner Server	CCX22 (4 vCPU, 16GB)	~€24/mo
R2 Storage	Backup (~10 GB)	$0.15/mo
R2 Egress	Zero	$0.00
Total		~~€24/mo (~~$26)