c1d00dcdc4
Remove distributed R2/Iceberg/SSH pipeline architecture in favor of
local subprocess execution with NVMe storage. Landing data backed up
to R2 via rclone timer.
- Strip Iceberg catalog, httpfs, boto3, paramiko, prefect, pyarrow
- Pipelines run via subprocess.run() with bounded timeouts
- Extract writes to {LANDING_DIR}/psd/{year}/{month}/{etag}.csv.gzip
- SQLMesh reads LANDING_DIR variable, writes to DUCKDB_PATH
- Delete unused provider stubs (ovh, scaleway, oracle)
- Add rclone systemd timer for R2 backup every 6h
- Update supervisor to run pipelines with env vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
86 lines
2.3 KiB
Markdown
86 lines
2.3 KiB
Markdown
# Materia Infrastructure
|
|
|
|
Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Hetzner Server (NVMe)
|
|
├── /opt/materia/ # Git repo, code, uv environment
|
|
├── /data/materia/landing/ # Extracted USDA data (year/month subdirs)
|
|
├── /data/materia/lakehouse.duckdb # SQLMesh output database
|
|
└── systemd services:
|
|
├── materia-supervisor # Pulls git, runs extract + transform daily
|
|
└── materia-backup.timer # Syncs landing/ to R2 every 6 hours
|
|
```
|
|
|
|
## Data Flow
|
|
|
|
1. **Extract**: USDA API → `/data/materia/landing/psd/{year}/{month}/{etag}.csv.gzip`
|
|
2. **Transform**: SQLMesh reads landing CSVs → writes to `/data/materia/lakehouse.duckdb`
|
|
3. **Backup**: rclone syncs `/data/materia/landing/` → R2 `materia-raw/landing/`
|
|
4. **Web**: Reads `lakehouse.duckdb` (read-only)
|
|
|
|
## Setup
|
|
|
|
### Prerequisites
|
|
|
|
- Hetzner server with NVMe storage
|
|
- Pulumi ESC configured (`beanflows/prod` environment)
|
|
- `GITLAB_READ_TOKEN` and `PULUMI_ACCESS_TOKEN` set
|
|
|
|
### Bootstrap
|
|
|
|
```bash
|
|
# From local machine or CI:
|
|
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh
|
|
```
|
|
|
|
This installs dependencies, clones the repo, creates data directories, and starts the supervisor service.
|
|
|
|
### R2 Backup
|
|
|
|
1. Install rclone: `apt install rclone`
|
|
2. Copy and configure: `cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf`
|
|
3. Fill in R2 credentials from Pulumi ESC
|
|
4. Install systemd units:
|
|
|
|
```bash
|
|
cp infra/backup/materia-backup.service /etc/systemd/system/
|
|
cp infra/backup/materia-backup.timer /etc/systemd/system/
|
|
systemctl daemon-reload
|
|
systemctl enable --now materia-backup.timer
|
|
```
|
|
|
|
## Pulumi IaC
|
|
|
|
Still manages Cloudflare R2 buckets and can provision Hetzner instances:
|
|
|
|
```bash
|
|
cd infra
|
|
pulumi login
|
|
pulumi stack select prod
|
|
pulumi up
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
```bash
|
|
# Supervisor status and logs
|
|
systemctl status materia-supervisor
|
|
journalctl -u materia-supervisor -f
|
|
|
|
# Backup timer status
|
|
systemctl list-timers materia-backup.timer
|
|
journalctl -u materia-backup -f
|
|
```
|
|
|
|
## Cost
|
|
|
|
| Resource | Type | Cost |
|
|
|----------|------|------|
|
|
| Hetzner Server | CCX22 (4 vCPU, 16GB) | ~€24/mo |
|
|
| R2 Storage | Backup (~10 GB) | $0.15/mo |
|
|
| R2 Egress | Zero | $0.00 |
|
|
| **Total** | | **~€24/mo (~$26)** |
|