deemanone/beanflows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(infra): use beanflows_service for supervisor

Browse Source 
- materia-supervisor.service: User=root → User=beanflows_service,
  add PATH so uv (~/.local/bin) is found without a login shell
- setup_server.sh: full rewrite — creates beanflows_service (nologin),
  generates SSH deploy key + age keypair as service user at XDG path
  (~/.config/sops/age/keys.txt), installs age/sops/rclone as root,
  prints both public keys + numbered next-step instructions
- bootstrap_supervisor.sh: full rewrite — removes GITLAB_READ_TOKEN
  requirement, clones via SSH as service user, installs uv as service
  user, decrypts with SOPS auto-discovery, uv sync as service user,
  systemctl as root
- web/deploy.sh: remove self-contained sops/age install + keypair
  generation; replace with simple sops check (exit if missing) and
  SOPS auto-discovery decrypt (no explicit key file needed)
- infra/readme.md: update architecture diagram for beanflows_service
  paths, update setup steps to match new scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-26 21:33:31 +01:00
parent b27f06d811
commit 0317cb885f
5 changed files with 227 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
infra/readme.md
View File

@@ -6,15 +6,17 @@ Single-server local-first setup for BeanFlows.coffee on Hetzner NVMe.
```
Hetzner Server (NVMe)
├── /opt/materia/ # Git repo (checked out at latest release tag)
├── /opt/materia/age-key.txt # Server age keypair (chmod 600, gitignored)
├── /opt/materia/.env # Decrypted from .env.prod.sops at deploy time
├── /data/materia/landing/ # Extracted raw data (immutable, content-addressed)
├── /data/materia/lakehouse.duckdb # SQLMesh exclusive write
├── /data/materia/analytics.duckdb # Read-only serving copy for web app
├── beanflows_service (system user, nologin)
│ ├── ~/.ssh/materia_deploy # ed25519 deploy key for GitLab read access
│ └── ~/.config/sops/age/keys.txt # age keypair (auto-discovered by SOPS)
├── /opt/materia/ # Git repo (owned by beanflows_service, latest release tag)
├── /opt/materia/.env # Decrypted from .env.prod.sops at deploy time
├── /data/materia/landing/ # Extracted raw data (immutable, content-addressed)
├── /data/materia/lakehouse.duckdb # SQLMesh exclusive write
├── /data/materia/analytics.duckdb # Read-only serving copy for web app
└── systemd services:
├── materia-supervisor # Python supervisor: extract → transform → export → deploy
└── materia-backup.timer # rclone: syncs landing/ to R2 every 6 hours
├── materia-supervisor # Python supervisor: extract → transform → export → deploy
└── materia-backup.timer # rclone: syncs landing/ to R2 every 6 hours
```
## Data Flow
@@ -33,15 +35,16 @@ Hetzner Server (NVMe)
bash infra/setup_server.sh
```
This creates data directories, installs age, and generates the server age keypair at `/opt/materia/age-key.txt`. It prints the server's age public key.
This creates the `beanflows_service` user, data directories, installs age + sops + rclone, generates an ed25519 SSH deploy key and an age keypair (both as the service user). It prints both public keys.
### 2. Add the server key to SOPS
On your workstation:
### 2. Add keys to GitLab and SOPS
```bash
# Add the server public key to .sops.yaml
# Then re-encrypt prod secrets to include the server key:
# Add the SSH deploy key to GitLab:
# → Repository Settings → Deploy Keys → Add key (read-only)
# Add the server age public key to .sops.yaml on your workstation,
# then re-encrypt prod secrets to include the server key:
sops updatekeys .env.prod.sops
git add .sops.yaml .env.prod.sops
git commit -m "chore: add server age key"
@@ -51,18 +54,19 @@ git push
### 3. Bootstrap the supervisor
```bash
# Requires GITLAB_READ_TOKEN (GitLab project access token, read-only)
export GITLAB_READ_TOKEN=<token>
ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh
```
This installs uv + sops + age, clones the repo, decrypts secrets, installs Python dependencies, and starts the supervisor service.
This installs uv (as service user), clones the repo via SSH, decrypts secrets, installs Python dependencies, and starts the supervisor service. No access tokens required — access is via the SSH deploy key.
### 4. Set up R2 backup
```bash
apt install rclone
cp infra/backup/rclone.conf.example /root/.config/rclone/rclone.conf
# Configure rclone as the service user (used by the backup timer):
sudo -u beanflows_service mkdir -p /home/beanflows_service/.config/rclone
sudo -u beanflows_service cp infra/backup/rclone.conf.example \
/home/beanflows_service/.config/rclone/rclone.conf
# Fill in R2 credentials from .env.prod.sops (ACCESS_KEY_ID, SECRET_ACCESS_KEY, bucket endpoint)
cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
@@ -90,6 +94,7 @@ make secrets-edit-prod
`bootstrap_supervisor.sh` decrypts `.env.prod.sops` → `/opt/materia/.env` during setup.
`web/deploy.sh` re-decrypts on every deploy (so secret rotations take effect automatically).
SOPS auto-discovers the service user's age key at `~/.config/sops/age/keys.txt` (XDG default).
## Deploy model (pull-based)
@@ -109,7 +114,7 @@ systemctl status materia-supervisor
journalctl -u materia-supervisor -f
# Workflow status table
cd /opt/materia && uv run python src/materia/supervisor.py status
cd /opt/materia && sudo -u beanflows_service uv run python src/materia/supervisor.py status
# Backup timer status
systemctl list-timers materia-backup.timer