deemanone/beanflows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update secrets

Browse Source
This commit is contained in:
Deeman
2026-02-27 13:30:53 +01:00
parent 37b48d8f1c
commit 5e22f2e1ae
7 changed files with 82 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
infra/readme.md
View File

@@ -24,7 +24,7 @@ Hetzner Server (NVMe)
1. **Extract** — Supervisor runs due extractors per `infra/supervisor/workflows.toml`
2. **Transform** — SQLMesh reads landing → writes `lakehouse.duckdb`
3. **Export** — `export_serving` copies `serving.*` → `analytics.duckdb` (atomic rename)
4. **Backup** — rclone syncs `/data/materia/landing/` → R2 `materia-raw/landing/`
4. **Backup** — rclone syncs `/data/materia/landing/` → R2 `backup/materia/landing/`
5. **Web** — Web app reads `analytics.duckdb` read-only (per-thread connections)
## Setup (new server)
@@ -59,20 +59,7 @@ ssh root@<server_ip> 'bash -s' < infra/bootstrap_supervisor.sh
This clones the repo via SSH, decrypts secrets, installs Python dependencies, and starts the supervisor service. No access tokens required — access is via the SSH deploy key. (All tools must already be installed by setup_server.sh.)
### 4. Set up R2 backup
```bash
apt install rclone
# Configure rclone as the service user (used by the backup timer):
sudo -u beanflows_service mkdir -p /home/beanflows_service/.config/rclone
sudo -u beanflows_service cp infra/backup/rclone.conf.example \
/home/beanflows_service/.config/rclone/rclone.conf
# Fill in R2 credentials from .env.prod.sops (ACCESS_KEY_ID, SECRET_ACCESS_KEY, bucket endpoint)
cp infra/backup/materia-backup.service /etc/systemd/system/
cp infra/backup/materia-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now materia-backup.timer
```
If `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, and `R2_ENDPOINT` are present in `.env.prod.sops`, bootstrap also generates `rclone.conf` and enables `materia-backup.timer` automatically. No manual R2 setup step needed.
## Secrets management