diff --git a/CLAUDE.md b/CLAUDE.md index 9a6bb27..40aac0e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,42 @@ pre-commit install uv add ``` +## Secrets Management with Pulumi ESC + +All secrets are managed via Pulumi ESC (Environment, Secrets, and Configuration). The production environment is `beanflows/prod`. + +**Load secrets into your shell:** +```bash +# Login to Pulumi ESC (one-time) +esc login + +# Load secrets as environment variables +eval $(esc env open beanflows/prod --format shell) + +# Now all secrets are available as env vars +echo $R2_ENDPOINT # Example: access R2 endpoint +``` + +**Run commands with ESC secrets:** +```bash +# Run a command with secrets loaded +esc run beanflows/prod -- uv run extract_psd + +# Run multiple commands +esc run beanflows/prod -- bash -c " + uv run extract_psd + cd transform/sqlmesh_materia && uv run sqlmesh plan prod +" +``` + +**Available secrets in `beanflows/prod`:** +- R2 storage: `R2_ENDPOINT`, `R2_BUCKET`, `R2_ACCESS_KEY`, `R2_SECRET_KEY` +- Hetzner Cloud: `HETZNER_TOKEN`, SSH keys +- GitLab: `GITLAB_READ_TOKEN` +- Iceberg catalog credentials + +**Note:** Never hardcode secrets! Always use Pulumi ESC or environment variables. + ## Project Structure This is a uv workspace with three main components: @@ -208,9 +244,11 @@ pytest --cov=./ --cov-report=xml - Runs on every master push - Creates/updates Hetzner CPX11 supervisor instance (~€4.49/mo) - Uses Pulumi ESC (`beanflows/prod`) for all secrets -- **`deploy:supervisor`**: Checks supervisor status - - Verifies supervisor is bootstrapped - - Supervisor auto-updates via `git pull` every 15 minutes (no CI/CD deployment needed) +- **`deploy:supervisor`**: Bootstraps and monitors supervisor + - Checks if supervisor is already bootstrapped (`test -d /opt/materia/.git`) + - If not bootstrapped: Runs `infra/bootstrap_supervisor.sh` automatically + - If already bootstrapped: Verifies service status + - After bootstrap: Supervisor auto-updates via `git pull` every 15 minutes **Note:** No build artifacts! Supervisor pulls code directly from git and runs via `uv`. @@ -236,12 +274,15 @@ pytest --cov=./ --cov-report=xml - Uses systemd service for automatic restart on failure - Pulls secrets from Pulumi ESC -**Bootstrap (one-time):** -```bash -# Get supervisor IP from Pulumi -cd infra && pulumi stack output supervisor_ip -s prod +**Bootstrap:** +Bootstrapping happens automatically in CI/CD (`deploy:supervisor` stage). The pipeline: +1. Checks if supervisor is already bootstrapped +2. If not: Runs `infra/bootstrap_supervisor.sh` with secrets injected +3. If yes: Verifies systemd service status -# Run bootstrap script +Manual bootstrap (if needed): +```bash +cd infra && pulumi stack output supervisor_ip -s prod export PULUMI_ACCESS_TOKEN= ssh root@ 'bash -s' < infra/bootstrap_supervisor.sh ```