From 57f290900115e0b8d9a88c7b0775815e05157d7a Mon Sep 17 00:00:00 2001
From: Deeman <hendriknote@gmail.com>
Date: Mon, 20 Oct 2025 22:07:24 +0200
Subject: [PATCH] Update documentation: Pulumi ESC usage and CI/CD bootstrap
 clarification
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Changes

1. **Added Pulumi ESC section**
   - How to login and load secrets into shell
   - `esc run` command for running commands with secrets
   - List of available secrets in `beanflows/prod` environment
   - Examples for common use cases

2. **Fixed supervisor bootstrap documentation**
   - Clarified that bootstrapping happens automatically in CI/CD
   - Pipeline checks if supervisor is already bootstrapped
   - Runs bootstrap script automatically only if needed
   - Removed misleading "one-time" manual bootstrap instructions
   - Added note that it's only needed manually in exceptional cases

3. **Updated deploy:supervisor stage description**
   - More accurate description of the bootstrap check logic
   - Explains the conditional execution (bootstrap vs status check)

These updates make the documentation more accurate and helpful for both
local development (with ESC) and understanding the production deployment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 CLAUDE.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 49 insertions(+), 8 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 9a6bb27..40aac0e 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -29,6 +29,42 @@ pre-commit install
 uv add <package-name>
 ```
 
+## Secrets Management with Pulumi ESC
+
+All secrets are managed via Pulumi ESC (Environment, Secrets, and Configuration). The production environment is `beanflows/prod`.
+
+**Load secrets into your shell:**
+```bash
+# Login to Pulumi ESC (one-time)
+esc login
+
+# Load secrets as environment variables
+eval $(esc env open beanflows/prod --format shell)
+
+# Now all secrets are available as env vars
+echo $R2_ENDPOINT  # Example: access R2 endpoint
+```
+
+**Run commands with ESC secrets:**
+```bash
+# Run a command with secrets loaded
+esc run beanflows/prod -- uv run extract_psd
+
+# Run multiple commands
+esc run beanflows/prod -- bash -c "
+  uv run extract_psd
+  cd transform/sqlmesh_materia && uv run sqlmesh plan prod
+"
+```
+
+**Available secrets in `beanflows/prod`:**
+- R2 storage: `R2_ENDPOINT`, `R2_BUCKET`, `R2_ACCESS_KEY`, `R2_SECRET_KEY`
+- Hetzner Cloud: `HETZNER_TOKEN`, SSH keys
+- GitLab: `GITLAB_READ_TOKEN`
+- Iceberg catalog credentials
+
+**Note:** Never hardcode secrets! Always use Pulumi ESC or environment variables.
+
 ## Project Structure
 
 This is a uv workspace with three main components:
@@ -208,9 +244,11 @@ pytest --cov=./ --cov-report=xml
   - Runs on every master push
   - Creates/updates Hetzner CPX11 supervisor instance (~€4.49/mo)
   - Uses Pulumi ESC (`beanflows/prod`) for all secrets
-- **`deploy:supervisor`**: Checks supervisor status
-  - Verifies supervisor is bootstrapped
-  - Supervisor auto-updates via `git pull` every 15 minutes (no CI/CD deployment needed)
+- **`deploy:supervisor`**: Bootstraps and monitors supervisor
+  - Checks if supervisor is already bootstrapped (`test -d /opt/materia/.git`)
+  - If not bootstrapped: Runs `infra/bootstrap_supervisor.sh` automatically
+  - If already bootstrapped: Verifies service status
+  - After bootstrap: Supervisor auto-updates via `git pull` every 15 minutes
 
 **Note:** No build artifacts! Supervisor pulls code directly from git and runs via `uv`.
 
@@ -236,12 +274,15 @@ pytest --cov=./ --cov-report=xml
 - Uses systemd service for automatic restart on failure
 - Pulls secrets from Pulumi ESC
 
-**Bootstrap (one-time):**
-```bash
-# Get supervisor IP from Pulumi
-cd infra && pulumi stack output supervisor_ip -s prod
+**Bootstrap:**
+Bootstrapping happens automatically in CI/CD (`deploy:supervisor` stage). The pipeline:
+1. Checks if supervisor is already bootstrapped
+2. If not: Runs `infra/bootstrap_supervisor.sh` with secrets injected
+3. If yes: Verifies systemd service status
 
-# Run bootstrap script
+Manual bootstrap (if needed):
+```bash
+cd infra && pulumi stack output supervisor_ip -s prod
 export PULUMI_ACCESS_TOKEN=<your-token>
 ssh root@<supervisor-ip> 'bash -s' < infra/bootstrap_supervisor.sh
 ```