b899bcbad4
Split the single lakehouse.duckdb into two files to eliminate the exclusive write-lock conflict between SQLMesh (pipeline) and the Quart web app (reader): lakehouse.duckdb — SQLMesh exclusive (all pipeline layers) serving.duckdb — web app reads (serving tables only, atomically swapped) Changes: web/src/beanflows/analytics.py - Replace persistent global _conn with per-thread connections (threading.local) - Add _get_conn(): opens read_only=True on first call per thread, reopens automatically on inode change (~1μs os.stat) to pick up atomic file swaps - Switch env var from DUCKDB_PATH → SERVING_DUCKDB_PATH - Add module docstring documenting architecture + DuckLake migration path web/src/beanflows/app.py - Startup check: use SERVING_DUCKDB_PATH - Health check: use _db_path instead of _conn src/materia/export_serving.py (new) - Reads all serving.* tables from lakehouse.duckdb (read_only) - Writes to serving_new.duckdb, then os.rename → serving.duckdb (atomic) - ~50 lines; runs after each SQLMesh transform src/materia/pipelines.py - Add export_serving pipeline entry (uv run python -c ...) infra/supervisor/supervisor.sh - Add SERVING_DUCKDB_PATH env var comment - Add export step: uv run materia pipeline run export_serving infra/supervisor/materia-supervisor.service - Add Environment=SERVING_DUCKDB_PATH=/data/materia/serving.duckdb infra/bootstrap_supervisor.sh - Add SERVING_DUCKDB_PATH to .env template web/.env.example + web/docker-compose.yml - Document both env vars; switch web service to SERVING_DUCKDB_PATH web/src/beanflows/dashboard/templates/settings.html - Minor settings page fix from prior session Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
70 lines
2.7 KiB
Bash
70 lines
2.7 KiB
Bash
#!/bin/sh
|
|
# Materia Supervisor - Continuous pipeline orchestration
|
|
# Inspired by TigerBeetle's CFO supervisor: simple, resilient, easy to understand
|
|
# https://github.com/tigerbeetle/tigerbeetle/blob/main/src/scripts/cfo_supervisor.sh
|
|
#
|
|
# Environment variables (set in systemd EnvironmentFile):
|
|
# LANDING_DIR — local path for extracted landing data
|
|
# DUCKDB_PATH — path to DuckDB lakehouse file (SQLMesh pipeline DB)
|
|
# SERVING_DUCKDB_PATH — path to serving-only DuckDB (web app reads from here)
|
|
# ALERT_WEBHOOK_URL — optional ntfy.sh / Slack / Telegram webhook for failure alerts
|
|
|
|
set -eu
|
|
|
|
readonly REPO_DIR="/opt/materia"
|
|
|
|
while true
|
|
do
|
|
(
|
|
# Clone repo if missing
|
|
if ! [ -d "$REPO_DIR/.git" ]
|
|
then
|
|
echo "Repository not found, bootstrap required!"
|
|
exit 1
|
|
fi
|
|
|
|
cd "$REPO_DIR"
|
|
|
|
# Update code from git
|
|
git fetch origin master
|
|
git switch --discard-changes --detach origin/master
|
|
uv sync
|
|
|
|
# Extract all data sources
|
|
LANDING_DIR="${LANDING_DIR:-/data/materia/landing}" \
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
uv run materia pipeline run extract
|
|
|
|
LANDING_DIR="${LANDING_DIR:-/data/materia/landing}" \
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
uv run materia pipeline run extract_cot
|
|
|
|
LANDING_DIR="${LANDING_DIR:-/data/materia/landing}" \
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
uv run materia pipeline run extract_prices
|
|
|
|
LANDING_DIR="${LANDING_DIR:-/data/materia/landing}" \
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
uv run materia pipeline run extract_ice
|
|
|
|
# Transform all data sources
|
|
LANDING_DIR="${LANDING_DIR:-/data/materia/landing}" \
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
uv run materia pipeline run transform
|
|
|
|
# Export serving tables to serving.duckdb (atomic swap).
|
|
# The web app reads from SERVING_DUCKDB_PATH and picks up the new file
|
|
# automatically via inode-based connection reopen — no restart needed.
|
|
DUCKDB_PATH="${DUCKDB_PATH:-/data/materia/lakehouse.duckdb}" \
|
|
SERVING_DUCKDB_PATH="${SERVING_DUCKDB_PATH:-/data/materia/serving.duckdb}" \
|
|
uv run materia pipeline run export_serving
|
|
|
|
) || {
|
|
# Notify on failure if webhook is configured, then sleep to avoid busy-loop
|
|
if [ -n "${ALERT_WEBHOOK_URL:-}" ]; then
|
|
curl -s -d "Materia pipeline failed at $(date)" "$ALERT_WEBHOOK_URL" 2>/dev/null || true
|
|
fi
|
|
sleep 600 # Sleep 10 min on failure
|
|
}
|
|
done
|