Simplify SQLMesh to use single prod gateway with virtual environments

- Remove dev gateway (local DuckDB file no longer needed)
- Single prod gateway connects to R2 Iceberg catalog
- Use virtual environments for dev isolation (e.g., dev_<username>)
- Update CLAUDE.md with new workflow and environment strategy
- Create comprehensive transform/sqlmesh_materia/README.md

Benefits:
- Simpler configuration (one gateway instead of two)
- All environments use same R2 Iceberg catalog
- SQLMesh handles environment isolation automatically
- No need to maintain local 13GB materia_dev.db file
- before_all hooks only run for prod gateway (no conditional logic needed)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Deeman
2025-10-13 21:47:04 +02:00
parent 6536724e00
commit d2352c1876
3 changed files with 121 additions and 29 deletions

View File

@@ -1,18 +1,8 @@
# --- Gateway Connection ---
# Single gateway connecting to R2 Iceberg catalog
# Local dev uses virtual environments (e.g., dev_<username>)
# Production uses the 'prod' environment
gateways:
dev:
connection:
# For more information on configuring the connection to your execution engine, visit:
# https://sqlmesh.readthedocs.io/en/stable/reference/configuration/#connection
# https://sqlmesh.readthedocs.io/en/stable/integrations/engines/duckdb/#connection-options
type: duckdb
database: materia_dev.db
extensions:
- name: zipfs
- name: httpfs
- name: iceberg
prod:
connection:
type: duckdb
@@ -21,8 +11,7 @@ gateways:
- name: httpfs
- name: iceberg
default_gateway: dev
default_gateway: prod
# --- Hooks ---
# Run initialization SQL before all plans/runs