padelnomics/transform/sqlmesh_padelnomics

Padelnomics Transform (SQLMesh)

3-layer SQL transformation pipeline using SQLMesh + DuckDB. Reads from the landing zone, produces analytics-ready tables consumed by the web app via an atomically-swapped serving DB.

Running

# From repo root — plan all changes (shows what will run)
uv run sqlmesh -p transform/sqlmesh_padelnomics plan

# Apply to production
uv run sqlmesh -p transform/sqlmesh_padelnomics plan prod

# Run model tests
uv run sqlmesh -p transform/sqlmesh_padelnomics test

# Format SQL
uv run sqlmesh -p transform/sqlmesh_padelnomics format

# Export serving tables to analytics.duckdb (run after SQLMesh)
DUCKDB_PATH=data/lakehouse.duckdb SERVING_DUCKDB_PATH=data/analytics.duckdb \
    uv run python -m padelnomics.export_serving

3-layer architecture

landing/                    ← raw files (extraction output)
  ├── overpass/*/*/courts.json.gz
  ├── eurostat/*/*/urb_cpop1.json.gz
  └── playtomic/*/*/tenants.json.gz

staging/                    ← reads landing files directly, type casting, dedup
  ├── staging.stg_padel_courts
  ├── staging.stg_playtomic_venues
  └── staging.stg_population

foundation/                 ← business logic, dimensions, facts
  ├── foundation.dim_venues
  └── foundation.dim_cities

serving/                    ← pre-aggregated for web app
  ├── serving.city_market_profile
  └── serving.planner_defaults

staging/ — read landing files + type casting

• Reads landing zone JSON files directly with read_json(..., format='auto', filename=true)
• Uses @LANDING_DIR variable for file path discovery
• Casts all columns to correct types: TRY_CAST(... AS DOUBLE)
• Deduplicates where source produces duplicates (ROW_NUMBER partitioned on ID)
• Validates coordinates, nulls, and data quality inline
• Naming: staging.stg_<source>

foundation/ — business logic

• Dimensions (dim_*): slowly changing attributes, one row per entity
• Facts (fact_*): events and measurements, one row per event
• May join across multiple staging models from different sources
• Naming: foundation.dim_<entity>, foundation.fact_<event>

serving/ — analytics-ready aggregates

• Pre-aggregated for specific web app query patterns
• These are the only tables the web app reads (via analytics.duckdb)
• Queried from analytics.py via fetch_analytics()
• Naming: serving.<purpose>

Two-DuckDB architecture

data/lakehouse.duckdb       ← SQLMesh exclusive write (DUCKDB_PATH)
  ├── staging.*
  ├── foundation.*
  └── serving.*

data/analytics.duckdb       ← web app read-only (SERVING_DUCKDB_PATH)
  └── serving.*             ← atomically replaced by export_serving.py

SQLMesh holds an exclusive write lock on lakehouse.duckdb during plan/run. The web app needs read-only access at all times. export_serving.py copies serving.* tables to a temp file, then atomically renames it to analytics.duckdb. The web app detects the inode change on next query — no restart needed.

Never point DUCKDB_PATH and SERVING_DUCKDB_PATH to the same file.

Adding a new data source

1. Add an extractor in extract/padelnomics_extract/ (see extraction README)
2. Add a staging model: models/staging/stg_<source>.sql that reads landing files directly
3. Join into foundation or serving models as needed

Model materialization

Layer	Default kind	Rationale
staging	FULL	Re-reads all landing files; cheap with DuckDB parallel scan
foundation	FULL	Business logic rarely changes; recompute is fast
serving	FULL	Small aggregates; web app needs latest at all times

For large historical tables, switch to kind INCREMENTAL_BY_TIME_RANGE with a time partition column.

Environment variables

Variable	Default	Description
LANDING_DIR	data/landing	Root of the landing zone
DUCKDB_PATH	data/lakehouse.duckdb	DuckDB file (SQLMesh exclusive write access)
SERVING_DUCKDB_PATH	data/analytics.duckdb	Serving DB (web app reads from here)