deemanone/padelnomics
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sqlmesh): add transform workspace member with 4-layer DuckDB pipeline

Browse Source 
Adds sqlmesh_padelnomics UV workspace member at transform/sqlmesh_padelnomics/.
DuckDB gateway, LANDING_DIR variable, @daily cron on all models.

Raw layer (reads landing zone gzip JSON):
  raw_overpass_courts    — OSM padel court elements (nodes with lat/lon/tags)
  raw_playtomic_tenants  — Playtomic venue records (tenant_id, location, name)
  raw_eurostat_population — Eurostat urb_cpop1 city population (unpivoted)

Staging layer (typed, deduped, country-resolved):
  stg_padel_courts       — OSM nodes only, ~100m bbox country approximation
  stg_playtomic_venues   — deduplicated Playtomic venues
  stg_population         — city population by year with integer types

Foundation layer:
  dim_venues             — deduped union of OSM + Playtomic (~100m grid)
  dim_cities             — Eurostat cities with population + venue counts

Serving layer (consumed by web app and SEO generation):
  city_market_profile    — OBT: market score, venue density, population per city
  planner_defaults       — per-city calculator pre-fill values with country median
                           fallbacks and competitive-pressure rate adjustments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Deeman
2026-02-21 22:03:59 +01:00
parent b8471c7f31
commit 9f8ca82505
15 changed files with 1610 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
CHANGELOG.md
View File

@@ -7,6 +7,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
### Added
- `transform/sqlmesh_padelnomics` workspace member: SQLMesh 4-layer model pipeline over DuckDB
- Raw: `raw_overpass_courts`, `raw_playtomic_tenants`, `raw_eurostat_population`
- Staging: `stg_padel_courts`, `stg_playtomic_venues`, `stg_population`
- Foundation: `dim_venues` (OSM + Playtomic deduped), `dim_cities` (with Eurostat population)
- Serving: `city_market_profile` (market score OBT), `planner_defaults` (per-city calculator pre-fill)
- `extract/padelnomics_extract` workspace member: Overpass API (padel courts via OSM), Eurostat city demographics (`urb_cpop1`, `ilc_di03`), and Playtomic unauthenticated tenant search extractors
- Landing zone structure at `data/landing/` with per-source subdirectories: `overpass/`, `eurostat/`, `playtomic/`
- `.env.example` entries for `DUCKDB_PATH` and `LANDING_DIR`