Update bootstrap_supervisor.sh and setup_server.sh to use
git.padelnomics.io:2222 instead of gitlab.com.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
'run' requires the prod environment to already exist. 'plan --auto-apply'
initializes the environment on first run and applies pending changes on
subsequent runs — fully self-healing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without the 'prod' argument sqlmesh defaults to dev_<username>, which
doesn't exist on the server (padelnomics_service user).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without -R, a manual uv sync or git operation run as root would create
files under /opt/padelnomics owned by root, breaking uv for the service
user (Permission denied on .venv/bin/python3).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
alpine/git sets ENTRYPOINT ["git"], so GitLab's shell executor was invoking
`git sh <script>` instead of `sh <script>`. Override with entrypoint: [""].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`credit_ledger cl` joined with `suppliers s` — both have `id`, so
SQLite raised OperationalError. Qualify as `cl.id` and `cl.supplier_id`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New route GET /admin/pipeline/lineage/schema/<model> — returns JSON
with columns+types (from information_schema for serving models),
row count, upstream and downstream model lists. Validates model
against _DAG to prevent arbitrary table access.
- Precomputes _DOWNSTREAM map at import time from _DAG.
- Lineage template: replaces minimal edge-highlight JS with full UX —
hover triggers schema prefetch + floating tooltip (layer badge, top 4
columns, "+N more" note); click opens 320px slide-in panel showing
row count, full schema table, upstream/downstream dep lists.
Dep items in panel are clickable to navigate between models.
Schema responses are cached client-side to avoid repeat fetches.
Staging/foundation models show "schema in lakehouse.duckdb only".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Propagates the conformed city key (city_slug) from dim_venues through the
full pricing pipeline, eliminating 3 fragile LOWER(TRIM(...)) fuzzy string
joins with deterministic key joins.
Changes (cascading, task-by-task):
- dim_venues: add city_slug computed column (REGEXP_REPLACE slug derivation)
- dim_venue_capacity: join foundation.dim_venues instead of stg_playtomic_venues;
carry city_slug alongside country_code/city
- fct_daily_availability: carry city_slug from dim_venue_capacity
- venue_pricing_benchmarks: carry city_slug from fct_daily_availability;
add to venue_stats GROUP BY and final SELECT/GROUP BY
- city_market_profile: join vpb on city_slug = city_slug (was LOWER(TRIM))
- planner_defaults: add city_slug to city_benchmarks CTE; join on city_slug
- pseo_city_pricing: join city_market_profile on city_slug (was LOWER(TRIM))
- pipeline_routes._DAG: dim_venue_capacity now depends on dim_venues, not stg_playtomic_venues
Result: dim_venues.city_slug → dim_cities.(country_code, city_slug) forms a
fully conformed geographic hierarchy with no fuzzy string comparisons.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a 5th tab to the admin pipeline page showing the full 3-layer
SQLMesh data lineage: 28 models, 35 edges across staging / foundation /
serving swim lanes.
- _DAG: canonical model dependency dict in pipeline_routes.py;
update when models are added/removed
- _classify_layer(): derives layer from name prefix (stg_/dim_fct_/rest)
- _render_lineage_svg(): pure Python SVG generator — 3-column swim lane
layout, bezier edges, color-coded per layer (green/blue/amber),
no external dependencies
- /lineage route: HTMX tab handler
- pipeline_lineage.html: partial with SVG embed + vanilla JS hover
effects (highlight connected edges, dim unrelated)
- pipeline.html: 5th "Lineage" tab button
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Redirect pipeline_mark_stale to pipeline_dashboard (full page) instead
of pipeline_extractions (partial), fixing the broken CSS on form submit
- pipeline_trigger_extract accepts optional 'extractor' POST field;
validates against workflows.toml names to prevent injection, passes
as payload to enqueue("run_extraction")
- handle_run_extraction dispatches to per-extractor CLI entry point
(extract-overpass, extract-eurostat, etc.) when extractor is set,
falls back to umbrella 'extract' command otherwise
- pipeline_overview.html: add Run button to each workflow card header,
posting extractor name with CSRF token to pipeline_trigger_extract
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 2a: NUTS-1 regional income for Germany (16 Bundesländer via admin1→NUTS-1 mapping)
Phase 2b: EU-wide NUTS-2 via GISCO spatial join + US Census ACS state income
- All EU-27+EFTA+UK locations now auto-resolve to NUTS-2 via ST_Contains
- Germany gets sub-Bundesland (38 Regierungsbezirke) differentiation
- US gets state-level income with PPS normalisation
- Income cascade: NUTS-2 → NUTS-1 → US state → country-level
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests imported make_sticky_selector but it was never implemented.
Hash-based (MD5) consistent selector — same key always returns the
same proxy, distributes across the pool.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5-pass editorial pipeline across 11 cornerstone articles (6 DE + 5 EN)
and 3 bilingual pSEO templates. All pieces scored ≥4.4 and cleared the
publish threshold.
Critical/High fixes applied:
- Ceiling height inconsistency: 7m → 8m in build guide tables (EN + DE)
- HTML <span> tags removed from meta_description_pattern in all 3 templates
- German gendering violations fixed in padel-halle-bauen-de (4 instances)
- Grammatical gender fix: "Das häufigste Vorabend-Fehler" → "Der häufigste Fehler"
- Noun capitalisation: "sport" → "Sport" in padel-standort-analyse-de
Medium fixes applied:
- Varied repeated "well-run padel halls" phrase in EN investment risks article
- Orphaned F&B note elevated to bold callout
- Colloquial idiom replaced in EN cost guide
- "analyze" → "analyse" (British English) in EN location guide
P4-A resolved: replaced static German city-tier lists in both location
guide articles with a universal "market maturity stages" framework section
(established / growth / emerging markets). Articles are now country-agnostic
and link to pSEO country overview pages for live market data.
7 open improvement items remain (P1-A/B, P2-A/B/C, P3-A, P4-B/C) — none
are publish blockers. See docs/editorial-review-2026-02.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- eurostat.py: add nama_10r_2hhinc dataset config; append filter params to
request URL so server pre-filters the large cube before download
- stg_regional_income.sql: new staging model — reads nama_10r_2hhinc.json.gz,
filters to NUTS-1 codes (3-char), normalises EL→GR / UK→GB
- dim_locations.sql: add admin1_to_nuts1 VALUES CTE (16 German Bundesländer)
+ regional_income CTE; final SELECT uses COALESCE(regional, country) income
- init_landing_seeds.py: add empty seed for nama_10r_2hhinc.json.gz
Munich/Bayern now scores ~29K PPS vs Chemnitz/Sachsen ~19K PPS instead of
both inheriting the same national average (~25.5K PPS).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 0 — income ceiling fix (opportunity_score):
PPS normalisation /200→/35000; economic power now differentiates
countries (DE 13.2, ES 10.7, SE 14.3 pts; was 20.0 everywhere)
Phase 1b — overpass_tennis in workflows.toml:
Monthly schedule added; was only in combined extractor
Phase 2b — dim_cities spatial population fallback:
GeoNames spatial CTE (ST_Distance_Sphere, 0.14° bbox) resolves
localization mismatches: Wien→1.69M, Milano→1.37M, München→1.49M
Coverage: 70.5% → 98.5% (5,401/5,481 cities with population)
Adds a coordinate-based population lookup as a fallback when string name
matching fails (~29% of cities). Uses bbox pre-filter (0.14° ≈ 15 km) then
ST_Distance_Sphere to find the nearest GeoNames location in the same country.
Fixes localization mismatches: Milano≠Milan, Wien≠Vienna, München≠Munich.
Population cascade: Eurostat EU > US Census > ONS UK > GeoNames string >
GeoNames spatial > 0.
Coverage: 70.5% → 98.5% (5,401 / 5,481 cities with population > 0).
Key cities before/after:
Wien: 0 → 1,691,468
Milano: 0 → 1,371,498
München: already matched by string; verified still correct at 1,488,719
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_draft/future_article: route intentionally redirects to parent (302) instead
of bare 404 — rename tests and update assertion accordingly
- test_dashboard_has_content_links: /admin/templates and /admin/scenarios are
subnav links shown only on content section pages, not the main dashboard;
test now only checks /admin/articles which is always in the sidebar
- test_seo_sidebar_link: sidebar labels the link "Analytics" (not "SEO Hub"
which is the page title); test now checks for /admin/seo URL presence
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove body_html from _sync_static_articles INSERT (no such column in articles table)
- Remove empty report_q1_stat*_unit keys from EN+DE locales (i18n parity test forbids empty values)
- Update report_landing.html to remove stats-strip__unit spans referencing deleted keys
- Fix 0020_articles_unique_url_language migration to preserve group_key when recreating articles table (migration clobbered the column added by the preceding 0020_articles_group_key migration)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the existing GitLab CI: one test job (pytest + ruff) gated by a
tag job that creates v<run_number> on master. Supervisor polls for new
tags to deploy — no SSH keys or deploy credentials in CI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tennis extraction was missing from workflows.toml — only ran via the combined
`uv run extract` command, not automatically in production.
Schedule: monthly (same cadence as padel courts, OSM tennis data updates slowly).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PPS values are 18k–37k but /200 normalisation caused LEAST(1.0, 115)=1.0
for ALL countries — 20pts flat uplift, zero differentiation.
Fix: /35000 creates real country spread:
LU 20.0pts, DE 15.2pts, ES 12.8pts, GB 10.5pts (vs 20.0 everywhere before)
Default for missing data 100→15000 (developing-market assumption, ~0.43).
Header comment updated to document v2 formula behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Feature 1 — group_key for static article admin grouping:
- Migration 0020: group_key TEXT column + index on articles table
- _sync_static_articles(): auto-upserts data/content/articles/*.md on
every /admin/articles load, reads cornerstone → group_key
- _get_article_list_grouped(): COALESCE(group_key, url_path) as group_id,
so EN/DE static cornerstones pair into one row (pSEO unchanged)
Feature 2 — Email-gated State of Padel report PDF:
- data/content/articles/state-of-padel-q1-2026-{en,de}.md → reports/
- New reports/ blueprint: GET/POST /<lang>/reports/<slug> (email gate),
GET /<lang>/reports/<slug>/download (PDF serve)
- Premium PDF: full-bleed navy cover, Padelnomics wordmark watermark at
3.5% opacity (position:fixed, every page), gold/teal accents, Georgia
headings, WeasyPrint CSS3 (no JS)
- make report-pdf target to build PDFs
- i18n EN + DE (26 keys each, native German via linguistic-mediation)
- /reports added to RESERVED_PREFIXES, data/content/reports/_build/ gitignored
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- report.css: full-bleed navy cover, Padelnomics logo watermark at 3.5%
opacity (position:fixed, repeats every page), gold/teal accents, Georgia
headings, running headers via CSS named strings, metric boxes, insight-box
- report.html: Jinja2 template with cover stats, TOC, body, disclaimer
- build_report_pdf.py: builds EN+DE PDFs from data/content/reports/*.md
(WeasyPrint, mistune, PyYAML; reads logo as file:// URI for watermark)
- Makefile: report-pdf target
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _sync_static_articles(): auto-upserts data/content/articles/*.md into
DB on every /admin/articles load; reads cornerstone → group_key
- _get_article_list_grouped(): now groups by COALESCE(group_key, url_path)
so static EN/DE cornerstone articles pair into one row
- articles() route: calls _sync_static_articles() before listing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Raises supply gap ceiling from 4/100k to 8/100k in
location_opportunity_profile.sql. The original 4/100k hard cliff
truncated opportunity scores to 0 for any city with ≥4 courts/100k,
but our data undercounts ~87% of real courts (FIP: 17,300 Spanish
courts vs 2,239 in our DB). Raising to 8/100k gives a gentler gradient
and fairer partial credit when density data is incomplete.
Documents existing formula behaviour discovered during analysis:
- Income PPS: country-level constants (18k-37k range) saturate the
/200 ceiling — all EU countries get flat 20/20 pts until city-level
income data lands.
- Catchment NULL: DuckDB LEAST(1.0, NULL) = 1.0 (ignores nulls), so
NULL nearest_padel_court_km already yields full 15 pts. COALESCE
fallback is dead code but harmless.
- Tennis courts within 25km: dim_locations data is empty (all 0 rows)
— 10-court threshold is correct for when data arrives, contributes
0 pts everywhere for now.
Effective score impact: minimal (99% of locations have 0 courts/100k,
so supply gap was already at max). Only ~1,050 dense-court cities
see a score increase (from 0 gap pts to partial gap pts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
