Commit Graph

758 Commits

Author SHA1 Message Date
Deeman
6d44c116aa feat(maps): single-color markers with Padelnomics Score
Phase B: Remove dual-ring marker design (ring/core), replace with
single-color markers colored by opportunity_score ("Padelnomics Score").
Simplify CSS, update tooltips to show one score line on all maps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 12:00:13 +01:00
Deeman
bda2f85fd6 fix(pipeline): CAST snapshot_date to DATE in venue_pricing_benchmarks
Phase A: defensive CAST for incremental time_column comparison.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 11:55:44 +01:00
Deeman
116a4272f1 feat(maps): dual-ring markers with 5-tier color scale
Replace flat circle markers with nested dual-ring structure encoding
two scores per marker (core = primary, ring = secondary). Upgrade
from 3-tier to 5-tier colorblind-safe scale. Add pulse animation for
high-opportunity markers (>=75). Extract shared PNMarkers module from
3 duplicated implementations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 22:16:45 +01:00
Deeman
291fb2abd9 feat(geo): use CF headers on opportunity map + country overview maps
Pre-select user's country on opportunity map dropdown (CF-IPCountry),
auto-load the map on page load. Highlight user's city on country
overview maps with a blue ring (CF-IPCity best-effort match). Unify
opportunity score color scale to red/amber/green (was using blue for
low scores). Inject window.__GEO global for client-side geo access.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:43:52 +01:00
Deeman
40d8c75b81 feat: stash CF-RegionCode and CF-IPCity headers in g for future use
All three Cloudflare geo headers now available:
- g.user_country (CF-IPCountry) — used by geo-sorted article listing
- g.user_region (CF-RegionCode) — available for within-country sorting
- g.user_city (CF-IPCity) — available for city-level proximity

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:26:50 +01:00
Deeman
d7bd053dc6 chore: update CHANGELOG for score v6, bubble UX, and geo-localization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:26:28 +01:00
Deeman
d379dc7551 feat(markets): geo-localize article + map sorting via CF-IPCountry
- Read Cloudflare CF-IPCountry header into g.user_country (before_request)
- _filter_articles() sorts user's country first, nearby countries second,
  then rest — falls back to published_at DESC when header is absent
- map_countries sorted so user's country bubble renders on top (Leaflet z-order)
- Nearby-country mapping covers DACH, Iberia, Nordics, Benelux, UK/IE, Americas

Prerequisite: Nginx must forward CF-IPCountry header to Quart (same as Umami).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:25:43 +01:00
Deeman
814e8290a2 fix(markets): add map legend + unify bubble color scales
- Add legend below map: bubble size = venue count, color = Market Score
- Unify opportunity score color to use same green/orange/red scale
  (was using blue for low scores, inconsistent with market score)
- Add mkt_legend_size / mkt_legend_color i18n keys (EN + DE)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:24:21 +01:00
Deeman
67fbfde53d feat(scoring): Opportunity Score v5 → v6 — calibrate for saturated markets
- Lower density ceiling 8→5/100k (Spain at 6-16/100k now hits zero-gap)
- Increase supply deficit weight 35→40 pts (primary differentiator)
- Reduce addressable market 25→20 pts (less weight on population alone)
- Invert market validation → market headroom (high country maturity = less opportunity)

Target: Spain avg opportunity drops from ~78 to ~50-60 range.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:23:08 +01:00
Deeman
3c135051fd feat(scoring): Score v6 — World Bank global economic data for non-EU countries
Non-EU countries (AR, MX, AE, AU, etc.) previously got NULL for
median_income_pps and pli_construction, falling back to EU-calibrated
defaults (15K PPS, PLI=100) that produced wrong scores.

New World Bank WDI extractor fetches GNI per capita PPP and price level
ratio for 215 countries. dim_countries uses Germany as calibration anchor
to scale WB values into the Eurostat range (dynamic ratio, self-corrects
as both sources update). EU countries keep exact Eurostat values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:17:33 +01:00
Deeman
fcef47cb22 chore: update CHANGELOG + admin dependency graph for score v4/v5
- CHANGELOG.md: document Market Score v4 and Opportunity Score v5 changes
- pipeline_routes.py: add dim_countries to location_profiles dependency list

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:32:06 +01:00
Deeman
118c2c0fc7 feat(scoring): Opportunity Score v4 → v5 — fix correlated components
- Merge supply gap (30pts) + catchment gap (15pts) → supply deficit (35pts, GREATEST)
  Eliminates ~80% correlated double-count on a single signal.
- Add sports culture signal (10pts): tennis court density as racquet-sport adoption proxy.
  Ceiling 50 courts/25km. Harmless when tennis data is zero (contributes 0).
- Add construction affordability (5pts): income relative to PLI construction costs.
  Joins dim_countries.pli_construction. High income + low build cost = high score.
- Reduce economic power from 20 → 15pts to make room.

New weights: addressable market 25, economic power 15, supply deficit 35,
sports culture 10, construction affordability 5, market validation 10.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:30:04 +01:00
Deeman
cd6d950233 feat(scoring): Market Score v3 → v4 — fix Spain underscoring
- Lower count gate threshold: 5 → 3 venues (3 establishes a market pattern)
- Lower density ceiling: LN(21) → LN(11) (10/100k is reachable for mature markets)
- Better demand fallback: 0.4 → 0.65 multiplier + 0.3 floor (venues = demand evidence)
- Fix economic context: income/200 → income/25000 (actual discrimination vs free 10 pts)

Expected: Spain avg market score rises from ~54 to ~65-75.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 15:22:48 +01:00
Deeman
28e44384ef merge: opportunity map HTMX data islands + remove dead API endpoint
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
# Conflicts:
#	transform/sqlmesh_padelnomics/models/serving/location_opportunity_profile.sql
#	web/src/padelnomics/api.py
#	web/src/padelnomics/public/templates/opportunity_map.html
v202603072014
2026-03-07 21:05:52 +01:00
Deeman
b1e008a2a4 refactor(maps): opportunity map → HTMX data islands, remove dead API endpoint
- Delete opportunity() JSON endpoint from api.py (dead after this refactor)
- Add GET /opportunity-map/data route returning HTML partial with two JSON
  data islands (opp_points + ref_points from serving.location_profiles)
- Create partials/opportunity_map_data.html (2-line data island partial)
- Rewrite opportunity_map.html: HTMX attrs on <select>, invisible #map-data
  swap target, htmx:afterSwap listener replaces fetch()-based loadCountry()

city_venues endpoint stays public (article-maps.js calls it on public pages).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:56:45 +01:00
Deeman
d556ceecee fix(api): restore public access to country_cities + opportunity endpoints
opportunity_map.html (public page) still fetches these. Only countries.json
and city_venues.json are no longer called from any public page, so those two
keep @login_required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:41:12 +01:00
Deeman
f215ea8e3a fix: supply gap inflation + inline map data + guard API endpoints
A. location_profiles.sql: supply gap now uses GREATEST(catchment_padel_courts,
   COALESCE(city_padel_venue_count, 0)) so Playtomic venues prevent cities like
   Murcia/Cordoba/Gijon from receiving a full 30-pt supply gap bonus when their
   OSM catchment count is zero. Expected ~10-15 pt drop for affected ES cities.

B. pseo_country_overview.sql: add population-weighted lat/lon centroid columns
   so the markets map can use accurate country positions from this table.

C/D. content/routes.py + markets.html: query pseo_country_overview in the route
   and pass as map_countries to the template, replacing the fetch('/api/...') call
   with inline JSON. Map scores now match pseo_country_overview (pop-weighted),
   and the page loads without an extra round-trip.

E. api.py: add @login_required to all 4 endpoints. Unauthenticated callers get
   a 302 redirect to login instead of data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:33:31 +01:00
Deeman
b2ffad055b fix(supervisor): use file path for export_serving (not -m module syntax)
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
-m padelnomics.export_serving resolves to web package, not src/padelnomics.
src/padelnomics is not a uv workspace member so it's not importable by name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071710
2026-03-07 18:08:46 +01:00
Deeman
544891611f feat(transform): opportunity score v4 — market validation + population-weighted aggregation
All checks were successful
CI / test (push) Successful in 57s
CI / tag (push) Successful in 2s
Two targeted fixes for inflated country scores (ES 83, SE 77):

1. pseo_country_overview: replace AVG() with population-weighted averages
   for avg_opportunity_score and avg_market_score. Madrid/Barcelona now
   dominate Spain's average instead of hundreds of 30K-town white-space
   towns. Expected ES drop from ~83 to ~55-65.

2. location_profiles: replace dead sports culture component (10 pts,
   tennis data all zeros) with market validation signal.
   Split scored CTE into: market_scored → country_market → scored.
   country_market aggregates AVG(market_score) per country from cities
   with padel courts (market_score > 0), so zero-court locations don't
   dilute the signal. ES (~60/100) → ~6 pts. SE (~35/100) → ~3.5 pts.
   NULL → 0.5 neutral → 5 pts (untested market, not penalised).

Score budget unchanged: 25+20+30+15+10 = 100 pts.
No new models, no new data sources, no cycles.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071634
2026-03-07 17:23:11 +01:00
Deeman
c30a7943aa feat(transform): opportunity score v4 — market validation + population-weighted aggregation
Two targeted fixes for inflated country scores (ES 83, SE 77):

1. pseo_country_overview: replace AVG() with population-weighted averages
   for avg_opportunity_score and avg_market_score. Madrid/Barcelona now
   dominate Spain's average instead of hundreds of 30K-town white-space
   towns. Expected ES drop from ~83 to ~55-65.

2. location_opportunity_profile: replace dead sports culture component
   (10 pts, tennis data all zeros) with market validation signal.
   New country_market CTE aggregates city_market_profile per country_code.
   ES (~60/100) → ~6 pts (proven demand). SE (~35/100) → ~3.5 pts
   (struggling market). NULL → 0.5 neutral → 5 pts (untested market).

Score budget unchanged: 25+20+30+15+10 = 100 pts.
New dependency: location_opportunity_profile → serving.city_market_profile (no cycle).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 15:36:51 +01:00
Deeman
b071199895 fix(docker): copy content/ directory into image
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 2s
content/articles/ holds the cornerstone .md source files which
_sync_static_articles() reads on every /admin/articles load.
Without this COPY they were absent from the container.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071404
2026-03-07 15:03:44 +01:00
Deeman
af536f22ea refactor: introduce REPO_ROOT in core.py, replace all CWD-relative paths
All checks were successful
CI / test (push) Successful in 56s
CI / tag (push) Successful in 2s
v202603071353
2026-03-07 14:52:38 +01:00
Deeman
c320bef83e refactor: introduce REPO_ROOT in core.py, replace all CWD-relative paths
Defines REPO_ROOT = Path(__file__).parents[3] once in core.py.
Replaces Path(__file__).parent.parent...parent chains and Path("data/...")
CWD-relative references in admin/routes.py, content/__init__.py,
content/routes.py, and worker.py (4x local repo_root variables).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 14:51:34 +01:00
Deeman
2938661ae7 refactor: move article .md sources from data/ to content/articles/
All checks were successful
CI / test (push) Successful in 55s
CI / tag (push) Successful in 3s
data/ is gitignored (pipeline artifacts). Article .md files are source
content and must be version-controlled. Moved to content/articles/ at
repo root. Also updates _ARTICLES_DIR and all Path("data/content/articles")
references in admin/routes.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071348
2026-03-07 14:47:49 +01:00
Deeman
9f8afdbda7 test(admin): regression tests — article delete never removes .md source
Three cases: single delete, bulk by IDs, bulk apply_to_all.
Also extends _create_article() helper with article_type param.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 14:10:41 +01:00
Deeman
055cc23482 test(admin): regression tests — article delete never removes .md source
All checks were successful
CI / test (push) Successful in 1m3s
CI / tag (push) Successful in 3s
v202603071320
2026-03-07 14:10:41 +01:00
Deeman
66353b3da1 fix(admin): article delete only removes build file + DB row, never .md source 2026-03-07 13:52:24 +01:00
Deeman
15378b1804 fix(admin): article delete only removes build file + DB row, never .md source
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:52:17 +01:00
Deeman
03fdec7297 feat(admin): article type tabs + fix affiliate delete buttons
- Migration 0029: article_type column (cornerstone/editorial/generated)
- Tab bar on /admin/articles with per-type counts
- Template filter only on Generated tab; delete guard uses article_type
- Type dropdown in article_new/edit form
- Fix: affiliate program and product Delete buttons had missing text/tag
2026-03-07 13:50:44 +01:00
Deeman
608f0356a5 fix(admin): affiliate program + product delete buttons missing text/closing tag
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:50:35 +01:00
Deeman
39225d6cfd feat(admin): article type tabs (cornerstone / editorial / generated)
- Migration 0029: ADD COLUMN article_type + backfill + index
- Tab bar on /admin/articles with per-type counts
- _build_article_where, _get_article_list, _get_article_list_grouped, and
  all routes now accept and thread article_type filter
- Template dropdown only shown on Generated tab
- Bulk form and matching-count endpoint carry article_type
- Delete guard uses article_type == 'generated' (not template_slug check)
- _sync_static_articles derives article_type from cornerstone frontmatter field
- generate_articles() upserts with article_type = 'generated'
- article_new / article_edit: Type dropdown (Editorial / Cornerstone)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 12:21:07 +01:00
Deeman
e537bfd9d3 fix(admin): protect cornerstone .md files from bulk delete + fix PDF 500
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071014
2026-03-07 11:13:23 +01:00
Deeman
a27da79705 fix(admin): protect cornerstone .md files from bulk delete + fix PDF 500
- Bulk delete (both explicit-IDs and apply_to_all paths) now only unlinks
  source .md files for generated articles (template_slug IS NOT NULL).
  Manual cornerstone articles keep their .md source on disk.

- _sync_static_articles() now also renders markdown → HTML and writes to
  BUILD_DIR/<lang>/<slug>.html after upserting the DB row, so cornerstones
  are immediately servable after a sync without a separate rebuild step.

- scenario_pdf(): replace d = json.loads(scenario["calc_json"]) with
  d = calc(state) so all current calc fields (moic, dscr, cashOnCash, …)
  are present and the PDF route no longer 500s on stale stored JSON.

- Restored data/content/articles/ cornerstone .md files via git checkout.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 11:09:19 +01:00
Deeman
8d86669360 merge: article bulk select-all matching + cornerstone filter
All checks were successful
CI / test (push) Successful in 59s
CI / tag (push) Successful in 3s
v202603062249
2026-03-06 23:48:26 +01:00
Deeman
7d523250f7 feat(admin): article bulk select-all matching + cornerstone filter
- Extract _build_article_where() helper, eliminating duplicated WHERE
  logic from _get_article_list() and _get_article_list_grouped()
- Add template_slug='__manual__' sentinel → filters template_slug IS NULL
  (cornerstone / hand-written articles without a pSEO template)
- Add GET /articles/matching-count endpoint returning count of articles
  matching current filter params (for the Gmail-style select-all banner)
- Extend POST /articles/bulk with apply_to_all=true mode: builds WHERE
  from filter params instead of explicit IDs; rebuild capped at 2,000,
  delete at 5,000
- Add "Manual" option to Template filter dropdown
- Add Gmail-style "select all matching" banner: appears when select-all
  checkbox is checked, fetches total count, lets user switch to
  apply_to_all mode with confirmation dialog
- Sync filter hidden inputs into bulk form on filter change; changing
  filters resets apply-to-all state and clears selection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:47:10 +01:00
Deeman
fee0d6913b fix(pipeline): use sqlmesh plan --auto-apply instead of run
All checks were successful
CI / test (push) Successful in 56s
CI / tag (push) Successful in 3s
v202603062136
2026-03-06 22:34:58 +01:00
Deeman
71e08a5fa6 fix(pipeline): also update supervisor.py to use plan --auto-apply
Missed the Python supervisor module — same fix as supervisor.sh and
worker.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:33:59 +01:00
Deeman
27e86db6a1 fix(pipeline): use sqlmesh plan --auto-apply instead of sqlmesh run
`sqlmesh run` only re-evaluates intervals for already-planned models —
it does not detect new, modified, or deleted models. Switching to
`plan prod --auto-apply` ensures schema changes (like the new
location_profiles model) are picked up automatically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:33:17 +01:00
Deeman
90754b8d9f chore: move ci.py to ~/.claude/scripts (uv inline script, no project dep)
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 2s
Script now lives globally as a uv inline-dependency script.
Removes per-project scripts/ci.py and the msgspec dev dependency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603061452
2026-03-06 15:51:36 +01:00
Deeman
277c92e507 chore: add scripts/ci.py for Gitea CI pipeline status
Copies ci.py from beanflows (same script, shared across projects).
Adds msgspec dev dependency required by the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 15:38:42 +01:00
Deeman
77ec3a289f feat(transform): H3 catchment index, res 5 k_ring(1) ~24km radius
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
Merges worktree-h3-catchment-index. dim_locations now computes h3_cell_res5
(res 5, ~8.5km edge). location_profiles and dim_locations updated;
old location_opportunity_profile.sql already removed on master.

Conflict: location_opportunity_profile.sql deleted on master, kept deletion
and applied h3_cell_res4→res5 rename to location_profiles instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603061413
2026-03-06 14:45:45 +01:00
Deeman
f81d5f19da fix(transform): tighten H3 catchment to res 5 (~24km radius)
Res 4 + k_ring(1) gave ~50-60km effective radius, causing Oldenburg to
absorb Bremen (40km away) and destroying score differentiation.

Res 5 + k_ring(1) gives ~24km — captures adjacent Gemeinden (Delmenhorst
at 15km) without bleeding into unrelated cities at 40km+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 14:34:56 +01:00
Deeman
4d29ecf1d6 merge: unified location_profiles serving model + both scores on map tooltips
All checks were successful
CI / test (push) Successful in 55s
CI / tag (push) Successful in 3s
# Conflicts:
#	CHANGELOG.md
#	transform/sqlmesh_padelnomics/models/serving/location_opportunity_profile.sql
v202603061305
2026-03-06 14:03:55 +01:00
Deeman
a3b4e1fab6 docs: update CHANGELOG, CLAUDE.md, and comments for location_profiles
Update transform CLAUDE.md source integration map and conformed
dimensions table. Update CHANGELOG with unified model + tooltip
changes. Fix stale comments in dim_cities.sql and serving README.

Subtask 5/5: documentation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:45:08 +01:00
Deeman
8b794d24a6 feat(maps): show both scores in all map tooltips
Country map: avg Market Score + avg Opportunity Score.
City map: Market Score + Opportunity Score per city.
Opportunity map: Opportunity Score + Market Score per location.

Subtask 4/5: tooltip updates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:42:36 +01:00
Deeman
688f2dd1ee refactor(web): update all references to location_profiles
Update api.py (3 endpoints), public/routes.py, analytics.py docstring,
pipeline_routes.py DAG, pipeline_query.html placeholder, and
test_pipeline.py fixtures to use the new unified model.

Subtask 3/5: web app references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:41:42 +01:00
Deeman
81b556b205 refactor(serving): replace old models with location_profiles
Delete city_market_profile.sql and location_opportunity_profile.sql.
Update downstream models (planner_defaults, pseo_city_costs_de,
pseo_city_pricing) to read from location_profiles instead.

Subtask 2/5: delete old models + update downstream SQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:39:52 +01:00
Deeman
cda94c9ee4 feat(serving): add unified location_profiles model
Combines city_market_profile and location_opportunity_profile into a
single serving model at (country_code, geoname_id) grain. Both Market
Score and Opportunity Score computed per location. City data enriched
via LEFT JOIN dim_cities on geoname_id.

Subtask 1/5: create new model (old models not yet removed).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:36:36 +01:00
Deeman
4fbd91b59b merge: automate h3 community extension install via sqlmesh config 2026-03-06 10:27:03 +01:00
Deeman
159d1b5b9a fix(transform): use community repository for h3 extension install
SQLMesh's extensions config supports dict form with 'repository' key,
which runs INSTALL h3 FROM community + LOAD h3 automatically at connect
time. No manual one-time install needed per machine.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:26:56 +01:00