Commit Graph

745 Commits

Author SHA1 Message Date
Deeman
28e44384ef merge: opportunity map HTMX data islands + remove dead API endpoint
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
# Conflicts:
#	transform/sqlmesh_padelnomics/models/serving/location_opportunity_profile.sql
#	web/src/padelnomics/api.py
#	web/src/padelnomics/public/templates/opportunity_map.html
v202603072014
2026-03-07 21:05:52 +01:00
Deeman
b1e008a2a4 refactor(maps): opportunity map → HTMX data islands, remove dead API endpoint
- Delete opportunity() JSON endpoint from api.py (dead after this refactor)
- Add GET /opportunity-map/data route returning HTML partial with two JSON
  data islands (opp_points + ref_points from serving.location_profiles)
- Create partials/opportunity_map_data.html (2-line data island partial)
- Rewrite opportunity_map.html: HTMX attrs on <select>, invisible #map-data
  swap target, htmx:afterSwap listener replaces fetch()-based loadCountry()

city_venues endpoint stays public (article-maps.js calls it on public pages).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:56:45 +01:00
Deeman
d556ceecee fix(api): restore public access to country_cities + opportunity endpoints
opportunity_map.html (public page) still fetches these. Only countries.json
and city_venues.json are no longer called from any public page, so those two
keep @login_required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:41:12 +01:00
Deeman
f215ea8e3a fix: supply gap inflation + inline map data + guard API endpoints
A. location_profiles.sql: supply gap now uses GREATEST(catchment_padel_courts,
   COALESCE(city_padel_venue_count, 0)) so Playtomic venues prevent cities like
   Murcia/Cordoba/Gijon from receiving a full 30-pt supply gap bonus when their
   OSM catchment count is zero. Expected ~10-15 pt drop for affected ES cities.

B. pseo_country_overview.sql: add population-weighted lat/lon centroid columns
   so the markets map can use accurate country positions from this table.

C/D. content/routes.py + markets.html: query pseo_country_overview in the route
   and pass as map_countries to the template, replacing the fetch('/api/...') call
   with inline JSON. Map scores now match pseo_country_overview (pop-weighted),
   and the page loads without an extra round-trip.

E. api.py: add @login_required to all 4 endpoints. Unauthenticated callers get
   a 302 redirect to login instead of data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 20:33:31 +01:00
Deeman
b2ffad055b fix(supervisor): use file path for export_serving (not -m module syntax)
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
-m padelnomics.export_serving resolves to web package, not src/padelnomics.
src/padelnomics is not a uv workspace member so it's not importable by name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071710
2026-03-07 18:08:46 +01:00
Deeman
544891611f feat(transform): opportunity score v4 — market validation + population-weighted aggregation
All checks were successful
CI / test (push) Successful in 57s
CI / tag (push) Successful in 2s
Two targeted fixes for inflated country scores (ES 83, SE 77):

1. pseo_country_overview: replace AVG() with population-weighted averages
   for avg_opportunity_score and avg_market_score. Madrid/Barcelona now
   dominate Spain's average instead of hundreds of 30K-town white-space
   towns. Expected ES drop from ~83 to ~55-65.

2. location_profiles: replace dead sports culture component (10 pts,
   tennis data all zeros) with market validation signal.
   Split scored CTE into: market_scored → country_market → scored.
   country_market aggregates AVG(market_score) per country from cities
   with padel courts (market_score > 0), so zero-court locations don't
   dilute the signal. ES (~60/100) → ~6 pts. SE (~35/100) → ~3.5 pts.
   NULL → 0.5 neutral → 5 pts (untested market, not penalised).

Score budget unchanged: 25+20+30+15+10 = 100 pts.
No new models, no new data sources, no cycles.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071634
2026-03-07 17:23:11 +01:00
Deeman
c30a7943aa feat(transform): opportunity score v4 — market validation + population-weighted aggregation
Two targeted fixes for inflated country scores (ES 83, SE 77):

1. pseo_country_overview: replace AVG() with population-weighted averages
   for avg_opportunity_score and avg_market_score. Madrid/Barcelona now
   dominate Spain's average instead of hundreds of 30K-town white-space
   towns. Expected ES drop from ~83 to ~55-65.

2. location_opportunity_profile: replace dead sports culture component
   (10 pts, tennis data all zeros) with market validation signal.
   New country_market CTE aggregates city_market_profile per country_code.
   ES (~60/100) → ~6 pts (proven demand). SE (~35/100) → ~3.5 pts
   (struggling market). NULL → 0.5 neutral → 5 pts (untested market).

Score budget unchanged: 25+20+30+15+10 = 100 pts.
New dependency: location_opportunity_profile → serving.city_market_profile (no cycle).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 15:36:51 +01:00
Deeman
b071199895 fix(docker): copy content/ directory into image
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 2s
content/articles/ holds the cornerstone .md source files which
_sync_static_articles() reads on every /admin/articles load.
Without this COPY they were absent from the container.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071404
2026-03-07 15:03:44 +01:00
Deeman
af536f22ea refactor: introduce REPO_ROOT in core.py, replace all CWD-relative paths
All checks were successful
CI / test (push) Successful in 56s
CI / tag (push) Successful in 2s
v202603071353
2026-03-07 14:52:38 +01:00
Deeman
c320bef83e refactor: introduce REPO_ROOT in core.py, replace all CWD-relative paths
Defines REPO_ROOT = Path(__file__).parents[3] once in core.py.
Replaces Path(__file__).parent.parent...parent chains and Path("data/...")
CWD-relative references in admin/routes.py, content/__init__.py,
content/routes.py, and worker.py (4x local repo_root variables).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 14:51:34 +01:00
Deeman
2938661ae7 refactor: move article .md sources from data/ to content/articles/
All checks were successful
CI / test (push) Successful in 55s
CI / tag (push) Successful in 3s
data/ is gitignored (pipeline artifacts). Article .md files are source
content and must be version-controlled. Moved to content/articles/ at
repo root. Also updates _ARTICLES_DIR and all Path("data/content/articles")
references in admin/routes.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071348
2026-03-07 14:47:49 +01:00
Deeman
9f8afdbda7 test(admin): regression tests — article delete never removes .md source
Three cases: single delete, bulk by IDs, bulk apply_to_all.
Also extends _create_article() helper with article_type param.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 14:10:41 +01:00
Deeman
055cc23482 test(admin): regression tests — article delete never removes .md source
All checks were successful
CI / test (push) Successful in 1m3s
CI / tag (push) Successful in 3s
v202603071320
2026-03-07 14:10:41 +01:00
Deeman
66353b3da1 fix(admin): article delete only removes build file + DB row, never .md source 2026-03-07 13:52:24 +01:00
Deeman
15378b1804 fix(admin): article delete only removes build file + DB row, never .md source
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:52:17 +01:00
Deeman
03fdec7297 feat(admin): article type tabs + fix affiliate delete buttons
- Migration 0029: article_type column (cornerstone/editorial/generated)
- Tab bar on /admin/articles with per-type counts
- Template filter only on Generated tab; delete guard uses article_type
- Type dropdown in article_new/edit form
- Fix: affiliate program and product Delete buttons had missing text/tag
2026-03-07 13:50:44 +01:00
Deeman
608f0356a5 fix(admin): affiliate program + product delete buttons missing text/closing tag
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:50:35 +01:00
Deeman
39225d6cfd feat(admin): article type tabs (cornerstone / editorial / generated)
- Migration 0029: ADD COLUMN article_type + backfill + index
- Tab bar on /admin/articles with per-type counts
- _build_article_where, _get_article_list, _get_article_list_grouped, and
  all routes now accept and thread article_type filter
- Template dropdown only shown on Generated tab
- Bulk form and matching-count endpoint carry article_type
- Delete guard uses article_type == 'generated' (not template_slug check)
- _sync_static_articles derives article_type from cornerstone frontmatter field
- generate_articles() upserts with article_type = 'generated'
- article_new / article_edit: Type dropdown (Editorial / Cornerstone)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 12:21:07 +01:00
Deeman
e537bfd9d3 fix(admin): protect cornerstone .md files from bulk delete + fix PDF 500
All checks were successful
CI / test (push) Successful in 1m0s
CI / tag (push) Successful in 3s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603071014
2026-03-07 11:13:23 +01:00
Deeman
a27da79705 fix(admin): protect cornerstone .md files from bulk delete + fix PDF 500
- Bulk delete (both explicit-IDs and apply_to_all paths) now only unlinks
  source .md files for generated articles (template_slug IS NOT NULL).
  Manual cornerstone articles keep their .md source on disk.

- _sync_static_articles() now also renders markdown → HTML and writes to
  BUILD_DIR/<lang>/<slug>.html after upserting the DB row, so cornerstones
  are immediately servable after a sync without a separate rebuild step.

- scenario_pdf(): replace d = json.loads(scenario["calc_json"]) with
  d = calc(state) so all current calc fields (moic, dscr, cashOnCash, …)
  are present and the PDF route no longer 500s on stale stored JSON.

- Restored data/content/articles/ cornerstone .md files via git checkout.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 11:09:19 +01:00
Deeman
8d86669360 merge: article bulk select-all matching + cornerstone filter
All checks were successful
CI / test (push) Successful in 59s
CI / tag (push) Successful in 3s
v202603062249
2026-03-06 23:48:26 +01:00
Deeman
7d523250f7 feat(admin): article bulk select-all matching + cornerstone filter
- Extract _build_article_where() helper, eliminating duplicated WHERE
  logic from _get_article_list() and _get_article_list_grouped()
- Add template_slug='__manual__' sentinel → filters template_slug IS NULL
  (cornerstone / hand-written articles without a pSEO template)
- Add GET /articles/matching-count endpoint returning count of articles
  matching current filter params (for the Gmail-style select-all banner)
- Extend POST /articles/bulk with apply_to_all=true mode: builds WHERE
  from filter params instead of explicit IDs; rebuild capped at 2,000,
  delete at 5,000
- Add "Manual" option to Template filter dropdown
- Add Gmail-style "select all matching" banner: appears when select-all
  checkbox is checked, fetches total count, lets user switch to
  apply_to_all mode with confirmation dialog
- Sync filter hidden inputs into bulk form on filter change; changing
  filters resets apply-to-all state and clears selection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:47:10 +01:00
Deeman
fee0d6913b fix(pipeline): use sqlmesh plan --auto-apply instead of run
All checks were successful
CI / test (push) Successful in 56s
CI / tag (push) Successful in 3s
v202603062136
2026-03-06 22:34:58 +01:00
Deeman
71e08a5fa6 fix(pipeline): also update supervisor.py to use plan --auto-apply
Missed the Python supervisor module — same fix as supervisor.sh and
worker.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:33:59 +01:00
Deeman
27e86db6a1 fix(pipeline): use sqlmesh plan --auto-apply instead of sqlmesh run
`sqlmesh run` only re-evaluates intervals for already-planned models —
it does not detect new, modified, or deleted models. Switching to
`plan prod --auto-apply` ensures schema changes (like the new
location_profiles model) are picked up automatically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 22:33:17 +01:00
Deeman
90754b8d9f chore: move ci.py to ~/.claude/scripts (uv inline script, no project dep)
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 2s
Script now lives globally as a uv inline-dependency script.
Removes per-project scripts/ci.py and the msgspec dev dependency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603061452
2026-03-06 15:51:36 +01:00
Deeman
277c92e507 chore: add scripts/ci.py for Gitea CI pipeline status
Copies ci.py from beanflows (same script, shared across projects).
Adds msgspec dev dependency required by the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 15:38:42 +01:00
Deeman
77ec3a289f feat(transform): H3 catchment index, res 5 k_ring(1) ~24km radius
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
Merges worktree-h3-catchment-index. dim_locations now computes h3_cell_res5
(res 5, ~8.5km edge). location_profiles and dim_locations updated;
old location_opportunity_profile.sql already removed on master.

Conflict: location_opportunity_profile.sql deleted on master, kept deletion
and applied h3_cell_res4→res5 rename to location_profiles instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603061413
2026-03-06 14:45:45 +01:00
Deeman
f81d5f19da fix(transform): tighten H3 catchment to res 5 (~24km radius)
Res 4 + k_ring(1) gave ~50-60km effective radius, causing Oldenburg to
absorb Bremen (40km away) and destroying score differentiation.

Res 5 + k_ring(1) gives ~24km — captures adjacent Gemeinden (Delmenhorst
at 15km) without bleeding into unrelated cities at 40km+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 14:34:56 +01:00
Deeman
4d29ecf1d6 merge: unified location_profiles serving model + both scores on map tooltips
All checks were successful
CI / test (push) Successful in 55s
CI / tag (push) Successful in 3s
# Conflicts:
#	CHANGELOG.md
#	transform/sqlmesh_padelnomics/models/serving/location_opportunity_profile.sql
v202603061305
2026-03-06 14:03:55 +01:00
Deeman
a3b4e1fab6 docs: update CHANGELOG, CLAUDE.md, and comments for location_profiles
Update transform CLAUDE.md source integration map and conformed
dimensions table. Update CHANGELOG with unified model + tooltip
changes. Fix stale comments in dim_cities.sql and serving README.

Subtask 5/5: documentation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:45:08 +01:00
Deeman
8b794d24a6 feat(maps): show both scores in all map tooltips
Country map: avg Market Score + avg Opportunity Score.
City map: Market Score + Opportunity Score per city.
Opportunity map: Opportunity Score + Market Score per location.

Subtask 4/5: tooltip updates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:42:36 +01:00
Deeman
688f2dd1ee refactor(web): update all references to location_profiles
Update api.py (3 endpoints), public/routes.py, analytics.py docstring,
pipeline_routes.py DAG, pipeline_query.html placeholder, and
test_pipeline.py fixtures to use the new unified model.

Subtask 3/5: web app references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:41:42 +01:00
Deeman
81b556b205 refactor(serving): replace old models with location_profiles
Delete city_market_profile.sql and location_opportunity_profile.sql.
Update downstream models (planner_defaults, pseo_city_costs_de,
pseo_city_pricing) to read from location_profiles instead.

Subtask 2/5: delete old models + update downstream SQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:39:52 +01:00
Deeman
cda94c9ee4 feat(serving): add unified location_profiles model
Combines city_market_profile and location_opportunity_profile into a
single serving model at (country_code, geoname_id) grain. Both Market
Score and Opportunity Score computed per location. City data enriched
via LEFT JOIN dim_cities on geoname_id.

Subtask 1/5: create new model (old models not yet removed).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:36:36 +01:00
Deeman
4fbd91b59b merge: automate h3 community extension install via sqlmesh config 2026-03-06 10:27:03 +01:00
Deeman
159d1b5b9a fix(transform): use community repository for h3 extension install
SQLMesh's extensions config supports dict form with 'repository' key,
which runs INSTALL h3 FROM community + LOAD h3 automatically at connect
time. No manual one-time install needed per machine.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:26:56 +01:00
Deeman
fcd0c9b007 docs: update CHANGELOG with H3 catchment score v3 2026-03-06 10:20:15 +01:00
Deeman
f841ae105a merge: use full trademarked score names in map tooltips 2026-03-06 10:19:56 +01:00
Deeman
dec4f07fbb merge: H3 catchment index for Marktpotenzial-Score v3 2026-03-06 10:19:51 +01:00
Deeman
4e4ff61699 feat(transform): H3 catchment index for Marktpotenzial-Score v3
Add H3 res-4 regional catchment metrics (~15-18km radius, cell + 6
neighbours) to both the addressable market (25pts) and supply gap
(30pts) components of location_opportunity_profile.

Changes:
- config.yaml: add h3 to DuckDB extensions (requires one-time
  INSTALL h3 FROM community on each machine)
- dim_locations: add h3_cell_res4 column via h3_latlng_to_cell()
- location_opportunity_profile: add hex_stats + catchment CTEs;
  update score formula to use catchment_population and
  catchment_padel_courts; expose catchment_population,
  catchment_padel_courts, catchment_venues_per_100k as output cols

Motivation: local population underestimates functional market for
mid-size cities (e.g. Oldenburg ~170K misses surrounding Gemeinden).
H3 k_ring(1) captures the realistic driving-distance catchment
(~462km²) consistently across both score components.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:19:43 +01:00
Deeman
f907f2cd60 fix(maps): use full trademarked score names in all map tooltips
"Score X/100" → "Padelnomics Market Score: X/100" on country map (markets
hub), city map (country overview). Opportunity map uses "Padelnomics
Opportunity Score: X/100". Consistent branding across all three map views.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:19:08 +01:00
Deeman
3ad2885c84 merge: fix map bubble styling + improve hover UX 2026-03-06 10:11:26 +01:00
Deeman
e2f54552b0 fix(maps): restore score colors for non-article cities, improve hover UX
Non-article cities were fully gray (#9CA3AF), stripping informational value.
Now all cities show score-based colors (green/amber/red). Non-article cities
are differentiated via lower opacity, dashed border, desaturation, and
default cursor (no click handler). Tooltips show scores for all cities —
article cities get "Click to explore →", non-article cities get "Coming soon".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:09:58 +01:00
Deeman
07ca1ce15b merge: custom 404/500 error pages + smarter map city clicks 2026-03-06 10:01:50 +01:00
Deeman
be9b10c13f docs: update CHANGELOG with error pages and map improvements
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:59:29 +01:00
Deeman
82d6333517 feat: differentiate cities with/without articles on country map
Cities without published articles appear in muted gray and are not
clickable. The cities.json API endpoint now queries SQLite for
published articles and adds a has_article boolean to each city row.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:57:01 +01:00
Deeman
ed48936dad feat: add styled 404/500 error pages with i18n support
Custom error templates extending base.html with centered layout.
404 is context-aware: detects /markets/{country}/{city} paths and
shows city-specific message with link back to country overview.
Both pages support EN/DE translations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:55:13 +01:00
Deeman
e3bda5b816 merge: fix admin template preview UX issues (maps, article_stats route, dev debug mode) 2026-03-06 09:35:45 +01:00
Deeman
831233cb29 fix(admin): add missing article_stats route, 500 handler, dev debug mode
- Add /admin/articles/stats HTMX partial endpoint that was referenced
  by article_stats.html but never created (caused 500 during generation)
- Add @app.errorhandler(500) to log exceptions with traceback
- Switch dev_run.sh from Granian to Quart debug mode for browser
  tracebacks and auto-reload

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:34:22 +01:00