Commit Graph

719 Commits

Author SHA1 Message Date
Deeman
277c92e507 chore: add scripts/ci.py for Gitea CI pipeline status
Copies ci.py from beanflows (same script, shared across projects).
Adds msgspec dev dependency required by the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 15:38:42 +01:00
Deeman
77ec3a289f feat(transform): H3 catchment index, res 5 k_ring(1) ~24km radius
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
Merges worktree-h3-catchment-index. dim_locations now computes h3_cell_res5
(res 5, ~8.5km edge). location_profiles and dim_locations updated;
old location_opportunity_profile.sql already removed on master.

Conflict: location_opportunity_profile.sql deleted on master, kept deletion
and applied h3_cell_res4→res5 rename to location_profiles instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603061413
2026-03-06 14:45:45 +01:00
Deeman
f81d5f19da fix(transform): tighten H3 catchment to res 5 (~24km radius)
Res 4 + k_ring(1) gave ~50-60km effective radius, causing Oldenburg to
absorb Bremen (40km away) and destroying score differentiation.

Res 5 + k_ring(1) gives ~24km — captures adjacent Gemeinden (Delmenhorst
at 15km) without bleeding into unrelated cities at 40km+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 14:34:56 +01:00
Deeman
4d29ecf1d6 merge: unified location_profiles serving model + both scores on map tooltips
All checks were successful
CI / test (push) Successful in 55s
CI / tag (push) Successful in 3s
# Conflicts:
#	CHANGELOG.md
#	transform/sqlmesh_padelnomics/models/serving/location_opportunity_profile.sql
v202603061305
2026-03-06 14:03:55 +01:00
Deeman
a3b4e1fab6 docs: update CHANGELOG, CLAUDE.md, and comments for location_profiles
Update transform CLAUDE.md source integration map and conformed
dimensions table. Update CHANGELOG with unified model + tooltip
changes. Fix stale comments in dim_cities.sql and serving README.

Subtask 5/5: documentation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:45:08 +01:00
Deeman
8b794d24a6 feat(maps): show both scores in all map tooltips
Country map: avg Market Score + avg Opportunity Score.
City map: Market Score + Opportunity Score per city.
Opportunity map: Opportunity Score + Market Score per location.

Subtask 4/5: tooltip updates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:42:36 +01:00
Deeman
688f2dd1ee refactor(web): update all references to location_profiles
Update api.py (3 endpoints), public/routes.py, analytics.py docstring,
pipeline_routes.py DAG, pipeline_query.html placeholder, and
test_pipeline.py fixtures to use the new unified model.

Subtask 3/5: web app references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:41:42 +01:00
Deeman
81b556b205 refactor(serving): replace old models with location_profiles
Delete city_market_profile.sql and location_opportunity_profile.sql.
Update downstream models (planner_defaults, pseo_city_costs_de,
pseo_city_pricing) to read from location_profiles instead.

Subtask 2/5: delete old models + update downstream SQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:39:52 +01:00
Deeman
cda94c9ee4 feat(serving): add unified location_profiles model
Combines city_market_profile and location_opportunity_profile into a
single serving model at (country_code, geoname_id) grain. Both Market
Score and Opportunity Score computed per location. City data enriched
via LEFT JOIN dim_cities on geoname_id.

Subtask 1/5: create new model (old models not yet removed).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:36:36 +01:00
Deeman
4fbd91b59b merge: automate h3 community extension install via sqlmesh config 2026-03-06 10:27:03 +01:00
Deeman
159d1b5b9a fix(transform): use community repository for h3 extension install
SQLMesh's extensions config supports dict form with 'repository' key,
which runs INSTALL h3 FROM community + LOAD h3 automatically at connect
time. No manual one-time install needed per machine.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:26:56 +01:00
Deeman
fcd0c9b007 docs: update CHANGELOG with H3 catchment score v3 2026-03-06 10:20:15 +01:00
Deeman
f841ae105a merge: use full trademarked score names in map tooltips 2026-03-06 10:19:56 +01:00
Deeman
dec4f07fbb merge: H3 catchment index for Marktpotenzial-Score v3 2026-03-06 10:19:51 +01:00
Deeman
4e4ff61699 feat(transform): H3 catchment index for Marktpotenzial-Score v3
Add H3 res-4 regional catchment metrics (~15-18km radius, cell + 6
neighbours) to both the addressable market (25pts) and supply gap
(30pts) components of location_opportunity_profile.

Changes:
- config.yaml: add h3 to DuckDB extensions (requires one-time
  INSTALL h3 FROM community on each machine)
- dim_locations: add h3_cell_res4 column via h3_latlng_to_cell()
- location_opportunity_profile: add hex_stats + catchment CTEs;
  update score formula to use catchment_population and
  catchment_padel_courts; expose catchment_population,
  catchment_padel_courts, catchment_venues_per_100k as output cols

Motivation: local population underestimates functional market for
mid-size cities (e.g. Oldenburg ~170K misses surrounding Gemeinden).
H3 k_ring(1) captures the realistic driving-distance catchment
(~462km²) consistently across both score components.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:19:43 +01:00
Deeman
f907f2cd60 fix(maps): use full trademarked score names in all map tooltips
"Score X/100" → "Padelnomics Market Score: X/100" on country map (markets
hub), city map (country overview). Opportunity map uses "Padelnomics
Opportunity Score: X/100". Consistent branding across all three map views.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:19:08 +01:00
Deeman
3ad2885c84 merge: fix map bubble styling + improve hover UX 2026-03-06 10:11:26 +01:00
Deeman
e2f54552b0 fix(maps): restore score colors for non-article cities, improve hover UX
Non-article cities were fully gray (#9CA3AF), stripping informational value.
Now all cities show score-based colors (green/amber/red). Non-article cities
are differentiated via lower opacity, dashed border, desaturation, and
default cursor (no click handler). Tooltips show scores for all cities —
article cities get "Click to explore →", non-article cities get "Coming soon".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:09:58 +01:00
Deeman
07ca1ce15b merge: custom 404/500 error pages + smarter map city clicks 2026-03-06 10:01:50 +01:00
Deeman
be9b10c13f docs: update CHANGELOG with error pages and map improvements
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:59:29 +01:00
Deeman
82d6333517 feat: differentiate cities with/without articles on country map
Cities without published articles appear in muted gray and are not
clickable. The cities.json API endpoint now queries SQLite for
published articles and adds a has_article boolean to each city row.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:57:01 +01:00
Deeman
ed48936dad feat: add styled 404/500 error pages with i18n support
Custom error templates extending base.html with centered layout.
404 is context-aware: detects /markets/{country}/{city} paths and
shows city-specific message with link back to country overview.
Both pages support EN/DE translations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:55:13 +01:00
Deeman
e3bda5b816 merge: fix admin template preview UX issues (maps, article_stats route, dev debug mode) 2026-03-06 09:35:45 +01:00
Deeman
831233cb29 fix(admin): add missing article_stats route, 500 handler, dev debug mode
- Add /admin/articles/stats HTMX partial endpoint that was referenced
  by article_stats.html but never created (caused 500 during generation)
- Add @app.errorhandler(500) to log exceptions with traceback
- Switch dev_run.sh from Granian to Quart debug mode for browser
  tracebacks and auto-reload

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:34:22 +01:00
Deeman
c5327c4012 fix(maps): move VENUE_ICON creation after Leaflet loads
L.divIcon() was called at IIFE top level before the dynamic Leaflet
script loaded, throwing ReferenceError and preventing all maps from
rendering. Move icon creation into script.onload callback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:01:54 +01:00
Deeman
4426ab2cb6 fix(admin): render Leaflet maps in template preview 2026-03-05 22:58:34 +01:00
Deeman
93c9408f6b fix(admin): render Leaflet maps in template preview
The .card wrapper has overflow:hidden which clips Leaflet's
absolutely-positioned tile layers. Override to overflow:visible
on the rendered-article card. Add .catch() to map fetch calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:58:27 +01:00
Deeman
84128a3a64 merge: fix map scripts in template preview 2026-03-05 22:33:16 +01:00
Deeman
e9b4faa05c fix(admin): move map scripts inline in template preview
Put Leaflet init scripts inside admin_content block instead of relying
on the scripts block inheritance chain through base_admin → base.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:33:16 +01:00
Deeman
a834bb481d merge: load Leaflet maps in admin template preview 2026-03-05 22:29:13 +01:00
Deeman
9515ec8ae9 fix(admin): load Leaflet maps in template preview page
The /admin/templates/<slug>/preview/<key> page renders article HTML
directly but never loaded Leaflet CSS/JS, so country-map and city-map
divs appeared empty.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:29:00 +01:00
Deeman
fb99d6e0db merge: fix sqlmesh worker command + article preview maps 2026-03-05 22:14:37 +01:00
Deeman
4ee80603ef fix(articles): load Leaflet maps in article editor preview
The admin article preview iframe was missing Leaflet CSS/JS and had
scripts blocked by the sandbox policy, so map shortcodes rendered as
empty divs.

- Extract inline map script to static/js/article-maps.js (shared
  between article_detail.html and admin preview)
- Replace f-string preview doc with a proper Jinja template that
  includes Leaflet assets
- Add allow-scripts to iframe sandbox on both initial load and HTMX
  preview updates

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:03:50 +01:00
Deeman
2e42245ad5 fix(worker): use sqlmesh run prod instead of plan prod --auto-apply
`plan --auto-apply` only detects SQL model changes and won't re-run
for new data. `run prod` evaluates missing cron intervals and picks
up newly extracted data — matching the fix already applied to the
supervisor.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 21:49:51 +01:00
Deeman
2f47d1e589 fix(pipeline): make availability chain incremental + fix supervisor
Convert the availability chain (stg_playtomic_availability →
fct_availability_slot → fct_daily_availability) from FULL to
INCREMENTAL_BY_TIME_RANGE so sqlmesh run processes only new daily
intervals instead of re-reading all files.

Supervisor changes:
- run_transform(): plan prod --auto-apply → run prod (evaluates
  missing cron intervals, picks up new data)
- git_pull_and_sync(): add plan prod --auto-apply before re-exec
  so model code changes are applied on deploy
- supervisor.sh: same plan → run change

Staging model uses a date-scoped glob (@start_ds) to read only
the current interval's files. snapshot_date cast to DATE (was
VARCHAR) as required by time_column.

Clean up redundant TRY_CAST(snapshot_date AS DATE) in
venue_pricing_benchmarks since it's already DATE from foundation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 21:34:02 +01:00
Deeman
ead12c4552 fix(planner): prevent chart containers from overflowing on small screens
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 2s
v202603051728
2026-03-05 18:27:33 +01:00
Deeman
c54eb50004 fix(planner): prevent chart containers from overflowing on small screens
Grid children default to min-width:auto, letting the Chart.js canvas
push the container wider than its grid track. Adding min-width:0 and
overflow:hidden constrains charts to their column width.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 18:27:27 +01:00
Deeman
5d7fcec17a chore: change GISCO extraction schedule from monthly to yearly
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
v202603051651
2026-03-05 17:50:19 +01:00
Deeman
f7faf7ab57 chore: change GISCO extraction schedule from monthly to yearly
NUTS2 boundaries rarely change; yearly (Jan 1) is sufficient.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:50:14 +01:00
Deeman
add5f8ddfa fix(extract): correct lc_lci_lev lcstruct filter value
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 3s
v202603051640
2026-03-05 17:39:37 +01:00
Deeman
15ca316682 fix(extract): correct lc_lci_lev lcstruct filter value
D1_D2_A_HW doesn't exist in the API; use D1_D4_MD5 (total labour cost
= compensation + taxes - subsidies).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:32:49 +01:00
Deeman
103ef73cf5 fix(pipeline): eurostat filter bugs + supervisor uses sqlmesh plan
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 3s
v202603051620
2026-03-05 17:19:21 +01:00
Deeman
aa27f14f3c fix(pipeline): eurostat filter bugs + supervisor uses sqlmesh plan
- nrg_pc_203: add missing unit=KWH filter (API returns 2 units)
- lc_lci_lev: fix currency→unit filter dimension name
- supervisor: use `sqlmesh plan prod --auto-apply` instead of
  `sqlmesh run` so new/changed models are detected automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:19:12 +01:00
Deeman
8205744444 chore: remove accidentally committed .claire/ worktree directory
All checks were successful
CI / test (push) Successful in 56s
CI / tag (push) Successful in 3s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603051612
2026-03-05 17:10:48 +01:00
Deeman
1cbefe349c add env var 2026-03-05 17:08:52 +01:00
Deeman
003f19e071 fix(pipeline): handle DuckDB catalog naming in diagnostic script 2026-03-05 17:07:52 +01:00
Deeman
c3f15535b8 fix(pipeline): handle DuckDB catalog naming in diagnostic script
The lakehouse.duckdb file uses catalog "lakehouse" not "local", causing
SQLMesh logical views to break. Script now auto-detects the catalog via
USE and falls back to physical tables when views fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:06:44 +01:00
Deeman
fcb8ec4227 merge: pipeline diagnostic script + extraction card UX improvements
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
v202603051445
2026-03-05 15:40:16 +01:00
Deeman
6b7fa45bce feat(admin): add pipeline diagnostic script + extraction card UX improvements
- Add scripts/check_pipeline.py: read-only diagnostic for pricing pipeline
  row counts, date range analysis, HAVING filter impact, join coverage
- Add description field to all 12 workflows in workflows.toml
- Parse and display descriptions on extraction status cards
- Show spinner + "Running" state with blue-tinted card border
- Display start time with "running..." text for active extractions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 15:40:12 +01:00
Deeman
0d8687859d fix(docker): copy workflows.toml into container for admin pipeline view
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 3s
The admin Extraction Status page reads infra/supervisor/workflows.toml
but the Dockerfile only copied web/ into the image. Adding the COPY
so the file exists at /app/infra/supervisor/workflows.toml in the
container.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603051417
2026-03-05 15:16:07 +01:00