Commit Graph

692 Commits

Author SHA1 Message Date
Deeman
688f2dd1ee refactor(web): update all references to location_profiles
Update api.py (3 endpoints), public/routes.py, analytics.py docstring,
pipeline_routes.py DAG, pipeline_query.html placeholder, and
test_pipeline.py fixtures to use the new unified model.

Subtask 3/5: web app references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:41:42 +01:00
Deeman
81b556b205 refactor(serving): replace old models with location_profiles
Delete city_market_profile.sql and location_opportunity_profile.sql.
Update downstream models (planner_defaults, pseo_city_costs_de,
pseo_city_pricing) to read from location_profiles instead.

Subtask 2/5: delete old models + update downstream SQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:39:52 +01:00
Deeman
cda94c9ee4 feat(serving): add unified location_profiles model
Combines city_market_profile and location_opportunity_profile into a
single serving model at (country_code, geoname_id) grain. Both Market
Score and Opportunity Score computed per location. City data enriched
via LEFT JOIN dim_cities on geoname_id.

Subtask 1/5: create new model (old models not yet removed).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 11:36:36 +01:00
Deeman
f907f2cd60 fix(maps): use full trademarked score names in all map tooltips
"Score X/100" → "Padelnomics Market Score: X/100" on country map (markets
hub), city map (country overview). Opportunity map uses "Padelnomics
Opportunity Score: X/100". Consistent branding across all three map views.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:19:08 +01:00
Deeman
e2f54552b0 fix(maps): restore score colors for non-article cities, improve hover UX
Non-article cities were fully gray (#9CA3AF), stripping informational value.
Now all cities show score-based colors (green/amber/red). Non-article cities
are differentiated via lower opacity, dashed border, desaturation, and
default cursor (no click handler). Tooltips show scores for all cities —
article cities get "Click to explore →", non-article cities get "Coming soon".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 10:09:58 +01:00
Deeman
be9b10c13f docs: update CHANGELOG with error pages and map improvements
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:59:29 +01:00
Deeman
82d6333517 feat: differentiate cities with/without articles on country map
Cities without published articles appear in muted gray and are not
clickable. The cities.json API endpoint now queries SQLite for
published articles and adds a has_article boolean to each city row.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:57:01 +01:00
Deeman
ed48936dad feat: add styled 404/500 error pages with i18n support
Custom error templates extending base.html with centered layout.
404 is context-aware: detects /markets/{country}/{city} paths and
shows city-specific message with link back to country overview.
Both pages support EN/DE translations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:55:13 +01:00
Deeman
831233cb29 fix(admin): add missing article_stats route, 500 handler, dev debug mode
- Add /admin/articles/stats HTMX partial endpoint that was referenced
  by article_stats.html but never created (caused 500 during generation)
- Add @app.errorhandler(500) to log exceptions with traceback
- Switch dev_run.sh from Granian to Quart debug mode for browser
  tracebacks and auto-reload

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:34:22 +01:00
Deeman
c5327c4012 fix(maps): move VENUE_ICON creation after Leaflet loads
L.divIcon() was called at IIFE top level before the dynamic Leaflet
script loaded, throwing ReferenceError and preventing all maps from
rendering. Move icon creation into script.onload callback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 09:01:54 +01:00
Deeman
93c9408f6b fix(admin): render Leaflet maps in template preview
The .card wrapper has overflow:hidden which clips Leaflet's
absolutely-positioned tile layers. Override to overflow:visible
on the rendered-article card. Add .catch() to map fetch calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:58:27 +01:00
Deeman
e9b4faa05c fix(admin): move map scripts inline in template preview
Put Leaflet init scripts inside admin_content block instead of relying
on the scripts block inheritance chain through base_admin → base.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:33:16 +01:00
Deeman
9515ec8ae9 fix(admin): load Leaflet maps in template preview page
The /admin/templates/<slug>/preview/<key> page renders article HTML
directly but never loaded Leaflet CSS/JS, so country-map and city-map
divs appeared empty.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:29:00 +01:00
Deeman
4ee80603ef fix(articles): load Leaflet maps in article editor preview
The admin article preview iframe was missing Leaflet CSS/JS and had
scripts blocked by the sandbox policy, so map shortcodes rendered as
empty divs.

- Extract inline map script to static/js/article-maps.js (shared
  between article_detail.html and admin preview)
- Replace f-string preview doc with a proper Jinja template that
  includes Leaflet assets
- Add allow-scripts to iframe sandbox on both initial load and HTMX
  preview updates

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 22:03:50 +01:00
Deeman
2e42245ad5 fix(worker): use sqlmesh run prod instead of plan prod --auto-apply
`plan --auto-apply` only detects SQL model changes and won't re-run
for new data. `run prod` evaluates missing cron intervals and picks
up newly extracted data — matching the fix already applied to the
supervisor.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 21:49:51 +01:00
Deeman
2f47d1e589 fix(pipeline): make availability chain incremental + fix supervisor
Convert the availability chain (stg_playtomic_availability →
fct_availability_slot → fct_daily_availability) from FULL to
INCREMENTAL_BY_TIME_RANGE so sqlmesh run processes only new daily
intervals instead of re-reading all files.

Supervisor changes:
- run_transform(): plan prod --auto-apply → run prod (evaluates
  missing cron intervals, picks up new data)
- git_pull_and_sync(): add plan prod --auto-apply before re-exec
  so model code changes are applied on deploy
- supervisor.sh: same plan → run change

Staging model uses a date-scoped glob (@start_ds) to read only
the current interval's files. snapshot_date cast to DATE (was
VARCHAR) as required by time_column.

Clean up redundant TRY_CAST(snapshot_date AS DATE) in
venue_pricing_benchmarks since it's already DATE from foundation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 21:34:02 +01:00
Deeman
c54eb50004 fix(planner): prevent chart containers from overflowing on small screens
Grid children default to min-width:auto, letting the Chart.js canvas
push the container wider than its grid track. Adding min-width:0 and
overflow:hidden constrains charts to their column width.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 18:27:27 +01:00
Deeman
f7faf7ab57 chore: change GISCO extraction schedule from monthly to yearly
NUTS2 boundaries rarely change; yearly (Jan 1) is sufficient.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:50:14 +01:00
Deeman
15ca316682 fix(extract): correct lc_lci_lev lcstruct filter value
D1_D2_A_HW doesn't exist in the API; use D1_D4_MD5 (total labour cost
= compensation + taxes - subsidies).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:32:49 +01:00
Deeman
aa27f14f3c fix(pipeline): eurostat filter bugs + supervisor uses sqlmesh plan
- nrg_pc_203: add missing unit=KWH filter (API returns 2 units)
- lc_lci_lev: fix currency→unit filter dimension name
- supervisor: use `sqlmesh plan prod --auto-apply` instead of
  `sqlmesh run` so new/changed models are detected automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:19:12 +01:00
Deeman
c3f15535b8 fix(pipeline): handle DuckDB catalog naming in diagnostic script
The lakehouse.duckdb file uses catalog "lakehouse" not "local", causing
SQLMesh logical views to break. Script now auto-detects the catalog via
USE and falls back to physical tables when views fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 17:06:44 +01:00
Deeman
6b7fa45bce feat(admin): add pipeline diagnostic script + extraction card UX improvements
- Add scripts/check_pipeline.py: read-only diagnostic for pricing pipeline
  row counts, date range analysis, HAVING filter impact, join coverage
- Add description field to all 12 workflows in workflows.toml
- Parse and display descriptions on extraction status cards
- Show spinner + "Running" state with blue-tinted card border
- Display start time with "running..." text for active extractions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 15:40:12 +01:00
Deeman
0d8687859d fix(docker): copy workflows.toml into container for admin pipeline view
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 3s
The admin Extraction Status page reads infra/supervisor/workflows.toml
but the Dockerfile only copied web/ into the image. Adding the COPY
so the file exists at /app/infra/supervisor/workflows.toml in the
container.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603051417
2026-03-05 15:16:07 +01:00
Deeman
b064e18aa1 fix(admin): resolve workflows.toml path via CWD instead of __file__
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
In prod the package is installed in a venv, so __file__.parents[4] doesn't
reach the repo root. Use CWD (repo root in both dev and prod via systemd
WorkingDirectory) with REPO_ROOT env var override.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603051340
2026-03-05 14:39:30 +01:00
Deeman
dc68976148 docs(marketing): add GTM, social posts, Reddit plan, and SEO calendar
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603051044
2026-03-05 11:43:11 +01:00
Deeman
60fa2bc720 test(billing): add Stripe E2E test scripts for sandbox validation
- test_stripe_sandbox.py: API-only validation of all 17 products (67 tests)
- stripe_e2e_setup.py: webhook endpoint registration via ngrok
- stripe_e2e_test.py: live webhook tests with real DB verification (67 tests)
- stripe_e2e_checkout_test.py: checkout webhook tests for credit packs,
  sticky boosts, and business plan PDF purchases (40 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 10:50:26 +01:00
Deeman
66c2dfce66 fix(billing): fetch line items for checkout.session.completed webhooks
_extract_line_items() was returning [] for all checkout sessions, which
meant _handle_transaction_completed never processed credit packs, sticky
boosts, or business plan PDF purchases. Now fetches line items from the
Stripe API using the session ID, with a fallback to embedded line_items.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 10:49:41 +01:00
Deeman
6e3c5554aa fix(admin): enable bulk actions in grouped articles view
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
- dev_run.sh: also remove app.db-shm and app.db-wal on reset to fix
  SQLite disk I/O error from stale WAL/SHM files
- articles bulk: add checkboxes to grouped rows (data-ids holds all
  variant IDs); checking a group selects EN+DE together
- restore select-all checkbox in grouped <th>
- add toggleArticleGroupSelect() JS function
- fix htmx:afterSwap to re-check group checkboxes correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202603050849
2026-03-05 09:48:54 +01:00
Deeman
ad02140594 fix(quote): add missing required asterisk and error hint to step 4
All checks were successful
CI / test (push) Successful in 54s
CI / tag (push) Successful in 3s
Step 4 (Project Phase) required location_status server-side but had no
visual "*" indicator and no error message when submitting without a
selection. All other steps already had both.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603042122
2026-03-04 22:21:57 +01:00
Deeman
5bcd87d7e5 fix(ci): replace non-existent quote.wizard endpoint with leads.quote_request
All checks were successful
CI / test (push) Successful in 53s
CI / tag (push) Successful in 3s
The CRO homepage overhaul (f4f8a45) introduced url_for('quote.wizard')
in landing.html, but that endpoint never existed — the actual route is
leads.quote_request. This broke CI runs #99–#109.

Also adds landing_vs_col_us to i18n allowlist (brand name, same in both
languages).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v202603042113
2026-03-04 22:12:45 +01:00
Deeman
77772b7ea4 feat(maps): beanflows-style divIcon bubbles + feature flag gate
Replace L.circleMarker with L.divIcon + .pn-marker CSS class (white
border, box-shadow, hover scale) matching the beanflows growing
conditions map pattern. Dark .map-tooltip CSS override (no arrow,
dark navy background). Small venue dots use .pn-venue class.

Add _require_maps_flag() to all 4 API endpoints (default=True so
dev works without seeding the flag row). Gate /opportunity-map route
the same way.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 20:51:00 +01:00
Deeman
59f1f0d699 merge(worktree): interactive maps for market pages
Self-hosted Leaflet 1.9.4 maps across 4 placements: markets hub
country bubbles, country overview city bubbles, city venue dots, and
a standalone opportunity map. New /api blueprint with 4 JSON endpoints.
New city_venue_locations SQLMesh serving model. No CDN — GDPR-safe.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

# Conflicts:
#	CHANGELOG.md
2026-03-04 15:36:41 +01:00
Deeman
0a89ba2213 docs: update CHANGELOG with interactive maps feature
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 15:33:35 +01:00
Deeman
6e936dbb95 feat(maps): Phase 5 — standalone opportunity map page
New route GET /<lang>/opportunity-map renders a full-width Leaflet map
with a country selector. On country change, fetches
/api/opportunity/{slug}.json and renders opportunity circles
(color-coded by score, sized by population) plus existing-venue gray
reference dots from /api/markets/{country}/cities.json.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 15:32:56 +01:00
Deeman
edf678ac4e feat(maps): Phase 4 — city venue dot map
New serving model: city_venue_locations joins dim_venues + dim_cities
to expose lat/lon/court_count per venue for the city dot map endpoint.

pseo_city_costs_de.sql: add c.lat, c.lon so city-cost articles have
city coordinates for the #city-map data attributes.

city-cost-de.md.jinja: add #city-map div (both DE and EN sections)
after the stats strip. Leaflet init handled by article_detail.html.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 15:07:06 +01:00
Deeman
0eef455543 feat(maps): Phase 3 — country overview city bubble map + article_detail Leaflet loader
Add #country-map div to country-overview.md.jinja (both DE/EN).
article_detail.html: always include Leaflet CSS, conditionally load
Leaflet JS only when #country-map or #city-map divs are present.
Initializes country city-bubble map and city venue-dot map from
/api/markets/{slug}/cities.json and /api/markets/{country}/{city}/venues.json.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 13:15:41 +01:00
Deeman
8e53fda283 feat(maps): Phase 2 — markets hub country bubble map
Add Leaflet map to /markets with country-level bubbles sized by
total_venues and colored by avg_market_score. Click navigates to
country overview page.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 13:04:40 +01:00
Deeman
db0d7cfee9 feat(maps): Phase 1 — Leaflet vendor files, API blueprint, app registration
Self-host Leaflet 1.9.4 JS/CSS/images in static/vendor/leaflet/.
Create api.py blueprint with 4 JSON endpoints for map data.
Register api_bp at /api in app.py (before content catch-all).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 12:46:13 +01:00
Deeman
61c197d233 merge(worktree): individualise article costs with per-country Eurostat data + tiered proxy tenant work
# Conflicts:
#	CHANGELOG.md
#	transform/sqlmesh_padelnomics/models/foundation/dim_cities.sql
#	transform/sqlmesh_padelnomics/models/foundation/dim_locations.sql
2026-03-04 12:44:56 +01:00
Deeman
2e68cfbe4f feat(transform): individualise article costs with per-country Eurostat data
Add real per-country cost data to ~30 calculator fields so pSEO articles
show country-specific CAPEX/OPEX instead of hardcoded DE defaults.

Extractor:
- eurostat.py: add 8 new datasets (nrg_pc_205, nrg_pc_203, lc_lci_lev,
  5×prc_ppp_ind variants); add optional `dataset_code` field so multiple
  dict entries can share one Eurostat API endpoint

Staging (4 new models):
- stg_electricity_prices — EUR/kWh by country, semi-annual
- stg_gas_prices         — EUR/GJ by country, semi-annual
- stg_labour_costs       — EUR/hour by country, annual (future staffed scenario)
- stg_price_levels       — PLI indices (EU27=100) for 5 categories, annual

Foundation:
- dim_countries (new) — conformed country dimension; eliminates ~50-line CASE
  blocks duplicated in dim_cities/dim_locations; computes ~29 calculator cost
  override columns from PLI ratios and energy price ratios vs DE baseline;
  NULL for DE so calculator falls through to DEFAULTS unchanged
- dim_cities — replace country_name/slug CASE blocks + country_income CTE
  with JOIN dim_countries
- dim_locations — same refactor as dim_cities

Serving:
- pseo_city_costs_de — JOIN dim_countries; add 29 camelCase override columns
  auto-applied by calculator (electricity, heating, rentSqm, hallCostSqm, …)
- planner_defaults — JOIN dim_countries; same 29 cost columns flow through
  to /api/market-data endpoint

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-04 10:09:48 +01:00
Deeman
7af6f32a2b merge: bulk actions for articles and leads
Some checks failed
CI / test (push) Failing after 33s
CI / tag (push) Has been skipped
2026-03-04 09:55:19 +01:00
Deeman
53fdbd9fd5 docs: update CHANGELOG with bulk actions feature
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 09:55:04 +01:00
Deeman
81487d6f01 feat(admin): bulk actions for articles and leads
Add bulk selection checkboxes and action bars to the articles and leads
admin pages, replicating the existing supplier bulk pattern.

Articles: publish, unpublish, toggle noindex, rebuild, delete (with
confirmation dialog). Leads: set status, set heat. Both re-render the
results partial after action via HTMX, preserving current filters.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 09:40:26 +01:00
Deeman
477f635bc5 test(billing): Stripe E2E webhook lifecycle tests
Some checks failed
CI / test (push) Failing after 29s
CI / tag (push) Has been skipped
2026-03-03 18:17:10 +01:00
Deeman
4dbded74ca test(billing): add Stripe E2E webhook lifecycle tests
16 tests covering the full Stripe webhook flow through /billing/webhook/stripe:
- Subscription creation (customer.subscription.created → DB row)
- Period end extraction from items (Stripe API 2026-02+ compatibility)
- Billing customer creation
- Status updates (active, past_due, trialing)
- Cancellation (customer.subscription.deleted → cancelled)
- Payment failure (invoice.payment_failed → past_due)
- One-time payments (checkout.session.completed mode=payment)
- Full lifecycle: create → update → recover → cancel
- Edge cases: missing metadata, empty items, invalid JSON, bad signatures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:17:05 +01:00
Deeman
230406f34f fix(billing): period_end from Stripe items + test 2026-03-03 18:06:01 +01:00
Deeman
7da6a4737d fix(billing): extract current_period_end from Stripe subscription items
Stripe API 2026-02+ moved current_period_end from subscription to
subscription items. Add _get_period_end() helper that falls back to
items[0].current_period_end when the subscription-level field is None.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:05:55 +01:00
Deeman
710e21a186 fix(billing): handle customer.subscription.created + test isolation 2026-03-03 17:58:15 +01:00
Deeman
72c4de91b0 fix(billing): handle customer.subscription.created webhook + test isolation
- Add customer.subscription.created → subscription.activated mapping in
  stripe.parse_webhook so direct API subscription creation also creates DB rows
- Add customer.subscription.created to setup_stripe.py enabled_events
- Pin PAYMENT_PROVIDER=paddle and STRIPE_WEBHOOK_SECRET="" in test conftest
  so billing tests don't hit real Stripe API when env has Stripe keys
- Add 8 unit tests for stripe.parse_webhook covering all event types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 17:29:13 +01:00
Deeman
046be665db merge: fix remaining request_options in stripe.py 2026-03-03 16:46:48 +01:00