Commit Graph

214 Commits

Author SHA1 Message Date
Deeman
3667616f3d feat(content): improve city-cost-de template; add country-overview + city-pricing templates
city-cost-de.md.jinja:
- Lead with market score hook instead of raw venue count
- Stats strip hero (venues, market score, peak rate, population)
- Better section headings ("What Does a Padel Investment Cost in X?")
- Mid-body planner CTA after financial cards
- Expanded FAQ (6 questions incl. ROI and country comparison)
- Footer cross-link to country overview page
- Fixed url_pattern to use country_slug directly

country-overview.md.jinja:
- Country hub page at /markets/{country_slug}
- Aggregates: total venues, cities, avg market score, pricing
- Top-5 cities table with internal links to city pages
- Hub-and-spoke internal linking architecture

city-pricing.md.jinja:
- Per-city pricing deep-dive at /markets/{country_slug}/{city_slug}/court-prices
- Stats strip: peak rate, off-peak, P25-P75 range, occupancy
- Pricing table + market context (above/below national median)
- Occupancy-driven pricing explanation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:38:06 +01:00
Deeman
cb95ecc9e6 feat(web): add .stats-strip CSS component for pSEO article hero metrics
Grid layout (2-col mobile, 4-col sm+) with label/value/unit slots.
Baked into static HTML at article generation time — no JS needed.
output.css is git-ignored (rebuild with: bin/tailwindcss -i ... -o ...).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:37:58 +01:00
Deeman
b3afd414a4 feat(transform): add three pSEO serving models — city costs, country overview, city pricing
- pseo_city_costs_de: unblocks city-cost-de template (~600 city pages),
  joins city_market_profile + planner_defaults, includes camelCase calc
  override columns (ratePeak, rateOffPeak, utilTarget, dblCourts, country)
- pseo_country_overview: per-country hub aggregating from pseo_city_costs_de,
  includes top_city_slugs/names lists for internal linking
- pseo_city_pricing: per-city pricing pages requiring >= 2 Playtomic venues,
  includes P25/P75 price range and occupancy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:37:50 +01:00
Deeman
b517e3e58d feat(transform): add country_name_en + country_slug to dim_cities, pass through city_market_profile
Prerequisite for all pSEO serving models. Adds CASE-based country_name_en
and URL-safe country_slug to foundation.dim_cities, then selects them through
serving.city_market_profile so downstream models inherit them automatically.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:37:43 +01:00
Deeman
1c195f3c05 fix: run_shell returns bool, not tuple
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:36:46 +01:00
Deeman
e35e01edb1 test: add 18 e2e tests for billing, checkout, supplier signup/dashboard, export
- Pricing page (EN/DE, plan cards, no-auth access)
- Checkout success (auth required, renders for authed user)
- Supplier signup wizard (step 1, plan cards, DE variant, success page)
- Supplier dashboard (overview stats, boosts/credit packs, listing, leads tabs)
- Business plan export (auth required, form renders)

Also fixes:
- E2e server init_db mock scope — before_serving was calling real init_db
  outside the patch context, overwriting the in-memory DB (fixes 3
  pre-existing failures: markets_hub, markets_results, signup_page)
- Add _seed_billing_data() for supplier + feature flags in e2e server
- Mock RESEND_API_KEY="" in conftest + e2e server to prevent real emails

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:34:39 +01:00
Deeman
8558fd6b40 feat: no-root deploy, remove CI deploy stage, deploy alerts
- deploy.sh installs sops/age to ./bin/ (no root/sudo needed)
- Remove CI deploy stage — supervisor auto-pulls and deploys
  (zero CI secrets: no SSH keys, no deploy credentials)
- Supervisor sends alert on deploy success/failure via webhook

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:23:19 +01:00
Deeman
e4bd9378f5 feat: self-provisioning deploy.sh — auto-installs sops+age, generates key
On first deploy to a new server, deploy.sh:
1. Installs age and sops binaries if missing
2. Generates an age keypair if missing
3. Prints the public key and exits with instructions

All checks are idempotent — subsequent deploys skip to decryption.
Removed duplicate sops/age setup from setup_server.sh (deploy.sh handles it).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:13:06 +01:00
Deeman
dcc1c15d05 Merge branch 'worktree-sops-secrets' 2026-02-23 17:21:01 +01:00
Deeman
18fab02d1e docs: add SOPS secrets section to CLAUDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:19:58 +01:00
Deeman
4ff0a0cce8 docs: update CHANGELOG and PROJECT.md for SOPS secrets migration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:19:09 +01:00
Deeman
5b074b8508 chore: delete .env.example, clarify .gitignore for sops files
.env.example replaced by .env.dev.sops. Added comment explaining that
.env.*.sops files are encrypted and committed, while .env is a
decrypted artifact that stays gitignored.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:18:39 +01:00
Deeman
fcf66104cb feat: install sops + age in setup_server.sh
Installs age and sops binaries, generates an age keypair at
/opt/padelnomics/age-key.txt, and prints the public key in next
steps so it can be added to .sops.yaml.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:15:22 +01:00
Deeman
5abd67aca5 Merge branch 'master' into worktree-sops-secrets 2026-02-23 17:14:52 +01:00
Deeman
944131535e refactor: remove CI heredoc — secrets now in encrypted sops files
deploy.sh handles decryption on the server. CI only needs SSH credentials
(SSH_PRIVATE_KEY, SSH_KNOWN_HOSTS, DEPLOY_USER, DEPLOY_HOST). All app
secrets removed from GitLab CI variables. Dead ADMIN_PASSWORD removed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:04:41 +01:00
Deeman
d91fd40cd2 feat: decrypt sops secrets in deploy.sh before docker compose
Reads age key from /opt/padelnomics/age-key.txt (overridable via
SOPS_AGE_KEY_FILE env var). Decrypts .env.prod.sops → .env with
chmod 600.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:04:17 +01:00
Deeman
9dcf237f6f feat: add encrypted prod secrets (.env.prod.sops)
Placeholder values (CHANGE_ME) for all secrets — fill via `make secrets-edit-prod`.
Includes all new supervisor, extraction, and SEO vars. Removes dead ADMIN_PASSWORD
and deprecated WAITLIST_MODE.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:03:59 +01:00
Deeman
f4a8ca7a74 fix: update waitlist tests for email i18n + audience restructuring
Pre-existing test failures after merge: enqueue payloads now include
'lang' key, and audience names changed from 'waitlist-auth' to
'newsletter' and 'waitlist-suppliers' to 'suppliers'.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:03:21 +01:00
Deeman
04ef8deaa1 feat: add encrypted dev secrets (.env.dev.sops) and Makefile targets
Dev env file replaces .env.example — decrypt with `make secrets-decrypt-dev`.
Makefile provides secrets-decrypt-dev, secrets-decrypt-prod, secrets-edit-dev,
secrets-edit-prod targets (wraps sops with --input-type dotenv).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:01:28 +01:00
Deeman
f735e36522 chore: add .sops.yaml for age-encrypted secrets
Maps .env.*.sops files to age public key for SOPS encryption.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:53:07 +01:00
Deeman
84229e50f7 Merge branch 'worktree-supervisor-flags'
Python supervisor + DB-backed feature flags

- supervisor.py replaces supervisor.sh (topological wave scheduling, croniter)
- workflows.toml workflow registry (5 extractors, cron presets, depends_on)
- proxy.py round-robin + sticky proxy rotation via PROXY_URLS
- Feature flags: migration 0019, is_flag_enabled(), feature_gate() decorator
- Admin /admin/flags UI with toggle (admin-only)
- lead_unlock gate on unlock_lead route
- 59 new tests (test_supervisor.py + test_feature_flags.py)
- Fix is_flag_enabled bug (fetch_one instead of execute_fetchone)

# Conflicts:
#	CHANGELOG.md
#	web/pyproject.toml
2026-02-23 15:29:43 +01:00
Deeman
8b7d474ede docs: update CHANGELOG and PROJECT.md for supervisor + feature flags
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:27:23 +01:00
Deeman
b5c9a4e573 test: e2e + unit tests for supervisor, proxy, and feature flags
- test_supervisor.py: 28 tests covering load_workflows, resolve_schedule,
  is_due, topological_waves, and proxy round-robin / sticky selection
- test_feature_flags.py: 31 tests covering migration 0019, is_flag_enabled,
  feature_gate decorator, admin toggle routes, and full toggle e2e flows
- conftest.py: seed feature flags with production defaults (markets=1,
  others=0) so all routes behave consistently in tests
- Fix is_flag_enabled bug: replace non-existent db.execute_fetchone()
  with fetch_one() helper
- Update 4 test_waitlist / test_businessplan tests that relied on
  WAITLIST_MODE patches — now enable the relevant DB flag instead

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:26:40 +01:00
Deeman
024feeaac4 feat: SEO/GEO admin hub — GSC, Bing, Umami sync + search/funnel/scorecard views
# Conflicts:
#	CHANGELOG.md
#	uv.lock
#	web/src/padelnomics/admin/templates/admin/base_admin.html
#	web/src/padelnomics/core.py
2026-02-23 15:23:03 +01:00
Deeman
36b90eb4df docs: update CHANGELOG and PROJECT.md for SEO/GEO hub
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:09:25 +01:00
Deeman
4bdccb65e9 test: add 41 tests for SEO/GEO hub — sync, queries, admin routes
Covers all query functions (search perf, funnel, scorecard),
sync functions (umami with mocked httpx, bing/gsc skip tests),
admin route rendering, CSRF-protected sync POST, and boundary
validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:08:13 +01:00
Deeman
325b897f38 Merge branch 'worktree-landing-backup'
# Conflicts:
#	CHANGELOG.md
2026-02-23 15:01:32 +01:00
Deeman
ccf03db9a3 feat: SEO/GEO admin hub — migration, sync module, routes, templates
- Migration 0019: seo_search_metrics, seo_analytics_metrics, seo_sync_log tables
- seo/ module: GSC, Bing, Umami sync + query functions (search perf, funnel, scorecard)
- Admin routes: /admin/seo hub with HTMX tabs + manual sync trigger
- Admin templates: hub page, search/funnel/scorecard partials, sidebar nav entry
- Worker: sync_gsc, sync_bing, sync_umami, cleanup_seo_metrics tasks + daily scheduler
- Config: GSC_SERVICE_ACCOUNT_PATH, GSC_SITE_URL, BING_WEBMASTER_API_KEY, BING_SITE_URL
- Deps: httpx, google-api-python-client, google-auth

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:00:36 +01:00
Deeman
76814dade7 feat: landing zone backup to R2 via rclone + Litestream
Landing files (append-only JSON.gz) synced to R2 every 30 min via
systemd timer + rclone. Extraction state DB (.state.sqlite) continuously
replicated via Litestream (second DB entry). Auto-restore on container
startup for both app.db and .state.sqlite. Reuses existing R2 bucket
and credentials — no new env vars needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:06:16 +01:00
Deeman
a1faddbed6 feat: Python supervisor + feature flags
Supervisor (replaces supervisor.sh):
- supervisor.py — cron-based pipeline orchestration, reads workflows.toml
  on every tick, runs due extractors in topological waves with parallel
  execution, then SQLMesh transform + serving export
- workflows.toml — workflow registry: overpass (monthly), eurostat (monthly),
  playtomic_tenants (weekly), playtomic_availability (daily),
  playtomic_recheck (hourly 6–23)
- padelnomics-supervisor.service — updated ExecStart to Python supervisor

Extraction enhancements:
- proxy.py — optional round-robin/sticky proxy rotation via PROXY_URLS env
- playtomic_availability.py — parallel fetch (EXTRACT_WORKERS), recheck mode
  (main_recheck) re-queries imminent slots for accurate occupancy measurement
- _shared.py — realistic browser User-Agent on all extractor sessions
- stg_playtomic_availability.sql — reads morning + recheck snapshots, tags each
- fct_daily_availability.sql — prefers recheck over morning for same slot

Feature flags (replaces WAITLIST_MODE env var):
- migration 0019 — feature_flags table, 5 initial flags:
  markets (on), payments/planner_export/supplier_signup/lead_unlock (off)
- core.py — is_flag_enabled() + feature_gate() decorator
- routes — payments, markets, planner_export, supplier_signup, lead_unlock gated
- admin flags UI — /admin/flags toggle page + nav link
- app.py — flag() injected as Jinja2 global

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:53:45 +01:00
Deeman
f77ec421e1 chore: add missing gitignore entries for worktrees, duckdb, logs, sqlmesh cache
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:21:14 +01:00
Deeman
49cadf6995 Merge branch 'worktree-sitemap-improvement'
# Conflicts:
#	web/src/padelnomics/admin/routes.py
2026-02-23 13:15:21 +01:00
Deeman
488e47b4b4 docs: rewrite CMS.md for SSG architecture
Replace the old CSV-upload CMS documentation with the new SSG system:
git templates, DuckDB data sources, generation pipeline, SEO pipeline
(hreflang, JSON-LD, canonical, OG), admin routes, and step-by-step
guide for adding new pSEO ideas.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:03:30 +01:00
Deeman
454b362c88 feat: admin email hub — sent log, inbox, compose, audiences, delivery tracking
Add full email management at /admin/emails with:
- email_log table tracking all outgoing emails with resend_id + delivery events
- inbound_emails table for Resend webhook-received messages
- Resend webhook handler (/webhooks/resend) updating delivery status in real-time
- send_email() returns resend_id (str|None) instead of bool; all 9 worker
  handlers pass email_type= for per-type filtering
- Admin UI: sent log with HTMX filters, email detail with API-enriched HTML
  preview, inbox with unread badges + reply, compose with branded wrapping,
  audience management with contact list/remove
- Sidebar Email section with unread badge via blueprint context processor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:00:23 +01:00
Deeman
1a6eae20d5 feat: pSEO CMS — SSG architecture with git templates + DuckDB
# Conflicts:
#	web/pyproject.toml
2026-02-23 12:51:30 +01:00
Deeman
afd46398af test: add E2E and unit tests for pSEO CMS flows
38 new tests covering all previously untested code paths:

Content module unit tests:
- TestDiscoverTemplates: discovery, empty dir, invalid frontmatter
- TestLoadTemplate: config+body loading, missing template, schema normalization
- TestExtractFaqPairs: FAQ extraction, no FAQ section, end-of-doc
- TestBuildBreadcrumbs: path segments, root, hyphenated labels
- TestBuildJsonld: BreadcrumbList, Article, FAQPage, headline truncation
- TestPreviewArticle: rendering, unknown row, language, unknown template

Admin route E2E tests:
- TestAdminTemplateDetail: config view, columns, sample data, unknown slug
- TestAdminTemplatePreview: rendered article, bad key/template redirects
- TestAdminTemplateGenerate: form display, article+scenario creation, unknown
- TestAdminTemplateRegenerate: idempotent update, unknown template redirect
- TestAdminTemplates: list shows discovered templates from disk

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:48:26 +01:00
Deeman
f1181342ad feat: SSG-inspired pSEO CMS — git templates + DuckDB direct reads
Replace the old CSV-upload-based CMS with an SSG architecture where
templates live in git as .md.jinja files with YAML frontmatter and
data comes directly from DuckDB serving tables. Only articles and
published_scenarios remain in SQLite for routing/state.

- Content module: discover, load, generate, preview functions
- Migration 0018: drop article_templates + template_data, recreate
  articles + published_scenarios without FK references, add
  template_slug/language/date_modified/seo_head columns
- Admin routes: read-only template views with generate/regenerate/preview
- SEO pipeline: canonical URLs, hreflang (EN+DE), JSON-LD (Article,
  FAQPage, BreadcrumbList), Open Graph tags baked at generation time
- Example template: city-cost-de.md.jinja for German city market data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:25:44 +01:00
Deeman
cb1f00baf0 test: add live Resend integration tests (delivered@resend.dev)
9 tests exercise the full handler→wrap→send_email→Resend API path
using Resend's @resend.dev test addresses. Skipped when RESEND_API_KEY
is not set.

With a full_access API key, tests also retrieve the sent email via
resend.Emails.get() and assert on the rendered HTML (wordmark, links,
project details, heat badges). With a sending_access key, send is
verified but HTML assertions are skipped gracefully.

Includes bounce handling test and 0.6s inter-test delay for Resend's
2 req/sec rate limit.

Run with: RESEND_API_KEY=re_xxx pytest -k resend_live

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:25:12 +01:00
Deeman
aafb3cfc94 test: add e2e tests for all 9 email handlers (54 tests)
Mock send_email and call each handler directly. Covers recipient,
subject content, HTML design elements (wordmark, preheader, heat
badges), from_addr, skip-on-missing-data guards, and email_sent_at
timestamp updates.

Also fixes IndexError in handle_send_welcome when payload has no name
("".split()[0] → safe fallback).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:08:58 +01:00
Deeman
894fd0c719 feat: email design & copy upgrade for all 9 transactional emails
Redesigned _email_wrap(): lowercase wordmark header matching website,
3px blue accent border, preheader text support, HR separators.
_email_button() now full-width block for mobile tap targets.

Rewrote copy: improved subject lines, urgency cues, quick-start links
in welcome, styled project recap in quote verify, heat badges on lead
forward, "what happens next" in lead matched, secondary CTAs.

~30 new/updated translation keys in both EN and DE.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:00:49 +01:00
Deeman
7737b79230 fix: DuckDB compat issues in Playtomic pipeline + export_serving
- Add maximum_object_size=128MB to read_json for 14K-venue tenants file
- Rewrite opening_hours to use UNION ALL unpivot (DuckDB struct dynamic access)
- Add seed file guard for availability model (empty result on first run)
- Fix snapshot_date VARCHAR→DATE comparison in venue_pricing_benchmarks
- Fix export_serving to resolve SQLMesh physical tables from view definitions
  (SQLMesh views reference "local" catalog unavailable outside its context)
- Add pyarrow dependency for Arrow-based cross-connection data transfer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:27:51 +01:00
Deeman
a055660cd2 fix: replace broken bbox pagination with global page-based extraction
Playtomic API ignores bbox params (min_latitude, etc.) and offset param.
Discovered that `page` param works correctly for global enumeration.

Result: 14,202 venues across 82 countries (was 100 with bbox approach).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:16:35 +01:00
Deeman
4e8d94de47 feat: email i18n + Resend audience restructuring
Add lang parameter to all enqueue() calls for email internationalization.
Restructure Resend audiences to 3 named audiences (owners, suppliers, waitlist).
Use _t() translation function in all email template handlers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:06:03 +01:00
Deeman
13c86ebf84 Merge branch 'worktree-extraction-overhaul'
# Conflicts:
#	transform/sqlmesh_padelnomics/models/foundation/dim_cities.sql
#	transform/sqlmesh_padelnomics/models/staging/stg_playtomic_venues.sql
2026-02-23 01:01:26 +01:00
Deeman
343808a276 fix: move Markets to demand side of nav + translate mobile sections
Markets now sits left of logo with Planner and Quotes (investor/demand
side). Mobile section headers use i18n keys instead of hardcoded English.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:57:51 +01:00
Deeman
79f7fc6fad feat: Playtomic pricing/occupancy pipeline + email i18n + audience restructure
Three workstreams:

1. Playtomic full data extraction & transform pipeline:
   - Expand venue bounding boxes from 4 to 23 regions (global coverage)
   - New staging models for court resources, opening hours, and slot-level
     availability with real prices from the Playtomic API
   - Foundation fact tables for venue capacity and daily occupancy/revenue
   - City-level pricing benchmarks replacing hardcoded country estimates
   - Planner defaults now use 3-tier cascade: city data → country → fallback

2. Transactional email i18n:
   - _t() helper in worker.py with ~70 translation keys (EN + DE)
   - All 8 email handlers translated, lang passed in task payloads

3. Resend audiences restructured to 3 named audiences (free plan limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:54:53 +01:00
Deeman
9aa8a796e5 Merge branch 'worktree-sitemap-improvement'
# Conflicts:
#	web/tests/conftest.py
2026-02-23 00:39:38 +01:00
Deeman
5b6c4182f7 fix: sort imports in remaining test files (ruff I001)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:22:51 +01:00
Deeman
6f81ffbc45 fix: sort imports in test files (ruff I001)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:19:19 +01:00
Deeman
e270d54f62 feat: sitemap hreflang alternates, caching, and lastmod cleanup
Extract sitemap generation to sitemap.py with xhtml:link hreflang
alternates (en/de/x-default) on every URL entry. Add 1-hour in-memory
TTL cache with Cache-Control header. Include supplier pages in both
languages (were EN-only). Drop misleading "today" lastmod from static
pages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:13:32 +01:00