Commit Graph

30 Commits

Author SHA1 Message Date
Deeman
84229e50f7 Merge branch 'worktree-supervisor-flags'
Python supervisor + DB-backed feature flags

- supervisor.py replaces supervisor.sh (topological wave scheduling, croniter)
- workflows.toml workflow registry (5 extractors, cron presets, depends_on)
- proxy.py round-robin + sticky proxy rotation via PROXY_URLS
- Feature flags: migration 0019, is_flag_enabled(), feature_gate() decorator
- Admin /admin/flags UI with toggle (admin-only)
- lead_unlock gate on unlock_lead route
- 59 new tests (test_supervisor.py + test_feature_flags.py)
- Fix is_flag_enabled bug (fetch_one instead of execute_fetchone)

# Conflicts:
#	CHANGELOG.md
#	web/pyproject.toml
2026-02-23 15:29:43 +01:00
Deeman
b5c9a4e573 test: e2e + unit tests for supervisor, proxy, and feature flags
- test_supervisor.py: 28 tests covering load_workflows, resolve_schedule,
  is_due, topological_waves, and proxy round-robin / sticky selection
- test_feature_flags.py: 31 tests covering migration 0019, is_flag_enabled,
  feature_gate decorator, admin toggle routes, and full toggle e2e flows
- conftest.py: seed feature flags with production defaults (markets=1,
  others=0) so all routes behave consistently in tests
- Fix is_flag_enabled bug: replace non-existent db.execute_fetchone()
  with fetch_one() helper
- Update 4 test_waitlist / test_businessplan tests that relied on
  WAITLIST_MODE patches — now enable the relevant DB flag instead

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:26:40 +01:00
Deeman
024feeaac4 feat: SEO/GEO admin hub — GSC, Bing, Umami sync + search/funnel/scorecard views
# Conflicts:
#	CHANGELOG.md
#	uv.lock
#	web/src/padelnomics/admin/templates/admin/base_admin.html
#	web/src/padelnomics/core.py
2026-02-23 15:23:03 +01:00
Deeman
4bdccb65e9 test: add 41 tests for SEO/GEO hub — sync, queries, admin routes
Covers all query functions (search perf, funnel, scorecard),
sync functions (umami with mocked httpx, bing/gsc skip tests),
admin route rendering, CSRF-protected sync POST, and boundary
validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:08:13 +01:00
Deeman
ccf03db9a3 feat: SEO/GEO admin hub — migration, sync module, routes, templates
- Migration 0019: seo_search_metrics, seo_analytics_metrics, seo_sync_log tables
- seo/ module: GSC, Bing, Umami sync + query functions (search perf, funnel, scorecard)
- Admin routes: /admin/seo hub with HTMX tabs + manual sync trigger
- Admin templates: hub page, search/funnel/scorecard partials, sidebar nav entry
- Worker: sync_gsc, sync_bing, sync_umami, cleanup_seo_metrics tasks + daily scheduler
- Config: GSC_SERVICE_ACCOUNT_PATH, GSC_SITE_URL, BING_WEBMASTER_API_KEY, BING_SITE_URL
- Deps: httpx, google-api-python-client, google-auth

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:00:36 +01:00
Deeman
a1faddbed6 feat: Python supervisor + feature flags
Supervisor (replaces supervisor.sh):
- supervisor.py — cron-based pipeline orchestration, reads workflows.toml
  on every tick, runs due extractors in topological waves with parallel
  execution, then SQLMesh transform + serving export
- workflows.toml — workflow registry: overpass (monthly), eurostat (monthly),
  playtomic_tenants (weekly), playtomic_availability (daily),
  playtomic_recheck (hourly 6–23)
- padelnomics-supervisor.service — updated ExecStart to Python supervisor

Extraction enhancements:
- proxy.py — optional round-robin/sticky proxy rotation via PROXY_URLS env
- playtomic_availability.py — parallel fetch (EXTRACT_WORKERS), recheck mode
  (main_recheck) re-queries imminent slots for accurate occupancy measurement
- _shared.py — realistic browser User-Agent on all extractor sessions
- stg_playtomic_availability.sql — reads morning + recheck snapshots, tags each
- fct_daily_availability.sql — prefers recheck over morning for same slot

Feature flags (replaces WAITLIST_MODE env var):
- migration 0019 — feature_flags table, 5 initial flags:
  markets (on), payments/planner_export/supplier_signup/lead_unlock (off)
- core.py — is_flag_enabled() + feature_gate() decorator
- routes — payments, markets, planner_export, supplier_signup, lead_unlock gated
- admin flags UI — /admin/flags toggle page + nav link
- app.py — flag() injected as Jinja2 global

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:53:45 +01:00
Deeman
49cadf6995 Merge branch 'worktree-sitemap-improvement'
# Conflicts:
#	web/src/padelnomics/admin/routes.py
2026-02-23 13:15:21 +01:00
Deeman
454b362c88 feat: admin email hub — sent log, inbox, compose, audiences, delivery tracking
Add full email management at /admin/emails with:
- email_log table tracking all outgoing emails with resend_id + delivery events
- inbound_emails table for Resend webhook-received messages
- Resend webhook handler (/webhooks/resend) updating delivery status in real-time
- send_email() returns resend_id (str|None) instead of bool; all 9 worker
  handlers pass email_type= for per-type filtering
- Admin UI: sent log with HTMX filters, email detail with API-enriched HTML
  preview, inbox with unread badges + reply, compose with branded wrapping,
  audience management with contact list/remove
- Sidebar Email section with unread badge via blueprint context processor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:00:23 +01:00
Deeman
1a6eae20d5 feat: pSEO CMS — SSG architecture with git templates + DuckDB
# Conflicts:
#	web/pyproject.toml
2026-02-23 12:51:30 +01:00
Deeman
f1181342ad feat: SSG-inspired pSEO CMS — git templates + DuckDB direct reads
Replace the old CSV-upload-based CMS with an SSG architecture where
templates live in git as .md.jinja files with YAML frontmatter and
data comes directly from DuckDB serving tables. Only articles and
published_scenarios remain in SQLite for routing/state.

- Content module: discover, load, generate, preview functions
- Migration 0018: drop article_templates + template_data, recreate
  articles + published_scenarios without FK references, add
  template_slug/language/date_modified/seo_head columns
- Admin routes: read-only template views with generate/regenerate/preview
- SEO pipeline: canonical URLs, hreflang (EN+DE), JSON-LD (Article,
  FAQPage, BreadcrumbList), Open Graph tags baked at generation time
- Example template: city-cost-de.md.jinja for German city market data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:25:44 +01:00
Deeman
aafb3cfc94 test: add e2e tests for all 9 email handlers (54 tests)
Mock send_email and call each handler directly. Covers recipient,
subject content, HTML design elements (wordmark, preheader, heat
badges), from_addr, skip-on-missing-data guards, and email_sent_at
timestamp updates.

Also fixes IndexError in handle_send_welcome when payload has no name
("".split()[0] → safe fallback).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:08:58 +01:00
Deeman
894fd0c719 feat: email design & copy upgrade for all 9 transactional emails
Redesigned _email_wrap(): lowercase wordmark header matching website,
3px blue accent border, preheader text support, HR separators.
_email_button() now full-width block for mobile tap targets.

Rewrote copy: improved subject lines, urgency cues, quick-start links
in welcome, styled project recap in quote verify, heat badges on lead
forward, "what happens next" in lead matched, secondary CTAs.

~30 new/updated translation keys in both EN and DE.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:00:49 +01:00
Deeman
4e8d94de47 feat: email i18n + Resend audience restructuring
Add lang parameter to all enqueue() calls for email internationalization.
Restructure Resend audiences to 3 named audiences (owners, suppliers, waitlist).
Use _t() translation function in all email template handlers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:06:03 +01:00
Deeman
13c86ebf84 Merge branch 'worktree-extraction-overhaul'
# Conflicts:
#	transform/sqlmesh_padelnomics/models/foundation/dim_cities.sql
#	transform/sqlmesh_padelnomics/models/staging/stg_playtomic_venues.sql
2026-02-23 01:01:26 +01:00
Deeman
343808a276 fix: move Markets to demand side of nav + translate mobile sections
Markets now sits left of logo with Planner and Quotes (investor/demand
side). Mobile section headers use i18n keys instead of hardcoded English.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:57:51 +01:00
Deeman
79f7fc6fad feat: Playtomic pricing/occupancy pipeline + email i18n + audience restructure
Three workstreams:

1. Playtomic full data extraction & transform pipeline:
   - Expand venue bounding boxes from 4 to 23 regions (global coverage)
   - New staging models for court resources, opening hours, and slot-level
     availability with real prices from the Playtomic API
   - Foundation fact tables for venue capacity and daily occupancy/revenue
   - City-level pricing benchmarks replacing hardcoded country estimates
   - Planner defaults now use 3-tier cascade: city data → country → fallback

2. Transactional email i18n:
   - _t() helper in worker.py with ~70 translation keys (EN + DE)
   - All 8 email handlers translated, lang passed in task payloads

3. Resend audiences restructured to 3 named audiences (free plan limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 00:54:53 +01:00
Deeman
9aa8a796e5 Merge branch 'worktree-sitemap-improvement'
# Conflicts:
#	web/tests/conftest.py
2026-02-23 00:39:38 +01:00
Deeman
e270d54f62 feat: sitemap hreflang alternates, caching, and lastmod cleanup
Extract sitemap generation to sitemap.py with xhtml:link hreflang
alternates (en/de/x-default) on every URL entry. Add 1-hour in-memory
TTL cache with Cache-Control header. Include supplier pages in both
languages (were EN-only). Drop misleading "today" lastmod from static
pages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:13:32 +01:00
Deeman
0521e89d7c fix: CI test failure — skip WeasyPrint tests when native libs unavailable
- Add requires_weasyprint marker to TestGenerateBusinessPlan and TestWorkerHandler
  (these need libgobject/pango/cairo which CI python:3.12-slim lacks)
- Fix export route tests: use opaque tokens instead of integer IDs
- Replace deprecated datetime.utcnow() with datetime.now(UTC)
- Add missing jsonify/Response imports to admin routes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 23:07:04 +01:00
Deeman
76695f3902 feat: admin scenario PDF download + business plan export tests
Add /scenarios/<id>/pdf admin route for direct PDF generation via WeasyPrint.
Fix plan.html Jinja template: .items → ['items'] to avoid dict method collision.
Add scenario fixture in conftest.py and comprehensive test suite for business
plan sections, PDF generation, worker handler, and export routes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 21:27:32 +01:00
Deeman
2db66efe77 feat: migrate transform to 3-layer architecture with per-layer schemas
Remove raw/ layer — staging models now read landing JSON directly.
Rename all model schemas from padelnomics.* to staging.*/foundation.*/serving.*.
Web app queries updated to serving.planner_defaults via SERVING_DUCKDB_PATH.
Supervisor gets daily sleep interval between pipeline runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:04:40 +01:00
Deeman
ea86940b78 feat: copier update v0.9.0 → v0.10.0
Pulls in template changes: export_serving.py for atomic DuckDB swap,
supervisor export step, SQLMesh glob macro, server provisioning script,
imprint template, and formatting improvements.

Template scaffold SQL models excluded (padelnomics has real models).
Web app routes/analytics unchanged (padelnomics-specific customizations).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 17:50:36 +01:00
Deeman
59306d5a90 Merge branch 'worktree-planner-mobile-redesign' 2026-02-22 17:08:10 +01:00
Deeman
a36aef7116 fix(planner): mobile polish — z-index fixes, sidebar alignment, table layout, German i18n
- Fix quote sidebar z-index (behind tab nav) and align top with tab content
- Fix bottom nav sticky positioning (move outside .planner-app)
- Fix wizard footer fixed positioning and width on mobile
- Fix bottom nav active state visibility (hardcoded colors outside CSS var scope)
- Fix country pills overflow with flex-wrap
- Fix tooltip clipping in collapsible sections
- Hide feedback button on mobile planner
- Add cache busting for static assets (_ASSET_VERSION)
- Convert export CTA to full clickable button
- Add CAPEX table section header, sort doughnut chart by size
- Cap data tables at 640px centered, horizontal scroll for wide tables
- Replace CAPEX jargon with plain German (Gesamtinvestition, Kostenaufschlüsselung)
- Update FAQ/landing copy to global language (not Europe-specific)
- Update default court sizes to realistic values (court + walkway only)
- Add missing planner_export_inline translation key (en + de)
- Revert wizard nav to client-side (HTMX broke on lang-prefixed routes)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 17:07:31 +01:00
Deeman
815fb7d98a feat: replace sequential IDs with opaque tokens in public URLs
Sequential IDs in /planner/export/<id> and /leads/<id>/unlock leaked
business volume (e.g. export_id=47 reveals ~47 PDFs sold). Replace with
22-char URL-safe tokens that carry no countable information.

- Migration 0017: adds `token TEXT` to business_plan_exports and
  lead_requests, backfills existing rows with secrets.token_urlsafe(16),
  creates unique indexes for fast lookups
- billing/routes.py: INSERT into business_plan_exports includes token
- leads/routes.py: INSERT into lead_requests includes token; enqueue
  payload includes lead_token; verify_quote() looks up by token
- planner/routes.py: /export/<token> route (was /export/<int:export_id>)
- suppliers/routes.py: /leads/<token>/unlock (was /leads/<int:lead_id>)
- worker.py: email links use token for both export and verify URLs
- Templates: url_for() calls use token= param
- test_phase0.py: _submit_guest_quote() returns (lead_id, auth_token,
  lead_token); verify URL tests use opaque lead token

Integer PKs unchanged; admin routes unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 17:03:13 +01:00
Deeman
37caf3db66 feat(planner): mobile-first redesign with bottom nav, collapsible sections, CTA bar
- Add sticky bottom tab bar on mobile (<768px) with 5 tabs (Setup, CAPEX, P&L, Cash, Returns)
- Merge Metrics tab into Returns as collapsible <details> section
- Wrap wizard input groups in collapsible <details> elements to reduce scroll fatigue
- Add contextual CTA bar above bottom nav showing CAPEX estimate + "Get Quotes" button
- Simplify desktop sidebar CTA (remove checklist, add text export link)
- Convert loadScenario/resetToDefaults/saveScenario from client-side JS to HTMX/navigation
- Convert wizard nav buttons to server-rendered partial (removes i18n from JS)
- Remove 3 unused window.__*__ globals, reduce planner.js from 208 to 131 lines
- Increase slider thumb size to 20px on mobile for better touch targets
- Add bottom padding to main content for bottom nav clearance

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 15:26:55 +01:00
Deeman
35fe934fec fix: dashboard quote link points to quote wizard instead of suppliers page
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 14:41:51 +01:00
Deeman
95888dedb9 fix(leads): fix undefined lang variable in quote validation error path
Used g.get("lang", "en") consistent with the rest of the file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 13:04:36 +01:00
Deeman
625c089284 fix: post-flatten dev scripts, lead form validation, copy improvements
- Fix dev_run.sh and dev_setup.sh cd path (../.. after repo flatten)
- Quote form: re-render step 9 inline on validation error instead of
  flash + redirect to step 1; phone/email errors now show field-level
- Supplier FAQ: move differentiation Q to top, fix Q10 email to
  hello@ (was leads@), rename Q1 to "How do I get listed?"
- Replace Innenhalle → Indoorhalle throughout DE locale and seed script

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 10:29:59 +01:00
Deeman
4ae00b35d1 refactor: flatten padelnomics/padelnomics/ → repo root
git mv all tracked files from the nested padelnomics/ workspace
directory to the git repo root. Merged .gitignore files.
No code changes — pure path rename.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 00:44:40 +01:00