Commit Graph

163 Commits

Author SHA1 Message Date
Deeman
ad1da5c335 feat: outreach follow-up scheduling, activity timeline, and pSEO noindex (migration 0025)
Feature A — Outreach follow-up + activity timeline:
- follow_up_at column on suppliers (migration 0025)
- HTMX date picker on outreach rows, POST /admin/outreach/<id>/follow-up
- Amber due-today banner on /admin/outreach with ?follow_up=due filter
- get_follow_up_due_count() for dashboard widget
- Activity timeline on /admin/suppliers/<id>: merges sent + received emails by contact_email

Feature B — pSEO article noindex:
- noindex column on articles (migration 0025)
- NOINDEX_THRESHOLDS per-template lambdas in content/__init__.py
- generate_articles() evaluates threshold and stores noindex=1 for thin-data articles
- <meta name="robots" content="noindex, follow"> in article_detail.html
- Sitemap excludes noindex articles (AND noindex = 0)
- pSEO dashboard noindex count card + article row badge

Tests: 49 new tests (29 outreach, 20 noindex), 1377 total, 0 failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 17:51:38 +01:00
Deeman
ea06dd0689 feat(outreach+pseo): follow-up scheduling, activity timeline, noindex articles (subtasks 1-9)
Feature A — Outreach follow-up scheduling + activity timeline:
- Migration 0025: follow_up_at column on suppliers
- POST /admin/outreach/<id>/follow-up route (HTMX date picker, updates row)
- get_follow_up_due_count() query + amber banner on /admin/outreach
- ?follow_up=due / ?follow_up=set filters in get_outreach_suppliers()
- Follow-up column in outreach_results.html + outreach_row.html date input
- Activity timeline on supplier_detail.html — merges email_log (sent outreach)
  and inbound_emails (received) by contact_email, sorted by date

Feature B — pSEO article noindex:
- Migration 0025: noindex column on articles (default 0)
- NOINDEX_THRESHOLDS dict in content/__init__.py (per-template thresholds)
- generate_articles() upsert now stores noindex = 1 for thin-data articles
- <meta name="robots" content="noindex, follow"> in article_detail.html (conditional)
- sitemap.py excludes noindex=1 articles from sitemap.xml
- pSEO dashboard noindex count card; article_row.html noindex badge
- 73 new tests (test_outreach.py + test_noindex.py), 1377 total, 0 failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 16:12:21 +01:00
Deeman
606d6f7d4c merge(outreach): admin outreach pipeline + separate sending domain
# Conflicts:
#	CHANGELOG.md
2026-02-25 15:29:20 +01:00
Deeman
efaba2cb76 feat(outreach): admin outreach pipeline + separate sending domain (all subtasks)
Adds cold B2B supplier outreach pipeline isolated from transactional emails.

Subtask 1 — Migration + constants:
- Migration 0024: 4 new columns on suppliers (outreach_status, outreach_notes,
  last_contacted_at, outreach_sequence_step); NULL status = not in pipeline
- EMAIL_ADDRESSES["outreach"] = hello.padelnomics.io (separate reputation domain)
- "outreach" added to EMAIL_TYPES

Subtask 2 — Query functions + routes:
- get_outreach_pipeline() — counts by status for pipeline cards
- get_outreach_suppliers() — filtered list with status/country/search
- GET /admin/outreach — pipeline dashboard
- GET /admin/outreach/results — HTMX partial
- POST /admin/outreach/<id>/status — inline status update
- POST /admin/outreach/<id>/note — inline note edit
- POST /admin/outreach/add-prospects — bulk set from supplier list

Subtask 3 — CSV import:
- GET/POST /admin/outreach/import
- Accepts name+contact_email (required), country_code/category/website (optional)
- Deduplicates by contact_email, auto-generates slug, capped at 500 rows

Subtask 4 — Templates:
- outreach.html (pipeline cards + HTMX filter + results table)
- outreach_import.html (CSV upload form)
- partials/outreach_results.html, partials/outreach_row.html
- base_admin.html: Outreach sidebar link
- suppliers.html + supplier_results.html: checkbox column + bulk action bar

Subtask 5 — Compose integration:
- email_compose() GET: ?from_key=outreach&email_type=outreach&supplier_id=<id>
  pre-fills from-addr, stores hidden fields, defaults wrap=0 (plain text)
- email_compose() POST: on outreach send, advances prospect→contacted,
  increments outreach_sequence_step, sets last_contacted_at
- email_compose.html: hidden email_type + supplier_id fields, outreach banner
- supplier_detail.html: outreach card (status, step, last contact, send button)

Subtask 6 — Tests:
- 44 tests in web/tests/test_outreach.py covering: constants, access control,
  query functions, dashboard, HTMX partial, status update, note update,
  add-prospects, CSV import, compose pre-fill, compose pipeline update

Subtask 7 — Docs:
- CHANGELOG.md and PROJECT.md updated

Manual step after deploy: add hello.padelnomics.io in Resend dashboard + DNS.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 14:06:53 +01:00
Deeman
de67d41fd4 merge: bp-and-articles — 12 cornerstone articles + KfW PDF overhaul
From worktree-bp-and-articles:

Content (12 articles, Batch 1):
  C2 Cost Bible (DE+EN), C3 Business Plan for Banks (DE+EN),
  C5 Location Guide (DE+EN), C6 Financing Guide (DE+EN),
  C7 Risk Register (DE+EN), C8 Build Guide (DE+EN)
  All written natively (linguistic-mediation for DE), frontmatter complete.

CMS fix:
  Article form now includes language selector; seo_head generated +
  stored for manually created articles; build path is lang-prefixed.

Business Plan PDF overhaul (KfW Gründerkredit-ready):
  - compute_sensitivity() extracted as reusable function
  - matplotlib SVG charts (P&L + 12-month cash flow)
  - Opening balance sheet, use-of-funds, sensitivity analysis
  - Market analysis auto-populated from DuckDB city data
  - Pre-export details form (/planner/export/details)
  - Migration 0020: bp_details_json on scenarios table
  - Complete PDF redesign: Precision Finance aesthetic
    (navy/gold, Georgia headings, cover page, TOC, 15 sections)
  - 28 new translation keys in en.json + de.json

Docs:
  SPORTPLATZWELT_RESEARCH.md + CUSTOMER_CHANNELS.md updated
  with verified contacts and trade show dates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

# Conflicts:
#	web/src/padelnomics/admin/routes.py
#	web/src/padelnomics/locales/de.json
#	web/src/padelnomics/locales/en.json
2026-02-25 13:58:16 +01:00
Deeman
37d4886e79 feat(pdf): KfW Gründerkredit-ready business plan overhaul
Part B of the bp-and-articles plan. Full overhaul of the PDF generation
engine and template to produce bank-ready documents.

New sections added (all required by KfW application):
- Founder/Management Profile (Gründerprofil)
- Business Description (Vorhabensbeschreibung)
- Market Analysis (auto-populated from DuckDB city data when available)
- Use of Funds (Mittelverwendungsplan)
- Opening Balance Sheet (Eröffnungsbilanz)
- Sensitivity Analysis (utilization + price tables)
- Risk Analysis (templated 6-risk register)

Data & computation:
- Extract compute_sensitivity() from augment_d() → reusable function
  shared by web planner and PDF generator
- Add matplotlib chart generation: _generate_pnl_chart_svg() +
  _generate_cashflow_chart_svg() — SVG embedded in WeasyPrint HTML
- Add _compute_opening_balance() + _compute_use_of_funds() helpers
- Add _fetch_market_data() — queries DuckDB serving.city_market_overview

Business plan details form:
- New /planner/export/details route (GET/POST)
- New export_details.html template — 5 narrative sections with placeholders
- Migration 0020: bp_details_json TEXT column on scenarios table
- generate_business_plan() loads bp_details_json + calls _fetch_market_data()

Design (plan.html + plan.css):
- Complete redesign: "Precision Finance" aesthetic
- Navy (#0F2651) + warm gold (#C9922C) + Georgia serif headings
- Full-bleed branded cover page with sidebar
- Table of contents page
- Gold-accented section headers (h2 with left border)
- Professional @page running headers/footers
- Executive summary 3-column card grid
- 2-column opening balance sheet
- Sensitivity tables with highlight-row for target/base values
- All tables: navy thead, alternating rows, professional total rows

Translations: 28 new bp_ keys in en.json and de.json for all new
section headings and table labels.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:53:32 +01:00
Deeman
6cb0fb32ec feat(cms): add language field + seo_head to manual article creation
- Add language selector (en/de) to article create/edit form
- Store language and generated seo_head in articles table on CREATE and UPDATE
- Write HTML build to BUILD_DIR/{lang}/{slug}.html (consistent with pSEO)
- article_detail.html: render article.seo_head when present (canonical,
  hreflang, OG, JSON-LD Article) — falls back to inline for legacy articles

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 13:25:17 +01:00
Deeman
d637687795 feat(pipeline): tests, docs, and ruff fixes (subtask 6/6)
- Add 29-test suite for all pipeline routes, data helpers, and query
  execution (test_pipeline.py); all 1333 tests pass
- Fix ruff UP041: asyncio.TimeoutError → TimeoutError in analytics.py
- Fix ruff UP036/F401: replace sys.version_info tomllib block with
  plain `import tomllib` (project requires Python 3.11+)
- Fix ruff F841: remove unused `cutoff` variable in pipeline_overview
- Update CHANGELOG.md with Pipeline Console entry
- Update PROJECT.md: add Pipeline Console to Admin Panel done list

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 13:02:51 +01:00
Deeman
8f8f7f7acb feat(pipeline): query editor tab templates
- partials/pipeline_query.html: dark-themed SQL textarea (navy bg, Commit
  Mono, 12px border-radius, electric blue focus glow) + schema sidebar
  (collapsible per-table column lists with types) + controls bar (Execute,
  Clear, limit/timeout note) + Tab-key indent + Cmd/Ctrl+Enter submit
- partials/pipeline_query_results.html: results table with sticky headers,
  horizontal scroll, row count + elapsed time metadata, truncation warning,
  error display in red monospace card

Subtask 5 of 6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:55:20 +01:00
Deeman
5b48a11e01 feat(pipeline): catalog tab templates
- partials/pipeline_catalog.html: accordion list of serving tables with
  row count badges, column count, click-to-expand lazy-loaded detail
- partials/pipeline_table_detail.html: column schema grid + sticky-header
  sample data table (10 rows, truncated values with title attribute)
- JS: toggleCatalogTable() + htmx.trigger(content, 'revealed') for
  lazy-loading detail only on first open

Subtask 4 of 6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:54:21 +01:00
Deeman
947a1a778e feat(pipeline): extractions tab template
- Filterable extraction run history table (extractor + status dropdowns,
  HTMX live filter via 'change' trigger)
- Status badges with stale row highlighting (amber background)
- 'Mark Failed' button for stuck 'running' rows (with confirm dialog)
- 'Run All Extractors' trigger button
- Pagination via hx-vals

Subtask 3 of 6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:53:36 +01:00
Deeman
cac876e48f feat(pipeline): dashboard + overview tab templates
- pipeline.html: 4 stat cards (total runs, success rate, serving tables,
  last export) + stale-run warning banner + tab bar (Overview/Extractions/
  Catalog/Query) + tab container (lazy-loaded via HTMX on page load)
- partials/pipeline_overview.html: extraction status grid (one card per
  workflow with status dot, schedule, last run timestamp, error preview),
  serving freshness table (row counts per table), landing zone file stats

Subtask 2 of 6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:53:02 +01:00
Deeman
060cb9b32e feat(pipeline): scaffold Pipeline Console blueprint + sidebar + app registration
- New pipeline_routes.py blueprint (url_prefix=/admin/pipeline) with:
  - All 9 routes (dashboard, overview, extractions, catalog, query editor)
  - Data access functions: state DB (sync+to_thread), serving meta, landing FS, workflows.toml
  - execute_user_query() added to analytics.py (columns+rows+error+elapsed_ms)
  - Query security: blocklist regex, 10k char limit, 1000 row cap, 10s timeout
- Add 'Pipeline' sidebar section to base_admin.html (between Analytics and System)
- Register pipeline_bp in app.py
- Add run_extraction task handler to worker.py

Subtask 1 of 6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:44:03 +01:00
Deeman
578a409893 feat(email-templates): tests, docs, and fix quote_verification sample data (subtask 8)
- Add 50 tests in test_email_templates.py:
  - TestRenderEmailTemplate: all 11 registry templates render in EN + DE
    without error; checks DOCTYPE, wordmark, font, CTA color, template-
    specific content (heat badges, brief rows, weekly digest loop, etc.)
    and registry structure
  - TestEmailGalleryRoutes: access control, gallery list (all labels
    present, preview links), preview pages (EN/DE/nonexistent/invalid-lang),
    compose preview endpoint (plain + wrapped + empty body)
- Fix _quote_verification_sample: add missing recap_parts key — StrictUndefined
  raised on the {% if recap_parts %} check when the variable was absent
- Update CHANGELOG.md: document email template system (renderer, base,
  macros, 11 templates, registry, gallery, compose preview, removed helpers)
- Update PROJECT.md: add email template system + gallery to Done section

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 12:24:52 +01:00
Deeman
fb83f432db feat(emails): subtask 7 — remove _email_wrap() and _email_button() from worker.py
All 10 email handlers now use render_email_template(). The two legacy
inline-HTML helpers are no longer needed and have been removed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:20:32 +01:00
Deeman
4fafd3e80e feat(emails): subtask 6 — admin gallery (routes, templates, sidebar link)
- Add GET /admin/emails/gallery — card grid of all 11 email types
- Add GET /admin/emails/gallery/<slug>?lang=en|de — preview with lang toggle
- Add email_gallery.html: 3-column responsive card grid
- Add email_gallery_preview.html: full-width iframe + EN/DE toggle + log link
- Add Gallery sidebar link to base_admin.html (admin_page == 'gallery')

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:13:35 +01:00
Deeman
536d5c8f40 feat(emails): subtask 5 — compose preview (admin_compose template + HTMX endpoint)
- Add emails/admin_compose.html: branded wrapper for ad-hoc compose body
- Update email_compose.html: two-column layout with HTMX live preview pane
  (hx-post, hx-trigger=input delay:500ms, hx-target=#preview-pane)
- Add partials/email_preview_frame.html: sandboxed iframe partial
- Add POST /admin/emails/compose/preview route (no CSRF — read-only render)
- Update email_compose POST handler to use render_email_template() instead
  of importing _email_wrap from worker

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:12:09 +01:00
Deeman
c31d4a71a0 feat(emails): subtask 4 — 4 complex templates (lead_forward, match_notify, digest, business_plan)
- Add lead_forward.html (brief table + contact table + optional CTA token link)
- Add lead_match_notify.html (new matching lead alert with heat badge)
- Add weekly_digest.html (leads table with Jinja2 for loop)
- Add business_plan.html (PDF ready notification with download CTA)
- Refactor 4 handlers in worker.py: send_lead_forward_email,
  notify_matching_suppliers, send_weekly_lead_digest, generate_business_plan

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:10:55 +01:00
Deeman
1c7cdc42f2 feat(emails): subtask 3 — 4 medium templates (quote_verification, waitlist, lead_matched)
- Add quote_verification.html (with optional project recap card)
- Add waitlist_supplier.html, waitlist_general.html
- Add lead_matched.html (with next-steps section + tip box)
- Refactor 3 handlers in worker.py: send_quote_verification,
  send_waitlist_confirmation, send_lead_matched_notification

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:08:55 +01:00
Deeman
daf1945d5b feat(emails): subtask 1-2 — email_templates.py foundation + 3 simple templates
- Add email_templates.py: standalone Jinja2 env, render_email_template(),
  EMAIL_TEMPLATE_REGISTRY with sample_data functions for all 11 email types
- Add templates/emails/_base.html: direct transliteration of _email_wrap()
- Add templates/emails/_macros.html: email_button, heat_badge, heat_badge_sm,
  section_heading, info_box macros
- Add magic_link.html, welcome.html, supplier_enquiry.html templates
- Refactor 3 handlers in worker.py to use render_email_template()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 12:08:55 +01:00
Deeman
e5960c08ff feat(admin): cross-section links across leads, suppliers, marketplace, emails
Lead detail:
- contact_email → 📧 email log (pre-filtered), mailto, Send Email compose
- country → leads list filtered by that country

Supplier detail:
- contact_email → 📧 email log (pre-filtered), mailto, Send Email compose
- claimed_by → user detail page (was plain "User #N")

Marketplace dashboard:
- Funnel card numbers are now links: Total → /leads, Verified New →
  /leads?status=new, Unlocked → /leads?status=forwarded, Won → /leads?status=closed_won
- Active suppliers number links to /suppliers

Marketplace activity stream:
- lead events → link to lead_detail
- unlock events → supplier name links to supplier_detail, "lead #N" links to lead_detail
- credit events → supplier name links to supplier_detail (query now joins
  suppliers table for name; ref2_id exposes supplier_id and lead_id per event)

Email detail:
- Reverse-lookup to_addr against lead_requests + suppliers; renders
  linked "Lead #N" / "Supplier Name" chips next to the To field

Email compose:
- Accepts ?to= query param to pre-fill recipient (enables Send Email links)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 10:15:25 +01:00
Deeman
3c0f57c0fd feat(leads): 2-hour admin review window before leads appear in supplier feed
New visible_from column on lead_requests set to NOW + 2h on both the
direct insert (logged-in user) and the email verification update.

Supplier feed, notify_matching_suppliers, and send_weekly_lead_digest
all filter on visible_from <= datetime('now'), so no lead surfaces to
suppliers before the window expires.

Migration 0023 adds the column and backfills existing verified leads
with created_at so they remain immediately visible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 09:53:19 +01:00
Deeman
5867c611f8 feat(admin): marketplace dashboard + HTMX lead management improvements
Admin marketplace (/admin/marketplace):
- Lead funnel cards: total / verified-new / unlocked / won / conversion rate
- Credit economy: issued / consumed / outstanding / 30-day burn
- Supplier engagement: active count / avg unlocks / response rate
- Feature flag toggles (lead_unlock, supplier_signup) with next= redirect
- Live activity stream (HTMX partial): last 50 lead / unlock / credit events

Admin leads list (/admin/leads):
- Summary cards: total / new+unverified / hot pipeline credits / forward rate
- Search filter (name, email, company) with HTMX live update
- Period pills: Today / 7d / 30d / All
- get_leads() now returns (rows, total_count); get_lead_stats() includes
  _total, _new_unverified, _hot_pipeline, _forward_rate

Admin lead detail (/admin/leads/<id>):
- Inline HTMX status change returning updated status badge partial
- Inline HTMX forward form returning updated forward history partial
  (replaces full-page reload on every status/forward action)
- Forward history table shows supplier, status, credit_cost, sent_at

Quote form extended with optional fields:
- build_context, glass_type, lighting_type, location_status,
  financing_status, services_needed, additional_info
  (captured in lead detail view but not required for heat scoring)

Sidebar nav: "Marketplace" tab added between Leads and Suppliers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 09:31:44 +01:00
Deeman
7af612504b feat(marketplace): lead matching notifications + weekly digest + CTA tracking
- notify_matching_suppliers task: on lead verification, finds growth/pro
  suppliers whose service_area matches the lead country and sends an
  instant alert email (LIMIT 20 suppliers per lead)
- send_weekly_lead_digest task: every Monday 08:00 UTC, sends paid
  suppliers a table of new matching leads from the past 7 days they
  haven't seen yet (LIMIT 5 per supplier)
- One-click CTA token: forward emails now include a "Mark as contacted"
  footer link; clicking sets forward status to 'contacted' immediately
- cta_token stored on lead_forwards after email send
- Supplier lead_respond endpoint: HTMX status update for forwarded leads
  (sent / viewed / contacted / quoted / won / lost / no_response)
- Supplier lead_cta_contacted endpoint: handles one-click email CTA,
  redirects to dashboard leads tab
- leads/routes.py: enqueue notify_matching_suppliers on quote verification

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 09:31:23 +01:00
Deeman
c84a5ffdd1 feat(db): migration 0022 — add response tracking to lead_forwards
Adds status_updated_at, supplier_note, and cta_token columns to the
lead_forwards table. cta_token gets a unique partial index for fast
one-click email CTA lookups.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 09:31:14 +01:00
Deeman
19dd9843af fix(dev): scope granian --reload-paths to web/src to stop DB WAL triggering reloads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:49:14 +01:00
Deeman
77d4c02db3 chore: run dev server with granian --reload for dev/prod parity
Replaces `python -m padelnomics.app` (Quart's built-in Hypercorn-based
dev runner) with granian directly. Adds granian[reload] extra which
pulls in watchfiles for file-change detection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:42:30 +01:00
Deeman
c95d66982b fix(logging): restore hypercorn logger silencing (still used by Quart dev server)
Quart depends on Hypercorn and uses it in app.run() → run_task().
Removing the silencing caused hypercorn.error noise in dev logs.
Keep both granian and hypercorn logger config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:40:23 +01:00
Deeman
fda7da7d59 chore: replace hypercorn with granian (Rust ASGI server)
Granian is ~3-5x faster than Hypercorn in benchmarks. No code changes
needed — Quart is standard ASGI so any ASGI server works.

- web/pyproject.toml: hypercorn → granian>=1.6.0 (installed: 2.7.1)
- Dockerfile CMD: hypercorn → granian --interface asgi
- core.py setup_logging(): silence granian loggers instead of hypercorn's

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:26:19 +01:00
Deeman
aa7a8bad99 test: sync i18n tests to current translation values
- wiz_summary_label DE: "Aktuelle Werte" → "Aktuelle Zusammenfassung"
- add mscore_reife_chip + mscore_potenzial_chip to identical-value allowlist (branded product names)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:24:37 +01:00
Deeman
e8fb8f51f7 feat(pseo): add pSEO Engine admin tab
Operational dashboard at /admin/pseo for the programmatic SEO system:
content gap detection, data freshness signals, article health checks
(hreflang orphans, missing build files, broken scenario refs), and
live generation job monitoring with HTMX progress bars.

- _serving_meta.json written by export_serving.py after atomic DB swap
- content/health.py: pure async query functions for all health checks
- Migration 0021: progress_current/total/error_log on tasks table
- generate_articles() writes progress every 50 articles + on completion
- admin/pseo_routes.py: 6 routes, standalone blueprint
- 5 HTML templates + sidebar nav + fromjson Jinja filter
- 45 tests (all passing); 2 bugs caught and fixed during testing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

# Conflicts:
#	src/padelnomics/export_serving.py
2026-02-24 21:00:00 +01:00
Deeman
ec15012d00 test: update mock_fetch_analytics to handle COUNT(*) queries
count_template_data() uses fetch_analytics with a COUNT(*) query.
The pseo_env test fixture's mock returned TEST_ROWS for any unrecognized
query, causing a KeyError on rows[0]["cnt"]. Add a COUNT(*) branch that
returns [{cnt: len(TEST_ROWS)}].

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:55:38 +01:00
Deeman
44c0dd0b8d refactor: minor TigerStyle cleanups
- export_serving.py: move `import re` to module level — was imported
  inside a loop body on every iteration
- sitemap.py: add comment documenting that the in-memory TTL cache is
  process-local (valid for single-worker deployment, Dockerfile --workers 1)
- playtomic_availability.py: use `or "10"` fallback for
  CIRCUIT_BREAKER_THRESHOLD env var to handle empty-string case

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:50:43 +01:00
Deeman
ee49862d91 test(pseo): add 45 tests for health checks + pSEO Engine admin routes
Covers content/health.py (get_template_stats, get_template_freshness,
get_content_gaps, check_hreflang_orphans, check_missing_build_files,
check_broken_scenario_refs, get_all_health_issues) and all 6 routes in
admin/pseo_routes.py (dashboard, health partial, gaps partial, generate
gaps, jobs list, job status polling).

Also fixes two bugs found while writing tests:
- check_hreflang_orphans: was grouping by url_path, but EN/DE articles
  have different paths. Now extracts natural key from slug pattern
  "{template_slug}-{lang}-{nk}" and groups by nk.
- pseo_job_status.html + pseo_jobs.html: | default('') | truncate() fails
  when completed_at is None (default() only handles undefined, not None).
  Fixed to (value or '') | truncate().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 20:50:03 +01:00
Deeman
83d148477d refactor: extract shared _query_scenarios() to remove duplication
scenarios() and scenario_results() both built the same WHERE clause and
ran the same filtered query. Extracted into _query_scenarios(search,
country, venue_type) -> (rows, total). Each handler is now ~10 lines
of param parsing + render_template.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:42:52 +01:00
Deeman
dd9ffd6d27 style: add units to variable names, name busy_timeout constant
- core.py: rename RATE_LIMIT_WINDOW → RATE_LIMIT_WINDOW_SECONDS (env var
  name RATE_LIMIT_WINDOW is unchanged — only the Python attribute)
- core.py: extract _BUSY_TIMEOUT_MS = 5000 local constant so the PRAGMA
  value is no longer a bare magic number
- worker.py: rename poll_interval → poll_interval_seconds

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:35:12 +01:00
Deeman
9107ba9bb8 perf: fix N+1 queries in templates(), handle_refill_monthly_credits()
templates() in admin:
- Replace per-template SELECT COUNT(*) articles queries with a single
  GROUP BY query before the loop — O(n) SQLite calls → O(1)
- Replace per-template SELECT * LIMIT 501 (for count) with a new
  count_template_data() that runs SELECT COUNT(*) — cheaper per call
- Add count_template_data() to content/__init__.py

handle_refill_monthly_credits() in worker:
- Replace N×3 per-supplier queries (fetch supplier, insert ledger,
  update balance) with 2 bulk SQL statements:
  1. INSERT INTO credit_ledger SELECT ... for all eligible suppliers
  2. UPDATE suppliers SET credit_balance = credit_balance + monthly_credits
- Wrap in single transaction() for atomicity
- Log total suppliers updated at INFO level

audiences() in admin:
- Add LIMIT 20 guard + comment explaining why one API call per audience
  is unavoidable (no bulk contacts endpoint in Resend)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:34:15 +01:00
Deeman
a051f9350f feat(pseo): create pSEO Engine admin templates + sidebar nav
- base_admin.html: add pSEO section with "pSEO Engine" link
- pseo_dashboard.html: template stats, freshness badges, HTMX gaps panels,
  recent jobs table, health issues HTMX-loaded section
- pseo_health.html: HTMX partial — hreflang orphans, missing build files,
  broken scenario refs (collapsible details with drill-down tables)
- pseo_gaps.html: HTMX partial — missing rows per template with generate button
- pseo_jobs.html: full jobs list with live progress bars (HTMX polling)
- pseo_job_status.html: HTMX partial — polls every 2s while job is pending
- app.py: add `fromjson` Jinja2 filter for displaying task payloads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 19:33:18 +01:00
Deeman
ef7fe6e079 perf: add missing indexes on lead_requests, suppliers, credit_ledger
New migration 0021 adds 7 indexes for columns used in WHERE clauses
across admin list routes and the worker refill handler:
- lead_requests(lead_type) — for all lead-type filters
- lead_requests(lead_type, status) — compound filter in lead queries
- lead_requests(lead_type, verified_at) — refill eligibility queries
- lead_requests(country) — country filter in lead results
- suppliers(tier) — tier filter in supplier admin list
- suppliers(claimed_by) — claimed/unclaimed filter
- credit_ledger(supplier_id) — SUM(delta) balance aggregation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:32:15 +01:00
Deeman
8a6fd61432 fix: bound unbounded operations — LIMIT on scenarios, timeouts on DuckDB and Resend
- admin/routes.py: add LIMIT 500 to scenarios() — was unbounded, could return
  arbitrarily large result sets and exhaust memory
- analytics.py: wrap asyncio.to_thread(DuckDB) in asyncio.wait_for with
  _QUERY_TIMEOUT_SECONDS=30 so a slow scan cannot permanently starve the
  asyncio thread pool
- core.py: replace resend.default_http_client with RequestsClient(timeout=10)
  so all Resend API calls are capped at 10 s (default was 30 s)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:31:34 +01:00
Deeman
04f59e9015 feat(pseo): add pseo_routes.py blueprint + register in app
New blueprint at /admin/pseo with:
- GET /admin/pseo/          → dashboard (stats, freshness, recent jobs)
- GET /admin/pseo/health    → HTMX partial: health issue lists
- GET /admin/pseo/gaps/<slug>             → HTMX partial: content gaps
- POST /admin/pseo/gaps/<slug>/generate  → enqueue gap-fill job
- GET /admin/pseo/jobs                   → full jobs list
- GET /admin/pseo/jobs/<id>/status       → HTMX polled progress bar

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 19:30:18 +01:00
Deeman
dc38972d68 fix: replace silent exception handlers with explicit error logging
Every bare `except Exception: pass` or `except Exception: return sentinel`
now logs via logger.exception() or logger.warning() so errors surface in
the application log instead of disappearing silently.

Changes per file:
- admin/routes.py: add logger; log in _inject_admin_sidebar_data(),
  email_detail() Resend enrichment, audiences() contact count loop,
  audience_contacts() Resend fetch
- core.py: log in _get_or_create_resend_audience(), capture_waitlist_email()
  DB insert, and capture_waitlist_email() Resend contact sync (warning level
  since that path is documented as non-critical)
- analytics.py: log DuckDB query failures before returning []

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:29:59 +01:00
Deeman
3dc7a7fc02 fix: add unbuffered python output in dev runner, cursor pattern in analytics
- dev_run.sh: add -u flag so log output is not buffered (real-time visibility)
- analytics.py: use explicit cursor() with try/finally close instead of
  calling execute() directly on the connection (thread-safe cursor lifecycle)
- .sops.yaml: add second age public key for local dev decryption access
- content/__init__.py: whitespace-only formatting fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:29:51 +01:00
Deeman
9cc853d38e feat(pseo): add generation progress tracking to tasks table
- Migration 0021: add progress_current, progress_total columns to tasks
- generate_articles(): accept task_id param, write progress every 50
  articles and once at completion via db_execute()
- worker.py handle_generate_articles: inject _task_id from process_task(),
  pass to generate_articles() so the pSEO dashboard can poll live progress

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 19:28:07 +01:00
Deeman
567100076f feat(pseo): add content/health.py — gap detection, freshness, health checks
New module with pure async query functions for the pSEO Engine dashboard:
- get_template_stats() — article counts by status/language per template
- get_template_freshness() — compare _serving_meta.json vs last article gen
- get_content_gaps() — DuckDB rows with no matching article per language
- check_hreflang_orphans() — published articles missing a sibling language
- check_missing_build_files() — published articles with no HTML on disk
- check_broken_scenario_refs() — articles referencing non-existent scenarios
- get_all_health_issues() — runs all checks, returns counts + detail lists

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 18:21:34 +01:00
Deeman
3a169d9443 feat: dual market score system — Marktreife + Marktpotenzial scores
Splits the single market score into two branded scores backed by a new
global data pipeline covering all GeoNames locations (pop ≥1K):

Data pipeline:
- GeoNames expanded: cities1000 (~140K locations) vs old cities15000
  (~24K). Added lat/lon/admin1/admin2. Feature codes include PPLA3/4/5.
- Tennis court Overpass extractor (extract-overpass-tennis → stg_tennis_courts)
- foundation.dim_locations: new conformed dim seeded from GeoNames,
  enriched with nearest_padel_court_km (ST_Distance_Sphere), padel venue
  count within 5km, tennis courts within 25km
- DuckDB spatial extension enabled (extensions: [spatial] in config.yaml)
- GEONAMES_USERNAME + CENSUS_API_KEY added to .env.dev.sops + .env.prod.sops

Scoring models:
- city_market_profile.sql (Marktreife-Score): adds x0.85 saturation
  discount when venues_per_100k > 8
- location_opportunity_profile.sql (Marktpotenzial-Score): new model,
  no filter on padel_venue_count, rewards supply gaps + catchment gaps

Methodology page:
- market_score.html: Two Scores intro, 5 Marktpotenzial component cards,
  score bands for both scores, FAQ 5-7, padelnomics wordmark spans on h2s
- en.json + de.json: 30+ new keys, native German (no calques), TM on chips

Docs: CHANGELOG, data-sources-inventory, SQLMesh CLAUDE.md, PROJECT.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 17:17:02 +01:00
Deeman
165eaf48bf fix(admin): live search not firing on text input + spinner always visible
hx-trigger bug:
  "from:find input" in hx-trigger attaches the event listener to the
  first <input> found in the form — which is the hidden CSRF token input.
  Typing in the visible search field never fires the listener on that
  element. Result: only Enter (form submit) triggered HTMX.
  Fix: drop "from:find input" so the listener is on the form itself,
  where input/change events from all children bubble naturally.

Spinner visibility bug:
  .search-spinner { opacity: 0 } relied on our compiled output.css.
  HTMX ships its own built-in CSS for .htmx-indicator (opacity:0 →
  opacity:1 on htmx-request). Using class="htmx-indicator search-spinner"
  delegates hide/show to HTMX's own stylesheet with no dependency on
  whether output.css has been rebuilt. Our .search-spinner only handles
  positioning and the spin animation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:09:49 +01:00
Deeman
caec0c4410 feat(ui): apply wordmark span to score h2 headings, add TM to chips
- 4 section h2 headings now render "padelnomics" in Bricolage Grotesque
  bold (same styled span as h1), matching the existing "padelnomics
  Market Score" wordmark pattern
- i18n h2 keys now contain only the suffix (e.g. "Marktreife-Score:
  What It Measures") since "padelnomics" is hardcoded in template
- Chip labels (primary score identification) get ™ suffix in both EN + DE

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 17:08:20 +01:00
Deeman
6d52a122e5 fix(i18n): apply padelnomics wordmark consistently to score names
Score names always appear as "padelnomics Marktreife-Score" and
"padelnomics Marktpotenzial-Score" in headings, chips, intro paragraphs,
and FAQ questions/answers — in both EN and DE locales.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 17:04:57 +01:00
Deeman
6d18c52983 merge: live search + loading indicators on all admin filter forms 2026-02-24 16:59:03 +01:00