Commit Graph

11 Commits

Author SHA1 Message Date
Deeman
fda7da7d59 chore: replace hypercorn with granian (Rust ASGI server)
Granian is ~3-5x faster than Hypercorn in benchmarks. No code changes
needed — Quart is standard ASGI so any ASGI server works.

- web/pyproject.toml: hypercorn → granian>=1.6.0 (installed: 2.7.1)
- Dockerfile CMD: hypercorn → granian --interface asgi
- core.py setup_logging(): silence granian loggers instead of hypercorn's

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 21:26:19 +01:00
Deeman
84229e50f7 Merge branch 'worktree-supervisor-flags'
Python supervisor + DB-backed feature flags

- supervisor.py replaces supervisor.sh (topological wave scheduling, croniter)
- workflows.toml workflow registry (5 extractors, cron presets, depends_on)
- proxy.py round-robin + sticky proxy rotation via PROXY_URLS
- Feature flags: migration 0019, is_flag_enabled(), feature_gate() decorator
- Admin /admin/flags UI with toggle (admin-only)
- lead_unlock gate on unlock_lead route
- 59 new tests (test_supervisor.py + test_feature_flags.py)
- Fix is_flag_enabled bug (fetch_one instead of execute_fetchone)

# Conflicts:
#	CHANGELOG.md
#	web/pyproject.toml
2026-02-23 15:29:43 +01:00
Deeman
024feeaac4 feat: SEO/GEO admin hub — GSC, Bing, Umami sync + search/funnel/scorecard views
# Conflicts:
#	CHANGELOG.md
#	uv.lock
#	web/src/padelnomics/admin/templates/admin/base_admin.html
#	web/src/padelnomics/core.py
2026-02-23 15:23:03 +01:00
Deeman
4bdccb65e9 test: add 41 tests for SEO/GEO hub — sync, queries, admin routes
Covers all query functions (search perf, funnel, scorecard),
sync functions (umami with mocked httpx, bing/gsc skip tests),
admin route rendering, CSRF-protected sync POST, and boundary
validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 15:08:13 +01:00
Deeman
a1faddbed6 feat: Python supervisor + feature flags
Supervisor (replaces supervisor.sh):
- supervisor.py — cron-based pipeline orchestration, reads workflows.toml
  on every tick, runs due extractors in topological waves with parallel
  execution, then SQLMesh transform + serving export
- workflows.toml — workflow registry: overpass (monthly), eurostat (monthly),
  playtomic_tenants (weekly), playtomic_availability (daily),
  playtomic_recheck (hourly 6–23)
- padelnomics-supervisor.service — updated ExecStart to Python supervisor

Extraction enhancements:
- proxy.py — optional round-robin/sticky proxy rotation via PROXY_URLS env
- playtomic_availability.py — parallel fetch (EXTRACT_WORKERS), recheck mode
  (main_recheck) re-queries imminent slots for accurate occupancy measurement
- _shared.py — realistic browser User-Agent on all extractor sessions
- stg_playtomic_availability.sql — reads morning + recheck snapshots, tags each
- fct_daily_availability.sql — prefers recheck over morning for same slot

Feature flags (replaces WAITLIST_MODE env var):
- migration 0019 — feature_flags table, 5 initial flags:
  markets (on), payments/planner_export/supplier_signup/lead_unlock (off)
- core.py — is_flag_enabled() + feature_gate() decorator
- routes — payments, markets, planner_export, supplier_signup, lead_unlock gated
- admin flags UI — /admin/flags toggle page + nav link
- app.py — flag() injected as Jinja2 global

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:53:45 +01:00
Deeman
1a6eae20d5 feat: pSEO CMS — SSG architecture with git templates + DuckDB
# Conflicts:
#	web/pyproject.toml
2026-02-23 12:51:30 +01:00
Deeman
f1181342ad feat: SSG-inspired pSEO CMS — git templates + DuckDB direct reads
Replace the old CSV-upload-based CMS with an SSG architecture where
templates live in git as .md.jinja files with YAML frontmatter and
data comes directly from DuckDB serving tables. Only articles and
published_scenarios remain in SQLite for routing/state.

- Content module: discover, load, generate, preview functions
- Migration 0018: drop article_templates + template_data, recreate
  articles + published_scenarios without FK references, add
  template_slug/language/date_modified/seo_head columns
- Admin routes: read-only template views with generate/regenerate/preview
- SEO pipeline: canonical URLs, hreflang (EN+DE), JSON-LD (Article,
  FAQPage, BreadcrumbList), Open Graph tags baked at generation time
- Example template: city-cost-de.md.jinja for German city market data

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 12:25:44 +01:00
Deeman
7737b79230 fix: DuckDB compat issues in Playtomic pipeline + export_serving
- Add maximum_object_size=128MB to read_json for 14K-venue tenants file
- Rewrite opening_hours to use UNION ALL unpivot (DuckDB struct dynamic access)
- Add seed file guard for availability model (empty result on first run)
- Fix snapshot_date VARCHAR→DATE comparison in venue_pricing_benchmarks
- Fix export_serving to resolve SQLMesh physical tables from view definitions
  (SQLMesh views reference "local" catalog unavailable outside its context)
- Add pyarrow dependency for Arrow-based cross-connection data transfer

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:27:51 +01:00
Deeman
53e9bbd66b feat: restructure extraction to one file per source
Split monolithic execute.py into per-source modules with separate CLI
entry points. Each extractor now uses the framework from utils.py:
- SQLite state tracking (start_run / end_run per extractor)
- Proper logging (replace print() with logger)
- Atomic gzip writes (write_gzip_atomic)
- Connection pooling (niquests.Session)
- Bounded pagination (MAX_PAGES_PER_BBOX = 500)

New entry points:
  extract              — run all 4 extractors sequentially
  extract-overpass     — OSM padel courts
  extract-eurostat     — city demographics (etag dedup)
  extract-playtomic-tenants      — venue listings
  extract-playtomic-availability — booking slots + pricing (NEW)

The availability extractor reads tenant IDs from the latest tenants.json.gz,
queries next-day slots for each venue, and stores daily consolidated snapshots.
Supports resumability via cursor and retry with backoff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 18:56:41 +01:00
Deeman
044dfd836b fix(deps): add duckdb to padelnomics production dependencies
analytics.py imports duckdb at the top level. The Dockerfile runs
`uv sync --package padelnomics` which only installs padelnomics deps —
duckdb was missing, so hypercorn failed to import padelnomics.app
entirely and never bound to port 5000. The health check timed out and
the container was marked unhealthy. Tests passed because uv sync in CI
syncs all workspace members (including transform/ which has duckdb).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 13:43:34 +01:00
Deeman
4ae00b35d1 refactor: flatten padelnomics/padelnomics/ → repo root
git mv all tracked files from the nested padelnomics/ workspace
directory to the git repo root. Merged .gitignore files.
No code changes — pure path rename.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-22 00:44:40 +01:00