Add real per-country cost data to ~30 calculator fields so pSEO articles
show country-specific CAPEX/OPEX instead of hardcoded DE defaults.
Extractor:
- eurostat.py: add 8 new datasets (nrg_pc_205, nrg_pc_203, lc_lci_lev,
5×prc_ppp_ind variants); add optional `dataset_code` field so multiple
dict entries can share one Eurostat API endpoint
Staging (4 new models):
- stg_electricity_prices — EUR/kWh by country, semi-annual
- stg_gas_prices — EUR/GJ by country, semi-annual
- stg_labour_costs — EUR/hour by country, annual (future staffed scenario)
- stg_price_levels — PLI indices (EU27=100) for 5 categories, annual
Foundation:
- dim_countries (new) — conformed country dimension; eliminates ~50-line CASE
blocks duplicated in dim_cities/dim_locations; computes ~29 calculator cost
override columns from PLI ratios and energy price ratios vs DE baseline;
NULL for DE so calculator falls through to DEFAULTS unchanged
- dim_cities — replace country_name/slug CASE blocks + country_income CTE
with JOIN dim_countries
- dim_locations — same refactor as dim_cities
Serving:
- pseo_city_costs_de — JOIN dim_countries; add 29 camelCase override columns
auto-applied by calculator (electricity, heating, rentSqm, hallCostSqm, …)
- planner_defaults — JOIN dim_countries; same 29 cost columns flow through
to /api/market-data endpoint
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add proxy_failure_limit param to make_tiered_cycler (default 3).
Individual proxies hitting the limit are marked dead and permanently
skipped. next_proxy() auto-escalates when all proxies in the active
tier are dead. Both mechanisms coexist: per-proxy dead tracking removes
broken individuals; tier-level threshold catches systemic failure.
- proxy.py: dead_proxies set + proxy_failure_counts dict in state;
next_proxy skips dead proxies with bounded loop; record_failure/
record_success accept optional proxy_url; dead_proxy_count() added
- playtomic_tenants.py: pass proxy_url to record_success/record_failure
- playtomic_availability.py: _worker returns (proxy_url, result);
serial loops in extract + extract_recheck capture proxy_url
- test_supervisor.py: 11 new tests in TestTieredCyclerDeadProxyTracking
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docker-compose.prod.yml: fix volume mount for all 6 web containers
from /opt/padelnomics/data (stale) → /data/padelnomics (live supervisor output);
add LANDING_DIR=/app/data/pipeline/landing so extraction/landing stats work
- pipeline_routes.py: fix _REPO_ROOT parents[5] → parents[4] so workflows.toml
is found in dev and pipeline overview shows workflow schedules
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 2a: NUTS-1 regional income for Germany (16 Bundesländer via admin1→NUTS-1 mapping)
Phase 2b: EU-wide NUTS-2 via GISCO spatial join + US Census ACS state income
- All EU-27+EFTA+UK locations now auto-resolve to NUTS-2 via ST_Contains
- Germany gets sub-Bundesland (38 Regierungsbezirke) differentiation
- US gets state-level income with PPS normalisation
- Income cascade: NUTS-2 → NUTS-1 → US state → country-level
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 0 — income ceiling fix (opportunity_score):
PPS normalisation /200→/35000; economic power now differentiates
countries (DE 13.2, ES 10.7, SE 14.3 pts; was 20.0 everywhere)
Phase 1b — overpass_tennis in workflows.toml:
Monthly schedule added; was only in combined extractor
Phase 2b — dim_cities spatial population fallback:
GeoNames spatial CTE (ST_Distance_Sphere, 0.14° bbox) resolves
localization mismatches: Wien→1.69M, Milano→1.37M, München→1.49M
Coverage: 70.5% → 98.5% (5,401/5,481 cities with population)
Records all Phase 1 deliverables: content gaps, data freshness,
health checks, generation job monitoring, 45 tests, bug fixes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces 94 occurrences of deprecated datetime.utcnow() and
datetime.utcfromtimestamp() across 22 files with utcnow()/utcnow_iso()
helpers. Zero DeprecationWarnings remain. All 1201 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Landing files (append-only JSON.gz) synced to R2 every 30 min via
systemd timer + rclone. Extraction state DB (.state.sqlite) continuously
replicated via Litestream (second DB entry). Auto-restore on container
startup for both app.db and .state.sqlite. Reuses existing R2 bucket
and credentials — no new env vars needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add full email management at /admin/emails with:
- email_log table tracking all outgoing emails with resend_id + delivery events
- inbound_emails table for Resend webhook-received messages
- Resend webhook handler (/webhooks/resend) updating delivery status in real-time
- send_email() returns resend_id (str|None) instead of bool; all 9 worker
handlers pass email_type= for per-type filtering
- Admin UI: sent log with HTMX filters, email detail with API-enriched HTML
preview, inbox with unread badges + reply, compose with branded wrapping,
audience management with contact list/remove
- Sidebar Email section with unread badge via blueprint context processor
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the old CSV-upload-based CMS with an SSG architecture where
templates live in git as .md.jinja files with YAML frontmatter and
data comes directly from DuckDB serving tables. Only articles and
published_scenarios remain in SQLite for routing/state.
- Content module: discover, load, generate, preview functions
- Migration 0018: drop article_templates + template_data, recreate
articles + published_scenarios without FK references, add
template_slug/language/date_modified/seo_head columns
- Admin routes: read-only template views with generate/regenerate/preview
- SEO pipeline: canonical URLs, hreflang (EN+DE), JSON-LD (Article,
FAQPage, BreadcrumbList), Open Graph tags baked at generation time
- Example template: city-cost-de.md.jinja for German city market data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redesigned _email_wrap(): lowercase wordmark header matching website,
3px blue accent border, preheader text support, HR separators.
_email_button() now full-width block for mobile tap targets.
Rewrote copy: improved subject lines, urgency cues, quick-start links
in welcome, styled project recap in quote verify, heat badges on lead
forward, "what happens next" in lead matched, secondary CTAs.
~30 new/updated translation keys in both EN and DE.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
