Adds a two-tier proxy system for the Playtomic availability extractor:
- Primary tier (PROXY_URLS): datacenter proxies, cheap and fast
- Fallback tier (PROXY_URLS_FALLBACK): residential rotating gateway, reliable
Circuit breaker opens after CIRCUIT_BREAKER_THRESHOLD (default: 10) consecutive
failures, permanently switching to the fallback tier for the rest of the run.
No auto-recovery — avoids flapping. If circuit opens with no fallback configured,
logs an error and writes partial results rather than continuing on a dead proxy pool.
Parallel mode submits futures in PARALLEL_BATCH_SIZE=100 batches so the circuit
breaker can stop new submissions after it opens.
New env vars added to .env.dev.sops (blank defaults):
PROXY_URLS_FALLBACK — residential/rotating gateway URL
CIRCUIT_BREAKER_THRESHOLD — consecutive failures before switching (default 10)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DuckDB spawns non-daemon background threads on import. Since
analytics.py was imported at module level (transitively by the
test suite), these threads kept the pytest process alive after
all tests completed. Moving the import into open_analytics_db()
means duckdb is only loaded when actually needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pytest completes all tests but process never exits in GitLab CI.
Disabling the faulthandler plugin fixes this.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The web containers had no access to the DuckDB serving layer —
analytics queries silently returned empty. Bind-mount the host
file read-only and set SERVING_DUCKDB_PATH in all app/worker/scheduler
services (both blue and green slots).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CI now creates v<pipeline_iid> tag after tests pass on master
- Supervisor fetches tags and only deploys when a newer tag is available;
skips if already on latest or no tags exist
- Fix test_seeds_markets_enabled: markets is seeded disabled (enabled=0),
test was asserting the wrong value
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three bugs causing 0 articles after generation:
1. Worker never called open_analytics_db() — fetch_analytics always
returned [] since _conn was None. Generation completed silently with
0 rows from DuckDB.
2. generate_articles upserted by url_path alone — after the URL prefix
fix, all languages share the same url_path (e.g. /markets/de/berlin).
The EN article was overwriting the DE article on every generation.
Fixed: deduplicate on (url_path, language).
3. article_page and _filter_articles didn't filter by language — with
shared url_paths a DE request could serve an EN article, and the
markets hub would show duplicate entries per city. Fixed: both now
filter by g.lang from the /<lang> blueprint prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- setup_server.sh now requires root, creates padelnomics_service user,
adds to docker group, generates deploy key in service user's home,
owns /opt/padelnomics and /data/padelnomics to service user
- supervisor service: User=padelnomics_service, updated PATH
- landing-backup service: User=padelnomics_service
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Unescaped HTML attribute quotes inside YAML double-quoted scalars broke
YAML parsing, causing discover_templates() to silently skip both
templates and generate nothing.
Fixed by replacing \" with \" in the span style attributes within the
YAML frontmatter. The wordmark span is preserved; YAML now parses
correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated city-cost-de and country-overview templates
- All user-facing "Market Score" mentions now use full branded name
- "padelnomics" rendered in Bricolage Grotesque 800 wordmark style
- Stat strip labels kept short (space-constrained)
- SQL column names unchanged (market_score stays as-is)
- Also seeds markets feature flag as disabled (0) by default
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pSEO template improvements:
- Fix double language prefix bug in article URLs
- Add German language to all 3 templates (city-cost-de, city-pricing, country-overview)
- Expand English content depth (~1500 words per template)
- Fix country-overview schema_type to [Article, FAQPage]
- Add cross-template links, scenario cross-references, extra FAQs, second CTAs
Subtask 5: Bilingual country-overview template with schema_type fix.
Schema fix:
- Changed schema_type from [Article] to [Article, FAQPage] — enables
FAQ rich results for existing FAQ content.
German variant:
- Full German prose: "Padel in {{ country_name_en }} — Marktüberblick"
- German stats-strip labels (Anlagen gesamt, Erfasste Städte, Ø Market Score,
Median Spitzenpreis)
- German market landscape analysis, pricing overview, FAQ (5 questions)
- German CTAs: "Zum Finanzplaner" throughout
- Peak/off-peak spread commentary for German readers
English expansion (~500 → ~1300 words):
- New "Market Landscape" section with market maturity analysis
(conditional on avg_market_score ranges)
- Expanded "Top Cities" with ranking methodology context
- Expanded "Pricing Overview" with peak/off-peak spread commentary
and utilization management implications
- Expanded "Build Your Business Plan" with more persuasive copy
- 2 new FAQ questions: "How fast is padel growing?" and "Which cities
have the best pricing data?"
- Planner links in FAQ answers + second CTA at bottom
- Language-prefixed internal links: /{{ language }}/markets/...
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Subtask 4: Bilingual city-pricing template with language conditionals.
German variant:
- Full German prose: "Padel-Court-Preise in {{ city_name }}"
- German stats-strip labels (Hauptzeit, Nebenzeit, Preisspanne, Auslastung)
- German pricing tables, comparison section, pricing drivers
- 5 German FAQ questions with planner links
- German CTAs throughout
English expansion (~400 → ~1500 words):
- New "Pricing Trends" section with time-of-day analysis
- Expanded city comparison with venue density benchmarks
- Expanded pricing drivers (facility quality, market maturity)
- Investment Outlook section with scenario cross-reference
[scenario:city-cost-de-{{ city_key }}:operating]
- 2 new FAQ questions (city comparison + price trends)
- Cross-links to city-cost-de and country-overview articles
- Second CTA at bottom + planner links in FAQ answers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Full German prose variant (natural Du/Dein, not literal translation)
- German stats-strip labels, table headers, FAQ, CTAs
- Expanded English content (~1500 words): analytical commentary after
scenario markers, Market Context section, venue density analysis
- Cross-link to city-pricing page in both languages
- Planner links in FAQ answers (both languages)
- Second CTA at bottom of each language block
- Language-conditional title_pattern and meta_description_pattern
Subtask 2+3 of pSEO template improvements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
generate_articles() was storing url_path with lang prefix (/en/markets/...)
but the content blueprint is registered at /<lang>, producing double-prefix
URLs like /en/en/markets/italy. Fix: store url_path without prefix, build
full_url with prefix for SEO tags (canonical, OG, hreflang, breadcrumbs).
Also removes /markets from RESERVED_PREFIXES since article sub-paths under
/markets/ are valid pSEO content URLs, not blueprint routes.
Subtask 1 of pSEO template improvements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
File has evolved from exploratory research into operational documentation
(pipeline status tracker, extractor refs, staging model grains, impl notes).
Aligns with other architectural docs in docs/ (USER_FLOWS.md, etc.).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace Priority Summary Table with Pipeline Status Tracker: status
(✅/🔲/⏸/—), score (1-5), credential requirements, and extractor refs
for all 30+ sources
- Add implementation notes to §1.1 (Overpass), §1.2 (Playtomic tenants +
availability), §5.1 (Eurostat urb_cpop1 + ilc_di03), §5.2 (Census), §5.3 (ONS)
- Update §8 DuckDB integration table with extractor names and status
- Add §10 FX / Currency Rates: ECB SDMX endpoint and Frankfurter.app wrapper,
proposed landing format and stg_fx_rates staging model design
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- bake_scenario_cards() accepts scenario_overrides dict for preview mode
(bypasses DB lookup when no published_scenario exists)
- preview_article() builds in-memory scenario dict and passes it through
- Fix double-encoded & in locale strings (was &amp; in rendered HTML)
- Fix ruff import sort in _datetimeformat
- Fix migration 0019 minor issue
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Publish/Unpublish returns updated <tr> partial via HTMX
- Delete returns empty string to remove row without page reload
- Extract article_row.html partial (used by both results table and
individual HTMX responses)
- article_results.html now includes article_row.html via loop
Subtask 7 of CMS admin improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- generate_articles() now writes body_md alongside body_html
to BUILD_DIR/{lang}/md/{slug}.md
- article_edit GET checks both manual and generated markdown paths
- Fix pre-existing ruff import sort in _datetimeformat
Subtask 5 of CMS admin improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add invalidate_sitemap_cache() to sitemap.py and call it from
article_publish, article_delete, and article_new admin routes.
Subtask 6 of CMS admin improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _get_article_list() with filters: status, template, language, search
- Add _get_article_stats() for header stats strip (total/live/scheduled/draft)
- Add /articles/results HTMX partial endpoint
- Add filter bar: search input + status/template/language dropdowns
- Paginate at 50 articles per page
- Add "View" link to live articles (opens public URL in new tab)
- Remove URL column (redundant), add Language column
Subtasks 2+3 of CMS admin improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Compute display_status (live/scheduled/draft) in SQL instead of broken
Jinja string comparison against undefined `now` variable
- Replace template_data_id (dropped in migration 0018) with template_slug
Subtask 1/8 of CMS admin improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add section 9 to data-sources-inventory.md covering live API quirks:
Eurostat SDMX city labels response shape, ONS CSV download path (observations
API 404s), US Census ACS place endpoint, GeoNames cities15000 bulk format
- Add population coverage summary table and DuckDB glob limitation note
- fix(extract): census_usa + geonames write empty placeholder when credentials
absent so SQLMesh staging models don't fail with "no files found"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eurostat_city_labels: API returns compact dimension JSON (category.label dict),
not SDMX 2.1 nested codelists structure. Fixed parser to read from
data["category"]["label"]. 1771 city codes fetched successfully.
ons_uk: observations endpoint (TS007A) is 404. Switched to CSV download via
/datasets/mid-year-pop-est/editions/mid-2022-england-wales — fetches ~68MB CSV,
filters to sex='all' + target year, aggregates population per LAD. 316 LADs ≥50K.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DuckDB rows can have NULL columns (e.g. market_score, median_peak_rate).
Replace None with 0 in render context so numeric Jinja2 filters like
round() and int don't crash with "NoneType doesn't define __round__".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add _datetimeformat Jinja2 filter to _render_pattern() — templates
use {{ 'now' | datetimeformat('%Y') }} but the filter was never
registered, causing "No filter named 'datetimeformat'" on preview.
- Move Preview button to first column in template detail data table
so it's visible without scrolling on wide tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Part B: Calculator improvements
Fix 1 — annualRevGrowth was dead code. Now applied as compound multiplier
from Year 2 onwards across all revenue streams (court, ancillary, membership,
F&B, coaching, retail).
Fix 2 — IRR initial outflow bug (HIGH). Was using -capex but NCFs are
post-debt-service (levered). Using capex as denominator while using levered
cash flows understates equity returns by the leverage ratio. Fix: use -equity
as outflow for equity IRR, add separate projectIrr (unlevered, uses -capex
with EBITDA flows).
Fix 3 — NPV at hurdle rate. Discounts equity NCFs and exit proceeds at
hurdleRate (default 12%). Reports npv and npvPositive. NPV > 0 iff equity
IRR > hurdle rate. Added hurdleRate slider (5–35%) to Exit settings.
Fix 4 — Remaining loan: replaces heuristic with correct amortization math
(PV of remaining payments: monthlyPayment × (1 - (1+r)^-(n-k)) / r).
Fix 5 — Exit EBITDA: uses terminal year EBITDA (holdYears - 1) instead of
hardcoded Year 3. exitValue now reflects actual exit year, not always Y3.
Fix 6 — MOIC: moic is now equity MOIC (total equity CFs / equity invested).
projectMoic is the project-level multiple. Waterfall updated to show both.
Fix 7 — Return decomposition / value bridge. Standard PE attribution:
EBITDA growth value (operational alpha) + debt paydown (financial leverage).
Displayed in tab_returns.html as an attribution table.
Fix 8 — OPEX growth rate. annualOpexGrowth (default 2%) inflates utilities,
staff, insurance from Year 2 onwards. Without this Y4-Y5 EBITDA was
systematically overstated. Added annualOpexGrowth slider to Exit settings.
Fix 9 — LTV and DSCR warnings. ltvWarning (>75%) and dscrWarning (<1.25x)
with inline warnings in tab_metrics.html.
Fix 10 — Interest-only period. interestOnlyMonths (0–24) reduces early NCF
drag. Added slider to Financing section.
Updated test: test_stab_ebitda_is_year3 → test_stab_ebitda_is_exit_year.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Part A: Data Layer — Sprints 1-5
Sprint 1 — Eurostat SDMX city labels (unblocks EU population):
- New extractor: eurostat_city_labels.py — fetches ESTAT/CITIES codelist
(city_code → city_name mapping) with ETag dedup
- New staging model: stg_city_labels.sql — grain city_code
- Updated dim_cities.sql — joins Eurostat population via city code lookup;
replaces hardcoded 0::BIGINT population
Sprint 2 — Market score formula v2:
- city_market_profile.sql: 30pt population (LN/1M), 25pt income PPS (/200),
30pt demand (occupancy or density), 15pt data confidence
- Moved venue_pricing_benchmarks join into base CTE so median_occupancy_rate
is available to the scoring formula
Sprint 3 — US Census ACS extractor:
- New extractor: census_usa.py — ACS 5-year place population (vintage 2023)
- New staging model: stg_population_usa.sql — grain (place_fips, ref_year)
Sprint 4 — ONS UK extractor:
- New extractor: ons_uk.py — 2021 Census LAD population via ONS beta API
- New staging model: stg_population_uk.sql — grain (lad_code, ref_year)
Sprint 5 — GeoNames global extractor:
- New extractor: geonames.py — cities15000.zip bulk download, filtered to ≥50K pop
- New staging model: stg_population_geonames.sql — grain geoname_id
- dim_cities: 5-source population cascade (Eurostat > Census > ONS > GeoNames > 0)
with case/whitespace-insensitive city name matching
Registered all 4 new CLI entrypoints in pyproject.toml and all.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously the admin role was only granted during dev-login via
ensure_admin_role(), but dev_run.sh resets the DB on each start,
so the role was never present when first visiting /admin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The multi-line python3 -c heredoc in the Makefile caused
"missing separator" errors since Make runs each recipe line
in a separate shell. Moved to web/scripts/init_landing_seeds.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NTFY_TOKEN support to send_alert() — sends Authorization header
when token is set, backwards-compatible with plain webhook URLs
- Set ALERT_WEBHOOK_URL and NTFY_TOKEN in .env.prod.sops
- Add NTFY_TOKEN= placeholder in .env.dev.sops
- Topic: gWMeiHxj8ZqLbbqT (hard-to-guess, token-gated)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three deviations from the quart_saas_boilerplate methodology corrected:
1. Fix dim_cities LIKE join (data quality bug)
- Old: FROM eurostat_cities LEFT JOIN venue_counts LIKE '%country_code%'
→ cartesian product (2.6M rows vs ~5500 expected)
- New: FROM venue_cities (dim_venues) as primary table, Eurostat for
enrichment only. grain (country_code, city_slug).
- Also fixes REGEXP_REPLACE to LOWER() before regex so uppercase city
names aren't stripped to '-'
2. Rename fct_venue_capacity → dim_venue_capacity
- Static venue attributes with no time key are a dimension, not a fact
- No SQL logic changes; update fct_daily_availability reference
3. Add fct_availability_slot at event grain
- New: grain (snapshot_date, tenant_id, resource_id, slot_start_time)
- Recheck dedup logic moves here from fct_daily_availability
- fct_daily_availability now reads fct_availability_slot (cleaner DAG)
Downstream fixes:
- city_market_profile, planner_defaults grain → (country_code, city_slug)
- pseo_city_costs_de, pseo_city_pricing add city_key composite natural key
(country_slug || '-' || city_slug) to avoid URL collisions across countries
- planner_defaults join in pseo_city_costs_de uses both country_code + city_slug
- Templates updated: natural_key city_slug → city_key
Added transform/sqlmesh_padelnomics/CLAUDE.md documenting data modeling rules,
conformed dimension map, and source integration architecture.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add 18 new E2E tests from master: pricing, checkout, supplier signup,
supplier dashboard, and business plan export (sections J-N)
- Force WAITLIST_MODE=false in visual server subprocess — the root .env
sets WAITLIST_MODE=true, and since Config class attributes evaluate at
import time (before fork), the subprocess inherits the parent's value.
Patching both os.environ and core.config directly ensures feature pages
render instead of waitlist templates.
- All 77 visual tests now pass in ~59 seconds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
data/ is gitignored so seed files are not tracked. The recheck glob in
stg_playtomic_availability requires at least one matching file or DuckDB
throws IOException. Run 'make init-landing-seeds' after a fresh clone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
stg_playtomic_availability:
- Add maximum_object_size = 134217728 (128 MB) to both read_json calls;
daily files exceed the 16 MB default as venue count grows
- Add seed recheck file (1970-01-01_recheck_00.json.gz, gitignored with data/)
to avoid READ_JSON IOException when no recheck files exist
pseo_city_costs_de + pseo_city_pricing:
- Add QUALIFY ROW_NUMBER() OVER (PARTITION BY city_slug ...) = 1 to
deduplicate rows caused by dim_cities' loose LIKE join; reduces
pseo_city_costs_de from 2.6M → 222 rows (one per unique city)
content/__init__.py:
- DuckDB lowercases all column names at rest ("ratePeak" → "ratepeak"),
so calc_overrides dict comprehension never matched DEFAULTS keys.
Fix: build case-insensitive reverse map {k.lower(): k} and normalise
row keys before lookup. Applied in both generate_articles() and
preview_article().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
