- Migration 0021: add progress_current, progress_total columns to tasks
- generate_articles(): accept task_id param, write progress every 50
articles and once at completion via db_execute()
- worker.py handle_generate_articles: inject _task_id from process_task(),
pass to generate_articles() so the pSEO dashboard can poll live progress
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New module with pure async query functions for the pSEO Engine dashboard:
- get_template_stats() — article counts by status/language per template
- get_template_freshness() — compare _serving_meta.json vs last article gen
- get_content_gaps() — DuckDB rows with no matching article per language
- check_hreflang_orphans() — published articles missing a sibling language
- check_missing_build_files() — published articles with no HTML on disk
- check_broken_scenario_refs() — articles referencing non-existent scenarios
- get_all_health_issues() — runs all checks, returns counts + detail lists
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Records exported_at_utc timestamp and per-table row counts immediately
after export_serving.py completes. The pSEO Engine dashboard reads this
file to show data freshness without querying file mtimes.
Also moves the inline `import re` to the top-level imports.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Splits the single market score into two branded scores backed by a new
global data pipeline covering all GeoNames locations (pop ≥1K):
Data pipeline:
- GeoNames expanded: cities1000 (~140K locations) vs old cities15000
(~24K). Added lat/lon/admin1/admin2. Feature codes include PPLA3/4/5.
- Tennis court Overpass extractor (extract-overpass-tennis → stg_tennis_courts)
- foundation.dim_locations: new conformed dim seeded from GeoNames,
enriched with nearest_padel_court_km (ST_Distance_Sphere), padel venue
count within 5km, tennis courts within 25km
- DuckDB spatial extension enabled (extensions: [spatial] in config.yaml)
- GEONAMES_USERNAME + CENSUS_API_KEY added to .env.dev.sops + .env.prod.sops
Scoring models:
- city_market_profile.sql (Marktreife-Score): adds x0.85 saturation
discount when venues_per_100k > 8
- location_opportunity_profile.sql (Marktpotenzial-Score): new model,
no filter on padel_venue_count, rewards supply gaps + catchment gaps
Methodology page:
- market_score.html: Two Scores intro, 5 Marktpotenzial component cards,
score bands for both scores, FAQ 5-7, padelnomics wordmark spans on h2s
- en.json + de.json: 30+ new keys, native German (no calques), TM on chips
Docs: CHANGELOG, data-sources-inventory, SQLMesh CLAUDE.md, PROJECT.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hx-trigger bug:
"from:find input" in hx-trigger attaches the event listener to the
first <input> found in the form — which is the hidden CSRF token input.
Typing in the visible search field never fires the listener on that
element. Result: only Enter (form submit) triggered HTMX.
Fix: drop "from:find input" so the listener is on the form itself,
where input/change events from all children bubble naturally.
Spinner visibility bug:
.search-spinner { opacity: 0 } relied on our compiled output.css.
HTMX ships its own built-in CSS for .htmx-indicator (opacity:0 →
opacity:1 on htmx-request). Using class="htmx-indicator search-spinner"
delegates hide/show to HTMX's own stylesheet with no dependency on
whether output.css has been rebuilt. Our .search-spinner only handles
positioning and the spin animation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- 4 section h2 headings now render "padelnomics" in Bricolage Grotesque
bold (same styled span as h1), matching the existing "padelnomics
Market Score" wordmark pattern
- i18n h2 keys now contain only the suffix (e.g. "Marktreife-Score:
What It Measures") since "padelnomics" is hardcoded in template
- Chip labels (primary score identification) get ™ suffix in both EN + DE
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Score names always appear as "padelnomics Marktreife-Score" and
"padelnomics Marktpotenzial-Score" in headings, chips, intro paragraphs,
and FAQ questions/answers — in both EN and DE locales.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scenarios:
- Convert from plain GET form to HTMX live search (scenario_results
route already existed, just needed wiring)
- Replace Filter submit button with JS-reset Clear button
- Update is_generating banner to match article_results.html style
Users:
- Add /admin/users/results HTMX partial route
- Extract user table into partials/user_results.html with HTMX pagination
- Convert search form to live-search (input delay:300ms)
Loading indicator (all 6 forms):
- Add hx-indicator pointing to a small arc spinner SVG
- Spinner fades in while the debounce + request is in flight
- CSS .search-spinner class in input.css (opacity 0 → 1 on htmx-request,
spin-icon animation only runs when visible)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The search/country/venue-type inputs used class="input" which has no
definition in input.css — falls back to the browser's default focus
outline. Replaced with form-input to get the consistent focus ring
(ring-2 / ring-electric / border-electric) used everywhere else in admin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Spinner:
- article_results.html: replace hidden polling div with a visible
animated spinner banner; CSS spin keyframe added to input.css
Batch commits:
- generate_articles() now commits every 200 articles instead of
holding one giant transaction; articles appear in the admin UI
progressively without waiting for the full run
Performance (pre-compiled Jinja templates):
- Create one Environment + compile url/title/meta/body templates once
before the loop instead of calling _render_pattern() per iteration;
eliminates ~4 × N Environment() constructions and re-parses of the
same template strings (N = articles, typically 500+)
- Reuse url_tmpl for hreflang alt-lang rendering
Scenario override passthrough:
- Pass just-computed scenario data directly to bake_scenario_cards()
via scenario_overrides, avoiding a DB SELECT that reads an uncommitted
row from a potentially separate connection
Timing instrumentation:
- Accumulate time spent in calc / render / bake phases per run
- Log totals at completion: "done — 500 total | calc=1.2s render=4.3s bake=0.1s"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SELECT-then-INSERT/UPDATE pairs in generate_articles() with
INSERT ... ON CONFLICT DO UPDATE statements, and wrap the entire loop in
a single transaction context manager. Eliminates ~1,500 individual SQLite
commits for a 500-article run (one commit per row replaced by one total).
Also fix _get_article_stats() returning None for live/scheduled/draft counts
when the articles table is empty: wrap SUM expressions in COALESCE(..., 0)
so they always return integers regardless of row count.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
articles.country stores "CH"/"DE"/etc., not English names. Update
get_country_name() to try the input as an uppercase code first, falling
back to the reverse-name lookup for any English-name values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add GEONAMES_USERNAME=padelnomics and CENSUS_API_KEY to .env.dev.sops and .env.prod.sops
- Enable DuckDB spatial extension in SQLMesh config.yaml (ST_Distance_Sphere for distance calcs + future map features)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a two-tier proxy system for the Playtomic availability extractor:
- Primary tier (PROXY_URLS): datacenter proxies, cheap and fast
- Fallback tier (PROXY_URLS_FALLBACK): residential rotating gateway, reliable
Circuit breaker opens after CIRCUIT_BREAKER_THRESHOLD (default: 10) consecutive
failures, permanently switching to the fallback tier for the rest of the run.
No auto-recovery — avoids flapping. If circuit opens with no fallback configured,
logs an error and writes partial results rather than continuing on a dead proxy pool.
Parallel mode submits futures in PARALLEL_BATCH_SIZE=100 batches so the circuit
breaker can stop new submissions after it opens.
New env vars added to .env.dev.sops (blank defaults):
PROXY_URLS_FALLBACK — residential/rotating gateway URL
CIRCUIT_BREAKER_THRESHOLD — consecutive failures before switching (default 10)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- feat(i18n): add country name translations for article country badges
- feat(content): convert article FAQ sections to collapsible details/summary
- feat(content): rebrand stats-strip Market Score with padelnomics wordmark + color coding
- fix(i18n): improve German translation quality across 94 keys
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Systematic review of de.json: fix unnatural calques from English, inconsistent
register (Du/Sie mixing), awkward phrasing, and machine-translation artifacts.
Market Score and product names intentionally kept in English as brand names.
Du (capitalized) maintained consistently as product voice throughout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace plain "Ø Market Score" / "Market Score" / "Avg Market Score" labels with
the branded padelnomics wordmark (Bricolage Grotesque bold). Add color-coded value:
green (≥65), amber (40–64), red (<40). Applied to country-overview.md.jinja (DE+EN)
and city-cost-de.md.jinja (DE+EN). Articles need Rebuild All to regenerate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace **Bold question?** / Answer markdown pattern with HTML <details>/<summary>
in all three article templates (city-pricing, city-cost-de, country-overview),
both DE and EN sections. Add .article-body details CSS for styled accordion look.
Articles need Rebuild All to regenerate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move COUNTRY_LABELS to i18n.py (shared). Add get_country_name(country_str, lang)
that maps English DB values (e.g. "Germany") to localised names via existing
dir_country_* translation keys. Register as Jinja filter country_name.
Apply to market_results.html country badge.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
hypercorn sets its own level on child loggers directly, so silencing
the parent 'hypercorn' logger alone isn't sufficient.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Zero print() calls remain in the app and worker (scripts use
basicConfig for clean CLI output). Setup_logging() in core.py
reads LOG_LEVEL env var and configures the root logger once.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
# Conflicts:
# web/src/padelnomics/core.py
# web/src/padelnomics/worker.py
- migrations/migrate.py: module logger, basicConfig in __main__
- scripts/seed_dev_data.py: module logger, convert all 19 prints
- scripts/seed_content.py: module logger, convert all 13 prints
- scripts/refresh_from_daas.py: module logger, convert all 11 prints
- scripts/setup_paddle.py: module logger, convert all 20 prints
All scripts use basicConfig(level=INFO, format='%(levelname)-8s %(message)s')
in their __main__ blocks for clean CLI output without timestamps.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add module logger (padelnomics.worker) and scheduler_logger
(padelnomics.worker.scheduler)
- Call setup_logging() at start of run_worker() and run_scheduler()
- Convert all 26 print() calls — drop manual [WORKER]/[SCHEDULER] prefixes
- Magic link + quote verification debug prints → logger.debug() (only
shown when LOG_LEVEL=DEBUG)
- Errors with exception context use logger.error() with %s formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add logging import and setup_logging() that reads LOG_LEVEL env var
(defaults DEBUG in dev, INFO in prod), sets format with timestamp +
level + logger name, silences hypercorn/asyncio noise
- Add module-level logger to core.py
- Convert 3 [EMAIL] print() calls to logger.info / logger.error
- Call setup_logging() from app.py at import time
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces 94 occurrences of deprecated datetime.utcnow() and
datetime.utcfromtimestamp() across 22 files with utcnow()/utcnow_iso()
helpers. Zero DeprecationWarnings remain. All 1201 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Show API errors and network failures in a red inline div below the
export form instead of browser alert() dialogs. Error div is hidden
on each new submit attempt so stale messages don't linger.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
utcnow_iso() now produces 'YYYY-MM-DD HH:MM:SS' (space separator) matching
SQLite's datetime('now') so lexicographic comparisons like
'published_at <= datetime(now)' work correctly.
Also add `id DESC` tiebreaker to get_ledger() ORDER BY to preserve
insertion order when multiple credits are added within the same second.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Also fixes test_supplier_webhooks.py fromisoformat() comparisons:
expires (naive, from DB) now compared against datetime.now(UTC).replace(tzinfo=None)
to avoid mixing naive/aware datetimes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrates 15 source files from the deprecated datetime.utcnow() API.
Uses utcnow() for in-memory math and utcnow_iso() (strftime format)
for SQLite TEXT column writes to preserve lexicographic sort order.
Also fixes datetime.utcfromtimestamp() in seo/_bing.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add _is_generating() helper — queries tasks table for pending generate_articles tasks
- Pass is_generating to article_results partial (both full page and HTMX route)
- article_results.html: render invisible hx-trigger="every 3s" div when generating;
polling stops naturally once generation completes and div is absent
- Add /admin/scenarios/results HTMX partial route with same is_generating logic
- Extract scenario table into admin/partials/scenario_results.html partial
- scenarios.html: wrap table in #scenario-results div, include partial
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
