PROXY_URLS_DATACENTER was missing the scheme prefix, causing SSL
handshake failures on the Rayobyte HTTP-only proxy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Validates each URL in PROXY_URLS_DATACENTER / PROXY_URLS_RESIDENTIAL:
logs a warning and skips any entry missing an http:// or https:// scheme
instead of passing malformed URLs that cause SSL or connection errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds HTMX live polling to the Overview tab (stops when quiet) and a new
Transform tab for managing the SQLMesh + export steps of the ELT pipeline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add proxy_failure_limit param to make_tiered_cycler (default 3).
Individual proxies hitting the limit are marked dead and permanently
skipped. next_proxy() auto-escalates when all proxies in the active
tier are dead. Both mechanisms coexist: per-proxy dead tracking removes
broken individuals; tier-level threshold catches systemic failure.
- proxy.py: dead_proxies set + proxy_failure_counts dict in state;
next_proxy skips dead proxies with bounded loop; record_failure/
record_success accept optional proxy_url; dead_proxy_count() added
- playtomic_tenants.py: pass proxy_url to record_success/record_failure
- playtomic_availability.py: _worker returns (proxy_url, result);
serial loops in extract + extract_recheck capture proxy_url
- test_supervisor.py: 11 new tests in TestTieredCyclerDeadProxyTracking
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the tenants extractor flattened all proxy tiers into a single
round-robin list, bypassing the circuit breaker entirely. When the free
Webshare tier runs out of bandwidth (402), all 20 free proxies fail and
the batch crashes — the paid datacenter/residential proxies are never tried.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the tenants extractor flattened all proxy tiers into a single
round-robin list, bypassing the circuit breaker entirely. When the free
Webshare tier runs out of bandwidth (402), all 20 free proxies fail and
the batch crashes — the paid datacenter/residential proxies are never tried.
Changes:
- Replace make_round_robin_cycler with make_tiered_cycler (same as availability)
- Add _fetch_page_via_cycler: retries per page across tiers, records
success/failure in cycler so circuit breaker can escalate
- Fix batch_size to BATCH_SIZE=20 constant (was len(all_proxies) ≈ 22)
- Check cycler.is_exhausted() before each batch; catch RuntimeError mid-batch
and write partial results rather than crashing with nothing
- CIRCUIT_BREAKER_THRESHOLD from env (default 10), matching availability
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docker-compose.prod.yml: fix volume mount for all 6 web containers
from /opt/padelnomics/data (stale) → /data/padelnomics (live supervisor output);
add LANDING_DIR=/app/data/pipeline/landing so extraction/landing stats work
- pipeline_routes.py: fix _REPO_ROOT parents[5] → parents[4] so workflows.toml
is found in dev and pipeline overview shows workflow schedules
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace ABS() bbox predicates with BETWEEN in all three spatial CTEs
(nearest_padel, padel_local, tennis_nearby). BETWEEN enables DuckDB's
IEJoin (interval join) which is O((N+M) log M) vs the previous O(N×M)
nested-loop cross-join.
Add country pre-filters to restrict the left side from ~140K global
locations to ~20K rows for padel/tennis CTEs (~8 countries each).
Expected: ~50-200x speedup on the spatial CTE portion of the model.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stg_population_geonames → dim_locations → location_opportunity_profile
were all 0 rows in prod because the GeoNames extractor was never
scheduled. First run will backfill cities1000 to landing zone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Centralises retailer config in affiliate_programs table (URL template,
tracking tag, commission %). Products now use program dropdown + product
identifier instead of manual URL baking. URL assembled at redirect time
via build_affiliate_url() — changing a tag propagates to all products
instantly. Backward compatible: legacy baked-URL products fall through
unchanged. Amazon OneLink (configured in Associates dashboard) handles
geo-redirect to local marketplaces with no additional programs needed.
Also fixes _rebuild_article() frontmatter rendering bug.
Commits: fix frontmatter, migration 0027, program CRUD functions,
redirect update, admin CRUD + templates, product form update, tests.
41 tests, all passing. Ruff clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the manual affiliate URL field with a program selector and
product identifier input. JS toggles visibility between program mode and
manual (custom URL) mode. retailer field is auto-populated from the
program name on save. INSERT/UPDATE statements include new program_id
and product_identifier columns. Validation accepts program+ID or manual
URL as the URL source.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds program list, create, edit, delete routes with appropriate guards
(delete blocked if products reference the program). Adds "Programs" tab
to the affiliate subnav. New templates: affiliate_programs.html,
affiliate_program_form.html, partials/affiliate_program_results.html.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Program-based products now get URLs assembled from the template at
redirect time. Changing a program's tracking_tag propagates instantly
to all its products without rebuilding. Legacy products (no program_id)
still use their baked affiliate_url via fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds get_all_programs(), get_program(), get_program_by_slug() for admin
CRUD. Adds build_affiliate_url() that assembles URLs from program template
+ product identifier, with fallback to baked affiliate_url for legacy
products. Updates get_product() to JOIN affiliate_programs so _program
dict is available at redirect time. _parse_product() extracts program
fields into nested _program key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes a bug where manual article previews rendered raw frontmatter
(title:, slug:, etc.) as visible text. Now strips the --- block using
the existing _FRONTMATTER_RE before passing the body to mistune.html().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HEAD~1..HEAD always shows the same diff after os.execv reloads the
process — every tick triggers deploy.sh if the last commit touched web/.
Fix: track the last-seen HEAD in a module-level variable. On first call
(fresh process after os.execv), fall back to HEAD~1 so the newly-deployed
commit is evaluated once. Recording HEAD before returning means the same
commit never fires twice, regardless of how many ticks pass.
Also remove two unused imports (json, urllib.request) caught by ruff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Live chips now open the article in a new tab. Draft/scheduled chips are
non-clickable spans (informational only). The Edit button is the sole
path to the edit page, removing the redundant double-link.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EN: replace cliché phrase "pointing to pockets of underserved demand"
→ "leaving genuine supply gaps even in established markets" (more precise)
DE version already had a cleaner equivalent — no change needed there.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EN prose: tighten intro paragraph — "The question investors actually need
answered is:" → "The question that matters:" (DE version already had the
cleaner formulation; now aligned)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docker-compose.prod.yml: replace file bind mount for analytics.duckdb
with directory bind mount (/opt/padelnomics/data:/app/data/pipeline:ro)
so os.rename() on the host is visible inside the container
- Override SERVING_DUCKDB_PATH to /app/data/pipeline/analytics.duckdb in
all 6 blue/green services (removes dependency on .env value)
- analytics.py: track file inode; call _check_and_reopen() at start of
each query — transparently picks up new analytics.duckdb without restart
when export_serving.py atomically replaces it after each pipeline run
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Align contingency figure: 10% → 10–20% range (consistent with C7/C8)
- Add native German bridge before KfW section
- Add "Das spüren Banken." to close personal guarantees section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The invisible trigger div was inside the CSS grid, occupying the first cell
(1fr) and pushing the form into the 380px column and the preview below it.
Moved it before the grid with display:none so it has no layout impact.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix gendered pronoun: "he'll" → "they'll"
- Align contingency figure: 10% → 10–20% (consistent with C7/C8 guidance)
- "despite the fact that" → "even though"
- Add bridge sentence before KfW section connecting to section 9 of plan framework
- Sharpen personal guarantees closer: "That comes across in a bank conversation"
→ "Banks can tell."
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hx-trigger="load, input from:..." fires the preview POST as soon as the page
opens, so editing an existing product shows its card without needing to
touch any field first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add native German bridge sentence before Bürgschaften section,
matching the EN improvement: abrupt transition now contextualised
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The form was posting to the save route on every input change (which would
save the product on every keystroke). Added a dedicated POST
/admin/affiliate/preview route that renders the product_card.html partial
from form data without touching the database.
Form now keeps action pointing to the save route; an invisible hx-div
triggers preview-only POSTs via hx-include="#affiliate-form".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add bridge sentence before Personal Guarantee section — this key topic
was abrupt without introduction; now connects cleanly from the debt
structure discussion above
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fixed Even so: colon to em dash (punctuation)
- Tightened Risk 5 advice opener (Model this explicitly.)
- Removed double pronoun in F&B note (before committing)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lender reference: made active sentence
- Fixed grammar: Ihr persoenlicher Track Record (nominative)
- Added closing thought before Was-erfolgreiche-Bauprojekte section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
