Eliminates confirmAction() entirely. One code path: all confirmations
go through showConfirm() called by the htmx:confirm interceptor.
14 template files converted to hx-boost + hx-confirm pattern.
Pipeline endpoints updated to exclude HX-Boosted requests from the
HTMX partial path.
# Conflicts:
# web/src/padelnomics/admin/templates/admin/affiliate_form.html
# web/src/padelnomics/admin/templates/admin/affiliate_program_form.html
# web/src/padelnomics/admin/templates/admin/base_admin.html
# web/src/padelnomics/admin/templates/admin/partials/affiliate_program_results.html
# web/src/padelnomics/admin/templates/admin/partials/affiliate_row.html
Task 3/6: lead_forward and lead_forward_htmx shared ~20 lines of
identical DB logic. Extracted into _forward_lead() that returns an
error string or None. Both routes now call the helper and differ
only in response format (redirect vs HTMX partial).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Task 2/6: Adds count_where(table_where, params) to core.py that
compresses the fetch_one + null-check COUNT(*) pattern. Applied
across admin/routes.py — dashboard stats shrinks from ~75 to ~25
lines, plus 10 more call sites compressed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate `confirmAction()` and the duplicate `cloneNode` hack entirely.
One code path: everything goes through `showConfirm()` called by the
`htmx:confirm` interceptor.
Dialog HTML:
- `<form method="dialog">` for native close semantics; button `value`
becomes `dialog.returnValue` — no manual event listener reassignment.
JS:
- `showConfirm(message)` — Promise-based, listens for `close` once.
- `htmx:confirm` handler calls `showConfirm()` and calls `issueRequest`
if confirmed. Replaces both the old HTMX handler and `confirmAction()`.
Templates (Padelnomics, 14 files):
- All `onclick=confirmAction(...)` and `onclick=confirm()` removed.
- Form-submit buttons: added `hx-boost="true"` to form + `hx-confirm`
on the submit button.
- Pure HTMX buttons (pipeline_transform, pipeline_overview): `hx-confirm`
replaces `onclick=if(!confirm(...))return false;`.
Pipeline routes (pipeline_trigger_extract, pipeline_trigger_transform):
- `is_htmx` now excludes `HX-Boosted: true` requests — boosted form
POSTs get the normal redirect instead of the inline partial.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The handler called evt.preventDefault() unconditionally, so auto-poll
requests (hx-trigger="every 5s", no hx-confirm) caused an empty dialog
to pop up every 5 seconds. Add an early return when evt.detail.question
is falsy so only actual hx-confirm interactions are intercepted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Free Webshare proxies were timing out and exhausting the circuit breaker
before datacenter/residential proxies got a chance to run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When all proxy tiers are exhausted and 0 venues are fetched, the working
file is empty and compress_jsonl_atomic asserts non-empty. Return early
with a warning instead of crashing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add global htmx:confirm handler in base_admin.html that intercepts
hx-confirm attributes and shows #confirm-dialog instead of window.confirm()
- Convert 4 pipeline HTMX buttons (Run Transform, Run Export, Run Full
Pipeline, Run extractor) from onclick+confirm() to hx-confirm
- Convert 4 affiliate form/list delete buttons from onclick+confirm()
to confirmAction() via event.preventDefault()
- Add scrollbar-width:none + ::-webkit-scrollbar{display:none} to
.pipeline-tabs to suppress spurious horizontal scrollbar
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
start_time is "HH:MM:SS" (time only), not a full ISO datetime. Combining
with resource's start_date to get "YYYY-MM-DDTHH:MM:SS" before parsing.
The ValueError was silently caught on every slot → 0 venues found → recheck
never actually ran since it was first deployed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace inline LITESTREAM_R2_* credentials in the backup service with
the named [r2-landing] rclone remote and R2_LANDING_* env vars, matching
the beanflows pattern. Add rclone.conf setup to bootstrap_supervisor.sh
so the remote is written from env on each bootstrap run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CI runs on Gitea only. GitLab is a passive push mirror — no runners,
no tagging, no deploy involvement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PROXY_URLS_* and other secrets were defined in .env but never loaded,
causing availability to run in slow serial mode (1 req/s) instead of
parallel mode with proxies.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
With parallel workers, threads that fetch a proxy just before escalation
can report failures after the tier has already changed — those failures
were silently counting against the new tier, immediately exhausting it
before it ever got tried (Rayobyte being skipped entirely in favour of
DataImpulse because 10 in-flight Webshare failures hit the threshold).
Fix: build a proxy_url → tier_idx reverse map at construction time and
skip the tier-level circuit breaker when the failing proxy belongs to an
already-escalated tier.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PROXY_URLS_DATACENTER was missing the scheme prefix, causing SSL
handshake failures on the Rayobyte HTTP-only proxy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Validates each URL in PROXY_URLS_DATACENTER / PROXY_URLS_RESIDENTIAL:
logs a warning and skips any entry missing an http:// or https:// scheme
instead of passing malformed URLs that cause SSL or connection errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds HTMX live polling to the Overview tab (stops when quiet) and a new
Transform tab for managing the SQLMesh + export steps of the ELT pipeline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add proxy_failure_limit param to make_tiered_cycler (default 3).
Individual proxies hitting the limit are marked dead and permanently
skipped. next_proxy() auto-escalates when all proxies in the active
tier are dead. Both mechanisms coexist: per-proxy dead tracking removes
broken individuals; tier-level threshold catches systemic failure.
- proxy.py: dead_proxies set + proxy_failure_counts dict in state;
next_proxy skips dead proxies with bounded loop; record_failure/
record_success accept optional proxy_url; dead_proxy_count() added
- playtomic_tenants.py: pass proxy_url to record_success/record_failure
- playtomic_availability.py: _worker returns (proxy_url, result);
serial loops in extract + extract_recheck capture proxy_url
- test_supervisor.py: 11 new tests in TestTieredCyclerDeadProxyTracking
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the tenants extractor flattened all proxy tiers into a single
round-robin list, bypassing the circuit breaker entirely. When the free
Webshare tier runs out of bandwidth (402), all 20 free proxies fail and
the batch crashes — the paid datacenter/residential proxies are never tried.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the tenants extractor flattened all proxy tiers into a single
round-robin list, bypassing the circuit breaker entirely. When the free
Webshare tier runs out of bandwidth (402), all 20 free proxies fail and
the batch crashes — the paid datacenter/residential proxies are never tried.
Changes:
- Replace make_round_robin_cycler with make_tiered_cycler (same as availability)
- Add _fetch_page_via_cycler: retries per page across tiers, records
success/failure in cycler so circuit breaker can escalate
- Fix batch_size to BATCH_SIZE=20 constant (was len(all_proxies) ≈ 22)
- Check cycler.is_exhausted() before each batch; catch RuntimeError mid-batch
and write partial results rather than crashing with nothing
- CIRCUIT_BREAKER_THRESHOLD from env (default 10), matching availability
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docker-compose.prod.yml: fix volume mount for all 6 web containers
from /opt/padelnomics/data (stale) → /data/padelnomics (live supervisor output);
add LANDING_DIR=/app/data/pipeline/landing so extraction/landing stats work
- pipeline_routes.py: fix _REPO_ROOT parents[5] → parents[4] so workflows.toml
is found in dev and pipeline overview shows workflow schedules
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace ABS() bbox predicates with BETWEEN in all three spatial CTEs
(nearest_padel, padel_local, tennis_nearby). BETWEEN enables DuckDB's
IEJoin (interval join) which is O((N+M) log M) vs the previous O(N×M)
nested-loop cross-join.
Add country pre-filters to restrict the left side from ~140K global
locations to ~20K rows for padel/tennis CTEs (~8 countries each).
Expected: ~50-200x speedup on the spatial CTE portion of the model.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
