Commit Graph

523 Commits

Author SHA1 Message Date
Deeman
53117094ee feat(affiliate): admin CRUD for affiliate programs
Adds program list, create, edit, delete routes with appropriate guards
(delete blocked if products reference the program). Adds "Programs" tab
to the affiliate subnav. New templates: affiliate_programs.html,
affiliate_program_form.html, partials/affiliate_program_results.html.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 22:32:45 +01:00
Deeman
6076a0b30f feat(affiliate): use build_affiliate_url() in /go/<slug> redirect
Program-based products now get URLs assembled from the template at
redirect time. Changing a program's tracking_tag propagates instantly
to all its products without rebuilding. Legacy products (no program_id)
still use their baked affiliate_url via fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 22:27:57 +01:00
Deeman
8dbbd0df05 feat(affiliate): add program CRUD functions + build_affiliate_url()
Adds get_all_programs(), get_program(), get_program_by_slug() for admin
CRUD. Adds build_affiliate_url() that assembles URLs from program template
+ product identifier, with fallback to baked affiliate_url for legacy
products. Updates get_product() to JOIN affiliate_programs so _program
dict is available at redirect time. _parse_product() extracts program
fields into nested _program key.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 22:23:53 +01:00
Deeman
b1eeb0a0ac feat(affiliate): add affiliate_programs table + migration 0027
Creates affiliate_programs for centralised retailer config (URL template,
tracking tag, commission %). Adds nullable program_id + product_identifier
to affiliate_products for backward compat. Seeds "Amazon" program with
oneLink template. Backfills existing products by extracting ASINs from
baked affiliate_url values.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 22:23:00 +01:00
Deeman
6aae92fc58 fix(admin): strip YAML frontmatter before mistune in _rebuild_article()
Fixes a bug where manual article previews rendered raw frontmatter
(title:, slug:, etc.) as visible text. Now strips the --- block using
the existing _FRONTMATTER_RE before passing the body to mistune.html().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 22:17:44 +01:00
Deeman
81ec8733c7 fix(admin): DE/EN chips in article list link to live article, not edit
Live chips now open the article in a new tab. Draft/scheduled chips are
non-clickable spans (informational only). The Edit button is the sole
path to the edit page, removing the redundant double-link.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:52:23 +01:00
Deeman
34065fa2ac fix(affiliate): move HTMX preview trigger outside grid container
The invisible trigger div was inside the CSS grid, occupying the first cell
(1fr) and pushing the form into the 380px column and the preview below it.
Moved it before the grid with display:none so it has no layout impact.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:40:21 +01:00
Deeman
c2dfefcc1e fix(affiliate): fire preview on page load so edit form shows card immediately
hx-trigger="load, input from:..." fires the preview POST as soon as the page
opens, so editing an existing product shows its card without needing to
touch any field first.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:37:35 +01:00
Deeman
8c4a4078f9 fix(affiliate): live preview uses dedicated /affiliate/preview endpoint
The form was posting to the save route on every input change (which would
save the product on every keystroke). Added a dedicated POST
/admin/affiliate/preview route that renders the product_card.html partial
from form data without touching the database.

Form now keeps action pointing to the save route; an invisible hx-div
triggers preview-only POSTs via hx-include="#affiliate-form".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:34:07 +01:00
Deeman
0984657e72 fix(affiliate): sidebar active state, subnav order, dev seed data
- base_admin.html: add 'affiliate_dashboard' to _section_map so Dashboard
  page stays under the Affiliate section (was falling through to 'overview')
- base_admin.html: sidebar Affiliate link now points to dashboard (first tab)
- base_admin.html: subnav order Dashboard | Products (was Products | Dashboard)
- seed_dev_data.py: add 10 affiliate products (4 rackets, 2 shoes, 1 ball,
  1 grip, 1 bag) + 236 click events spread over 30 days for dashboard charts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:32:20 +01:00
Deeman
5c22ea9780 feat(affiliate): tests, ruff cleanup, CHANGELOG + PROJECT.md (commit 9/9)
- 26 tests in web/tests/test_affiliate.py covering hash_ip determinism,
  daily rotation, product CRUD, bake_product_cards marker replacement,
  click redirect (302 + logged), inactive/unknown 404, multi-retailer
- ruff: fix E741 ambiguous var (l → line in _form_to_product), F401 unused
  import, I001 import sort in admin/routes.py
- CHANGELOG: affiliate product system entry
- PROJECT.md: affiliate system moved to Done, Wirecutter backlog item removed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:06:01 +01:00
Deeman
1fdd2d07a4 feat(affiliate): 10 German equipment review article scaffolds
Topics: bester Schläger, Anfänger, defensiv, Fortgeschrittene, unter 100€,
Bälle, Schuhe, Ausrüstung-Guide, Zubehör, Geschenke. Each includes
[product:slug] and [product-group:category] markers, German headings,
placeholder prose, and <details> FAQ sections. Ready for editorial fill-in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 20:51:05 +01:00
Deeman
2214d7a58f feat(affiliate): i18n strings — affiliate_cta_buy, disclosure, pros/cons labels
Added in both en.json and de.json. German uses generisches Maskulinum per
project standards. tformat-compatible {retailer} placeholder in at_retailer key.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:52:43 +01:00
Deeman
0f360fd230 feat(affiliate): admin dashboard — click stats, daily bar chart, top products/articles
Pure CSS bar chart (div heights via inline %). Stats computed server-side in SQL.
Days filter (7d/30d/90d). Estimated revenue shown as rough indicator (~3% CR × €80).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:51:15 +01:00
Deeman
bc7e40b531 feat(affiliate): admin CRUD — routes, list/form templates, sidebar entry
Routes: GET/POST affiliate, affiliate/results (HTMX), affiliate/new,
affiliate/<id>/edit, affiliate/<id>/delete, affiliate/<id>/toggle.
Templates: affiliate_products.html (filterable list), affiliate_form.html
(two-column with live preview slot), partials/affiliate_row.html,
partials/affiliate_results.html. Affiliate added to base_admin.html sidebar
and subnav (Products | Dashboard).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:50:25 +01:00
Deeman
ef85d3bb36 feat(affiliate): /go/<slug> click redirect with rate limiting + click logging
302 redirect (not 301) so every click is tracked. Extracts lang/article_slug
from Referer header best-effort. Rate-limited to 60/min per IP; clicks
above limit still redirect but are not logged to prevent amplification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:41:04 +01:00
Deeman
4d45b99cd8 feat(affiliate): product card baking — PRODUCT_RE, bake_product_cards(), templates
Adds [product:slug] and [product-group:category] marker replacement.
Templates: product_card.html (horizontal editorial callout) and
product_group.html (responsive comparison grid). Chained after
bake_scenario_cards() in generate_articles(), preview_article(),
article_new(), article_edit(), and _rebuild_article().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:40:27 +01:00
Deeman
b5db9d16b9 feat(affiliate): core affiliate module — product lookup, click logging, stats
Pure async functions: get_product(), get_products_by_category(), log_click(),
hash_ip() with daily-rotating GDPR salt, get_click_stats() with SQL aggregation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:36:31 +01:00
Deeman
2e149fc1db feat(affiliate): migration 0026 — affiliate_products + affiliate_clicks tables
Adds affiliate product catalog and click tracking tables.
UNIQUE(slug, language) mirrors articles schema for multi-language support.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:35:27 +01:00
Deeman
6fb1e990e3 merge: three-tier proxy + daily tenants + staging model cleanup
All checks were successful
CI / test (push) Successful in 48s
CI / tag (push) Successful in 3s
v10
2026-02-28 18:26:50 +01:00
Deeman
6edf8ba65e fix(transform): remove blob fallback CTEs, update tenants glob to daily partition depth
TigerStyle clean break — no backwards-compat shims for old file formats:

- stg_playtomic_{venues,opening_hours,resources}: glob updated from
  */*/tenants.jsonl.gz (2-level, old weekly) to */*/*/tenants.jsonl.gz
  (3-level, new daily YYYY/MM/DD partition); blob tenants.json.gz CTE removed
- stg_playtomic_availability: morning_blob and recheck_blob CTEs removed;
  only JSONL format (availability_*.jsonl.gz) is read going forward

Verified locally: stg_playtomic_venues evaluates to 14231 venues from
2026/02/28/tenants.jsonl.gz with 0 errors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 18:26:44 +01:00
Deeman
ed0a578050 add concurrency var
All checks were successful
CI / test (push) Successful in 49s
CI / tag (push) Successful in 3s
v9
2026-02-28 18:20:52 +01:00
Deeman
c1cdeec6be fix(extract): default worker count to 200 when proxies configured
All checks were successful
CI / test (push) Successful in 49s
CI / tag (push) Successful in 3s
Previously fell back to len(tiers[0]) (e.g. 10 for Webshare) when
PROXY_CONCURRENCY was unset. Default is now MAX_PROXY_CONCURRENCY=200
so single-URL rotating proxies (DC/residential) run at full concurrency
without needing an explicit env var.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202602281706 v8
2026-02-28 18:06:55 +01:00
Deeman
710624f417 fix(supervisor): re-decrypt .env.prod.sops on tag deploy
All checks were successful
CI / test (push) Successful in 49s
CI / tag (push) Successful in 3s
git_pull_and_sync() was missing the sops decrypt step, so .env on the
server was never updated when secrets changed. Now decrypts after
checkout, before uv sync.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202602281657 v7
2026-02-28 17:57:32 +01:00
Deeman
6cf98f44d4 fix(transform): remove blob compat CTE from stg_tennis_courts
All checks were successful
CI / test (push) Successful in 49s
CI / tag (push) Successful in 3s
The overpass_tennis extractor has written JSONL-only since it was added.
The dual-format UNION ALL was backwards-compat debt that broke the
transform once no courts.json.gz files exist on the server:

  IO Error: No files found that match the pattern
  "data/landing/overpass_tennis/*/*/courts.json.gz"

Remove blob_elements CTE and the UNION ALL. Only read JSONL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202602281640 v6
2026-02-28 17:39:11 +01:00
Deeman
60659a5ec5 merge: daily tenant snapshots with date-based partition 2026-02-28 17:30:33 +01:00
Deeman
beb4195f16 feat(extract): daily tenant snapshots with date-based partition
- playtomic_tenants: partition by YYYY/MM/DD instead of ISO week;
  schedule changed from weekly to daily in workflows.toml
- playtomic_availability: _load_tenant_ids now tries 3-level glob
  (*/*/*/tenants.jsonl.gz) first for daily files, falls back to
  2-level for old monthly/weekly data

Alphabetical sort would rank old monthly files above daily ones
('t' > '2' in ASCII), so the explicit fallback chain is required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 17:27:16 +01:00
Deeman
88cc857f3a merge: weekly tenant snapshots via ISO week partition 2026-02-28 17:19:25 +01:00
Deeman
9116625884 feat(extract): weekly tenant snapshots via ISO week partition
Tenants extractor now partitions by ISO week (e.g. 2026/W09) instead of
month (2026/02), so each weekly run writes a fresh file rather than
skipping for the rest of the month.

_load_tenant_ids() in playtomic_availability already globs */*/tenants.jsonl.gz
and sorts reverse — 'W09' > '02' alphabetically so weekly files take priority
over old monthly ones automatically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 17:19:19 +01:00
Deeman
1af65bb46f feat(extract): add PROXY_CONCURRENCY override for rotating single-URL proxies
When DC/residential tiers have a single rotating endpoint, worker_count
defaulted to 1 (one URL = one worker). PROXY_CONCURRENCY lets you set
an explicit thread count (e.g. 100) for providers that handle concurrent
connections on a single URL.

Capped at MAX_PROXY_CONCURRENCY=200 to avoid overloading the endpoint.
Falls back to len(tiers[0]) when unset (existing behaviour).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 17:06:53 +01:00
Deeman
9b0bfc478d merge: three-tier proxy system with Webshare auto-fetch 2026-02-28 17:00:10 +01:00
Deeman
adf22924f6 feat(extract): three-tier proxy system with Webshare auto-fetch
Replace two-tier proxy setup (PROXY_URLS / PROXY_URLS_FALLBACK) with
N-tier escalation: free → datacenter → residential.

- proxy.py: fetch_webshare_proxies() auto-fetches the Webshare download
  API on each run (no more stale manually-copied lists). load_proxy_tiers()
  assembles tiers from WEBSHARE_DOWNLOAD_URL, PROXY_URLS_DATACENTER,
  PROXY_URLS_RESIDENTIAL. make_tiered_cycler() generalised to list[list[str]]
  with N-level escalation; is_fallback_active() replaced by is_exhausted().
  Old load_proxy_urls() / load_fallback_proxy_urls() deleted.

- playtomic_availability.py: both extract() and extract_recheck() use
  load_proxy_tiers() + generalised cycler. _fetch_venues_parallel fallback_urls
  param removed. All is_fallback_active() checks → is_exhausted().

- playtomic_tenants.py: flattens tiers for simple round-robin.

- test_supervisor.py: TestLoadProxyUrls removed (function deleted).
  Added TestFetchWebshareProxies, TestLoadProxyTiers, TestTieredCyclerNTier
  (11 tests covering parse format, error handling, escalation, thread safety).

47 tests pass, ruff clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 16:57:07 +01:00
Deeman
09665b7786 update proxies 2026-02-28 16:51:40 +01:00
Deeman
93349923bd merge(better-alerts): improve supervisor alert messages 2026-02-28 12:27:14 +01:00
Deeman
642041b32b fix(supervisor): improve alert messages with category prefix and error snippet
Each alert now includes a neutral category tag ([extract], [transform],
[export], [deploy], [supervisor]) and the first line of the error, so
notifications are actionable without revealing tech stack details on the
public free ntfy tier.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 12:27:11 +01:00
Deeman
bb70a5372b docs: replace GitLab CI/CD section with Gitea pull-based deployment
All checks were successful
CI / test (push) Successful in 48s
CI / tag (push) Successful in 3s
Remove outdated SSH-push model referencing GitLab variables. Document
the actual pull-based flow: Gitea Actions → tag → supervisor polls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v5
2026-02-28 01:58:11 +01:00
Deeman
bc28d93662 fix: remove duplicate age key in .sops.yaml
All checks were successful
CI / test (push) Successful in 47s
CI / tag (push) Successful in 3s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v202602271834 v4
2026-02-27 18:30:31 +01:00
Deeman
81ce1d277a update key
Some checks failed
CI / test (push) Has been cancelled
CI / tag (push) Has been cancelled
2026-02-27 18:26:14 +01:00
Deeman
2012894eeb chore: migrate from GitLab to self-hosted Gitea
Some checks failed
CI / test (push) Has been cancelled
CI / tag (push) Has been cancelled
Update bootstrap_supervisor.sh and setup_server.sh to use
git.padelnomics.io:2222 instead of gitlab.com.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 18:23:20 +01:00
Deeman
143ad28854 fix(supervisor): use sqlmesh plan --auto-apply instead of run
Some checks failed
CI / test (push) Has been cancelled
CI / tag (push) Has been cancelled
'run' requires the prod environment to already exist. 'plan --auto-apply'
initializes the environment on first run and applies pending changes on
subsequent runs — fully self-healing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v1
2026-02-27 15:40:37 +01:00
Deeman
415d28afa9 fix(supervisor): run sqlmesh against prod environment
Without the 'prod' argument sqlmesh defaults to dev_<username>, which
doesn't exist on the server (padelnomics_service user).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:39:55 +01:00
Deeman
66d7cdea21 update 2026-02-27 15:39:39 +01:00
Deeman
9c2bf51c73 fix(infra): chown -R APP_DIR so service user owns full tree
Without -R, a manual uv sync or git operation run as root would create
files under /opt/padelnomics owned by root, breaking uv for the service
user (Permission denied on .venv/bin/python3).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:23:12 +01:00
Deeman
7e0b06a2ad prototype 2026-02-27 14:03:40 +01:00
Deeman
dca198c17d fix(ci): clear alpine/git entrypoint in tag job
alpine/git sets ENTRYPOINT ["git"], so GitLab's shell executor was invoking
`git sh <script>` instead of `sh <script>`. Override with entrypoint: [""].

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 13:59:50 +01:00
Deeman
49820391ab fix(admin): qualify ambiguous column name in marketplace_activity query
`credit_ledger cl` joined with `suppliers s` — both have `id`, so
SQLite raised OperationalError. Qualify as `cl.id` and `cl.supplier_id`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 13:59:30 +01:00
Deeman
f048e8276f style(admin): rename nav label "Pipeline" → "Data Platform"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 13:59:13 +01:00
Deeman
bcacc7aae6 merge(pipeline-lineage): conform geographic dimension hierarchy via city_slug 2026-02-27 13:31:44 +01:00
Deeman
00393933ca merge: lineage hover tooltip + click schema panel 2026-02-27 13:24:20 +01:00
Deeman
89ff931212 feat(lineage): hover tooltip + click-to-inspect schema panel
- New route GET /admin/pipeline/lineage/schema/<model> — returns JSON
  with columns+types (from information_schema for serving models),
  row count, upstream and downstream model lists. Validates model
  against _DAG to prevent arbitrary table access.
- Precomputes _DOWNSTREAM map at import time from _DAG.
- Lineage template: replaces minimal edge-highlight JS with full UX —
  hover triggers schema prefetch + floating tooltip (layer badge, top 4
  columns, "+N more" note); click opens 320px slide-in panel showing
  row count, full schema table, upstream/downstream dep lists.
  Dep items in panel are clickable to navigate between models.
  Schema responses are cached client-side to avoid repeat fetches.
  Staging/foundation models show "schema in lakehouse.duckdb only".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 13:23:54 +01:00