Commit Graph

16 Commits

Author SHA1 Message Date
Deeman
36bd815525 fix(secrets): add secrets-updatekeys-prod target, use --input-type dotenv
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:40:03 +01:00
Deeman
dc2428eea4 fix(infra): fix setup_server.sh summary — correct bootstrap command + sops format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:31:14 +01:00
Deeman
834f9cb702 fix(infra): guard SSH config write, add ROTATE_KEYS for key rotation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:12:14 +01:00
Deeman
5218717e8d refactor(infra): converge on setup+bootstrap pattern, fix systemd copy bug
- setup_server.sh: full rewrite to match materia/template pattern — adds Docker
  install, git/curl/ca-certificates apt install, age + sops install (arch-aware),
  uv install as service user, age keypair generation, SSH config write (root+chown);
  removes systemd unit copy (was buggy: copied before repo was cloned)
- NEW bootstrap_supervisor.sh: ~45 lines — age key check, clone/fetch, tag checkout,
  sops decrypt, uv sync, copy landing-backup + supervisor systemd units, enable + start
- deploy.sh: replace 53-line self-install preamble (sops/age install + keypair
  generation + exit-1 flow) with simple sops check + decrypt; Docker blue/green
  logic unchanged

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 06:57:00 +01:00
Deeman
d834bdc59a feat(extract): recheck every 30 min with 30-min window for accurate occupancy
Each slot is now rechecked once, at most 30 min before it starts.
Worst-case miss: a booking made 29 min before start.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 09:39:30 +01:00
Deeman
7f3bde56b6 fix(infra): guard chown calls to make setup_server.sh fully idempotent
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 02:53:50 +01:00
Deeman
619d1570ef fix(infra): run services as padelnomics_service user instead of root
- setup_server.sh now requires root, creates padelnomics_service user,
  adds to docker group, generates deploy key in service user's home,
  owns /opt/padelnomics and /data/padelnomics to service user
- supervisor service: User=padelnomics_service, updated PATH
- landing-backup service: User=padelnomics_service

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 02:51:09 +01:00
Deeman
189f04cc47 fix(infra): run rclone installer with sudo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 02:37:50 +01:00
Deeman
e4bd9378f5 feat: self-provisioning deploy.sh — auto-installs sops+age, generates key
On first deploy to a new server, deploy.sh:
1. Installs age and sops binaries if missing
2. Generates an age keypair if missing
3. Prints the public key and exits with instructions

All checks are idempotent — subsequent deploys skip to decryption.
Removed duplicate sops/age setup from setup_server.sh (deploy.sh handles it).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:13:06 +01:00
Deeman
fcf66104cb feat: install sops + age in setup_server.sh
Installs age and sops binaries, generates an age keypair at
/opt/padelnomics/age-key.txt, and prints the public key in next
steps so it can be added to .sops.yaml.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:15:22 +01:00
Deeman
84229e50f7 Merge branch 'worktree-supervisor-flags'
Python supervisor + DB-backed feature flags

- supervisor.py replaces supervisor.sh (topological wave scheduling, croniter)
- workflows.toml workflow registry (5 extractors, cron presets, depends_on)
- proxy.py round-robin + sticky proxy rotation via PROXY_URLS
- Feature flags: migration 0019, is_flag_enabled(), feature_gate() decorator
- Admin /admin/flags UI with toggle (admin-only)
- lead_unlock gate on unlock_lead route
- 59 new tests (test_supervisor.py + test_feature_flags.py)
- Fix is_flag_enabled bug (fetch_one instead of execute_fetchone)

# Conflicts:
#	CHANGELOG.md
#	web/pyproject.toml
2026-02-23 15:29:43 +01:00
Deeman
76814dade7 feat: landing zone backup to R2 via rclone + Litestream
Landing files (append-only JSON.gz) synced to R2 every 30 min via
systemd timer + rclone. Extraction state DB (.state.sqlite) continuously
replicated via Litestream (second DB entry). Auto-restore on container
startup for both app.db and .state.sqlite. Reuses existing R2 bucket
and credentials — no new env vars needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 14:06:16 +01:00
Deeman
a1faddbed6 feat: Python supervisor + feature flags
Supervisor (replaces supervisor.sh):
- supervisor.py — cron-based pipeline orchestration, reads workflows.toml
  on every tick, runs due extractors in topological waves with parallel
  execution, then SQLMesh transform + serving export
- workflows.toml — workflow registry: overpass (monthly), eurostat (monthly),
  playtomic_tenants (weekly), playtomic_availability (daily),
  playtomic_recheck (hourly 6–23)
- padelnomics-supervisor.service — updated ExecStart to Python supervisor

Extraction enhancements:
- proxy.py — optional round-robin/sticky proxy rotation via PROXY_URLS env
- playtomic_availability.py — parallel fetch (EXTRACT_WORKERS), recheck mode
  (main_recheck) re-queries imminent slots for accurate occupancy measurement
- _shared.py — realistic browser User-Agent on all extractor sessions
- stg_playtomic_availability.sql — reads morning + recheck snapshots, tags each
- fct_daily_availability.sql — prefers recheck over morning for same slot

Feature flags (replaces WAITLIST_MODE env var):
- migration 0019 — feature_flags table, 5 initial flags:
  markets (on), payments/planner_export/supplier_signup/lead_unlock (off)
- core.py — is_flag_enabled() + feature_gate() decorator
- routes — payments, markets, planner_export, supplier_signup, lead_unlock gated
- admin flags UI — /admin/flags toggle page + nav link
- app.py — flag() injected as Jinja2 global

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:53:45 +01:00
Deeman
2db66efe77 feat: migrate transform to 3-layer architecture with per-layer schemas
Remove raw/ layer — staging models now read landing JSON directly.
Rename all model schemas from padelnomics.* to staging.*/foundation.*/serving.*.
Web app queries updated to serving.planner_defaults via SERVING_DUCKDB_PATH.
Supervisor gets daily sleep interval between pipeline runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:04:40 +01:00
Deeman
ea86940b78 feat: copier update v0.9.0 → v0.10.0
Pulls in template changes: export_serving.py for atomic DuckDB swap,
supervisor export step, SQLMesh glob macro, server provisioning script,
imprint template, and formatting improvements.

Template scaffold SQL models excluded (padelnomics has real models).
Web app routes/analytics unchanged (padelnomics-specific customizations).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 17:50:36 +01:00
Deeman
18ee24818b feat: copier update v0.9.0 — extraction docs, state tracking, architecture guides
Sync template from 29ac25b → v0.9.0 (29 template commits). Due to
template's _subdirectory migration, new files were manually rendered
rather than auto-merged by copier.

New files:
- .claude/CLAUDE.md + coding_philosophy.md (agent instructions)
- extract utils.py: SQLite state tracking for extraction runs
- extract/transform READMEs: architecture & pattern documentation
- infra/supervisor: systemd service + orchestration script
- Per-layer model READMEs (raw, staging, foundation, serving)

Also fixes copier-answers.yml (adds 4 feature toggles, removes stale
payment_provider key) and scopes CLAUDE.md gitignore to root only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 15:44:48 +01:00