Commit Graph

252 Commits

Author SHA1 Message Date
Deeman
b5fae9d528 fix(docker): use python:3.13-slim base image, switch hypercorn → granian
Some checks failed
CI / test-cli (push) Failing after 7s
CI / test-sqlmesh (push) Failing after 6s
CI / test-web (push) Failing after 6s
CI / tag (push) Has been skipped
2026-02-28 23:52:00 +01:00
Deeman
d6bd5d927c fix: rename readme.md → README.md to match Dockerfile COPY
Some checks failed
CI / test-cli (push) Failing after 7s
CI / test-sqlmesh (push) Failing after 6s
CI / test-web (push) Failing after 6s
CI / tag (push) Has been skipped
2026-02-28 23:48:15 +01:00
Deeman
ba62214bbd chore: remove DUCKDB_PATH and SERVING_DUCKDB_PATH from secrets (deployment paths, not secrets)
Some checks failed
CI / test-cli (push) Failing after 10s
CI / test-sqlmesh (push) Failing after 8s
CI / test-web (push) Successful in 15s
CI / tag (push) Has been skipped
2026-02-28 23:45:02 +01:00
Deeman
a79c1cec7b chore: remove LANDING_DIR from secrets (deployment path, not a secret)
Some checks failed
CI / test-cli (push) Failing after 11s
CI / test-sqlmesh (push) Failing after 8s
CI / test-web (push) Successful in 14s
CI / tag (push) Has been skipped
2026-02-28 23:43:05 +01:00
Deeman
890fb0e693 fix(supervisor): disable SUPERVISOR_GIT_PULL until deploy.sh is set up
Some checks failed
CI / test-cli (push) Failing after 9s
CI / test-sqlmesh (push) Failing after 9s
CI / test-web (push) Successful in 14s
CI / tag (push) Has been skipped
2026-02-28 23:36:19 +01:00
Deeman
260c9058b3 chore: regenerate uv.lock after materia → beanflows-pipeline rename
Some checks failed
CI / test-cli (push) Failing after 10s
CI / test-sqlmesh (push) Failing after 9s
CI / tag (push) Has been cancelled
CI / test-web (push) Has been cancelled
2026-02-28 23:35:55 +01:00
Deeman
9201a4dca9 fix: rename secrets.py → vault.py to avoid shadowing stdlib secrets module
Some checks failed
CI / test-cli (push) Failing after 10s
CI / test-sqlmesh (push) Failing after 9s
CI / test-web (push) Successful in 15s
CI / tag (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 23:33:06 +01:00
Deeman
79ce3f2913 fix: rename root package to beanflows-pipeline to avoid workspace conflict
Some checks failed
CI / test-cli (push) Failing after 11s
CI / test-sqlmesh (push) Failing after 9s
CI / test-web (push) Successful in 15s
CI / tag (push) Has been skipped
The web package is already named 'beanflows'. Renaming the root CLI/infra
package to 'beanflows-pipeline' (src/beanflows_pipeline/) resolves the
uv workspace name conflict and Python namespace collision.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 23:32:07 +01:00
Deeman
d14990bb01 refactor: rename materia → beanflows throughout codebase
Some checks failed
CI / test-cli (push) Failing after 5s
CI / test-sqlmesh (push) Failing after 4s
CI / test-web (push) Failing after 5s
CI / tag (push) Has been skipped
- Rename src/materia/ → src/beanflows/ (Python package)
- Rename transform/sqlmesh_materia/ → transform/sqlmesh_beanflows/
- Rename infra/supervisor/materia-supervisor.service → beanflows-supervisor.service
- Rename infra/backup/materia-backup.{service,timer} → beanflows-backup.{service,timer}
- Update all path strings: /opt/materia → /opt/beanflows, /data/materia → /data/beanflows
- Update pyproject.toml: project name, CLI entrypoint, workspace source key
- Update all internal imports from materia.* → beanflows.*
- Update infra scripts: REPO_DIR, service names, systemctl references
- Fix docker-compose.prod.yml: /data/materia → /data/beanflows (bind mount path)

Intentionally left unchanged: Pulumi stack name (materia-infrastructure) and
Hetzner resource names ("materia-key", "managed_by: materia") — these reference
live cloud infrastructure and require separate cloud-side renames.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 23:00:52 +01:00
Deeman
9ea4f09600 fix(supervisor): improve alert messages with category prefix and error snippet
Mirrors the same fix applied to padelnomics. Each alert now includes a
neutral category tag ([extract], [transform], [export], [deploy],
[supervisor]) and the first line of the error for quick diagnosis without
revealing tech stack details on the public free ntfy tier.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 12:28:44 +01:00
Deeman
c8b86569ff chore: consolidate to single ruff config in root pyproject.toml
All checks were successful
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 14s
CI / test-web (push) Successful in 14s
CI / tag (push) Successful in 2s
- Merge web ruff settings (select E/F/I/UP, line-length 100) into root config
- Remove [tool.ruff] section from web/pyproject.toml
- Remove "web" from root ruff exclude list
- Simplify pre-commit hook to one command: ruff check .
- Update CI to use: uv run ruff check . (from repo root)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v12
2026-02-28 12:21:01 +01:00
Deeman
42c1309b20 chore: add pre-commit ruff hook with auto-fix
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 12s
CI / test-web (push) Failing after 14s
CI / tag (push) Has been skipped
- scripts/hooks/pre-commit: runs ruff --fix for root and web/ (matching CI)
  and re-stages any auto-fixed files so they land in the commit
- Makefile: add install-hooks target (run once after clone)
- pyproject.toml: exclude web/ from root ruff (web has its own config)
- Fix remaining import sort warnings caught by the new hook

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 10:19:29 +01:00
Deeman
c5a218490e chore(web): fix ruff warnings in src/ (unused imports, unsorted imports)
All checks were successful
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 12s
CI / test-web (push) Successful in 14s
CI / tag (push) Successful in 2s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v10
2026-02-28 10:11:41 +01:00
Deeman
52bd731fc3 chore: fix all ruff lint warnings (unused imports, unsorted imports, unused vars)
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 13s
CI / test-web (push) Failing after 14s
CI / tag (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 10:05:05 +01:00
Deeman
e85d0eab63 fix(api): lowercase default metric names to match ALLOWED_METRICS
Some checks failed
CI / test-cli (push) Successful in 12s
CI / test-sqlmesh (push) Successful in 13s
CI / test-web (push) Failing after 14s
CI / tag (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 09:55:13 +01:00
Deeman
e872ba0204 fix(tests): resolve all CI test failures (verified locally, 218 pass)
Some checks failed
CI / test-cli (push) Successful in 10s
CI / test-sqlmesh (push) Successful in 12s
CI / test-web (push) Failing after 12s
CI / tag (push) Has been skipped
- billing/routes: replace httpx calls with paddle_billing SDK; add
  _paddle_client() factory; switch webhook verification to
  Notifications.Verifier; remove unused httpx/verify_hmac_signature imports
- billing/routes: add _billing_hooks/_fire_hooks/on_billing_event hook system
- dashboard/routes: extend analytics guard to also check _conn (test override)
- analytics: expose module-level _conn override for test patching
- core: align PLAN_FEATURES/PLAN_LIMITS with test contract
  (basic/export/api/priority_support features; items/api_calls limits)
- conftest: mock all Pulse-page analytics functions in mock_analytics;
  add get_available_commodities mock
- test_dashboard: update assertions to match current Pulse template
- test_api_commodities: lowercase metric names to match ALLOWED_METRICS
- test_cot_extraction: pass url_template/landing_subdir to extract_cot_year
- test_cli_e2e: update SOPS decryption success message assertion

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 02:10:06 +01:00
Deeman
8d1dbace0f fix(analytics): add _conn module-level override for test patching
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 12s
CI / test-web (push) Failing after 12s
CI / tag (push) Has been skipped
Tests monkeypatch analytics._conn to inject a temp DuckDB connection.
The attribute didn't exist; fetch_analytics now uses it when set,
bypassing the _db_path / threading.local path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 02:00:11 +01:00
Deeman
cddcd4463e docs: update CI/CD references from GitLab to Gitea
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 13s
CI / test-web (push) Failing after 12s
CI / tag (push) Has been skipped
Replace .gitlab/.gitlab-ci.yml with .gitea/workflows/ci.yaml, update
CI_JOB_TOKEN → github.token, CI_PIPELINE_IID → github.run_number, and
update setup instructions to point to git.padelnomics.io deploy keys.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 01:58:09 +01:00
Deeman
efb5a165e7 fix(billing): add missing hook infrastructure (_billing_hooks, on_billing_event, _fire_hooks)
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 13s
CI / tag (push) Has been cancelled
CI / test-web (push) Has been cancelled
Tests expected a billing event hook system that was never implemented.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 01:57:48 +01:00
Deeman
d58fa67238 fix(tests): update test assertions to match refactored function signatures
Some checks failed
CI / test-cli (push) Successful in 11s
CI / test-sqlmesh (push) Successful in 13s
CI / test-web (push) Failing after 11s
CI / tag (push) Has been skipped
- Pass url_template and landing_subdir to extract_cot_year (signature changed to support both COT variants)
- Update secrets test assertion from 'ESC connection successful' to 'SOPS decryption successful'

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 01:56:13 +01:00
Deeman
66d484955d fix: correct Gitea repo name materia → beanflows
Some checks failed
CI / test-cli (push) Failing after 57s
CI / test-sqlmesh (push) Successful in 13s
CI / test-web (push) Failing after 11s
CI / tag (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 18:19:18 +01:00
Deeman
2e928de156 chore: migrate from GitLab to self-hosted Gitea
Some checks failed
CI / test-cli (push) Has been cancelled
CI / test-sqlmesh (push) Has been cancelled
CI / test-web (push) Has been cancelled
CI / tag (push) Has been cancelled
Update bootstrap_supervisor.sh and setup_server.sh to use
git.padelnomics.io:2222 instead of gitlab.com.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 18:05:54 +01:00
Deeman
1de5d69906 fix(supervisor): use sqlmesh plan prod --auto-apply instead of run
'run' requires the prod environment to already exist and defaults to
dev_<username> on first run. 'plan --auto-apply' initializes prod if
missing and applies pending changes — fully self-healing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:49:38 +01:00
Deeman
dd07b0218b fix(infra): chown -R APP_DIR so service user owns full tree
Without -R, a manual uv sync or git operation run as root would create
files under the app dir owned by root, breaking uv for the service user
(Permission denied on .venv/bin/python3).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 15:23:20 +01:00
Deeman
a5d2a61cfb fix(billing): add missing helper functions and fix upsert_subscription signature
- Add upsert_billing_customer / get_billing_customer (billing_customers table)
- Add record_transaction (idempotent on provider_transaction_id)
- Fix upsert_subscription: remove provider_customer_id param, key by
  provider_subscription_id instead of user_id (allows multi-sub)
- Update webhook handler to call upsert_billing_customer separately

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 14:43:14 +01:00
Deeman
3faa29d8e5 fix(ci): move .gitlab-ci.yml to repo root so GitLab picks it up
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 14:26:37 +01:00
Deeman
2d79627ca9 fix(infra): change host port to 5001 to avoid conflict with padelnomics
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 14:12:45 +01:00
Deeman
33c8b4edbd update kyes 2026-02-27 13:51:56 +01:00
Deeman
5e22f2e1ae update secrets 2026-02-27 13:30:53 +01:00
Deeman
37b48d8f1c chore: use git remote for copier _src_path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 11:03:18 +01:00
Deeman
dee0600ee8 chore: delete stale web/ deployment files (now at repo root)
Removes: web/Dockerfile, web/docker-compose.yml, web/docker-compose.prod.yml,
web/deploy.sh, web/litestream.yml, web/router/, web/.copier-answers.yml,
web/.env.example — all superseded by root-level counterparts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 10:26:26 +01:00
Deeman
3a8dd6ba00 refactor: add .copier-answers.yml at root + feature flags + .env.example
.copier-answers.yml (new, at repo root):
- Points to local template path (was GitLab remote)
- _commit: v0.19.0 (enables copier update)
- Reflects actual feature set: enable_cms, enable_daas, not directory/i18n/leads

web/src/beanflows/core.py:
- Added ENABLE_CMS/ENABLE_DAAS/ENABLE_DIRECTORY/ENABLE_LEADS/BUSINESS_MODEL
  to Config class (mirrors copier.yml questions for runtime feature gating)

.env.example (new, at repo root):
- Moved from web/.env.example; updated DUCKDB_PATH/SERVING_DUCKDB_PATH
  to root-relative paths (local.duckdb, analytics.duckdb)

.gitignore:
- Added web/src/beanflows/static/css/output.css (previously only in web/.gitignore)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 10:25:28 +01:00
Deeman
c4cb263f18 refactor: move deployment files from web/ to repo root
Moves Dockerfile, docker-compose.yml, docker-compose.prod.yml, deploy.sh,
litestream.yml, and router/ from web/ to the monorepo root so copier update
can manage them from the template.

Dockerfile updated for monorepo layout:
- CSS build stage: COPY web/src/ ./web/src/ (Tailwind input path updated)
- Python build stage: copies root uv.lock + web/pyproject.toml separately,
  runs `uv sync --package beanflows` (not full workspace sync)
- Runtime CSS copy path updated to web/src/beanflows/static/css/output.css

deploy.sh: fixed sops path ($APP_DIR/.env.prod.sops, was $APP_DIR/../)

supervisor.py:
- web_code_changed(): Dockerfile path is now root-level (was web/Dockerfile)
- tick(): deploy script is now ./deploy.sh (was ./web/deploy.sh)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 10:24:52 +01:00
Deeman
128c177401 feat(ci): add Gitea Actions workflow
Mirrors the existing GitLab CI: three parallel test jobs (cli, sqlmesh,
web) gated by a tag job that creates v<run_number> on master. Supervisor
polls for new tags to deploy — no SSH keys or deploy credentials in CI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 08:20:28 +01:00
Deeman
7c5235ff39 feat(admin): flat sidebar + horizontal subnav navigation
Replace grouped section labels + 9 individual links with 5 flat
section-level items (Dashboard, Manage, Content, Engagement, System)
and a horizontal tab strip for multi-page sections. Active state
derived via _section_map dict — no JS required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:53:07 +01:00
Deeman
2b2a7274ca fix(ci): always run tests, remove needs:[] so tag waits for passing tests
Two bugs in the previous CI:
- needs: [] on the tag job bypassed stage ordering — tags were created before
  tests finished, defeating the entire pull-based deploy safety guarantee
- changes: rules meant a push to infra/ or docs would skip all tests but still
  create a tag

Now matches the padelnomics pattern: all three test jobs always run on master
and MRs, tag job runs after the test stage completes (stage ordering, no needs).
Also use uv sync --all-packages consistently across all jobs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:47:02 +01:00
Deeman
54dbb296dd fix(secrets): add secrets-updatekeys-prod target, use --input-type dotenv
sops updatekeys doesn't inherit --input-type from context, so calling it bare
on .env.prod.sops causes "Error unmarshalling input json" (guesses JSON from
the .sops extension). Explicit --input-type dotenv fixes it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:40:03 +01:00
Deeman
7d3263a39c chore: add server age key 2026-02-27 07:37:36 +01:00
Deeman
fd164ca66a chore: add server age key 2026-02-27 07:31:56 +01:00
Deeman
a3ce707a5b fix(infra): fix setup_server.sh summary — correct bootstrap command + sops format
- Detect server IP at runtime (hostname -I) and print real ssh command
- Replace misleading >- yaml block + '+' notation with correct comma-separated
  age key format: age: <dev-key>,<server-key>
- Label next steps as "(run from your workstation)"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:31:14 +01:00
Deeman
d14b45f7d6 fix(infra): guard SSH config write, add ROTATE_KEYS for key rotation
setup_server.sh is now fully idempotent on re-runs:
- deploy key generation was already guarded; SSH config write was not
- SSH config now only written if it doesn't exist (content never changes)
- ROTATE_KEYS=1 deletes the old keypair before generation, prints new
  public key to add to GitLab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 07:12:09 +01:00
Deeman
c778903264 merge(infra): consolidate tool installs in setup, strip bootstrap to essentials
Merges worktree-sops-supervisor-docs → master.

Summary of changes:
- setup_server.sh: now installs all tools (git, curl, age, sops, rclone, uv) —
  single source of truth for server provisioning
- bootstrap_supervisor.sh: stripped to ~45 lines — zero tool installs, only
  clone/fetch + decrypt + uv sync + systemd enable
- readme.md: updated descriptions to reflect new responsibilities

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 06:57:09 +01:00
Deeman
cf65fa16b6 refactor(infra): consolidate tool installs in setup, strip bootstrap to essentials
- setup_server.sh: add git/curl/ca-certificates apt install, add uv install
  as service user, fix SSH config write (root + chown vs sudo heredoc), remove
  noise log lines after set -e makes them redundant
- bootstrap_supervisor.sh: remove all tool installs (apt, uv, sops, age) —
  setup_server.sh is now the single source of truth; strip to ~45 lines:
  age-key check, clone/fetch, tag checkout, decrypt, uv sync, systemd enable
- readme.md: update step 1 and step 3 descriptions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-26 22:25:31 +01:00
Deeman
0317cb885f feat(infra): use beanflows_service for supervisor
- materia-supervisor.service: User=root → User=beanflows_service,
  add PATH so uv (~/.local/bin) is found without a login shell
- setup_server.sh: full rewrite — creates beanflows_service (nologin),
  generates SSH deploy key + age keypair as service user at XDG path
  (~/.config/sops/age/keys.txt), installs age/sops/rclone as root,
  prints both public keys + numbered next-step instructions
- bootstrap_supervisor.sh: full rewrite — removes GITLAB_READ_TOKEN
  requirement, clones via SSH as service user, installs uv as service
  user, decrypts with SOPS auto-discovery, uv sync as service user,
  systemctl as root
- web/deploy.sh: remove self-contained sops/age install + keypair
  generation; replace with simple sops check (exit if missing) and
  SOPS auto-discovery decrypt (no explicit key file needed)
- infra/readme.md: update architecture diagram for beanflows_service
  paths, update setup steps to match new scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-26 21:33:31 +01:00
Deeman
b27f06d811 chore: remove stale ralph-loop session file
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 20:28:31 +01:00
Deeman
ba3994a63b chore: add secrets-encrypt-dev/prod targets to match template 2026-02-26 20:27:35 +01:00
Deeman
6d4921d1a6 chore: align Makefile with padelnomics (pinned tailwind version, help target, dev target, .PHONY) 2026-02-26 20:23:31 +01:00
Deeman
c469f585eb docs: add PROJECT.md with backlog (retry/backoff for ICE + yfinance) 2026-02-26 20:08:12 +01:00
Deeman
8f97c6b0c9 fix(positioning): prevent canvas collapse on type/range toggle
- Lock #positioning-canvas min-height to current offsetHeight before each
  HTMX swap, release it in htmx:afterSwap — prevents flash-to-zero during
  Chart.js initialization in the new content
- Add CSS min-height:200px fallback on all canvas containers so they never
  fully collapse even before JS runs
- Extract _swapCanvas() helper to deduplicate setRange/setType logic

Root cause of visual collapse: cot_positioning_combined table missing
(needs sqlmesh plan prod + export_serving to materialize).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 20:04:19 +01:00
Deeman
d3c9d95386 fix(analytics): wrap max_date in ANY_VALUE() in get_weather_stress_latest
DuckDB requires all selected columns to be aggregate expressions when there
is no GROUP BY. latest.max_date is a scalar CTE value but still needs
ANY_VALUE() wrapping to satisfy the binder.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 19:50:27 +01:00