Tests monkeypatch analytics._conn to inject a temp DuckDB connection.
The attribute didn't exist; fetch_analytics now uses it when set,
bypassing the _db_path / threading.local path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace .gitlab/.gitlab-ci.yml with .gitea/workflows/ci.yaml, update
CI_JOB_TOKEN → github.token, CI_PIPELINE_IID → github.run_number, and
update setup instructions to point to git.padelnomics.io deploy keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Pass url_template and landing_subdir to extract_cot_year (signature changed to support both COT variants)
- Update secrets test assertion from 'ESC connection successful' to 'SOPS decryption successful'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Update bootstrap_supervisor.sh and setup_server.sh to use
git.padelnomics.io:2222 instead of gitlab.com.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
'run' requires the prod environment to already exist and defaults to
dev_<username> on first run. 'plan --auto-apply' initializes prod if
missing and applies pending changes — fully self-healing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without -R, a manual uv sync or git operation run as root would create
files under the app dir owned by root, breaking uv for the service user
(Permission denied on .venv/bin/python3).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.copier-answers.yml (new, at repo root):
- Points to local template path (was GitLab remote)
- _commit: v0.19.0 (enables copier update)
- Reflects actual feature set: enable_cms, enable_daas, not directory/i18n/leads
web/src/beanflows/core.py:
- Added ENABLE_CMS/ENABLE_DAAS/ENABLE_DIRECTORY/ENABLE_LEADS/BUSINESS_MODEL
to Config class (mirrors copier.yml questions for runtime feature gating)
.env.example (new, at repo root):
- Moved from web/.env.example; updated DUCKDB_PATH/SERVING_DUCKDB_PATH
to root-relative paths (local.duckdb, analytics.duckdb)
.gitignore:
- Added web/src/beanflows/static/css/output.css (previously only in web/.gitignore)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors the existing GitLab CI: three parallel test jobs (cli, sqlmesh,
web) gated by a tag job that creates v<run_number> on master. Supervisor
polls for new tags to deploy — no SSH keys or deploy credentials in CI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace grouped section labels + 9 individual links with 5 flat
section-level items (Dashboard, Manage, Content, Engagement, System)
and a horizontal tab strip for multi-page sections. Active state
derived via _section_map dict — no JS required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs in the previous CI:
- needs: [] on the tag job bypassed stage ordering — tags were created before
tests finished, defeating the entire pull-based deploy safety guarantee
- changes: rules meant a push to infra/ or docs would skip all tests but still
create a tag
Now matches the padelnomics pattern: all three test jobs always run on master
and MRs, tag job runs after the test stage completes (stage ordering, no needs).
Also use uv sync --all-packages consistently across all jobs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sops updatekeys doesn't inherit --input-type from context, so calling it bare
on .env.prod.sops causes "Error unmarshalling input json" (guesses JSON from
the .sops extension). Explicit --input-type dotenv fixes it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Detect server IP at runtime (hostname -I) and print real ssh command
- Replace misleading >- yaml block + '+' notation with correct comma-separated
age key format: age: <dev-key>,<server-key>
- Label next steps as "(run from your workstation)"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
setup_server.sh is now fully idempotent on re-runs:
- deploy key generation was already guarded; SSH config write was not
- SSH config now only written if it doesn't exist (content never changes)
- ROTATE_KEYS=1 deletes the old keypair before generation, prints new
public key to add to GitLab
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merges worktree-sops-supervisor-docs → master.
Summary of changes:
- setup_server.sh: now installs all tools (git, curl, age, sops, rclone, uv) —
single source of truth for server provisioning
- bootstrap_supervisor.sh: stripped to ~45 lines — zero tool installs, only
clone/fetch + decrypt + uv sync + systemd enable
- readme.md: updated descriptions to reflect new responsibilities
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- setup_server.sh: add git/curl/ca-certificates apt install, add uv install
as service user, fix SSH config write (root + chown vs sudo heredoc), remove
noise log lines after set -e makes them redundant
- bootstrap_supervisor.sh: remove all tool installs (apt, uv, sops, age) —
setup_server.sh is now the single source of truth; strip to ~45 lines:
age-key check, clone/fetch, tag checkout, decrypt, uv sync, systemd enable
- readme.md: update step 1 and step 3 descriptions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- materia-supervisor.service: User=root → User=beanflows_service,
add PATH so uv (~/.local/bin) is found without a login shell
- setup_server.sh: full rewrite — creates beanflows_service (nologin),
generates SSH deploy key + age keypair as service user at XDG path
(~/.config/sops/age/keys.txt), installs age/sops/rclone as root,
prints both public keys + numbered next-step instructions
- bootstrap_supervisor.sh: full rewrite — removes GITLAB_READ_TOKEN
requirement, clones via SSH as service user, installs uv as service
user, decrypts with SOPS auto-discovery, uv sync as service user,
systemctl as root
- web/deploy.sh: remove self-contained sops/age install + keypair
generation; replace with simple sops check (exit if missing) and
SOPS auto-discovery decrypt (no explicit key file needed)
- infra/readme.md: update architecture diagram for beanflows_service
paths, update setup steps to match new scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lock #positioning-canvas min-height to current offsetHeight before each
HTMX swap, release it in htmx:afterSwap — prevents flash-to-zero during
Chart.js initialization in the new content
- Add CSS min-height:200px fallback on all canvas containers so they never
fully collapse even before JS runs
- Extract _swapCanvas() helper to deduplicate setRange/setType logic
Root cause of visual collapse: cot_positioning_combined table missing
(needs sqlmesh plan prod + export_serving to materialize).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DuckDB requires all selected columns to be aggregate expressions when there
is no GROUP BY. latest.max_date is a scalar CTE value but still needs
ANY_VALUE() wrapping to satisfy the binder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-part fix for charts going tiny on range changes (especially 3m) and
staying broken after subsequent navigations:
1. dashboard_base.html: global htmx:beforeSwap handler destroys any Chart.js
instances in the swap target before HTMX replaces the DOM. Without this,
the old chart's ResizeObserver remains attached to the parent container and
interferes with the newly created chart instance's dimension calculations.
2. All chart pages (positioning, supply, warehouse, weather): afterSwap handler
now wraps chart resize in requestAnimationFrame, ensuring the browser has
completed layout before Chart.js measures container dimensions. MA toggle
state is also restored inside the rAF callback after resize.
Root cause: chart init scripts run synchronously during innerHTML swap, before
browser layout is complete. Fast server responses (e.g. 3m = small dataset)
gave even less time for layout, making the timing issue reproducible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- analytics.py: add _cot_table() helper; add combined=False param to
get_cot_positioning_time_series(), get_cot_positioning_latest(),
get_cot_index_trend(); add get_cot_options_delta() for MM net delta
between combined and futures-only
- dashboard/routes.py: read ?type=fut|combined param; pass combined flag
to analytics calls; conditionally fetch options_delta when combined
- api/routes.py: add ?type= param to /positioning and /positioning/latest
endpoints; returned JSON includes type field
- positioning.html: add report type pill group (Futures / F+O Combined)
with setType() JS; setRange() and popstate now preserve the type param
- positioning_canvas.html: sync type pills on HTMX swap; show Opt Δ badge
on MM Net card when combined+options_delta available; conditional chart
title and subtitle reflect which report variant is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- extract/cftc_cot: refactor extract_cot_year() to accept url_template and
landing_subdir params; add _extract_cot() shared loop; add extract_cot_combined()
entry point using com_disagg_txt_{year}.zip → landing/cot_combined/
- pyproject.toml: add extract_cot_combined script entry point
- macros/__init__.py: add @cot_combined_glob() for cot_combined/**/*.csv.gzip
- fct_cot_positioning.sql: union cot_glob and cot_combined_glob in src CTE;
add report_type column (FutOnly_or_Combined) to cast_and_clean + deduplicated;
include FutOnly_or_Combined in hkey to avoid key collisions; add report_type to grain
- obt_cot_positioning.sql: add report_type = 'FutOnly' filter to preserve
existing serving behavior
- obt_cot_positioning_combined.sql: new serving model filtered to report_type =
'Combined'; identical analytics (COT index, net %, windows) on combined data
- pipelines.py: register extract_cot_combined; add to extract_all meta-pipeline
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace push-based SSH deploy (deploy:web stage with SSH credentials +
individual env var injection) with tag-based pull deploy:
- Add `tag` stage: creates v${CI_PIPELINE_IID} tag using CI_JOB_TOKEN
- Remove all SSH variables (SSH_PRIVATE_KEY, SSH_KNOWN_HOSTS, DEPLOY_USER,
DEPLOY_HOST) and all individual secret variables from CI
- Zero deploy secrets in CI — only CI_JOB_TOKEN (built-in) needed
Deployment is now handled by the on-server supervisor (src/materia/supervisor.py)
which polls for new v* tags every 60s and runs web/deploy.sh automatically.
Secrets live in .env.prod.sops (git-committed, age-encrypted), decrypted at
deploy time by deploy.sh — never stored in GitLab CI variables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
App containers need access to the serving DuckDB populated by the
pipeline supervisor. Bind-mounts /data/materia/analytics.duckdb as
read-only and sets SERVING_DUCKDB_PATH in container environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-install sops + age binaries to web/bin/ if not present
- Generate age keypair at repo root age-key.txt if missing (prints public
key with instructions to add to .sops.yaml, then exits)
- Decrypt .env.prod.sops → web/.env at deploy time (no CI secrets needed)
- Backup SQLite DB before migration (timestamped, keeps last 3)
- Rollback on health check failure: dump logs + restore DB backup
- Reset nginx router to current slot before --wait to avoid upstream errors
- Remove web/scripts/deploy.sh (duplicate)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Load .env from repo root first (created by `make secrets-decrypt-dev`),
falling back to web/.env for legacy setups. Also fixes import sort order
and removes unused httpx import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
secrets.py: replace Pulumi ESC (esc CLI) with SOPS decrypt. Reads
.env.prod.sops via `sops --decrypt`, parses dotenv output. Same public
API: get_secret(), list_secrets(), test_connection().
cli.py: update secrets subcommand help text and test command messaging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- .sops.yaml: creation rules matching .env.{dev,prod}.sops (dotenv format)
- .env.dev.sops: encrypted dev defaults (blank API keys, local paths)
- .env.prod.sops: encrypted prod template (placeholder values to fill in)
- Makefile: root Makefile with secrets-decrypt-dev/prod, secrets-edit-dev/prod, css-build/watch
- .gitignore: add age-key.txt
Dev workflow: make secrets-decrypt-dev → .env (repo root) → web app picks it up.
Server: deploy.sh will auto-decrypt .env.prod.sops on each deploy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- yfinance confirmed not viable (OPRA only, KC=F not covered)
- CFTC COT combined report is the free immediate path (URL change only)
- ICE Report Center settlement data viable with WebICE login automation
- Barchart OnDemand has correct coverage but requires paid subscription
- All OpenBB providers, Polygon.io, Nasdaq Data Link confirmed no KC=F coverage
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
