- Merge web ruff settings (select E/F/I/UP, line-length 100) into root config
- Remove [tool.ruff] section from web/pyproject.toml
- Remove "web" from root ruff exclude list
- Simplify pre-commit hook to one command: ruff check .
- Update CI to use: uv run ruff check . (from repo root)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scripts/hooks/pre-commit: runs ruff --fix for root and web/ (matching CI)
and re-stages any auto-fixed files so they land in the commit
- Makefile: add install-hooks target (run once after clone)
- pyproject.toml: exclude web/ from root ruff (web has its own config)
- Fix remaining import sort warnings caught by the new hook
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests monkeypatch analytics._conn to inject a temp DuckDB connection.
The attribute didn't exist; fetch_analytics now uses it when set,
bypassing the _db_path / threading.local path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.copier-answers.yml (new, at repo root):
- Points to local template path (was GitLab remote)
- _commit: v0.19.0 (enables copier update)
- Reflects actual feature set: enable_cms, enable_daas, not directory/i18n/leads
web/src/beanflows/core.py:
- Added ENABLE_CMS/ENABLE_DAAS/ENABLE_DIRECTORY/ENABLE_LEADS/BUSINESS_MODEL
to Config class (mirrors copier.yml questions for runtime feature gating)
.env.example (new, at repo root):
- Moved from web/.env.example; updated DUCKDB_PATH/SERVING_DUCKDB_PATH
to root-relative paths (local.duckdb, analytics.duckdb)
.gitignore:
- Added web/src/beanflows/static/css/output.css (previously only in web/.gitignore)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace grouped section labels + 9 individual links with 5 flat
section-level items (Dashboard, Manage, Content, Engagement, System)
and a horizontal tab strip for multi-page sections. Active state
derived via _section_map dict — no JS required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merges worktree-sops-supervisor-docs → master.
Summary of changes:
- setup_server.sh: now installs all tools (git, curl, age, sops, rclone, uv) —
single source of truth for server provisioning
- bootstrap_supervisor.sh: stripped to ~45 lines — zero tool installs, only
clone/fetch + decrypt + uv sync + systemd enable
- readme.md: updated descriptions to reflect new responsibilities
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- materia-supervisor.service: User=root → User=beanflows_service,
add PATH so uv (~/.local/bin) is found without a login shell
- setup_server.sh: full rewrite — creates beanflows_service (nologin),
generates SSH deploy key + age keypair as service user at XDG path
(~/.config/sops/age/keys.txt), installs age/sops/rclone as root,
prints both public keys + numbered next-step instructions
- bootstrap_supervisor.sh: full rewrite — removes GITLAB_READ_TOKEN
requirement, clones via SSH as service user, installs uv as service
user, decrypts with SOPS auto-discovery, uv sync as service user,
systemctl as root
- web/deploy.sh: remove self-contained sops/age install + keypair
generation; replace with simple sops check (exit if missing) and
SOPS auto-discovery decrypt (no explicit key file needed)
- infra/readme.md: update architecture diagram for beanflows_service
paths, update setup steps to match new scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lock #positioning-canvas min-height to current offsetHeight before each
HTMX swap, release it in htmx:afterSwap — prevents flash-to-zero during
Chart.js initialization in the new content
- Add CSS min-height:200px fallback on all canvas containers so they never
fully collapse even before JS runs
- Extract _swapCanvas() helper to deduplicate setRange/setType logic
Root cause of visual collapse: cot_positioning_combined table missing
(needs sqlmesh plan prod + export_serving to materialize).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DuckDB requires all selected columns to be aggregate expressions when there
is no GROUP BY. latest.max_date is a scalar CTE value but still needs
ANY_VALUE() wrapping to satisfy the binder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-part fix for charts going tiny on range changes (especially 3m) and
staying broken after subsequent navigations:
1. dashboard_base.html: global htmx:beforeSwap handler destroys any Chart.js
instances in the swap target before HTMX replaces the DOM. Without this,
the old chart's ResizeObserver remains attached to the parent container and
interferes with the newly created chart instance's dimension calculations.
2. All chart pages (positioning, supply, warehouse, weather): afterSwap handler
now wraps chart resize in requestAnimationFrame, ensuring the browser has
completed layout before Chart.js measures container dimensions. MA toggle
state is also restored inside the rAF callback after resize.
Root cause: chart init scripts run synchronously during innerHTML swap, before
browser layout is complete. Fast server responses (e.g. 3m = small dataset)
gave even less time for layout, making the timing issue reproducible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- analytics.py: add _cot_table() helper; add combined=False param to
get_cot_positioning_time_series(), get_cot_positioning_latest(),
get_cot_index_trend(); add get_cot_options_delta() for MM net delta
between combined and futures-only
- dashboard/routes.py: read ?type=fut|combined param; pass combined flag
to analytics calls; conditionally fetch options_delta when combined
- api/routes.py: add ?type= param to /positioning and /positioning/latest
endpoints; returned JSON includes type field
- positioning.html: add report type pill group (Futures / F+O Combined)
with setType() JS; setRange() and popstate now preserve the type param
- positioning_canvas.html: sync type pills on HTMX swap; show Opt Δ badge
on MM Net card when combined+options_delta available; conditional chart
title and subtitle reflect which report variant is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
App containers need access to the serving DuckDB populated by the
pipeline supervisor. Bind-mounts /data/materia/analytics.duckdb as
read-only and sets SERVING_DUCKDB_PATH in container environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-install sops + age binaries to web/bin/ if not present
- Generate age keypair at repo root age-key.txt if missing (prints public
key with instructions to add to .sops.yaml, then exits)
- Decrypt .env.prod.sops → web/.env at deploy time (no CI secrets needed)
- Backup SQLite DB before migration (timestamped, keeps last 3)
- Rollback on health check failure: dump logs + restore DB backup
- Reset nginx router to current slot before --wait to avoid upstream errors
- Remove web/scripts/deploy.sh (duplicate)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Load .env from repo root first (created by `make secrets-decrypt-dev`),
falling back to web/.env for legacy setups. Also fixes import sort order
and removes unused httpx import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polls /auth/dev-login until the app responds, then opens an incognito/private
window — same pattern as padelnomics. Tries flatpak Chrome → flatpak Firefox
→ system Chrome → Chromium → Firefox in that order.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 4 REST endpoints under /api/v1/weather/:
- GET /weather/locations — 12 locations with latest stress, sorted by severity
- GET /weather/locations/<id> — daily series for one location (?metrics, ?days)
- GET /weather/stress — global daily stress trend (?days)
- GET /weather/alerts — locations with active crop stress flags
All endpoints use @api_key_required(scopes=["read"]) and return {"data": ...}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ALLOWED_WEATHER_METRICS frozenset and 5 new query functions:
- get_weather_locations(): 12 locations with latest stress index for map/cards
- get_weather_location_series(): time series for one location (dynamic metrics)
- get_weather_stress_latest(): global snapshot for Pulse metric card
- get_weather_stress_trend(): daily global avg/max for chart and sparkline
- get_weather_active_alerts(): locations with active stress flags
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename env var to plural (CSV list) in CI yml to match the actual
config key. Add hendrik@beanflow.coffee and simon@beanflows.coffee
as hardcoded defaults so they get admin access without needing the
env var set explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dashboard/routes.py (4 places) and admin/routes.py still checked
analytics._conn is not None after _conn was removed in the two-file
refactor — causing AttributeError → 500 on every dashboard page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs fixed:
1. Cross-connection COPY: DuckDB doesn't support referencing another
connection's tables as src.serving.table. Replace with Arrow as
intermediate: src reads to Arrow, dst.register() + CREATE TABLE.
2. Catalog/schema name collision: naming the export file serving.duckdb
made DuckDB assign catalog name "serving" — same as the schema we
create inside it. Every serving.table query became ambiguous. Rename
to analytics.duckdb (catalog "analytics", schema "serving" = no clash).
SERVING_DUCKDB_PATH values updated: serving.duckdb → analytics.duckdb
in supervisor, service, bootstrap, dev_run.sh, .env.example, docker-compose.
3. Temp file: use _export.duckdb (not serving.duckdb.tmp) to avoid
the same catalog collision during the write phase.
Verified: 6 tables exported, serving.* queries work read-only.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On the first `./scripts/dev_run.sh` invocation (serving.duckdb absent),
automatically run extract → transform → export_serving from the repo root
so the dashboard is populated without any manual steps.
Subsequent runs skip the pipeline for a fast startup. Delete serving.duckdb
from the repo root to force a full pipeline re-run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The assert _db_path in fetch_analytics() would crash dashboard routes
locally when SERVING_DUCKDB_PATH is unset or serving.duckdb doesn't
exist yet. Change to graceful return [] so the app degrades cleanly.
Also add SERVING_DUCKDB_PATH=../serving.duckdb to local .env so the
web app will auto-connect once `materia pipeline run export_serving`
has been run for the first time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split the single lakehouse.duckdb into two files to eliminate the exclusive
write-lock conflict between SQLMesh (pipeline) and the Quart web app (reader):
lakehouse.duckdb — SQLMesh exclusive (all pipeline layers)
serving.duckdb — web app reads (serving tables only, atomically swapped)
Changes:
web/src/beanflows/analytics.py
- Replace persistent global _conn with per-thread connections (threading.local)
- Add _get_conn(): opens read_only=True on first call per thread, reopens
automatically on inode change (~1μs os.stat) to pick up atomic file swaps
- Switch env var from DUCKDB_PATH → SERVING_DUCKDB_PATH
- Add module docstring documenting architecture + DuckLake migration path
web/src/beanflows/app.py
- Startup check: use SERVING_DUCKDB_PATH
- Health check: use _db_path instead of _conn
src/materia/export_serving.py (new)
- Reads all serving.* tables from lakehouse.duckdb (read_only)
- Writes to serving_new.duckdb, then os.rename → serving.duckdb (atomic)
- ~50 lines; runs after each SQLMesh transform
src/materia/pipelines.py
- Add export_serving pipeline entry (uv run python -c ...)
infra/supervisor/supervisor.sh
- Add SERVING_DUCKDB_PATH env var comment
- Add export step: uv run materia pipeline run export_serving
infra/supervisor/materia-supervisor.service
- Add Environment=SERVING_DUCKDB_PATH=/data/materia/serving.duckdb
infra/bootstrap_supervisor.sh
- Add SERVING_DUCKDB_PATH to .env template
web/.env.example + web/docker-compose.yml
- Document both env vars; switch web service to SERVING_DUCKDB_PATH
web/src/beanflows/dashboard/templates/settings.html
- Minor settings page fix from prior session
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove 'Write' scope checkbox from API key creation form — BeanFlows
is a read-only data platform, write keys are meaningless to users.
Scope is now always 'read' via hidden input.
- Add try/except in billing.manage route so Paddle API failures (e.g.
no live credentials in dev) show a user-facing flash error instead
of a 500.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Apply button flow with immediate HTMX partial fetches:
- toggleCountry() does an optimistic UI update (row + badge) then
calls htmx.ajax() targeting #cc-canvas with swap=innerHTML
- URL is pushed to history on every selection change (bookmarkable)
- HX-Request now returns countries_canvas.html fragment (chips +
chart/empty + inline IIFE that re-syncs globals + re-inits Chart.js)
- Panel (dark) is never swapped; canvas fades during in-flight request
- PALETTE, buildRankings(), initChart() defined once on page load,
called by both initial render and partial IIFE after each swap
- Apply button removed; Clear triggers fetchCanvas() with empty codes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace generic multi-select + plain card with a two-panel layout:
- Dark espresso selector panel (sticky, searchable, click-to-toggle)
with country rows showing rank, name, production figure, checkbox
- Right canvas: metric segment tabs, selected-country chips (colored),
Chart.js line chart with dark espresso tooltip, and a JS-built
rankings table with proportional colored bars (latest year)
- Smooth fade-in animations, monospaced figures, copper accent palette
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- serving/ice_aging_stocks.sql: pass-through from foundation, parses age
bucket string to start/end days ints for correct sort order
- serving/ice_warehouse_stocks_by_port.sql: monthly by-port since 1996,
adds MoM change, MoM %, 12-month rolling average
- analytics.py: get_ice_aging_latest(), get_ice_aging_trend(),
get_ice_stocks_by_port_trend(), get_ice_stocks_by_port_latest()
- api/routes.py: GET /commodities/<code>/stocks/aging and
GET /commodities/<code>/stocks/by-port with auth + rate limiting
- dashboard/routes.py: add 3 new queries to asyncio.gather(), pass to template
- index.html: aging stacked bar chart (age buckets × port) with 4 metric
cards; by-port stacked area chart (30-year history) with 4 metric cards
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Create dashboard_base.html: standalone app shell with 56px sticky
header (logo + user email + sign out), 220px left sidebar with
Overview/Countries/Settings nav items (SVG icons, active state via
request.path), and fixed mobile bottom tab bar (md:hidden)
- Add CSS component classes: .app-shell, .app-header, .app-sidebar,
.sidebar-item, .app-content, .mobile-bottom-nav, .mobile-nav-item
- Extract feedback widget into _feedback_widget.html partial; include
from both base.html and dashboard_base.html
- Switch index.html, countries.html, settings.html to extend
dashboard_base.html; remove <main class="container-page"> wrappers
- Remove "Back to Dashboard" button from countries.html (sidebar
provides persistent navigation)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
