.copier-answers.yml (new, at repo root):
- Points to local template path (was GitLab remote)
- _commit: v0.19.0 (enables copier update)
- Reflects actual feature set: enable_cms, enable_daas, not directory/i18n/leads
web/src/beanflows/core.py:
- Added ENABLE_CMS/ENABLE_DAAS/ENABLE_DIRECTORY/ENABLE_LEADS/BUSINESS_MODEL
to Config class (mirrors copier.yml questions for runtime feature gating)
.env.example (new, at repo root):
- Moved from web/.env.example; updated DUCKDB_PATH/SERVING_DUCKDB_PATH
to root-relative paths (local.duckdb, analytics.duckdb)
.gitignore:
- Added web/src/beanflows/static/css/output.css (previously only in web/.gitignore)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace grouped section labels + 9 individual links with 5 flat
section-level items (Dashboard, Manage, Content, Engagement, System)
and a horizontal tab strip for multi-page sections. Active state
derived via _section_map dict — no JS required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merges worktree-sops-supervisor-docs → master.
Summary of changes:
- setup_server.sh: now installs all tools (git, curl, age, sops, rclone, uv) —
single source of truth for server provisioning
- bootstrap_supervisor.sh: stripped to ~45 lines — zero tool installs, only
clone/fetch + decrypt + uv sync + systemd enable
- readme.md: updated descriptions to reflect new responsibilities
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- materia-supervisor.service: User=root → User=beanflows_service,
add PATH so uv (~/.local/bin) is found without a login shell
- setup_server.sh: full rewrite — creates beanflows_service (nologin),
generates SSH deploy key + age keypair as service user at XDG path
(~/.config/sops/age/keys.txt), installs age/sops/rclone as root,
prints both public keys + numbered next-step instructions
- bootstrap_supervisor.sh: full rewrite — removes GITLAB_READ_TOKEN
requirement, clones via SSH as service user, installs uv as service
user, decrypts with SOPS auto-discovery, uv sync as service user,
systemctl as root
- web/deploy.sh: remove self-contained sops/age install + keypair
generation; replace with simple sops check (exit if missing) and
SOPS auto-discovery decrypt (no explicit key file needed)
- infra/readme.md: update architecture diagram for beanflows_service
paths, update setup steps to match new scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lock #positioning-canvas min-height to current offsetHeight before each
HTMX swap, release it in htmx:afterSwap — prevents flash-to-zero during
Chart.js initialization in the new content
- Add CSS min-height:200px fallback on all canvas containers so they never
fully collapse even before JS runs
- Extract _swapCanvas() helper to deduplicate setRange/setType logic
Root cause of visual collapse: cot_positioning_combined table missing
(needs sqlmesh plan prod + export_serving to materialize).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DuckDB requires all selected columns to be aggregate expressions when there
is no GROUP BY. latest.max_date is a scalar CTE value but still needs
ANY_VALUE() wrapping to satisfy the binder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-part fix for charts going tiny on range changes (especially 3m) and
staying broken after subsequent navigations:
1. dashboard_base.html: global htmx:beforeSwap handler destroys any Chart.js
instances in the swap target before HTMX replaces the DOM. Without this,
the old chart's ResizeObserver remains attached to the parent container and
interferes with the newly created chart instance's dimension calculations.
2. All chart pages (positioning, supply, warehouse, weather): afterSwap handler
now wraps chart resize in requestAnimationFrame, ensuring the browser has
completed layout before Chart.js measures container dimensions. MA toggle
state is also restored inside the rAF callback after resize.
Root cause: chart init scripts run synchronously during innerHTML swap, before
browser layout is complete. Fast server responses (e.g. 3m = small dataset)
gave even less time for layout, making the timing issue reproducible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- analytics.py: add _cot_table() helper; add combined=False param to
get_cot_positioning_time_series(), get_cot_positioning_latest(),
get_cot_index_trend(); add get_cot_options_delta() for MM net delta
between combined and futures-only
- dashboard/routes.py: read ?type=fut|combined param; pass combined flag
to analytics calls; conditionally fetch options_delta when combined
- api/routes.py: add ?type= param to /positioning and /positioning/latest
endpoints; returned JSON includes type field
- positioning.html: add report type pill group (Futures / F+O Combined)
with setType() JS; setRange() and popstate now preserve the type param
- positioning_canvas.html: sync type pills on HTMX swap; show Opt Δ badge
on MM Net card when combined+options_delta available; conditional chart
title and subtitle reflect which report variant is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
App containers need access to the serving DuckDB populated by the
pipeline supervisor. Bind-mounts /data/materia/analytics.duckdb as
read-only and sets SERVING_DUCKDB_PATH in container environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-install sops + age binaries to web/bin/ if not present
- Generate age keypair at repo root age-key.txt if missing (prints public
key with instructions to add to .sops.yaml, then exits)
- Decrypt .env.prod.sops → web/.env at deploy time (no CI secrets needed)
- Backup SQLite DB before migration (timestamped, keeps last 3)
- Rollback on health check failure: dump logs + restore DB backup
- Reset nginx router to current slot before --wait to avoid upstream errors
- Remove web/scripts/deploy.sh (duplicate)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Load .env from repo root first (created by `make secrets-decrypt-dev`),
falling back to web/.env for legacy setups. Also fixes import sort order
and removes unused httpx import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polls /auth/dev-login until the app responds, then opens an incognito/private
window — same pattern as padelnomics. Tries flatpak Chrome → flatpak Firefox
→ system Chrome → Chromium → Firefox in that order.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 4 REST endpoints under /api/v1/weather/:
- GET /weather/locations — 12 locations with latest stress, sorted by severity
- GET /weather/locations/<id> — daily series for one location (?metrics, ?days)
- GET /weather/stress — global daily stress trend (?days)
- GET /weather/alerts — locations with active crop stress flags
All endpoints use @api_key_required(scopes=["read"]) and return {"data": ...}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ALLOWED_WEATHER_METRICS frozenset and 5 new query functions:
- get_weather_locations(): 12 locations with latest stress index for map/cards
- get_weather_location_series(): time series for one location (dynamic metrics)
- get_weather_stress_latest(): global snapshot for Pulse metric card
- get_weather_stress_trend(): daily global avg/max for chart and sparkline
- get_weather_active_alerts(): locations with active stress flags
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename env var to plural (CSV list) in CI yml to match the actual
config key. Add hendrik@beanflow.coffee and simon@beanflows.coffee
as hardcoded defaults so they get admin access without needing the
env var set explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dashboard/routes.py (4 places) and admin/routes.py still checked
analytics._conn is not None after _conn was removed in the two-file
refactor — causing AttributeError → 500 on every dashboard page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs fixed:
1. Cross-connection COPY: DuckDB doesn't support referencing another
connection's tables as src.serving.table. Replace with Arrow as
intermediate: src reads to Arrow, dst.register() + CREATE TABLE.
2. Catalog/schema name collision: naming the export file serving.duckdb
made DuckDB assign catalog name "serving" — same as the schema we
create inside it. Every serving.table query became ambiguous. Rename
to analytics.duckdb (catalog "analytics", schema "serving" = no clash).
SERVING_DUCKDB_PATH values updated: serving.duckdb → analytics.duckdb
in supervisor, service, bootstrap, dev_run.sh, .env.example, docker-compose.
3. Temp file: use _export.duckdb (not serving.duckdb.tmp) to avoid
the same catalog collision during the write phase.
Verified: 6 tables exported, serving.* queries work read-only.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On the first `./scripts/dev_run.sh` invocation (serving.duckdb absent),
automatically run extract → transform → export_serving from the repo root
so the dashboard is populated without any manual steps.
Subsequent runs skip the pipeline for a fast startup. Delete serving.duckdb
from the repo root to force a full pipeline re-run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The assert _db_path in fetch_analytics() would crash dashboard routes
locally when SERVING_DUCKDB_PATH is unset or serving.duckdb doesn't
exist yet. Change to graceful return [] so the app degrades cleanly.
Also add SERVING_DUCKDB_PATH=../serving.duckdb to local .env so the
web app will auto-connect once `materia pipeline run export_serving`
has been run for the first time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split the single lakehouse.duckdb into two files to eliminate the exclusive
write-lock conflict between SQLMesh (pipeline) and the Quart web app (reader):
lakehouse.duckdb — SQLMesh exclusive (all pipeline layers)
serving.duckdb — web app reads (serving tables only, atomically swapped)
Changes:
web/src/beanflows/analytics.py
- Replace persistent global _conn with per-thread connections (threading.local)
- Add _get_conn(): opens read_only=True on first call per thread, reopens
automatically on inode change (~1μs os.stat) to pick up atomic file swaps
- Switch env var from DUCKDB_PATH → SERVING_DUCKDB_PATH
- Add module docstring documenting architecture + DuckLake migration path
web/src/beanflows/app.py
- Startup check: use SERVING_DUCKDB_PATH
- Health check: use _db_path instead of _conn
src/materia/export_serving.py (new)
- Reads all serving.* tables from lakehouse.duckdb (read_only)
- Writes to serving_new.duckdb, then os.rename → serving.duckdb (atomic)
- ~50 lines; runs after each SQLMesh transform
src/materia/pipelines.py
- Add export_serving pipeline entry (uv run python -c ...)
infra/supervisor/supervisor.sh
- Add SERVING_DUCKDB_PATH env var comment
- Add export step: uv run materia pipeline run export_serving
infra/supervisor/materia-supervisor.service
- Add Environment=SERVING_DUCKDB_PATH=/data/materia/serving.duckdb
infra/bootstrap_supervisor.sh
- Add SERVING_DUCKDB_PATH to .env template
web/.env.example + web/docker-compose.yml
- Document both env vars; switch web service to SERVING_DUCKDB_PATH
web/src/beanflows/dashboard/templates/settings.html
- Minor settings page fix from prior session
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove 'Write' scope checkbox from API key creation form — BeanFlows
is a read-only data platform, write keys are meaningless to users.
Scope is now always 'read' via hidden input.
- Add try/except in billing.manage route so Paddle API failures (e.g.
no live credentials in dev) show a user-facing flash error instead
of a 500.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Apply button flow with immediate HTMX partial fetches:
- toggleCountry() does an optimistic UI update (row + badge) then
calls htmx.ajax() targeting #cc-canvas with swap=innerHTML
- URL is pushed to history on every selection change (bookmarkable)
- HX-Request now returns countries_canvas.html fragment (chips +
chart/empty + inline IIFE that re-syncs globals + re-inits Chart.js)
- Panel (dark) is never swapped; canvas fades during in-flight request
- PALETTE, buildRankings(), initChart() defined once on page load,
called by both initial render and partial IIFE after each swap
- Apply button removed; Clear triggers fetchCanvas() with empty codes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace generic multi-select + plain card with a two-panel layout:
- Dark espresso selector panel (sticky, searchable, click-to-toggle)
with country rows showing rank, name, production figure, checkbox
- Right canvas: metric segment tabs, selected-country chips (colored),
Chart.js line chart with dark espresso tooltip, and a JS-built
rankings table with proportional colored bars (latest year)
- Smooth fade-in animations, monospaced figures, copper accent palette
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- serving/ice_aging_stocks.sql: pass-through from foundation, parses age
bucket string to start/end days ints for correct sort order
- serving/ice_warehouse_stocks_by_port.sql: monthly by-port since 1996,
adds MoM change, MoM %, 12-month rolling average
- analytics.py: get_ice_aging_latest(), get_ice_aging_trend(),
get_ice_stocks_by_port_trend(), get_ice_stocks_by_port_latest()
- api/routes.py: GET /commodities/<code>/stocks/aging and
GET /commodities/<code>/stocks/by-port with auth + rate limiting
- dashboard/routes.py: add 3 new queries to asyncio.gather(), pass to template
- index.html: aging stacked bar chart (age buckets × port) with 4 metric
cards; by-port stacked area chart (30-year history) with 4 metric cards
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Create dashboard_base.html: standalone app shell with 56px sticky
header (logo + user email + sign out), 220px left sidebar with
Overview/Countries/Settings nav items (SVG icons, active state via
request.path), and fixed mobile bottom tab bar (md:hidden)
- Add CSS component classes: .app-shell, .app-header, .app-sidebar,
.sidebar-item, .app-content, .mobile-bottom-nav, .mobile-nav-item
- Extract feedback widget into _feedback_widget.html partial; include
from both base.html and dashboard_base.html
- Switch index.html, countries.html, settings.html to extend
dashboard_base.html; remove <main class="container-page"> wrappers
- Remove "Back to Dashboard" button from countries.html (sidebar
provides persistent navigation)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- routes.py: return_exceptions=True on gather, log individual query failures
with per-result defaults so one bad query doesn't blank the whole page
- settings.html: fix billing.portal → billing.manage (correct blueprint route)
- vision.md: update current state to February 2026, document shipped features
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use return_exceptions=True so a CatalogException from a single query
(e.g. table not yet populated in a fresh env) degrades gracefully
instead of crashing the whole dashboard render.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data
Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Admin flow:
- Remove /admin/login (password-based) and /admin/dev-login routes entirely
- admin_required now checks only the 'admin' role; redirects to auth.login
- auth/dev-login with an ADMIN_EMAILS address redirects directly to /admin/
- .env.example: replace ADMIN_PASSWORD with ADMIN_EMAILS=admin@beanflows.coffee
Dev seeding:
- Add dev_seed.py: idempotent upsert of 4 fixed accounts (admin, free,
starter, pro) so every access tier is testable after dev_run.sh
- dev_run.sh: seed after migrations, show all 4 login shortcuts
Regression tests (37 passing):
- test_analytics.py: concurrent fetch_analytics calls return correct row
counts (cursor thread-safety regression), column names are lowercase
- test_roles.py TestAdminAuthFlow: password login routes return 404,
admin_required redirects to auth.login, dev-login grants admin role
and redirects to admin panel when email is in ADMIN_EMAILS
- conftest.py: add mock_analytics fixture (fixes 7 pre-existing dashboard
test errors); fix assertion text and lowercase metric param in tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_conn.execute() is not thread-safe for concurrent calls from multiple
threads. asyncio.gather submits each analytics query to the thread pool
via asyncio.to_thread, causing race conditions that silently returned
empty result sets. _conn.cursor() creates an independent cursor that is
safe to use from separate threads simultaneously.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLMesh normalizes unquoted identifiers to lowercase in physical tables,
so commodity_metrics columns are e.g. 'production' not 'Production'.
Update ALLOWED_METRICS, all analytics.py SQL queries, dashboard routes,
and both dashboard templates (Jinja + JS chart references) to use
lowercase column names consistently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
