- Merge web ruff settings (select E/F/I/UP, line-length 100) into root config
- Remove [tool.ruff] section from web/pyproject.toml
- Remove "web" from root ruff exclude list
- Simplify pre-commit hook to one command: ruff check .
- Update CI to use: uv run ruff check . (from repo root)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests monkeypatch analytics._conn to inject a temp DuckDB connection.
The attribute didn't exist; fetch_analytics now uses it when set,
bypassing the _db_path / threading.local path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.copier-answers.yml (new, at repo root):
- Points to local template path (was GitLab remote)
- _commit: v0.19.0 (enables copier update)
- Reflects actual feature set: enable_cms, enable_daas, not directory/i18n/leads
web/src/beanflows/core.py:
- Added ENABLE_CMS/ENABLE_DAAS/ENABLE_DIRECTORY/ENABLE_LEADS/BUSINESS_MODEL
to Config class (mirrors copier.yml questions for runtime feature gating)
.env.example (new, at repo root):
- Moved from web/.env.example; updated DUCKDB_PATH/SERVING_DUCKDB_PATH
to root-relative paths (local.duckdb, analytics.duckdb)
.gitignore:
- Added web/src/beanflows/static/css/output.css (previously only in web/.gitignore)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace grouped section labels + 9 individual links with 5 flat
section-level items (Dashboard, Manage, Content, Engagement, System)
and a horizontal tab strip for multi-page sections. Active state
derived via _section_map dict — no JS required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Lock #positioning-canvas min-height to current offsetHeight before each
HTMX swap, release it in htmx:afterSwap — prevents flash-to-zero during
Chart.js initialization in the new content
- Add CSS min-height:200px fallback on all canvas containers so they never
fully collapse even before JS runs
- Extract _swapCanvas() helper to deduplicate setRange/setType logic
Root cause of visual collapse: cot_positioning_combined table missing
(needs sqlmesh plan prod + export_serving to materialize).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DuckDB requires all selected columns to be aggregate expressions when there
is no GROUP BY. latest.max_date is a scalar CTE value but still needs
ANY_VALUE() wrapping to satisfy the binder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-part fix for charts going tiny on range changes (especially 3m) and
staying broken after subsequent navigations:
1. dashboard_base.html: global htmx:beforeSwap handler destroys any Chart.js
instances in the swap target before HTMX replaces the DOM. Without this,
the old chart's ResizeObserver remains attached to the parent container and
interferes with the newly created chart instance's dimension calculations.
2. All chart pages (positioning, supply, warehouse, weather): afterSwap handler
now wraps chart resize in requestAnimationFrame, ensuring the browser has
completed layout before Chart.js measures container dimensions. MA toggle
state is also restored inside the rAF callback after resize.
Root cause: chart init scripts run synchronously during innerHTML swap, before
browser layout is complete. Fast server responses (e.g. 3m = small dataset)
gave even less time for layout, making the timing issue reproducible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- analytics.py: add _cot_table() helper; add combined=False param to
get_cot_positioning_time_series(), get_cot_positioning_latest(),
get_cot_index_trend(); add get_cot_options_delta() for MM net delta
between combined and futures-only
- dashboard/routes.py: read ?type=fut|combined param; pass combined flag
to analytics calls; conditionally fetch options_delta when combined
- api/routes.py: add ?type= param to /positioning and /positioning/latest
endpoints; returned JSON includes type field
- positioning.html: add report type pill group (Futures / F+O Combined)
with setType() JS; setRange() and popstate now preserve the type param
- positioning_canvas.html: sync type pills on HTMX swap; show Opt Δ badge
on MM Net card when combined+options_delta available; conditional chart
title and subtitle reflect which report variant is shown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Load .env from repo root first (created by `make secrets-decrypt-dev`),
falling back to web/.env for legacy setups. Also fixes import sort order
and removes unused httpx import.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 4 REST endpoints under /api/v1/weather/:
- GET /weather/locations — 12 locations with latest stress, sorted by severity
- GET /weather/locations/<id> — daily series for one location (?metrics, ?days)
- GET /weather/stress — global daily stress trend (?days)
- GET /weather/alerts — locations with active crop stress flags
All endpoints use @api_key_required(scopes=["read"]) and return {"data": ...}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ALLOWED_WEATHER_METRICS frozenset and 5 new query functions:
- get_weather_locations(): 12 locations with latest stress index for map/cards
- get_weather_location_series(): time series for one location (dynamic metrics)
- get_weather_stress_latest(): global snapshot for Pulse metric card
- get_weather_stress_trend(): daily global avg/max for chart and sparkline
- get_weather_active_alerts(): locations with active stress flags
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename env var to plural (CSV list) in CI yml to match the actual
config key. Add hendrik@beanflow.coffee and simon@beanflows.coffee
as hardcoded defaults so they get admin access without needing the
env var set explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dashboard/routes.py (4 places) and admin/routes.py still checked
analytics._conn is not None after _conn was removed in the two-file
refactor — causing AttributeError → 500 on every dashboard page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The assert _db_path in fetch_analytics() would crash dashboard routes
locally when SERVING_DUCKDB_PATH is unset or serving.duckdb doesn't
exist yet. Change to graceful return [] so the app degrades cleanly.
Also add SERVING_DUCKDB_PATH=../serving.duckdb to local .env so the
web app will auto-connect once `materia pipeline run export_serving`
has been run for the first time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Split the single lakehouse.duckdb into two files to eliminate the exclusive
write-lock conflict between SQLMesh (pipeline) and the Quart web app (reader):
lakehouse.duckdb — SQLMesh exclusive (all pipeline layers)
serving.duckdb — web app reads (serving tables only, atomically swapped)
Changes:
web/src/beanflows/analytics.py
- Replace persistent global _conn with per-thread connections (threading.local)
- Add _get_conn(): opens read_only=True on first call per thread, reopens
automatically on inode change (~1μs os.stat) to pick up atomic file swaps
- Switch env var from DUCKDB_PATH → SERVING_DUCKDB_PATH
- Add module docstring documenting architecture + DuckLake migration path
web/src/beanflows/app.py
- Startup check: use SERVING_DUCKDB_PATH
- Health check: use _db_path instead of _conn
src/materia/export_serving.py (new)
- Reads all serving.* tables from lakehouse.duckdb (read_only)
- Writes to serving_new.duckdb, then os.rename → serving.duckdb (atomic)
- ~50 lines; runs after each SQLMesh transform
src/materia/pipelines.py
- Add export_serving pipeline entry (uv run python -c ...)
infra/supervisor/supervisor.sh
- Add SERVING_DUCKDB_PATH env var comment
- Add export step: uv run materia pipeline run export_serving
infra/supervisor/materia-supervisor.service
- Add Environment=SERVING_DUCKDB_PATH=/data/materia/serving.duckdb
infra/bootstrap_supervisor.sh
- Add SERVING_DUCKDB_PATH to .env template
web/.env.example + web/docker-compose.yml
- Document both env vars; switch web service to SERVING_DUCKDB_PATH
web/src/beanflows/dashboard/templates/settings.html
- Minor settings page fix from prior session
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove 'Write' scope checkbox from API key creation form — BeanFlows
is a read-only data platform, write keys are meaningless to users.
Scope is now always 'read' via hidden input.
- Add try/except in billing.manage route so Paddle API failures (e.g.
no live credentials in dev) show a user-facing flash error instead
of a 500.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Apply button flow with immediate HTMX partial fetches:
- toggleCountry() does an optimistic UI update (row + badge) then
calls htmx.ajax() targeting #cc-canvas with swap=innerHTML
- URL is pushed to history on every selection change (bookmarkable)
- HX-Request now returns countries_canvas.html fragment (chips +
chart/empty + inline IIFE that re-syncs globals + re-inits Chart.js)
- Panel (dark) is never swapped; canvas fades during in-flight request
- PALETTE, buildRankings(), initChart() defined once on page load,
called by both initial render and partial IIFE after each swap
- Apply button removed; Clear triggers fetchCanvas() with empty codes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace generic multi-select + plain card with a two-panel layout:
- Dark espresso selector panel (sticky, searchable, click-to-toggle)
with country rows showing rank, name, production figure, checkbox
- Right canvas: metric segment tabs, selected-country chips (colored),
Chart.js line chart with dark espresso tooltip, and a JS-built
rankings table with proportional colored bars (latest year)
- Smooth fade-in animations, monospaced figures, copper accent palette
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- serving/ice_aging_stocks.sql: pass-through from foundation, parses age
bucket string to start/end days ints for correct sort order
- serving/ice_warehouse_stocks_by_port.sql: monthly by-port since 1996,
adds MoM change, MoM %, 12-month rolling average
- analytics.py: get_ice_aging_latest(), get_ice_aging_trend(),
get_ice_stocks_by_port_trend(), get_ice_stocks_by_port_latest()
- api/routes.py: GET /commodities/<code>/stocks/aging and
GET /commodities/<code>/stocks/by-port with auth + rate limiting
- dashboard/routes.py: add 3 new queries to asyncio.gather(), pass to template
- index.html: aging stacked bar chart (age buckets × port) with 4 metric
cards; by-port stacked area chart (30-year history) with 4 metric cards
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Create dashboard_base.html: standalone app shell with 56px sticky
header (logo + user email + sign out), 220px left sidebar with
Overview/Countries/Settings nav items (SVG icons, active state via
request.path), and fixed mobile bottom tab bar (md:hidden)
- Add CSS component classes: .app-shell, .app-header, .app-sidebar,
.sidebar-item, .app-content, .mobile-bottom-nav, .mobile-nav-item
- Extract feedback widget into _feedback_widget.html partial; include
from both base.html and dashboard_base.html
- Switch index.html, countries.html, settings.html to extend
dashboard_base.html; remove <main class="container-page"> wrappers
- Remove "Back to Dashboard" button from countries.html (sidebar
provides persistent navigation)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- routes.py: return_exceptions=True on gather, log individual query failures
with per-result defaults so one bad query doesn't blank the whole page
- settings.html: fix billing.portal → billing.manage (correct blueprint route)
- vision.md: update current state to February 2026, document shipped features
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use return_exceptions=True so a CatalogException from a single query
(e.g. table not yet populated in a fresh env) degrades gracefully
instead of crashing the whole dashboard render.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- config.yaml: remove ambiguousorinvalidcolumn linter rule (false positives on read_csv TVFs)
- fct_cot_positioning: use TRY_CAST throughout — CFTC uses '.' as null in many columns
- raw/cot_disaggregated: add columns() declaration for 33 varchar cols
- dim_commodity: switch from SEED to FULL model with SQL VALUES to preserve leading zeros
Pandas auto-converts '083' → 83 even with varchar column declarations in SEED models
- seeds/dim_commodity.csv: correct cftc_commodity_code from '083731' (contract market code)
to '083' (3-digit CFTC commodity code); add CSV quoting
- test_cot_foundation.yaml: fix output key name, vars for time range, partial: true,
and correct cftc_commodity_code to '083'
- analytics.py: COFFEE_CFTC_CODE '083731' → '083' to match actual data
Result: serving.cot_positioning has 685 rows (2013-01-08 to 2026-02-17), 23/23 tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Admin flow:
- Remove /admin/login (password-based) and /admin/dev-login routes entirely
- admin_required now checks only the 'admin' role; redirects to auth.login
- auth/dev-login with an ADMIN_EMAILS address redirects directly to /admin/
- .env.example: replace ADMIN_PASSWORD with ADMIN_EMAILS=admin@beanflows.coffee
Dev seeding:
- Add dev_seed.py: idempotent upsert of 4 fixed accounts (admin, free,
starter, pro) so every access tier is testable after dev_run.sh
- dev_run.sh: seed after migrations, show all 4 login shortcuts
Regression tests (37 passing):
- test_analytics.py: concurrent fetch_analytics calls return correct row
counts (cursor thread-safety regression), column names are lowercase
- test_roles.py TestAdminAuthFlow: password login routes return 404,
admin_required redirects to auth.login, dev-login grants admin role
and redirects to admin panel when email is in ADMIN_EMAILS
- conftest.py: add mock_analytics fixture (fixes 7 pre-existing dashboard
test errors); fix assertion text and lowercase metric param in tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_conn.execute() is not thread-safe for concurrent calls from multiple
threads. asyncio.gather submits each analytics query to the thread pool
via asyncio.to_thread, causing race conditions that silently returned
empty result sets. _conn.cursor() creates an independent cursor that is
safe to use from separate threads simultaneously.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLMesh normalizes unquoted identifiers to lowercase in physical tables,
so commodity_metrics columns are e.g. 'production' not 'Production'.
Update ALLOWED_METRICS, all analytics.py SQL queries, dashboard routes,
and both dashboard templates (Jinja + JS chart references) to use
lowercase column names consistently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- admin_required now accepts users with 'admin' role (via g.user) in
addition to the password-based is_admin session flag, so both auth
methods grant access
- impersonate stores the admin's user_id (not True) in admin_impersonating
so stop-impersonating can restore the correct session
- stop_impersonating restores user_id from admin_impersonating instead of
just popping it
- remove s.stripe_customer_id from get_user_by_id (Paddle project, no
stripe_customer_id column in subscriptions)
Fixes 3 test_roles.py failures: test_admin_index_accessible_with_admin_role,
test_impersonate_stores_admin_id, test_stop_impersonating_restores_admin
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
g.subscription is explicitly set to None in load_user, so
g.get("subscription", {}) returns None (key exists), not {}.
Use (g.get(...) or {}) to coalesce None to an empty dict.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
